2024-04-12T02:38:10.903 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T02:38:10.903 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} --name yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588 --verbose -- /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/orig.config.yaml 2024-04-12T02:38:10.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T02:38:11.034 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588 branch: reef-release description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} email: null first_in_suite: false job_id: '7652588' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef-release ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef-release sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 6358 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTwpORqxF9iz3chL4tw4yrbtah//4i57koBpG6bhfbnaBL2JrJPBf4di8jSm+YAH/YD8q536515s1peMJjqEzg= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/9ghsepfvA6W427iDW4ZktMrfXGxKfbxzC0J3HL+ULgMBV5nXlZzI6yDzw85DCviyvol6Y+ZF+JXcH/Mlg7oY= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-11_20:30:01 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-12T02:38:11.034 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-12T02:38:11.035 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-12T02:38:11.035 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T02:38:11.037 INFO:teuthology.task.internal:Checking packages... 2024-04-12T02:38:11.058 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-12T02:38:11.059 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T02:38:11.059 INFO:teuthology.packaging:ref: None 2024-04-12T02:38:11.059 INFO:teuthology.packaging:tag: None 2024-04-12T02:38:11.059 INFO:teuthology.packaging:branch: reef-release 2024-04-12T02:38:11.059 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:38:11.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2024-04-12T02:38:11.292 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1187.gd540ebac 2024-04-12T02:38:11.293 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T02:38:11.300 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T02:38:11.300 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T02:38:11.308 INFO:teuthology.task.internal:Saving configuration 2024-04-12T02:38:11.321 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T02:38:11.327 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T02:38:11.358 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:32:59.595267', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTwpORqxF9iz3chL4tw4yrbtah//4i57koBpG6bhfbnaBL2JrJPBf4di8jSm+YAH/YD8q536515s1peMJjqEzg='} 2024-04-12T02:38:11.389 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:32:59.593702', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/9ghsepfvA6W427iDW4ZktMrfXGxKfbxzC0J3HL+ULgMBV5nXlZzI6yDzw85DCviyvol6Y+ZF+JXcH/Mlg7oY='} 2024-04-12T02:38:11.389 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T02:38:11.396 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-04-12T02:38:11.396 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-04-12T02:38:11.396 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T02:38:11.480 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8365ed9e50>, signals=[15]) 2024-04-12T02:38:11.480 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T02:38:11.487 INFO:teuthology.task.internal:Opening connections... 2024-04-12T02:38:11.487 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:38:11.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:38:11.573 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:38:11.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:38:11.671 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T02:38:11.677 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-04-12T02:38:11.698 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-04-12T02:38:11.698 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-04-12T02:38:11.757 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Red Hat Enterprise Linux" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8.6 (Ootpa)" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:ID="rhel" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="fedora" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8.6" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.redhat.com/" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T02:38:11.758 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:11.759 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T02:38:11.759 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-12T02:38:11.759 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-12T02:38:11.759 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-12T02:38:11.759 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-04-12T02:38:11.796 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-04-12T02:38:11.816 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-04-12T02:38:11.816 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux" 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.6 (Ootpa)" 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel" 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora" 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.6" 2024-04-12T02:38:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-12T02:38:11.876 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-12T02:38:11.877 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-04-12T02:38:11.920 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T02:38:11.929 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T02:38:11.935 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T02:38:11.935 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-04-12T02:38:11.938 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-04-12T02:38:11.955 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T02:38:11.962 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T02:38:11.962 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-04-12T02:38:11.997 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-04-12T02:38:12.035 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T02:38:12.107 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T02:38:12.122 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T02:38:12.122 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T02:38:12.123 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T02:38:12.123 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T02:38:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:38:12.142 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-04-12T02:38:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:38:12.143 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-04-12T02:38:12.200 INFO:teuthology.orchestra.run.smithi181.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:12.200 INFO:teuthology.task.kernel:Running kernel on smithi181: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:12.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-04-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:12.202 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:12.202 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-04-12T02:38:13.692 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:13.692 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-04-12T02:38:13.743 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:13.743 INFO:teuthology.orchestra.run.smithi181.stdout:Unable to read consumer identity 2024-04-12T02:38:13.809 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:13.809 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:13.809 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:13.866 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:13.866 INFO:teuthology.orchestra.run.smithi181.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:13.867 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-12T02:38:15.671 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-04-12T02:38:16.896 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-12T02:38:16.945 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-12T02:38:20.631 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 423 kB/s | 24 kB 00:00 2024-04-12T02:38:20.653 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 595 kB/s | 24 kB 00:00 2024-04-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-12T02:38:21.366 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-12T02:38:21.367 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:38:21.540 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2024-04-12T02:38:21.564 DEBUG:teuthology.orchestra.run.smithi181:> echo no | sudo yum reinstall kernel || true 2024-04-12T02:38:21.914 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:21.914 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-04-12T02:38:21.940 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:21.940 INFO:teuthology.orchestra.run.smithi181.stdout:Unable to read consumer identity 2024-04-12T02:38:21.992 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:21.992 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:21.993 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:22.012 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:22.012 INFO:teuthology.orchestra.run.smithi181.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:22.012 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Apr 2024 02:38:20 AM UTC. 2024-04-12T02:38:22.320 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Apr 2024 02:38:20 AM UTC. 2024-04-12T02:38:22.373 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:38:22.380 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:38:22.394 INFO:teuthology.orchestra.run.smithi181.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi181.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:38:22.419 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-04-12T02:38:22.436 INFO:teuthology.orchestra.run.smithi181.stderr:Error: No packages marked for reinstall. 2024-04-12T02:38:22.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2024-04-12T02:38:22.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum reinstall -y kernel || true 2024-04-12T02:38:22.798 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:22.798 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-04-12T02:38:22.816 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:38:22.816 INFO:teuthology.orchestra.run.smithi181.stdout:Unable to read consumer identity 2024-04-12T02:38:22.870 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:22.871 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:22.871 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:38:22.886 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:22.886 INFO:teuthology.orchestra.run.smithi181.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:38:22.886 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:03 ago on Fri 12 Apr 2024 02:38:20 AM UTC. 2024-04-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:03 ago on Fri 12 Apr 2024 02:38:20 AM UTC. 2024-04-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi181.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi181.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-04-12T02:38:23.293 INFO:teuthology.orchestra.run.smithi181.stderr:Error: No packages marked for reinstall. 2024-04-12T02:38:23.326 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T02:38:23.343 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T02:38:23.486 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:23.486 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:23.486 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-12T02:38:23.486 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-12T02:38:23.486 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T02:38:23.504 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:23.504 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:38:23.504 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-12T02:38:23.504 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-12T02:38:23.504 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T02:38:23.519 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-04-12T02:38:23.534 DEBUG:teuthology.parallel:result is None 2024-04-12T02:38:23.552 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-04-12T02:38:23.567 DEBUG:teuthology.parallel:result is None 2024-04-12T02:38:23.567 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T02:38:23.582 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T02:38:23.582 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T02:38:23.586 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T02:38:23.608 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T02:38:23.628 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T02:38:23.702 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T02:38:23.702 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T02:38:23.706 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T02:38:23.743 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T02:38:23.778 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T02:38:23.778 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T02:38:23.783 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T02:38:23.818 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:38:23.820 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:38:23.829 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:38:23.831 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:38:23.832 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T02:38:23.843 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T02:38:23.844 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T02:38:23.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T02:38:23.903 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T02:38:23.934 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T02:38:23.934 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T02:38:23.945 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T02:38:23.963 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:38:24.056 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:38:24.099 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:38:24.191 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:38:24.224 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:38:24.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T02:38:24.291 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:38:24.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:38:24.383 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:38:24.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:38:24.508 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:38:24.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T02:38:24.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-04-12T02:38:24.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-04-12T02:38:24.622 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T02:38:24.658 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T02:38:25.132 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T02:38:25.141 INFO:teuthology.task.internal:Starting timer... 2024-04-12T02:38:25.141 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T02:38:25.155 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T02:38:25.165 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-12T02:38:25.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-04-12T02:38:25.243 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-04-12T02:38:25.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-04-12T02:38:25.309 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-04-12T02:38:25.311 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T02:38:25.311 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-04-12T02:38:25.331 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-04-12T02:38:25.332 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-04-12T02:38:25.359 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-04-12T02:38:25.359 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-04-12T02:38:25.359 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T02:38:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:38:25.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T02:38:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:38:25.427 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T02:38:25.427 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-04-12T02:38:25.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-04-12T02:38:25.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T02:38:25.513 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-12T02:38:25.570 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-12T02:38:28.171 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T02:38:28.943 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T02:38:28.944 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T02:46:17.833 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-04-12T02:46:17.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-04-12T02:46:17.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:46:17.956 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-04-12T02:46:18.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-04-12T02:46:18.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-12T02:46:18.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:46:18.107 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-04-12T02:46:18.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-12T02:46:18.181 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T02:46:18.191 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T02:46:18.192 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T02:46:18.192 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T02:46:18.194 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T02:46:18.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T02:46:18.227 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T02:46:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T02:46:18.254 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T02:46:18.269 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T02:46:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-04-12T02:46:18.336 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-04-12T02:46:18.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-04-12T02:46:18.350 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-04-12T02:46:18.351 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T02:46:18.364 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T02:46:18.365 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T02:46:18.378 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T02:46:18.433 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.448 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.452 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T02:46:18.453 INFO:teuthology.run_tasks:Running task install... 2024-04-12T02:46:18.462 DEBUG:teuthology.task.install:project ceph 2024-04-12T02:46:18.462 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-12T02:46:18.462 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T02:46:18.462 INFO:teuthology.task.install:Using flavor: default 2024-04-12T02:46:18.468 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T02:46:18.468 INFO:teuthology.task.install:extra packages: [] 2024-04-12T02:46:18.468 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T02:46:18.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:46:18.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T02:46:18.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:46:18.708 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-12T02:46:18.708 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-12T02:46:18.723 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-12T02:46:18.723 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-12T02:46:19.010 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T02:46:19.010 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:46:19.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T02:46:19.039 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T02:46:19.039 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T02:46:19.083 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T02:46:19.083 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:46:19.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T02:46:19.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T02:46:19.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T02:46:19.114 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T02:46:19.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T02:46:19.184 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T02:46:19.250 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2024-04-12T02:46:19.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2024-04-12T02:46:19.262 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T02:46:19.329 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-04-12T02:46:19.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-04-12T02:46:19.627 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:46:19.707 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:46:19.990 INFO:teuthology.orchestra.run.smithi099.stdout:61 files removed 2024-04-12T02:46:20.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2024-04-12T02:46:20.064 INFO:teuthology.orchestra.run.smithi181.stdout:61 files removed 2024-04-12T02:46:20.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-04-12T02:46:20.374 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:46:20.445 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:46:21.728 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 91 kB/s | 80 kB 00:00 2024-04-12T02:46:21.844 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 89 kB/s | 80 kB 00:00 2024-04-12T02:46:22.337 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-12T02:46:22.428 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 21 kB/s | 11 kB 00:00 2024-04-12T02:46:22.695 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-04-12T02:46:22.778 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 4.6 kB/s | 1.5 kB 00:00 2024-04-12T02:46:22.879 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-04-12T02:46:22.921 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2024-04-12T02:46:23.563 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-04-12T02:46:24.513 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-04-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-04-12T02:46:28.056 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 552 kB/s | 24 kB 00:00 2024-04-12T02:46:28.448 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 67 MB 00:01 2024-04-12T02:46:29.223 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 67 MB 00:01 2024-04-12T02:46:39.073 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 60 MB 00:00 2024-04-12T02:46:39.157 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 25 MB/s | 8.8 MB 00:00 2024-04-12T02:46:41.916 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 60 MB 00:01 2024-04-12T02:46:49.439 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 26 MB/s | 8.8 MB 00:00 2024-04-12T02:46:56.672 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:46:56.674 INFO:teuthology.orchestra.run.smithi099.stdout:==================================================================================================== 2024-04-12T02:46:56.674 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:46:56.674 INFO:teuthology.orchestra.run.smithi099.stdout:==================================================================================================== 2024-04-12T02:46:56.674 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:46:56.674 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-12T02:46:56.675 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T02:46:56.676 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:==================================================================================================== 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:Install 21 Packages 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 59 M 2024-04-12T02:46:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:46:57.522 INFO:teuthology.orchestra.run.smithi099.stdout:(1/23): ceph-base-18.2.2-1187.gd540ebac.el8.x86 8.8 MB/s | 5.5 MB 00:00 2024-04-12T02:46:57.581 INFO:teuthology.orchestra.run.smithi099.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 418 kB/s | 24 kB 00:00 2024-04-12T02:46:57.706 INFO:teuthology.orchestra.run.smithi099.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 6.1 MB/s | 780 kB 00:00 2024-04-12T02:46:57.755 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout:==================================================================================================== 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout:==================================================================================================== 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-04-12T02:46:57.758 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-12T02:46:57.759 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout:==================================================================================================== 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout:Install 21 Packages 2024-04-12T02:46:57.760 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-04-12T02:46:57.761 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:46:57.761 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 59 M 2024-04-12T02:46:57.761 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:46:57.789 INFO:teuthology.orchestra.run.smithi099.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.6 MB/s | 474 kB 00:00 2024-04-12T02:46:58.006 INFO:teuthology.orchestra.run.smithi099.stdout:(5/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8. 12 MB/s | 13 MB 00:01 2024-04-12T02:46:58.073 INFO:teuthology.orchestra.run.smithi099.stdout:(6/23): python3-ceph-argparse-18.2.2-1187.gd540 671 kB/s | 45 kB 00:00 2024-04-12T02:46:58.157 INFO:teuthology.orchestra.run.smithi099.stdout:(7/23): python3-ceph-common-18.2.2-1187.gd540eb 1.5 MB/s | 129 kB 00:00 2024-04-12T02:46:58.232 INFO:teuthology.orchestra.run.smithi099.stdout:(8/23): python3-cephfs-18.2.2-1187.gd540ebac.el 2.6 MB/s | 197 kB 00:00 2024-04-12T02:46:58.299 INFO:teuthology.orchestra.run.smithi099.stdout:(9/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 13 MB/s | 6.6 MB 00:00 2024-04-12T02:46:58.324 INFO:teuthology.orchestra.run.smithi099.stdout:(10/23): python3-rados-18.2.2-1187.gd540ebac.el 4.0 MB/s | 372 kB 00:00 2024-04-12T02:46:58.349 INFO:teuthology.orchestra.run.smithi099.stdout:(11/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 7.1 MB/s | 364 kB 00:00 2024-04-12T02:46:58.375 INFO:teuthology.orchestra.run.smithi099.stdout:(12/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 2.1 MB/s | 107 kB 00:00 2024-04-12T02:46:58.633 INFO:teuthology.orchestra.run.smithi181.stdout:(1/23): ceph-base-18.2.2-1187.gd540ebac.el8.x86 7.7 MB/s | 5.5 MB 00:00 2024-04-12T02:46:58.666 INFO:teuthology.orchestra.run.smithi181.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 728 kB/s | 24 kB 00:00 2024-04-12T02:46:58.742 INFO:teuthology.orchestra.run.smithi181.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 10 MB/s | 780 kB 00:00 2024-04-12T02:46:58.808 INFO:teuthology.orchestra.run.smithi099.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 136 kB/s | 59 kB 00:00 2024-04-12T02:46:58.834 INFO:teuthology.orchestra.run.smithi181.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.0 MB/s | 474 kB 00:00 2024-04-12T02:46:59.050 INFO:teuthology.orchestra.run.smithi099.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 310 kB/s | 75 kB 00:00 2024-04-12T02:46:59.084 INFO:teuthology.orchestra.run.smithi099.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 417 kB/s | 306 kB 00:00 2024-04-12T02:46:59.151 INFO:teuthology.orchestra.run.smithi099.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-12T02:46:59.193 INFO:teuthology.orchestra.run.smithi099.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2024-04-12T02:46:59.235 INFO:teuthology.orchestra.run.smithi099.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-04-12T02:46:59.277 INFO:teuthology.orchestra.run.smithi099.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2024-04-12T02:46:59.342 INFO:teuthology.orchestra.run.smithi181.stdout:(5/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 13 MB/s | 6.6 MB 00:00 2024-04-12T02:46:59.434 INFO:teuthology.orchestra.run.smithi181.stdout:(6/23): python3-ceph-argparse-18.2.2-1187.gd540 488 kB/s | 45 kB 00:00 2024-04-12T02:46:59.494 INFO:teuthology.orchestra.run.smithi099.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.9 MB/s | 1.7 MB 00:00 2024-04-12T02:46:59.534 INFO:teuthology.orchestra.run.smithi181.stdout:(7/23): python3-ceph-common-18.2.2-1187.gd540eb 1.3 MB/s | 129 kB 00:00 2024-04-12T02:46:59.635 INFO:teuthology.orchestra.run.smithi181.stdout:(8/23): python3-cephfs-18.2.2-1187.gd540ebac.el 1.9 MB/s | 197 kB 00:00 2024-04-12T02:46:59.686 INFO:teuthology.orchestra.run.smithi099.stdout:(21/23): librados2-18.2.2-1187.gd540ebac.el8.x8 9.0 MB/s | 3.7 MB 00:00 2024-04-12T02:46:59.727 INFO:teuthology.orchestra.run.smithi181.stdout:(9/23): python3-rados-18.2.2-1187.gd540ebac.el8 4.0 MB/s | 372 kB 00:00 2024-04-12T02:46:59.819 INFO:teuthology.orchestra.run.smithi181.stdout:(10/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 3.9 MB/s | 364 kB 00:00 2024-04-12T02:46:59.902 INFO:teuthology.orchestra.run.smithi181.stdout:(11/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 1.3 MB/s | 107 kB 00:00 2024-04-12T02:47:00.061 INFO:teuthology.orchestra.run.smithi099.stdout:(22/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 6.3 MB/s | 3.6 MB 00:00 2024-04-12T02:47:00.228 INFO:teuthology.orchestra.run.smithi181.stdout:(12/23): ceph-common-18.2.2-1187.gd540ebac.el8. 9.4 MB/s | 22 MB 00:02 2024-04-12T02:47:00.436 INFO:teuthology.orchestra.run.smithi181.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 574 kB/s | 306 kB 00:00 2024-04-12T02:47:00.461 INFO:teuthology.orchestra.run.smithi181.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 253 kB/s | 59 kB 00:00 2024-04-12T02:47:00.486 INFO:teuthology.orchestra.run.smithi181.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-04-12T02:47:00.578 INFO:teuthology.orchestra.run.smithi181.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-04-12T02:47:00.621 INFO:teuthology.orchestra.run.smithi181.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2024-04-12T02:47:00.663 INFO:teuthology.orchestra.run.smithi181.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-04-12T02:47:00.705 INFO:teuthology.orchestra.run.smithi181.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.4 MB/s | 354 kB 00:00 2024-04-12T02:47:00.764 INFO:teuthology.orchestra.run.smithi181.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 5.7 MB/s | 1.7 MB 00:00 2024-04-12T02:47:01.080 INFO:teuthology.orchestra.run.smithi181.stdout:(21/23): librados2-18.2.2-1187.gd540ebac.el8.x8 9.8 MB/s | 3.7 MB 00:00 2024-04-12T02:47:01.139 INFO:teuthology.orchestra.run.smithi181.stdout:(22/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 9.5 MB/s | 3.6 MB 00:00 2024-04-12T02:47:04.786 INFO:teuthology.orchestra.run.smithi099.stdout:(23/23): ceph-common-18.2.2-1187.gd540ebac.el8. 2.7 MB/s | 22 MB 00:07 2024-04-12T02:47:04.789 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:47:04.790 INFO:teuthology.orchestra.run.smithi099.stdout:Total 7.3 MB/s | 59 MB 00:08 2024-04-12T02:47:04.837 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:47:04.882 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:47:04.882 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:47:05.211 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:47:05.220 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:47:05.847 INFO:teuthology.orchestra.run.smithi181.stdout:(23/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8 1.7 MB/s | 13 MB 00:07 2024-04-12T02:47:05.851 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:47:05.852 INFO:teuthology.orchestra.run.smithi181.stdout:Total 7.3 MB/s | 59 MB 00:08 2024-04-12T02:47:05.900 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:47:05.945 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:47:05.946 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:47:06.276 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:47:06.285 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:47:06.434 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:47:06.638 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-12T02:47:06.813 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T02:47:07.202 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T02:47:07.413 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:07.508 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:07.530 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:47:07.669 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T02:47:07.911 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-12T02:47:07.979 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-12T02:47:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T02:47:08.513 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T02:47:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T02:47:08.800 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:08.908 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T02:47:08.929 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:09.098 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T02:47:09.121 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:09.224 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:09.340 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-12T02:47:09.438 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T02:47:09.498 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T02:47:09.550 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T02:47:09.694 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-12T02:47:09.849 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-12T02:47:09.951 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T02:47:10.040 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T02:47:10.167 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:10.245 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:10.271 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T02:47:10.454 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T02:47:10.458 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T02:47:10.512 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T02:47:10.546 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T02:47:10.689 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-12T02:47:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-12T02:47:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-12T02:47:10.886 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-12T02:47:11.110 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T02:47:11.359 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T02:47:11.508 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T02:47:11.603 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T02:47:11.673 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-12T02:47:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-12T02:47:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T02:47:11.902 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T02:47:11.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-12T02:47:12.130 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-12T02:47:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-12T02:47:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-12T02:47:12.933 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T02:47:13.008 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T02:47:13.582 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-12T02:47:13.734 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-12T02:47:15.915 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:16.729 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:17.030 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T02:47:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T02:47:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:17.452 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:17.716 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:18.309 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T02:47:18.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T02:47:18.676 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T02:47:18.990 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:36.544 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:36.544 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2024-04-12T02:47:36.544 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2024-04-12T02:47:36.545 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:47:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T02:47:37.927 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T02:47:37.927 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T02:47:38.167 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T02:47:38.167 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /run 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-04-12T02:47:38.424 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-04-12T02:47:38.425 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-12T02:47:38.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-12T02:47:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-12T02:47:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T02:47:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-12T02:47:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T02:47:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T02:47:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T02:47:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T02:47:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:40.760 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T02:47:40.761 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T02:47:40.762 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-12T02:47:43.789 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:43.789 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.310 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.311 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T02:47:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:47:44.312 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:47:44.506 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2024-04-12T02:47:44.858 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:47:45.638 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 112 kB/s | 4.1 kB 00:00 2024-04-12T02:47:45.800 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T02:47:45.801 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:47:45.908 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 81 kB/s | 4.5 kB 00:00 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T02:47:46.380 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:47:46.382 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T02:47:46.382 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:47:46.382 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:47:46.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-04-12T02:47:46.918 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:47:47.718 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 121 kB/s | 4.1 kB 00:00 2024-04-12T02:47:48.135 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 134 kB/s | 4.5 kB 00:00 2024-04-12T02:47:48.499 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-12T02:47:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 48 M 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 213 M 2024-04-12T02:47:48.501 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:47:48.543 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2024-04-12T02:47:48.635 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-12T02:47:48.694 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2024-04-12T02:47:48.777 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-12T02:47:48.853 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 405 kB/s | 142 kB 00:00 2024-04-12T02:47:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-12T02:47:50.656 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 48 M 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 213 M 2024-04-12T02:47:50.657 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:47:50.701 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2024-04-12T02:47:50.801 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.5 MB/s | 323 kB 00:00 2024-04-12T02:47:50.828 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 20 MB/s | 47 MB 00:02 2024-04-12T02:47:50.830 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:47:50.830 INFO:teuthology.orchestra.run.smithi099.stdout:Total 21 MB/s | 48 MB 00:02 2024-04-12T02:47:50.843 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2024-04-12T02:47:50.870 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:47:50.885 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-04-12T02:47:50.891 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:47:50.892 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:47:50.968 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 458 kB/s | 142 kB 00:00 2024-04-12T02:47:51.159 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:47:51.168 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:47:51.630 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:47:51.778 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T02:47:51.848 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T02:47:52.048 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-12T02:47:52.254 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-12T02:47:52.468 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-12T02:47:52.637 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T02:47:53.127 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 19 MB/s | 47 MB 00:02 2024-04-12T02:47:53.129 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:47:53.129 INFO:teuthology.orchestra.run.smithi181.stdout:Total 19 MB/s | 48 MB 00:02 2024-04-12T02:47:53.169 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:47:53.190 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:47:53.190 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:47:53.458 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:47:53.467 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:47:53.972 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:47:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T02:47:54.430 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-12T02:47:54.661 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-12T02:47:54.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-12T02:47:55.052 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T02:47:57.109 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T02:47:57.383 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-12T02:47:57.645 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-12T02:47:59.553 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T02:47:59.841 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-12T02:48:00.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-12T02:48:02.395 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-12T02:48:02.395 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-7.el8.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:02.932 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:03.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2024-04-12T02:48:03.435 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:04.182 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 114 kB/s | 4.1 kB 00:00 2024-04-12T02:48:04.433 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:48:04.758 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-12T02:48:04.758 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:48:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:05.322 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:48:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-7.el8.x86_64 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:05.323 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:48:05.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-04-12T02:48:05.833 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:06.583 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 138 kB/s | 4.1 kB 00:00 2024-04-12T02:48:06.891 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 119 kB/s | 4.5 kB 00:00 2024-04-12T02:48:07.149 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:48:07.152 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================= 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================= 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-12T02:48:07.153 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-12T02:48:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================= 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:Install 32 Packages 2024-04-12T02:48:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 31 M 2024-04-12T02:48:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 124 M 2024-04-12T02:48:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:48:07.579 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 22 kB/s | 6.4 kB 00:00 2024-04-12T02:48:07.796 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 3.0 MB/s | 1.6 MB 00:00 2024-04-12T02:48:08.138 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 2.7 MB/s | 2.3 MB 00:00 2024-04-12T02:48:08.230 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 2.6 MB/s | 245 kB 00:00 2024-04-12T02:48:08.497 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 163 kB/s | 44 kB 00:00 2024-04-12T02:48:08.706 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86_ 20 MB/s | 18 MB 00:00 2024-04-12T02:48:08.731 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 794 kB/s | 185 kB 00:00 2024-04-12T02:48:08.898 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.3 MB/s | 384 kB 00:00 2024-04-12T02:48:08.973 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-04-12T02:48:09.049 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-04-12T02:48:09.074 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 470 kB/s | 173 kB 00:00 2024-04-12T02:48:09.124 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2024-04-12T02:48:09.158 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 700 kB/s | 59 kB 00:00 2024-04-12T02:48:09.217 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-12T02:48:09.242 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-04-12T02:48:09.292 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2024-04-12T02:48:09.367 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86 2.7 MB/s | 4.8 MB 00:01 2024-04-12T02:48:09.392 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 159 kB/s | 24 kB 00:00 2024-04-12T02:48:09.418 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-04-12T02:48:09.443 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-04-12T02:48:09.464 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================= 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================= 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-12T02:48:09.467 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T02:48:09.468 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T02:48:09.469 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================= 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout:Install 32 Packages 2024-04-12T02:48:09.470 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:09.471 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 M 2024-04-12T02:48:09.471 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 124 M 2024-04-12T02:48:09.471 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:48:09.502 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-04-12T02:48:09.536 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 930 kB/s | 85 kB 00:00 2024-04-12T02:48:09.577 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 325 kB/s | 23 kB 00:00 2024-04-12T02:48:09.628 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2024-04-12T02:48:09.661 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2024-04-12T02:48:09.695 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): python3-cryptography-3.2.1-7.el8_9.x86 8.2 MB/s | 559 kB 00:00 2024-04-12T02:48:09.728 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.7 MB/s | 457 kB 00:00 2024-04-12T02:48:09.754 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 663 kB/s | 39 kB 00:00 2024-04-12T02:48:09.787 INFO:teuthology.orchestra.run.smithi181.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2024-04-12T02:48:09.787 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 1.7 MB/s | 103 kB 00:00 2024-04-12T02:48:09.813 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2024-04-12T02:48:09.846 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 587 kB/s | 251 kB 00:00 2024-04-12T02:48:09.871 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 649 kB/s | 54 kB 00:00 2024-04-12T02:48:09.876 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:09.876 INFO:teuthology.orchestra.run.smithi099.stdout:Total 11 MB/s | 31 MB 00:02 2024-04-12T02:48:09.971 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:48:10.009 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:48:10.009 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:48:10.037 INFO:teuthology.orchestra.run.smithi181.stdout:(2/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 5.9 MB/s | 2.3 MB 00:00 2024-04-12T02:48:10.070 INFO:teuthology.orchestra.run.smithi181.stdout:(3/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 3.7 MB/s | 1.6 MB 00:00 2024-04-12T02:48:10.162 INFO:teuthology.orchestra.run.smithi181.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 2.6 MB/s | 245 kB 00:00 2024-04-12T02:48:10.233 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:48:10.242 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:48:10.671 INFO:teuthology.orchestra.run.smithi181.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 86 kB/s | 44 kB 00:00 2024-04-12T02:48:10.746 INFO:teuthology.orchestra.run.smithi181.stdout:(6/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86_ 5.0 MB/s | 4.8 MB 00:00 2024-04-12T02:48:11.030 INFO:teuthology.orchestra.run.smithi181.stdout:(7/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86_ 19 MB/s | 18 MB 00:00 2024-04-12T02:48:11.055 INFO:teuthology.orchestra.run.smithi181.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 483 kB/s | 185 kB 00:00 2024-04-12T02:48:11.181 INFO:teuthology.orchestra.run.smithi181.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-04-12T02:48:11.297 INFO:teuthology.orchestra.run.smithi181.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 151 kB/s | 18 kB 00:00 2024-04-12T02:48:11.340 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:48:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 418 kB/s | 49 kB 00:00 2024-04-12T02:48:11.456 INFO:teuthology.orchestra.run.smithi181.stdout:(12/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 243 kB/s | 173 kB 00:00 2024-04-12T02:48:11.532 INFO:teuthology.orchestra.run.smithi181.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 503 kB/s | 59 kB 00:00 2024-04-12T02:48:11.632 INFO:teuthology.orchestra.run.smithi181.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-04-12T02:48:11.674 INFO:teuthology.orchestra.run.smithi181.stdout:(15/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-04-12T02:48:11.707 INFO:teuthology.orchestra.run.smithi181.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 568 kB/s | 384 kB 00:00 2024-04-12T02:48:11.766 INFO:teuthology.orchestra.run.smithi181.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-04-12T02:48:11.783 INFO:teuthology.orchestra.run.smithi181.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 219 kB/s | 24 kB 00:00 2024-04-12T02:48:11.808 INFO:teuthology.orchestra.run.smithi181.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-04-12T02:48:11.875 INFO:teuthology.orchestra.run.smithi181.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-04-12T02:48:11.925 INFO:teuthology.orchestra.run.smithi181.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-04-12T02:48:11.951 INFO:teuthology.orchestra.run.smithi181.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2024-04-12T02:48:12.041 INFO:teuthology.orchestra.run.smithi181.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 518 kB/s | 85 kB 00:00 2024-04-12T02:48:12.060 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T02:48:12.068 INFO:teuthology.orchestra.run.smithi181.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 161 kB/s | 23 kB 00:00 2024-04-12T02:48:12.102 INFO:teuthology.orchestra.run.smithi181.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 725 kB/s | 109 kB 00:00 2024-04-12T02:48:12.135 INFO:teuthology.orchestra.run.smithi181.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2024-04-12T02:48:12.169 INFO:teuthology.orchestra.run.smithi181.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 7.3 MB/s | 457 kB 00:00 2024-04-12T02:48:12.194 INFO:teuthology.orchestra.run.smithi181.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 664 kB/s | 39 kB 00:00 2024-04-12T02:48:12.219 INFO:teuthology.orchestra.run.smithi181.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2024-04-12T02:48:12.245 INFO:teuthology.orchestra.run.smithi181.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2024-04-12T02:48:12.270 INFO:teuthology.orchestra.run.smithi181.stdout:(31/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2024-04-12T02:48:12.303 INFO:teuthology.orchestra.run.smithi181.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 2.4 MB/s | 559 kB 00:00 2024-04-12T02:48:12.308 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:12.308 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 31 MB 00:02 2024-04-12T02:48:12.402 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:48:12.439 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:48:12.440 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:48:12.456 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T02:48:12.604 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-12T02:48:12.665 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:48:12.674 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:48:12.832 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-12T02:48:13.169 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-12T02:48:13.362 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-12T02:48:13.553 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-12T02:48:13.934 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-12T02:48:14.251 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-12T02:48:14.334 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:48:14.406 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-12T02:48:14.582 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-12T02:48:14.587 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T02:48:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T02:48:14.838 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-12T02:48:14.960 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-12T02:48:15.061 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-12T02:48:15.201 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-12T02:48:15.271 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-12T02:48:15.461 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-12T02:48:15.555 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-12T02:48:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-12T02:48:15.794 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-12T02:48:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-12T02:48:15.970 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-12T02:48:15.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-12T02:48:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-12T02:48:16.356 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-12T02:48:16.401 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-12T02:48:16.645 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T02:48:16.668 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-12T02:48:16.831 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-12T02:48:16.901 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T02:48:17.024 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-12T02:48:17.144 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T02:48:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-12T02:48:17.382 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-12T02:48:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-12T02:48:17.672 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-12T02:48:17.805 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-12T02:48:17.977 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-12T02:48:18.026 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T02:48:18.201 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-12T02:48:18.376 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-12T02:48:18.401 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-12T02:48:18.554 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-12T02:48:18.563 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-12T02:48:18.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-12T02:48:19.170 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T02:48:19.719 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T02:48:20.004 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T02:48:20.203 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T02:48:20.258 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-12T02:48:20.438 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T02:48:20.546 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-12T02:48:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T02:48:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T02:48:21.122 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T02:48:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-12T02:48:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T02:48:21.471 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T02:48:21.506 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T02:48:21.524 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T02:48:21.646 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T02:48:22.042 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T02:48:22.042 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-12T02:48:22.042 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-12T02:48:22.042 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-12T02:48:22.042 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-12T02:48:22.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-12T02:48:22.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-12T02:48:23.110 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T02:48:23.329 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T02:48:23.809 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T02:48:23.997 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T02:48:24.211 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T02:48:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T02:48:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T02:48:24.546 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-12T02:48:24.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-12T02:48:24.941 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-12T02:48:24.942 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-12T02:48:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-12T02:48:27.301 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:27.828 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T02:48:27.829 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T02:48:27.830 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T02:48:27.831 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:27.831 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:28.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2024-04-12T02:48:28.369 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:29.786 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 119 kB/s | 4.1 kB 00:00 2024-04-12T02:48:30.044 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 134 kB/s | 4.5 kB 00:00 2024-04-12T02:48:30.054 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-12T02:48:30.054 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:48:31.296 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:31.296 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:48:31.296 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:31.296 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:31.296 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T02:48:31.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T02:48:31.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:31.299 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:48:32.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-04-12T02:48:32.455 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:48:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:48:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:48:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:32.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2024-04-12T02:48:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:33.296 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:48:33.596 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 kB/s | 4.5 kB 00:00 2024-04-12T02:48:33.997 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:48:34.256 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 kB/s | 4.5 kB 00:00 2024-04-12T02:48:35.926 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:48:36.110 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:48:36.111 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:48:36.111 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:48:36.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-04-12T02:48:36.593 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:36.855 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:48:36.855 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:36.855 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:48:36.855 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:36.855 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:36.856 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 170 k 2024-04-12T02:48:36.857 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 692 k 2024-04-12T02:48:36.857 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:48:37.052 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 872 kB/s | 170 kB 00:00 2024-04-12T02:48:37.052 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:37.053 INFO:teuthology.orchestra.run.smithi099.stdout:Total 867 kB/s | 170 kB 00:00 2024-04-12T02:48:37.053 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:48:37.060 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:48:37.061 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:48:37.063 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:48:37.071 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:48:37.112 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:48:37.438 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:48:37.746 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:48:38.784 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:39.163 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:40.239 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:48:40.239 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:40.239 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:48:40.239 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:48:40.240 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:40.241 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-04-12T02:48:40.241 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 692 k 2024-04-12T02:48:40.241 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:48:40.426 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 917 kB/s | 170 kB 00:00 2024-04-12T02:48:40.427 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:40.427 INFO:teuthology.orchestra.run.smithi181.stdout:Total 911 kB/s | 170 kB 00:00 2024-04-12T02:48:40.427 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:48:40.435 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:48:40.435 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:48:40.438 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:48:40.446 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:48:40.488 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:48:42.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:42.505 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:42.706 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:43.802 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:43.802 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:48:44.335 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:44.335 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:48:44.335 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:48:44.335 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:44.335 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:44.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T02:48:44.859 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:45.714 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 137 kB/s | 4.1 kB 00:00 2024-04-12T02:48:45.953 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 176 kB/s | 4.5 kB 00:00 2024-04-12T02:48:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T02:48:47.210 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:48:48.026 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:48.026 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:48:48.026 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:48:48.026 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:48.026 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:48:48.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T02:48:48.540 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:48.541 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 165 k 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 503 k 2024-04-12T02:48:48.542 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:48:48.545 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:49.018 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 348 kB/s | 165 kB 00:00 2024-04-12T02:48:49.019 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:49.019 INFO:teuthology.orchestra.run.smithi099.stdout:Total 347 kB/s | 165 kB 00:00 2024-04-12T02:48:49.019 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:48:49.030 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:48:49.030 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:48:49.083 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:48:49.092 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:48:49.205 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:48:49.353 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 115 kB/s | 4.1 kB 00:00 2024-04-12T02:48:49.366 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:48:49.646 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 126 kB/s | 4.5 kB 00:00 2024-04-12T02:48:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:48:52.148 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:48:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:48:52.150 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 165 k 2024-04-12T02:48:52.150 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 503 k 2024-04-12T02:48:52.150 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:48:52.451 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 550 kB/s | 165 kB 00:00 2024-04-12T02:48:52.452 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:48:52.452 INFO:teuthology.orchestra.run.smithi181.stdout:Total 547 kB/s | 165 kB 00:00 2024-04-12T02:48:52.452 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:48:52.463 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:48:52.463 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:48:52.517 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:48:52.526 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:48:52.639 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:48:52.818 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:48:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:48:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:48:54.581 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:48:55.138 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:55.138 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:48:55.138 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:48:55.138 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:48:55.139 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:55.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2024-04-12T02:48:55.641 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:48:56.611 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-12T02:48:56.939 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-12T02:48:59.455 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:48:59.641 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:48:59.642 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:48:59.642 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:48:59.786 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T02:49:00.131 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T02:49:00.272 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:49:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:49:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:49:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:49:00.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-04-12T02:49:01.326 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:02.005 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T02:49:02.336 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-12T02:49:03.085 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-04-12T02:49:03.409 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 109 kB/s | 4.5 kB 00:00 2024-04-12T02:49:04.922 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:49:04.923 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:49:04.923 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:49:04.923 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:Install 5 Packages 2024-04-12T02:49:04.924 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:04.925 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.8 M 2024-04-12T02:49:04.925 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 87 M 2024-04-12T02:49:04.925 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:49:05.265 INFO:teuthology.orchestra.run.smithi099.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1187.gd540 55 kB/s | 15 kB 00:00 2024-04-12T02:49:05.290 INFO:teuthology.orchestra.run.smithi099.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1187.gd54 83 kB/s | 24 kB 00:00 2024-04-12T02:49:05.348 INFO:teuthology.orchestra.run.smithi099.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 10 MB/s | 3.5 MB 00:00 2024-04-12T02:49:05.432 INFO:teuthology.orchestra.run.smithi099.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2024-04-12T02:49:05.490 INFO:teuthology.orchestra.run.smithi099.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 982 kB/s | 196 kB 00:00 2024-04-12T02:49:05.492 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:05.492 INFO:teuthology.orchestra.run.smithi099.stdout:Total 6.7 MB/s | 3.8 MB 00:00 2024-04-12T02:49:05.493 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:49:05.500 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:49:05.500 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:49:05.567 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:49:05.576 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:49:05.748 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:49:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:49:05.942 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:49:05.943 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:49:05.943 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:49:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-12T02:49:06.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T02:49:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-12T02:49:06.385 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T02:49:06.441 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:07.549 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-12T02:49:07.793 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T02:49:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T02:49:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-12T02:49:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-12T02:49:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T02:49:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-12T02:49:08.147 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T02:49:08.584 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-12T02:49:11.081 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:49:11.082 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:49:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:49:11.082 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:49:11.082 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:49:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout:Install 5 Packages 2024-04-12T02:49:11.083 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:11.084 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.8 M 2024-04-12T02:49:11.084 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 87 M 2024-04-12T02:49:11.084 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:49:11.338 INFO:teuthology.orchestra.run.smithi181.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1187.gd54 142 kB/s | 24 kB 00:00 2024-04-12T02:49:11.368 INFO:teuthology.orchestra.run.smithi181.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1187.gd540 74 kB/s | 15 kB 00:00 2024-04-12T02:49:11.543 INFO:teuthology.orchestra.run.smithi181.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 165 kB/s | 34 kB 00:00 2024-04-12T02:49:11.601 INFO:teuthology.orchestra.run.smithi181.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 8.1 MB/s | 3.5 MB 00:00 2024-04-12T02:49:11.652 INFO:teuthology.orchestra.run.smithi181.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 693 kB/s | 196 kB 00:00 2024-04-12T02:49:11.653 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:11.653 INFO:teuthology.orchestra.run.smithi181.stdout:Total 6.6 MB/s | 3.8 MB 00:00 2024-04-12T02:49:11.654 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:49:11.661 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:49:11.661 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:49:11.728 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:49:11.737 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:49:11.970 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:49:12.178 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-12T02:49:12.406 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-12T02:49:12.580 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T02:49:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-12T02:49:12.872 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:49:13.427 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:13.427 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:49:13.427 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:13.427 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:13.427 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:13.428 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T02:49:13.428 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T02:49:13.428 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:13.428 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:49:13.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T02:49:13.736 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-12T02:49:13.962 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T02:49:14.239 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T02:49:14.239 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-12T02:49:14.239 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-12T02:49:14.239 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T02:49:14.239 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-12T02:49:14.768 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 109 kB/s | 4.1 kB 00:00 2024-04-12T02:49:15.015 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 kB/s | 4.5 kB 00:00 2024-04-12T02:49:17.610 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:49:17.611 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================================= 2024-04-12T02:49:17.611 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:49:17.611 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================================= 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:49:17.612 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================================= 2024-04-12T02:49:17.613 INFO:teuthology.orchestra.run.smithi099.stdout:Install 8 Packages 2024-04-12T02:49:17.613 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:17.613 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 36 M 2024-04-12T02:49:17.613 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 207 M 2024-04-12T02:49:17.613 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:49:17.738 INFO:teuthology.orchestra.run.smithi099.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-04-12T02:49:17.772 INFO:teuthology.orchestra.run.smithi099.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-04-12T02:49:17.856 INFO:teuthology.orchestra.run.smithi099.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2024-04-12T02:49:17.881 INFO:teuthology.orchestra.run.smithi099.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-04-12T02:49:18.070 INFO:teuthology.orchestra.run.smithi099.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 24 MB/s | 5.2 MB 00:00 2024-04-12T02:49:18.307 INFO:teuthology.orchestra.run.smithi099.stdout:(6/8): ceph-mgr-diskprediction-local-18.2.2-118 11 MB/s | 7.4 MB 00:00 2024-04-12T02:49:18.415 INFO:teuthology.orchestra.run.smithi099.stdout:(7/8): openblas-0.3.15-6.el8.x86_64.rpm 9.5 MB/s | 5.0 MB 00:00 2024-04-12T02:49:18.599 INFO:teuthology.orchestra.run.smithi099.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 26 MB/s | 14 MB 00:00 2024-04-12T02:49:18.600 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:18.600 INFO:teuthology.orchestra.run.smithi099.stdout:Total 37 MB/s | 36 MB 00:00 2024-04-12T02:49:18.846 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:49:18.865 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:49:18.865 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:49:19.175 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:49:19.183 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:49:19.349 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-12T02:49:19.349 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:49:19.736 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:19.897 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:49:19.925 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T02:49:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T02:49:20.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T02:49:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:20.423 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:20.807 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T02:49:21.253 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 113 kB/s | 4.1 kB 00:00 2024-04-12T02:49:21.555 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 kB/s | 4.5 kB 00:00 2024-04-12T02:49:21.579 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T02:49:21.767 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T02:49:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T02:49:22.779 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T02:49:22.991 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T02:49:24.065 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================================= 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================================= 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:49:24.066 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================================= 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout:Install 8 Packages 2024-04-12T02:49:24.067 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:24.068 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 36 M 2024-04-12T02:49:24.068 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 207 M 2024-04-12T02:49:24.068 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:49:24.159 INFO:teuthology.orchestra.run.smithi181.stdout:(1/8): libgfortran-8.5.0-20.el8.x86_64.rpm 7.0 MB/s | 645 kB 00:00 2024-04-12T02:49:24.193 INFO:teuthology.orchestra.run.smithi181.stdout:(2/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-04-12T02:49:24.235 INFO:teuthology.orchestra.run.smithi181.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 5.3 MB/s | 225 kB 00:00 2024-04-12T02:49:24.310 INFO:teuthology.orchestra.run.smithi181.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2024-04-12T02:49:24.443 INFO:teuthology.orchestra.run.smithi181.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.2 MB 00:00 2024-04-12T02:49:24.527 INFO:teuthology.orchestra.run.smithi181.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-04-12T02:49:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T02:49:24.752 INFO:teuthology.orchestra.run.smithi181.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 44 MB/s | 14 MB 00:00 2024-04-12T02:49:25.227 INFO:teuthology.orchestra.run.smithi181.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-118 6.4 MB/s | 7.4 MB 00:01 2024-04-12T02:49:25.228 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:25.228 INFO:teuthology.orchestra.run.smithi181.stdout:Total 31 MB/s | 36 MB 00:01 2024-04-12T02:49:25.477 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:49:25.496 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:49:25.496 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:49:25.807 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:49:25.815 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:49:25.991 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-12T02:49:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T02:49:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-12T02:49:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-12T02:49:26.496 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:49:26.662 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T02:49:26.807 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T02:49:26.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:27.754 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T02:49:28.357 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T02:49:28.587 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T02:49:29.265 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T02:49:29.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T02:49:29.853 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T02:49:31.448 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-12T02:49:31.448 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:49:31.595 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T02:49:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:31.995 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:49:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-12T02:49:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:31.996 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:49:32.198 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2024-04-12T02:49:32.548 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:32.870 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-12T02:49:33.094 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T02:49:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-12T02:49:33.418 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-12T02:49:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 130 kB/s | 4.5 kB 00:00 2024-04-12T02:49:36.147 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-12T02:49:36.149 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:36.150 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout:Install 14 Packages 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 2.0 M 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 25 M 2024-04-12T02:49:36.151 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:49:36.339 INFO:teuthology.orchestra.run.smithi099.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 124 kB/s | 16 kB 00:00 2024-04-12T02:49:36.365 INFO:teuthology.orchestra.run.smithi099.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 211 kB/s | 33 kB 00:00 2024-04-12T02:49:36.456 INFO:teuthology.orchestra.run.smithi099.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 694 kB/s | 81 kB 00:00 2024-04-12T02:49:36.523 INFO:teuthology.orchestra.run.smithi099.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 966 kB/s | 64 kB 00:00 2024-04-12T02:49:36.599 INFO:teuthology.orchestra.run.smithi099.stdout:(5/14): python3-kubernetes-11.0.0-6.el8.noarch. 5.0 MB/s | 1.2 MB 00:00 2024-04-12T02:49:36.624 INFO:teuthology.orchestra.run.smithi099.stdout:(6/14): python3-websocket-client-0.56.0-5.el8.n 610 kB/s | 61 kB 00:00 2024-04-12T02:49:36.691 INFO:teuthology.orchestra.run.smithi099.stdout:(7/14): python3-jwt-1.6.1-2.el8.noarch.rpm 464 kB/s | 43 kB 00:00 2024-04-12T02:49:36.708 INFO:teuthology.orchestra.run.smithi099.stdout:(8/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.8 MB/s | 155 kB 00:00 2024-04-12T02:49:36.733 INFO:teuthology.orchestra.run.smithi099.stdout:(9/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 94 kB/s | 49 kB 00:00 2024-04-12T02:49:36.758 INFO:teuthology.orchestra.run.smithi099.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 632 kB/s | 43 kB 00:00 2024-04-12T02:49:36.792 INFO:teuthology.orchestra.run.smithi099.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 237 kB/s | 20 kB 00:00 2024-04-12T02:49:36.809 INFO:teuthology.orchestra.run.smithi099.stdout:(12/14): python3-pyasn1-modules-0.3.7-6.el8.noa 1.4 MB/s | 110 kB 00:00 2024-04-12T02:49:36.834 INFO:teuthology.orchestra.run.smithi099.stdout:(13/14): python3-jsonpatch-1.21-2.el8.noarch.rp 378 kB/s | 27 kB 00:00 2024-04-12T02:49:36.859 INFO:teuthology.orchestra.run.smithi099.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.8 MB/s | 126 kB 00:00 2024-04-12T02:49:36.860 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:36.860 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-04-12T02:49:36.930 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:49:36.946 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:49:36.946 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:49:37.027 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:49:37.035 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:49:37.551 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:49:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-12T02:49:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-12T02:49:38.055 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-12T02:49:38.055 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:49:38.178 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-12T02:49:38.403 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-12T02:49:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:38.614 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:38.615 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:49:38.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-04-12T02:49:39.125 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:39.283 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-12T02:49:39.581 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-12T02:49:39.843 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-12T02:49:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-12T02:49:40.074 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-12T02:49:40.197 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:49:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-12T02:49:40.469 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-12T02:49:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-12T02:49:41.814 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-12T02:49:42.039 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-12T02:49:42.195 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-12T02:49:42.467 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-12T02:49:42.468 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-12T02:49:42.468 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-12T02:49:42.468 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-12T02:49:42.468 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-12T02:49:42.717 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T02:49:42.719 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:Install 14 Packages 2024-04-12T02:49:42.720 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-04-12T02:49:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 25 M 2024-04-12T02:49:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:49:43.168 INFO:teuthology.orchestra.run.smithi181.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 127 kB/s | 16 kB 00:00 2024-04-12T02:49:43.227 INFO:teuthology.orchestra.run.smithi181.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 176 kB/s | 33 kB 00:00 2024-04-12T02:49:43.252 INFO:teuthology.orchestra.run.smithi181.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 978 kB/s | 81 kB 00:00 2024-04-12T02:49:43.294 INFO:teuthology.orchestra.run.smithi181.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2024-04-12T02:49:43.344 INFO:teuthology.orchestra.run.smithi181.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 1.2 MB/s | 61 kB 00:00 2024-04-12T02:49:43.378 INFO:teuthology.orchestra.run.smithi181.stdout:(6/14): python3-kubernetes-11.0.0-6.el8.noarch. 7.7 MB/s | 1.2 MB 00:00 2024-04-12T02:49:43.436 INFO:teuthology.orchestra.run.smithi181.stdout:(7/14): python3-jwt-1.6.1-2.el8.noarch.rpm 463 kB/s | 43 kB 00:00 2024-04-12T02:49:43.462 INFO:teuthology.orchestra.run.smithi181.stdout:(8/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 116 kB/s | 49 kB 00:00 2024-04-12T02:49:43.487 INFO:teuthology.orchestra.run.smithi181.stdout:(9/14): python3-requests-oauthlib-1.0.0-1.el8.n 844 kB/s | 43 kB 00:00 2024-04-12T02:49:43.538 INFO:teuthology.orchestra.run.smithi181.stdout:(10/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.0 MB/s | 155 kB 00:00 2024-04-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi181.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 199 kB/s | 20 kB 00:00 2024-04-12T02:49:43.588 INFO:teuthology.orchestra.run.smithi181.stdout:(12/14): python3-pyasn1-modules-0.3.7-6.el8.noa 1.1 MB/s | 110 kB 00:00 2024-04-12T02:49:43.613 INFO:teuthology.orchestra.run.smithi181.stdout:(13/14): python3-jsonpatch-1.21-2.el8.noarch.rp 352 kB/s | 27 kB 00:00 2024-04-12T02:49:43.638 INFO:teuthology.orchestra.run.smithi181.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-04-12T02:49:43.640 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:43.640 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-12T02:49:43.712 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:49:43.727 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:49:43.727 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:49:43.809 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:49:43.818 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:49:44.348 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:49:44.601 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-12T02:49:44.862 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-12T02:49:45.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-12T02:49:45.284 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-12T02:49:45.496 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-12T02:49:45.784 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-12T02:49:46.082 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-12T02:49:47.083 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-12T02:49:47.333 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-12T02:49:47.501 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-12T02:49:47.501 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:49:47.524 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-12T02:49:47.790 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T02:49:48.069 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:48.070 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:49:48.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T02:49:48.609 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:48.887 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-12T02:49:49.203 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-12T02:49:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T02:49:49.441 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 kB/s | 4.1 kB 00:00 2024-04-12T02:49:49.647 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T02:49:49.647 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-12T02:49:49.647 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-12T02:49:49.647 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-12T02:49:49.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-12T02:49:49.706 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:49:52.326 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:49:52.327 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================================== 2024-04-12T02:49:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================================== 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================================== 2024-04-12T02:49:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:Install 5 Packages 2024-04-12T02:49:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:49:52.329 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 6.0 M 2024-04-12T02:49:52.329 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2024-04-12T02:49:52.329 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:49:52.569 INFO:teuthology.orchestra.run.smithi099.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 410 kB/s | 60 kB 00:00 2024-04-12T02:49:52.653 INFO:teuthology.orchestra.run.smithi099.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2024-04-12T02:49:52.761 INFO:teuthology.orchestra.run.smithi099.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2024-04-12T02:49:52.795 INFO:teuthology.orchestra.run.smithi099.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.7 MB/s | 538 kB 00:00 2024-04-12T02:49:53.020 INFO:teuthology.orchestra.run.smithi099.stdout:(5/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 231 kB/s | 138 kB 00:00 2024-04-12T02:49:53.020 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:49:53.020 INFO:teuthology.orchestra.run.smithi099.stdout:Total 8.6 MB/s | 6.0 MB 00:00 2024-04-12T02:49:53.076 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:49:53.087 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:49:53.087 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:49:53.175 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:49:53.184 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:49:54.169 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:49:54.513 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-12T02:49:54.720 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-12T02:49:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:49:54.761 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-12T02:49:55.286 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T02:49:55.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T02:49:55.288 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T02:49:55.288 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T02:49:55.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:55.288 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:49:55.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T02:49:55.660 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T02:49:55.830 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:49:56.660 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:49:56.960 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:49:57.127 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T02:49:57.309 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T02:49:57.589 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T02:49:57.589 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-12T02:49:57.589 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T02:49:57.589 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T02:49:57.589 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-12T02:49:59.512 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================================== 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================================== 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:49:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================================== 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout:Install 5 Packages 2024-04-12T02:49:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:49:59.515 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 6.0 M 2024-04-12T02:49:59.515 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-04-12T02:49:59.515 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:50:00.357 INFO:teuthology.orchestra.run.smithi181.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 122 kB/s | 60 kB 00:00 2024-04-12T02:50:00.524 INFO:teuthology.orchestra.run.smithi181.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2024-04-12T02:50:00.549 INFO:teuthology.orchestra.run.smithi181.stdout:(3/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 201 kB/s | 138 kB 00:00 2024-04-12T02:50:00.574 INFO:teuthology.orchestra.run.smithi181.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 669 kB/s | 476 kB 00:00 2024-04-12T02:50:00.599 INFO:teuthology.orchestra.run.smithi181.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.0 MB/s | 538 kB 00:00 2024-04-12T02:50:00.602 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:00.602 INFO:teuthology.orchestra.run.smithi181.stdout:Total 5.5 MB/s | 6.0 MB 00:01 2024-04-12T02:50:00.660 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:50:00.672 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:50:00.672 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:50:00.761 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:50:00.770 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:50:01.749 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:50:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-12T02:50:02.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-12T02:50:02.393 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-12T02:50:02.393 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:50:02.672 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T02:50:02.944 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:02.945 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:03.006 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T02:50:03.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2024-04-12T02:50:03.470 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:03.657 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T02:50:04.309 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T02:50:04.309 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-12T02:50:04.309 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T02:50:04.309 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T02:50:04.310 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-12T02:50:04.314 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 101 kB/s | 4.1 kB 00:00 2024-04-12T02:50:04.562 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 119 kB/s | 4.5 kB 00:00 2024-04-12T02:50:07.154 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-12T02:50:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 942 k 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 3.0 M 2024-04-12T02:50:07.156 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:50:07.340 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 452 kB/s | 83 kB 00:00 2024-04-12T02:50:08.782 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 528 kB/s | 859 kB 00:01 2024-04-12T02:50:08.783 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:08.783 INFO:teuthology.orchestra.run.smithi099.stdout:Total 579 kB/s | 942 kB 00:01 2024-04-12T02:50:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:50:08.820 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:50:08.820 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:50:08.910 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:50:08.919 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:50:09.087 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:50:09.339 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-12T02:50:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T02:50:09.891 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-12T02:50:09.891 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:50:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T02:50:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-12T02:50:10.447 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:10.447 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:50:10.447 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:50:10.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T02:50:10.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T02:50:10.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-12T02:50:10.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T02:50:10.448 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:10.448 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:50:10.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-04-12T02:50:11.001 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:11.924 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 109 kB/s | 4.1 kB 00:00 2024-04-12T02:50:12.218 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:50:14.733 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:50:14.733 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-04-12T02:50:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:14.735 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 942 k 2024-04-12T02:50:14.735 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 3.0 M 2024-04-12T02:50:14.735 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:50:14.756 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-12T02:50:14.756 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:50:14.803 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-04-12T02:50:15.195 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 1.8 MB/s | 859 kB 00:00 2024-04-12T02:50:15.195 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:15.196 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.0 MB/s | 942 kB 00:00 2024-04-12T02:50:15.207 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:50:15.229 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:50:15.229 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:50:15.309 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:15.309 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:50:15.309 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-12T02:50:15.309 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:15.309 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:15.313 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:50:15.322 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:50:15.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-volume 2024-04-12T02:50:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:50:15.779 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-12T02:50:15.833 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:15.970 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T02:50:16.356 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T02:50:16.356 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-12T02:50:16.664 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 99 kB/s | 4.1 kB 00:00 2024-04-12T02:50:16.936 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 4.5 kB 00:00 2024-04-12T02:50:19.540 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:19.541 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================================== 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 311 k 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.4 M 2024-04-12T02:50:19.542 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:50:19.664 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 395 kB/s | 48 kB 00:00 2024-04-12T02:50:19.940 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 662 kB/s | 263 kB 00:00 2024-04-12T02:50:19.940 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:19.940 INFO:teuthology.orchestra.run.smithi099.stdout:Total 780 kB/s | 311 kB 00:00 2024-04-12T02:50:19.951 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:50:19.963 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:50:19.963 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:50:19.999 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:50:20.008 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:50:20.137 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:50:20.434 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-12T02:50:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T02:50:21.121 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T02:50:21.121 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-12T02:50:23.218 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-12T02:50:23.218 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:50:23.780 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:23.781 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:50:23.781 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-12T02:50:23.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:23.781 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:50:23.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-04-12T02:50:24.349 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:25.191 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 kB/s | 4.1 kB 00:00 2024-04-12T02:50:25.491 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 119 kB/s | 4.5 kB 00:00 2024-04-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:26.773 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:26.930 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2024-04-12T02:50:27.292 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:50:28.002 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================================== 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-04-12T02:50:28.003 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:28.004 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 311 k 2024-04-12T02:50:28.004 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.4 M 2024-04-12T02:50:28.004 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:50:28.169 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 292 kB/s | 48 kB 00:00 2024-04-12T02:50:28.394 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 673 kB/s | 263 kB 00:00 2024-04-12T02:50:28.395 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:28.395 INFO:teuthology.orchestra.run.smithi181.stdout:Total 794 kB/s | 311 kB 00:00 2024-04-12T02:50:28.407 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:50:28.419 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:50:28.419 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:50:28.456 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:50:28.465 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:50:28.602 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:50:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-12T02:50:29.113 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-12T02:50:29.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T02:50:29.437 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-12T02:50:29.640 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T02:50:29.640 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-12T02:50:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:50:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:50:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:32.041 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 139 k 2024-04-12T02:50:32.041 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 513 k 2024-04-12T02:50:32.041 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:50:32.247 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 673 kB/s | 139 kB 00:00 2024-04-12T02:50:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Total 669 kB/s | 139 kB 00:00 2024-04-12T02:50:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:50:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:50:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:50:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:50:32.318 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:50:32.434 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:50:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:32.916 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:36.691 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-12T02:50:36.691 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:50:37.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-04-12T02:50:37.790 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:38.841 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 105 kB/s | 4.1 kB 00:00 2024-04-12T02:50:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:39.029 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:50:39.150 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 kB/s | 4.5 kB 00:00 2024-04-12T02:50:39.582 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:39.582 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:50:39.582 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:50:39.582 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:39.583 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:39.742 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2024-04-12T02:50:40.101 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:41.670 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:50:41.671 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:41.671 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 139 k 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 513 k 2024-04-12T02:50:41.672 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:50:41.908 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T02:50:42.094 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 330 kB/s | 139 kB 00:00 2024-04-12T02:50:42.094 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:42.094 INFO:teuthology.orchestra.run.smithi181.stdout:Total 329 kB/s | 139 kB 00:00 2024-04-12T02:50:42.095 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:50:42.100 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:50:42.100 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:50:42.158 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:50:42.167 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:50:42.273 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-12T02:50:42.293 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:50:42.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:42.749 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:44.796 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:50:44.987 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:50:44.989 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:50:44.989 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:45.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2024-04-12T02:50:45.497 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:47.366 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-12T02:50:47.686 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-12T02:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 31 k 2024-04-12T02:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 139 k 2024-04-12T02:50:50.283 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:50:50.537 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 121 kB/s | 31 kB 00:00 2024-04-12T02:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout:Total 121 kB/s | 31 kB 00:00 2024-04-12T02:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:50:50.542 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:50:50.542 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:50:50.554 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:50:50.562 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:50:50.624 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:50:50.716 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:50.717 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:50:50.803 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:51.048 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:51.267 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:50:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:50:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:50:51.267 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:50:51.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-04-12T02:50:51.791 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:52.745 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 kB/s | 4.1 kB 00:00 2024-04-12T02:50:53.129 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-12T02:50:55.445 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:50:55.633 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:50:55.634 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:50:55.634 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:50:55.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-04-12T02:50:56.135 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:50:58.030 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-12T02:50:58.401 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 kB/s | 4.5 kB 00:00 2024-04-12T02:50:58.465 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:50:58.465 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:50:59.008 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:59.008 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:50:59.008 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:50:59.009 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:50:59.009 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:50:59.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2024-04-12T02:50:59.577 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:01.042 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:51:01.043 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:01.044 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:51:01.044 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:01.044 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 k 2024-04-12T02:51:01.044 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 139 k 2024-04-12T02:51:01.044 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:51:01.190 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 212 kB/s | 31 kB 00:00 2024-04-12T02:51:01.190 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:01.191 INFO:teuthology.orchestra.run.smithi181.stdout:Total 210 kB/s | 31 kB 00:00 2024-04-12T02:51:01.191 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:51:01.195 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:51:01.195 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:51:01.207 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:51:01.215 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:51:01.299 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:51:01.473 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:01.535 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-12T02:51:01.762 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:01.921 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-12T02:51:04.330 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:04.519 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:04.520 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:04.520 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:04.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2024-04-12T02:51:05.034 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 90 kB/s | 4.1 kB 00:00 2024-04-12T02:51:07.022 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-12T02:51:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:09.051 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:51:09.540 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:09.616 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:09.617 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:51:09.617 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:51:09.617 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:09.617 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:09.734 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:09.812 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:09.812 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:09.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-04-12T02:51:09.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2024-04-12T02:51:10.172 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:10.237 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:11.041 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 105 kB/s | 4.1 kB 00:00 2024-04-12T02:51:11.120 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-12T02:51:11.341 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 118 kB/s | 4.5 kB 00:00 2024-04-12T02:51:11.378 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 4.5 kB 00:00 2024-04-12T02:51:13.773 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:13.784 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:13.964 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:13.965 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:13.965 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:13.973 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:13.974 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:13.974 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:14.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-04-12T02:51:14.272 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2024-04-12T02:51:14.467 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:14.629 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:15.339 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 111 kB/s | 4.1 kB 00:00 2024-04-12T02:51:15.503 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:51:15.643 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 114 kB/s | 4.5 kB 00:00 2024-04-12T02:51:15.753 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-12T02:51:17.960 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:18.152 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:18.153 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:18.154 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:18.161 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:18.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-04-12T02:51:18.350 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:18.352 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:18.352 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:18.505 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2024-04-12T02:51:18.645 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:18.863 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:19.597 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 117 kB/s | 4.1 kB 00:00 2024-04-12T02:51:19.674 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 122 kB/s | 4.1 kB 00:00 2024-04-12T02:51:19.897 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 126 kB/s | 4.5 kB 00:00 2024-04-12T02:51:19.915 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 138 kB/s | 4.5 kB 00:00 2024-04-12T02:51:22.291 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:22.317 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:22.481 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:22.482 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:22.482 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:22.510 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:22.511 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:22.511 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:22.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-04-12T02:51:22.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2024-04-12T02:51:22.978 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:23.012 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:23.764 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 116 kB/s | 4.1 kB 00:00 2024-04-12T02:51:23.778 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 144 kB/s | 4.1 kB 00:00 2024-04-12T02:51:24.011 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 163 kB/s | 4.5 kB 00:00 2024-04-12T02:51:24.062 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 137 kB/s | 4.5 kB 00:00 2024-04-12T02:51:26.399 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:26.454 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:26.592 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:26.593 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-04-12T02:51:26.593 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:26.643 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:26.644 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:26.645 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:26.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-04-12T02:51:26.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2024-04-12T02:51:27.135 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:27.271 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:28.024 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 145 kB/s | 4.1 kB 00:00 2024-04-12T02:51:28.119 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 146 kB/s | 4.1 kB 00:00 2024-04-12T02:51:28.325 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 153 kB/s | 4.5 kB 00:00 2024-04-12T02:51:28.352 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 166 kB/s | 4.5 kB 00:00 2024-04-12T02:51:30.649 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:30.842 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:30.843 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:30.843 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:30.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-04-12T02:51:31.044 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:51:31.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:31.046 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 95 k 2024-04-12T02:51:31.046 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 280 k 2024-04-12T02:51:31.046 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:51:31.353 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:31.425 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 251 kB/s | 95 kB 00:00 2024-04-12T02:51:31.425 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Total 250 kB/s | 95 kB 00:00 2024-04-12T02:51:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:51:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:51:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:51:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:51:31.503 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:51:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:51:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:32.115 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:32.126 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 137 kB/s | 4.1 kB 00:00 2024-04-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 155 kB/s | 4.5 kB 00:00 2024-04-12T02:51:34.774 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T02:51:34.963 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:34.964 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-12T02:51:34.964 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:35.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-04-12T02:51:35.459 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:36.214 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 159 kB/s | 4.1 kB 00:00 2024-04-12T02:51:36.506 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 171 kB/s | 4.5 kB 00:00 2024-04-12T02:51:36.788 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:36.788 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:51:37.336 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:37.336 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:51:37.336 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:51:37.337 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:37.337 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:37.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2024-04-12T02:51:37.856 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:38.647 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 116 kB/s | 4.1 kB 00:00 2024-04-12T02:51:38.895 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 135 kB/s | 4.5 kB 00:00 2024-04-12T02:51:39.032 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:51:39.033 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:39.034 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 95 k 2024-04-12T02:51:39.034 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 280 k 2024-04-12T02:51:39.034 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:51:39.364 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 287 kB/s | 95 kB 00:00 2024-04-12T02:51:39.365 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:39.365 INFO:teuthology.orchestra.run.smithi181.stdout:Total 286 kB/s | 95 kB 00:00 2024-04-12T02:51:39.365 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:51:39.376 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:51:39.376 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:51:39.431 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:51:39.440 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:51:39.594 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:51:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:40.063 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:41.492 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:51:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:41.494 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.5 M 2024-04-12T02:51:41.494 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 16 M 2024-04-12T02:51:41.494 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:51:42.020 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 6.6 MB/s | 3.5 MB 00:00 2024-04-12T02:51:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:Total 6.5 MB/s | 3.5 MB 00:00 2024-04-12T02:51:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:51:42.034 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:51:42.034 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:51:42.102 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:51:42.110 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:51:42.582 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:51:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:44.671 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:51:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:45.233 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:51:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:51:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:45.233 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:45.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-04-12T02:51:45.735 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:46.538 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 141 kB/s | 4.1 kB 00:00 2024-04-12T02:51:46.845 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 126 kB/s | 4.5 kB 00:00 2024-04-12T02:51:48.070 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:48.070 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:51:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:48.626 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:51:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:51:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:48.626 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:51:48.782 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2024-04-12T02:51:49.142 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:49.353 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:51:49.354 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:51:49.355 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:51:49.355 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:49.355 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.5 M 2024-04-12T02:51:49.355 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 16 M 2024-04-12T02:51:49.355 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:51:49.876 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 6.6 MB/s | 3.5 MB 00:00 2024-04-12T02:51:49.876 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:49.876 INFO:teuthology.orchestra.run.smithi181.stdout:Total 6.6 MB/s | 3.5 MB 00:00 2024-04-12T02:51:49.877 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:51:49.889 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:51:49.889 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:51:49.960 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:51:49.968 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:51:49.994 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-12T02:51:50.244 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 124 kB/s | 4.5 kB 00:00 2024-04-12T02:51:50.418 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:51:50.565 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:51.007 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:52.821 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-04-12T02:51:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:51:52.823 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 181 k 2024-04-12T02:51:52.823 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 572 k 2024-04-12T02:51:52.823 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-04-12T02:51:53.291 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 388 kB/s | 181 kB 00:00 2024-04-12T02:51:53.291 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:51:53.292 INFO:teuthology.orchestra.run.smithi099.stdout:Total 387 kB/s | 181 kB 00:00 2024-04-12T02:51:53.292 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-04-12T02:51:53.303 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-04-12T02:51:53.303 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-04-12T02:51:53.359 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-04-12T02:51:53.368 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-04-12T02:51:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-04-12T02:51:53.664 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:51:55.899 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:51:56.475 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:56.475 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:51:56.476 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:51:56.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:51:56.476 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:51:56.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-04-12T02:51:56.989 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-04-12T02:51:57.970 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-12T02:51:58.306 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 kB/s | 4.5 kB 00:00 2024-04-12T02:52:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:52:00.053 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-04-12T02:52:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:00.602 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-04-12T02:52:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:52:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:00.602 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-04-12T02:52:00.781 DEBUG:teuthology.parallel:result is None 2024-04-12T02:52:00.813 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-12T02:52:00.814 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:00.815 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 181 k 2024-04-12T02:52:00.815 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 572 k 2024-04-12T02:52:00.815 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-12T02:52:01.104 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 627 kB/s | 181 kB 00:00 2024-04-12T02:52:01.105 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-12T02:52:01.105 INFO:teuthology.orchestra.run.smithi181.stdout:Total 625 kB/s | 181 kB 00:00 2024-04-12T02:52:01.105 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-12T02:52:01.117 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-12T02:52:01.117 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-12T02:52:01.173 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-12T02:52:01.181 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-12T02:52:01.303 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-12T02:52:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:52:01.737 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:52:09.146 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T02:52:09.146 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-04-12T02:52:09.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:09.704 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-12T02:52:09.704 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T02:52:09.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:09.704 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-12T02:52:09.870 DEBUG:teuthology.parallel:result is None 2024-04-12T02:52:09.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:52:10.033 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T02:52:10.061 INFO:teuthology.orchestra.run.smithi099.stdout:18.2.2-1187.gd540ebac.el8 2024-04-12T02:52:10.062 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-12T02:52:10.062 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-12T02:52:10.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:52:10.289 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T02:52:10.312 INFO:teuthology.orchestra.run.smithi181.stdout:18.2.2-1187.gd540ebac.el8 2024-04-12T02:52:10.312 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-12T02:52:10.312 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-12T02:52:10.313 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T02:52:10.314 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:10.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T02:52:10.348 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:10.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T02:52:10.380 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T02:52:10.380 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:10.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T02:52:10.418 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T02:52:10.488 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:10.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T02:52:10.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T02:52:10.581 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T02:52:10.581 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:10.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T02:52:10.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T02:52:10.679 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:10.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T02:52:10.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T02:52:10.772 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T02:52:10.772 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:10.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T02:52:10.801 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T02:52:10.869 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:10.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T02:52:10.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T02:52:10.962 INFO:teuthology.run_tasks:Running task ceph... 2024-04-12T02:52:11.038 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-12T02:52:11.038 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2024-04-12T02:52:11.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-04-12T02:52:11.069 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-12T02:52:11.069 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-12T02:52:11.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-12T02:52:11.133 INFO:tasks.ceph:Creating extra log directories... 2024-04-12T02:52:11.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-12T02:52:11.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-12T02:52:11.211 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-12T02:52:11.211 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-12T02:52:11.211 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7652588', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi099.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTwpORqxF9iz3chL4tw4yrbtah//4i57koBpG6bhfbnaBL2JrJPBf4di8jSm+YAH/YD8q536515s1peMJjqEzg=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/9ghsepfvA6W427iDW4ZktMrfXGxKfbxzC0J3HL+ULgMBV5nXlZzI6yDzw85DCviyvol6Y+ZF+JXcH/Mlg7oY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-11_20:30:01', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-12T02:52:11.212 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-12T02:52:11.242 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-12T02:52:11.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T02:52:11.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T02:52:11.326 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:11.326 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T02:52:11.380 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T02:52:11.381 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 181326 Links: 1 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-04-12 02:51:42.937406751 +0000 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-04-12 02:45:35.880190638 +0000 2024-04-12T02:52:11.438 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-04-12 02:45:35.880190638 +0000 2024-04-12T02:52:11.439 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-04-12T02:52:11.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T02:52:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-04-12T02:52:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-04-12T02:52:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000262282 s, 2.0 MB/s 2024-04-12T02:52:11.503 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T02:52:11.561 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-04-12T02:52:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T02:52:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 179027 Links: 1 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-04-12 02:51:42.937406751 +0000 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-04-12 02:45:36.245182292 +0000 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-04-12 02:45:36.245182292 +0000 2024-04-12T02:52:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-04-12T02:52:11.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T02:52:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-04-12T02:52:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-04-12T02:52:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000237724 s, 2.2 MB/s 2024-04-12T02:52:11.685 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T02:52:11.742 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 181439 Links: 1 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-04-12 02:51:42.938406728 +0000 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-04-12 02:45:36.576174723 +0000 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-04-12 02:45:36.576174723 +0000 2024-04-12T02:52:11.800 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-04-12T02:52:11.800 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T02:52:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-04-12T02:52:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-04-12T02:52:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000186687 s, 2.7 MB/s 2024-04-12T02:52:11.866 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T02:52:11.924 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-04-12T02:52:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T02:52:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:11.981 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 179099 Links: 1 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-04-12 02:51:42.938406728 +0000 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-04-12 02:45:36.929166651 +0000 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-04-12 02:45:36.929166651 +0000 2024-04-12T02:52:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-04-12T02:52:11.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T02:52:12.046 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-04-12T02:52:12.046 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-04-12T02:52:12.046 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000224427 s, 2.3 MB/s 2024-04-12T02:52:12.047 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T02:52:12.105 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:12.105 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T02:52:12.105 INFO:tasks.ceph:Generating config... 2024-04-12T02:52:12.106 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-12T02:52:12.106 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-12T02:52:12.106 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-12T02:52:12.106 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-12T02:52:12.106 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client] setuser = ceph 2024-04-12T02:52:12.107 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-12T02:52:12.107 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-12T02:52:12.107 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-12T02:52:12.107 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-12T02:52:12.107 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-12T02:52:12.107 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:12.108 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:12.108 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-12T02:52:12.108 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-12T02:52:12.203 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.keyring 2024-04-12T02:52:12.206 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-12T02:52:12.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T02:52:12.276 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.99')] 2024-04-12T02:52:12.277 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-12T02:52:12.277 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:12.277 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T02:52:12.333 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --print /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:52:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:52:12.415 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid d7fe4a71-2c44-4ac1-9d99-d175c6f86930 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = pacific 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:fsid d7fe4a71-2c44-4ac1-9d99-d175c6f86930 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2024-04-12T02:52:12.415776+0000 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:created 2024-04-12T02:52:12.415776+0000 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 16 (pacific) 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 1 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2024-04-12T02:52:12.416 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-12T02:52:12.417 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T02:52:12.474 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID d7fe4a71-2c44-4ac1-9d99-d175c6f86930... 2024-04-12T02:52:12.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-12T02:52:12.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-12T02:52:12.560 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-04-12T02:52:12.560 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2024-04-12T02:52:12.560 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T02:52:12.560 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2024-04-12T02:52:12.560 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T02:52:12.561 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T02:52:12.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.99 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = d7fe4a71-2c44-4ac1-9d99-d175c6f86930 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2024-04-12T02:52:12.563 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2024-04-12T02:52:12.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-04-12T02:52:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:12.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2024-04-12T02:52:12.567 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2024-04-12T02:52:12.568 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2024-04-12T02:52:12.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-04-12T02:52:12.570 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.571 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.572 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = 172.21.15.99 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = d7fe4a71-2c44-4ac1-9d99-d175c6f86930 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.574 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: rocksdb delete range threshold = 0 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:12.575 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T02:52:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T02:52:12.577 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw = 20 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw sync = 20 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt require ssl = False 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt s3 kms backend = testing 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw curl low speed time = 300 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw data log num shards = 4 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw md log max shards = 4 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync data inject err probability = 0.1 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync log trim interval = 0 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync obj etag verify = True 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: setgroup = ceph 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout: setuser = ceph 2024-04-12T02:52:12.578 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout: rgw data notify interval msec = 0 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-04-12T02:52:12.579 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:12.582 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-12T02:52:12.583 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-12T02:52:12.678 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-12T02:52:12.678 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:12.678 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-12T02:52:12.695 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:12.695 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-12T02:52:12.752 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:52:12.752 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:12.752 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c1.keyring 2024-04-12T02:52:12.752 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T02:52:12.828 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:12.828 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:12.885 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:52:12.886 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:12.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/c1.keyring 2024-04-12T02:52:12.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T02:52:12.929 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:12.929 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:12.987 INFO:tasks.ceph:Setting up mon nodes... 2024-04-12T02:52:12.988 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-12T02:52:12.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-12T02:52:13.041 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-12T02:52:13.044 INFO:tasks.ceph:Setting up mds nodes... 2024-04-12T02:52:13.044 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-12T02:52:13.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-12T02:52:13.087 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-12T02:52:13.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-12T02:52:13.143 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-12T02:52:13.156 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-12T02:52:13.156 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T02:52:13.157 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-12T02:52:13.183 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:13.183 INFO:tasks.ceph:role: c1.osd.0 2024-04-12T02:52:13.183 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:52:13.183 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:13.257 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:13.258 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:13.258 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:13.695 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-04-12T02:52:13.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-04-12T02:52:13.697 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-12T02:52:13.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-12T02:52:13.915 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-12T02:52:13.982 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:13.982 INFO:tasks.ceph:role: c1.osd.1 2024-04-12T02:52:13.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:52:13.982 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:14.482 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-04-12T02:52:14.483 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-04-12T02:52:14.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-12T02:52:14.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-12T02:52:14.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-12T02:52:14.688 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:14.688 INFO:tasks.ceph:role: c1.osd.2 2024-04-12T02:52:14.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:52:14.689 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-12T02:52:14.762 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:14.763 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:15.215 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-04-12T02:52:15.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-04-12T02:52:15.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-12T02:52:15.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-12T02:52:15.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:15.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.509+0000 7f20b55247c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-12T02:52:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.512+0000 7f20b55247c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-12T02:52:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.512+0000 7f20b55247c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T02:52:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.512+0000 7f20b55247c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T02:52:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.512+0000 7f20b55247c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T02:52:15.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:15.519+0000 7f20b55247c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-12T02:52:17.608 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-12T02:52:17.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T02:52:17.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:17.732+0000 7f33523ad7c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-12T02:52:19.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-12T02:52:19.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.942+0000 7faa45ec37c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.942+0000 7faa45ec37c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.942+0000 7faa45ec37c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.942+0000 7faa45ec37c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.942+0000 7faa45ec37c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T02:52:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:52:19.943+0000 7faa45ec37c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-12T02:52:22.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-12T02:52:22.057 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-12T02:52:22.057 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.057 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-12T02:52:22.128 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.128 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-12T02:52:22.196 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-12T02:52:22.265 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-12T02:52:22.332 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.332 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-12T02:52:22.390 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:22.390 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-12T02:52:22.446 INFO:tasks.ceph:Adding keys to all mons... 2024-04-12T02:52:22.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/c1.keyring 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDdoRhmWE1+AhAAxf8xc1szKfgYSC4mREvvHg== 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDfoRhmbiJpHhAAv/5LZZxYmzhfxTGwD/VZDQ== 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDhoRhmYjKtKxAAMnm6sgDGRSpC3b8LgrK0hw== 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDjoRhmG6ozOBAAAWYTlhPZMIwGLaub0jD1HQ== 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-04-12T02:52:22.512 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDdoRhmQ/Y6BRAA7bzeAoQwZ6EkeInxwF9KfQ== 2024-04-12T02:52:22.513 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2024-04-12T02:52:22.513 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQDdoRhmsaCRCBAANGHvmPyjm3ah1FjYXNWX0w== 2024-04-12T02:52:22.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-12T02:52:22.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:22.660 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:22.713 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:22.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T02:52:22.822 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T02:52:22.879 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-12T02:52:22.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-12T02:52:22.905 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-12T02:52:23.006 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-12T02:52:23.033 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-12T02:52:23.090 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-12T02:52:23.090 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-12T02:52:23.090 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-12T02:52:23.133 INFO:tasks.ceph.c1.mon.a:Started 2024-04-12T02:52:23.133 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-12T02:52:23.133 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-12T02:52:23.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-12T02:52:23.135 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-12T02:52:23.135 INFO:tasks.ceph:Setting crush tunables to default 2024-04-12T02:52:23.135 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd crush tunables default 2024-04-12T02:52:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-04-12T02:52:23.371 INFO:tasks.ceph:check_enable_crimson: False 2024-04-12T02:52:23.371 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-12T02:52:23.371 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:23.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-12T02:52:23.398 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:23.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-12T02:52:23.465 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:23.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-12T02:52:23.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new a7d888d9-e20b-4d68-9cc2-461f48533470 0 2024-04-12T02:52:23.804 INFO:teuthology.orchestra.run.smithi099.stdout:0 2024-04-12T02:52:23.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new 13df6200-730f-4c7a-a0cb-41b9beec2059 1 2024-04-12T02:52:23.855 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:23.854+0000 7f980e938200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T02:52:23.938 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:23.937+0000 7f980e938200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T02:52:24.013 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.012+0000 7f980e938200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T02:52:24.031 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-04-12T02:52:24.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new 050efab0-6a1a-459e-a9ab-5292e40d6be4 2 2024-04-12T02:52:24.087 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.086+0000 7f980e938200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T02:52:24.270 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.269+0000 7f980e938200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T02:52:24.303 INFO:teuthology.orchestra.run.smithi099.stdout:2 2024-04-12T02:52:24.317 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-12T02:52:24.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-12T02:52:24.319 INFO:tasks.ceph.c1.osd.0:Started 2024-04-12T02:52:24.319 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-12T02:52:24.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-12T02:52:24.321 INFO:tasks.ceph.c1.osd.1:Started 2024-04-12T02:52:24.321 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-12T02:52:24.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-12T02:52:24.323 INFO:tasks.ceph.c1.osd.2:Started 2024-04-12T02:52:24.324 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T02:52:24.461 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.460+0000 7f980e938200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T02:52:24.552 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:24.552 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"fsid":"d7fe4a71-2c44-4ac1-9d99-d175c6f86930","created":"2024-04-12T02:52:23.261245+0000","modified":"2024-04-12T02:52:24.301083+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T02:52:24.301083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a7d888d9-e20b-4d68-9cc2-461f48533470","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"13df6200-730f-4c7a-a0cb-41b9beec2059","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"050efab0-6a1a-459e-a9ab-5292e40d6be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:24.565 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-12T02:52:24.565 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-12T02:52:24.577 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.576+0000 7f980e938200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T02:52:24.673 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.672+0000 7f980e938200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T02:52:24.750 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.749+0000 7f980e938200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T02:52:24.867 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-12T02:52:24.939 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2024-04-12T02:52:24.939+0000 7fc38c1307c0 -1 Falling back to public interface 2024-04-12T02:52:24.949 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2024-04-12T02:52:24.949+0000 7f94620b77c0 -1 Falling back to public interface 2024-04-12T02:52:24.970 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2024-04-12T02:52:24.970+0000 7f22fa2ba7c0 -1 Falling back to public interface 2024-04-12T02:52:24.979 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:24.978+0000 7f980e938200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T02:52:25.054 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:25.053+0000 7f980e938200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T02:52:25.080 INFO:teuthology.misc.health.smithi099.stdout: 2024-04-12T02:52:25.080 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":5,"fsid":"d7fe4a71-2c44-4ac1-9d99-d175c6f86930","created":"2024-04-12T02:52:23.261245+0000","modified":"2024-04-12T02:52:24.301083+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T02:52:24.301083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a7d888d9-e20b-4d68-9cc2-461f48533470","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"13df6200-730f-4c7a-a0cb-41b9beec2059","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"050efab0-6a1a-459e-a9ab-5292e40d6be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:25.092 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-12T02:52:25.439 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:25.438+0000 7f980e938200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T02:52:25.760 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:25.759+0000 7f980e938200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T02:52:25.921 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:25.920+0000 7f980e938200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T02:52:25.999 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:25.998+0000 7f980e938200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T02:52:26.708 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:26.707+0000 7f980e938200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T02:52:26.791 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:26.790+0000 7f980e938200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T02:52:26.924 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:26.923+0000 7f980e938200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T02:52:27.481 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2024-04-12T02:52:27.480+0000 7fc38c1307c0 -1 osd.1 0 log_to_monitors true 2024-04-12T02:52:27.491 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:27.490+0000 7f980e938200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T02:52:27.665 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:27.664+0000 7f980e938200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T02:52:27.681 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2024-04-12T02:52:27.681+0000 7f94620b77c0 -1 osd.0 0 log_to_monitors true 2024-04-12T02:52:27.753 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2024-04-12T02:52:27.752+0000 7f22fa2ba7c0 -1 osd.2 0 log_to_monitors true 2024-04-12T02:52:27.838 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:27.838+0000 7f980e938200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T02:52:28.663 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:28.662+0000 7f980e938200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T02:52:28.920 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:28.919+0000 7f980e938200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T02:52:28.998 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:28.997+0000 7f980e938200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T02:52:29.053 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2024-04-12T02:52:29.052+0000 7fc3804e8700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T02:52:29.067 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2024-04-12T02:52:29.066+0000 7fc3780c2700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:29.377 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:29.376+0000 7f980e938200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T02:52:30.387 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2024-04-12T02:52:30.386+0000 7f97c9bb2700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-12T02:52:30.406 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2024-04-12T02:52:30.406+0000 7f945646f700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T02:52:30.407 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2024-04-12T02:52:30.407+0000 7f22ee672700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T02:52:30.424 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2024-04-12T02:52:30.424+0000 7f22e624c700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:30.426 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2024-04-12T02:52:30.426+0000 7f944e049700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:31.395 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-12T02:52:31.724 INFO:teuthology.misc.health.smithi099.stdout: 2024-04-12T02:52:31.724 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":9,"fsid":"d7fe4a71-2c44-4ac1-9d99-d175c6f86930","created":"2024-04-12T02:52:23.261245+0000","modified":"2024-04-12T02:52:31.415499+0000","last_up_change":"2024-04-12T02:52:31.415499+0000","last_in_change":"2024-04-12T02:52:24.301083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a7d888d9-e20b-4d68-9cc2-461f48533470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":40694},{"type":"v1","addr":"172.21.15.99:6809","nonce":40694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":40694},{"type":"v1","addr":"172.21.15.99:6811","nonce":40694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":40694},{"type":"v1","addr":"172.21.15.99:6815","nonce":40694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":40694},{"type":"v1","addr":"172.21.15.99:6813","nonce":40694}]},"public_addr":"172.21.15.99:6809/40694","cluster_addr":"172.21.15.99:6811/40694","heartbeat_back_addr":"172.21.15.99:6815/40694","heartbeat_front_addr":"172.21.15.99:6813/40694","state":["exists","up"]},{"osd":1,"uuid":"13df6200-730f-4c7a-a0cb-41b9beec2059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":40693},{"type":"v1","addr":"172.21.15.99:6801","nonce":40693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":40693},{"type":"v1","addr":"172.21.15.99:6803","nonce":40693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":40693},{"type":"v1","addr":"172.21.15.99:6807","nonce":40693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":40693},{"type":"v1","addr":"172.21.15.99:6805","nonce":40693}]},"public_addr":"172.21.15.99:6801/40693","cluster_addr":"172.21.15.99:6803/40693","heartbeat_back_addr":"172.21.15.99:6807/40693","heartbeat_front_addr":"172.21.15.99:6805/40693","state":["exists","up"]},{"osd":2,"uuid":"050efab0-6a1a-459e-a9ab-5292e40d6be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":40713},{"type":"v1","addr":"172.21.15.99:6817","nonce":40713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":40713},{"type":"v1","addr":"172.21.15.99:6819","nonce":40713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":40713},{"type":"v1","addr":"172.21.15.99:6823","nonce":40713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":40713},{"type":"v1","addr":"172.21.15.99:6821","nonce":40713}]},"public_addr":"172.21.15.99:6817/40713","cluster_addr":"172.21.15.99:6819/40713","heartbeat_back_addr":"172.21.15.99:6823/40713","heartbeat_front_addr":"172.21.15.99:6821/40713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.455973+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:31.737 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-12T02:52:31.737 INFO:tasks.ceph:Creating RBD pool 2024-04-12T02:52:31.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-12T02:52:32.423 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'rbd' created 2024-04-12T02:52:32.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-12T02:52:33.438 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-12T02:52:33.450 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-12T02:52:33.451 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-12T02:52:33.451 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-12T02:52:33.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-12T02:52:33.451 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-12T02:52:33.839 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:33.851 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":40312},{"type":"v1","addr":"172.21.15.99:6825","nonce":40312}]},"active_addr":"172.21.15.99:6825/40312","active_change":"2024-04-12T02:52:29.378722+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":386776836}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":506128912}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":3614672470}]}]} 2024-04-12T02:52:33.853 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-12T02:52:33.853 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-12T02:52:33.853 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T02:52:34.176 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:34.176 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":11,"fsid":"d7fe4a71-2c44-4ac1-9d99-d175c6f86930","created":"2024-04-12T02:52:23.261245+0000","modified":"2024-04-12T02:52:33.423839+0000","last_up_change":"2024-04-12T02:52:31.415499+0000","last_in_change":"2024-04-12T02:52:24.301083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-12T02:52:32.078994+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-12T02:52:33.423399+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d888d9-e20b-4d68-9cc2-461f48533470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":40694},{"type":"v1","addr":"172.21.15.99:6809","nonce":40694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":40694},{"type":"v1","addr":"172.21.15.99:6811","nonce":40694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":40694},{"type":"v1","addr":"172.21.15.99:6815","nonce":40694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":40694},{"type":"v1","addr":"172.21.15.99:6813","nonce":40694}]},"public_addr":"172.21.15.99:6809/40694","cluster_addr":"172.21.15.99:6811/40694","heartbeat_back_addr":"172.21.15.99:6815/40694","heartbeat_front_addr":"172.21.15.99:6813/40694","state":["exists","up"]},{"osd":1,"uuid":"13df6200-730f-4c7a-a0cb-41b9beec2059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":40693},{"type":"v1","addr":"172.21.15.99:6801","nonce":40693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":40693},{"type":"v1","addr":"172.21.15.99:6803","nonce":40693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":40693},{"type":"v1","addr":"172.21.15.99:6807","nonce":40693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":40693},{"type":"v1","addr":"172.21.15.99:6805","nonce":40693}]},"public_addr":"172.21.15.99:6801/40693","cluster_addr":"172.21.15.99:6803/40693","heartbeat_back_addr":"172.21.15.99:6807/40693","heartbeat_front_addr":"172.21.15.99:6805/40693","state":["exists","up"]},{"osd":2,"uuid":"050efab0-6a1a-459e-a9ab-5292e40d6be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":40713},{"type":"v1","addr":"172.21.15.99:6817","nonce":40713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":40713},{"type":"v1","addr":"172.21.15.99:6819","nonce":40713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":40713},{"type":"v1","addr":"172.21.15.99:6823","nonce":40713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":40713},{"type":"v1","addr":"172.21.15.99:6821","nonce":40713}]},"public_addr":"172.21.15.99:6817/40713","cluster_addr":"172.21.15.99:6819/40713","heartbeat_back_addr":"172.21.15.99:6823/40713","heartbeat_front_addr":"172.21.15.99:6821/40713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.665532+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.455973+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.770363+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:34.190 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-12T02:52:34.190 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T02:52:34.519 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:34.519 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":12,"fsid":"d7fe4a71-2c44-4ac1-9d99-d175c6f86930","created":"2024-04-12T02:52:23.261245+0000","modified":"2024-04-12T02:52:34.427615+0000","last_up_change":"2024-04-12T02:52:31.415499+0000","last_in_change":"2024-04-12T02:52:24.301083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-12T02:52:32.078994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-12T02:52:33.423399+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d888d9-e20b-4d68-9cc2-461f48533470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":40694},{"type":"v1","addr":"172.21.15.99:6809","nonce":40694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":40694},{"type":"v1","addr":"172.21.15.99:6811","nonce":40694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":40694},{"type":"v1","addr":"172.21.15.99:6815","nonce":40694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":40694},{"type":"v1","addr":"172.21.15.99:6813","nonce":40694}]},"public_addr":"172.21.15.99:6809/40694","cluster_addr":"172.21.15.99:6811/40694","heartbeat_back_addr":"172.21.15.99:6815/40694","heartbeat_front_addr":"172.21.15.99:6813/40694","state":["exists","up"]},{"osd":1,"uuid":"13df6200-730f-4c7a-a0cb-41b9beec2059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":40693},{"type":"v1","addr":"172.21.15.99:6801","nonce":40693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":40693},{"type":"v1","addr":"172.21.15.99:6803","nonce":40693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":40693},{"type":"v1","addr":"172.21.15.99:6807","nonce":40693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":40693},{"type":"v1","addr":"172.21.15.99:6805","nonce":40693}]},"public_addr":"172.21.15.99:6801/40693","cluster_addr":"172.21.15.99:6803/40693","heartbeat_back_addr":"172.21.15.99:6807/40693","heartbeat_front_addr":"172.21.15.99:6805/40693","state":["exists","up"]},{"osd":2,"uuid":"050efab0-6a1a-459e-a9ab-5292e40d6be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":40713},{"type":"v1","addr":"172.21.15.99:6817","nonce":40713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":40713},{"type":"v1","addr":"172.21.15.99:6819","nonce":40713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":40713},{"type":"v1","addr":"172.21.15.99:6823","nonce":40713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":40713},{"type":"v1","addr":"172.21.15.99:6821","nonce":40713}]},"public_addr":"172.21.15.99:6817/40713","cluster_addr":"172.21.15.99:6819/40713","heartbeat_back_addr":"172.21.15.99:6823/40713","heartbeat_front_addr":"172.21.15.99:6821/40713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.665532+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.455973+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:28.770363+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:34.533 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-12T02:52:34.533 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-12T02:52:34.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-12T02:52:34.719 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2024-04-12T02:52:34.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-12T02:52:34.722 INFO:teuthology.orchestra.run.smithi099.stdout:38654705667 2024-04-12T02:52:34.723 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-12T02:52:34.799 INFO:teuthology.orchestra.run.smithi099.stdout:38654705667 2024-04-12T02:52:34.799 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-12T02:52:35.164 INFO:teuthology.orchestra.run.smithi099.stdout:34359738369 2024-04-12T02:52:35.178 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-04-12T02:52:35.207 INFO:teuthology.orchestra.run.smithi099.stdout:38654705666 2024-04-12T02:52:35.221 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.2 2024-04-12T02:52:35.229 INFO:teuthology.orchestra.run.smithi099.stdout:38654705666 2024-04-12T02:52:35.243 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2024-04-12T02:52:36.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-12T02:52:36.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-12T02:52:36.244 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-12T02:52:36.537 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2024-04-12T02:52:36.551 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-04-12T02:52:36.551 DEBUG:teuthology.parallel:result is None 2024-04-12T02:52:36.578 INFO:teuthology.orchestra.run.smithi099.stdout:38654705667 2024-04-12T02:52:36.593 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.2 2024-04-12T02:52:36.594 DEBUG:teuthology.parallel:result is None 2024-04-12T02:52:36.613 INFO:teuthology.orchestra.run.smithi099.stdout:38654705667 2024-04-12T02:52:36.627 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.0 2024-04-12T02:52:36.627 DEBUG:teuthology.parallel:result is None 2024-04-12T02:52:36.627 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-12T02:52:36.627 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-12T02:52:36.988 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:36.991 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-04-12T02:52:37.002 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-12T02:52:35.435053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82180,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490620,"statfs":{"total":322122547200,"available":322038394880,"internally_reserved":0,"allocated":761856,"data_stored":534930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.051627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.558061+0000","last_change":"2024-04-12T02:52:33.434637+0000","last_active":"2024-04-12T02:52:34.558061+0000","last_peered":"2024-04-12T02:52:34.558061+0000","last_clean":"2024-04-12T02:52:34.558061+0000","last_became_active":"2024-04-12T02:52:33.434372+0000","last_became_peered":"2024-04-12T02:52:33.434372+0000","last_unstale":"2024-04-12T02:52:34.558061+0000","last_undegraded":"2024-04-12T02:52:34.558061+0000","last_fullsized":"2024-04-12T02:52:34.558061+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:31:27.075420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557905+0000","last_change":"2024-04-12T02:52:33.435203+0000","last_active":"2024-04-12T02:52:34.557905+0000","last_peered":"2024-04-12T02:52:34.557905+0000","last_clean":"2024-04-12T02:52:34.557905+0000","last_became_active":"2024-04-12T02:52:33.434922+0000","last_became_peered":"2024-04-12T02:52:33.434922+0000","last_unstale":"2024-04-12T02:52:34.557905+0000","last_undegraded":"2024-04-12T02:52:34.557905+0000","last_fullsized":"2024-04-12T02:52:34.557905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:51:36.573433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.434590+0000","last_change":"2024-04-12T02:52:33.436119+0000","last_active":"2024-04-12T02:52:34.434590+0000","last_peered":"2024-04-12T02:52:34.434590+0000","last_clean":"2024-04-12T02:52:34.434590+0000","last_became_active":"2024-04-12T02:52:33.435864+0000","last_became_peered":"2024-04-12T02:52:33.435864+0000","last_unstale":"2024-04-12T02:52:34.434590+0000","last_undegraded":"2024-04-12T02:52:34.434590+0000","last_fullsized":"2024-04-12T02:52:34.434590+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:48:19.092189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557988+0000","last_change":"2024-04-12T02:52:33.435075+0000","last_active":"2024-04-12T02:52:34.557988+0000","last_peered":"2024-04-12T02:52:34.557988+0000","last_clean":"2024-04-12T02:52:34.557988+0000","last_became_active":"2024-04-12T02:52:33.434818+0000","last_became_peered":"2024-04-12T02:52:33.434818+0000","last_unstale":"2024-04-12T02:52:34.557988+0000","last_undegraded":"2024-04-12T02:52:34.557988+0000","last_fullsized":"2024-04-12T02:52:34.557988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:04:48.636664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.756928+0000","last_change":"2024-04-12T02:52:33.434462+0000","last_active":"2024-04-12T02:52:34.756928+0000","last_peered":"2024-04-12T02:52:34.756928+0000","last_clean":"2024-04-12T02:52:34.756928+0000","last_became_active":"2024-04-12T02:52:33.434194+0000","last_became_peered":"2024-04-12T02:52:33.434194+0000","last_unstale":"2024-04-12T02:52:34.756928+0000","last_undegraded":"2024-04-12T02:52:34.756928+0000","last_fullsized":"2024-04-12T02:52:34.756928+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:19:37.172281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'64","reported_seq":97,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.662631+0000","last_change":"2024-04-12T02:52:34.559769+0000","last_active":"2024-04-12T02:52:34.662631+0000","last_peered":"2024-04-12T02:52:34.662631+0000","last_clean":"2024-04-12T02:52:34.662631+0000","last_became_active":"2024-04-12T02:52:34.558838+0000","last_became_peered":"2024-04-12T02:52:34.558838+0000","last_unstale":"2024-04-12T02:52:34.662631+0000","last_undegraded":"2024-04-12T02:52:34.662631+0000","last_fullsized":"2024-04-12T02:52:34.662631+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:33.423839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:33.423839+0000","last_clean_scrub_stamp":"2024-04-12T02:52:33.423839+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:23:45.857988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557905+0000","last_change":"2024-04-12T02:52:33.437561+0000","last_active":"2024-04-12T02:52:34.557905+0000","last_peered":"2024-04-12T02:52:34.557905+0000","last_clean":"2024-04-12T02:52:34.557905+0000","last_became_active":"2024-04-12T02:52:33.437298+0000","last_became_peered":"2024-04-12T02:52:33.437298+0000","last_unstale":"2024-04-12T02:52:34.557905+0000","last_undegraded":"2024-04-12T02:52:34.557905+0000","last_fullsized":"2024-04-12T02:52:34.557905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:41:19.186988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557973+0000","last_change":"2024-04-12T02:52:33.435542+0000","last_active":"2024-04-12T02:52:34.557973+0000","last_peered":"2024-04-12T02:52:34.557973+0000","last_clean":"2024-04-12T02:52:34.557973+0000","last_became_active":"2024-04-12T02:52:33.435298+0000","last_became_peered":"2024-04-12T02:52:33.435298+0000","last_unstale":"2024-04-12T02:52:34.557973+0000","last_undegraded":"2024-04-12T02:52:34.557973+0000","last_fullsized":"2024-04-12T02:52:34.557973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:16:30.240255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.434613+0000","last_change":"2024-04-12T02:52:33.436053+0000","last_active":"2024-04-12T02:52:34.434613+0000","last_peered":"2024-04-12T02:52:34.434613+0000","last_clean":"2024-04-12T02:52:34.434613+0000","last_became_active":"2024-04-12T02:52:33.435792+0000","last_became_peered":"2024-04-12T02:52:33.435792+0000","last_unstale":"2024-04-12T02:52:34.434613+0000","last_undegraded":"2024-04-12T02:52:34.434613+0000","last_fullsized":"2024-04-12T02:52:34.434613+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:43:15.515772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056768,"data_stored":1049648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27944,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829656,"statfs":{"total":107374182400,"available":107345567744,"internally_reserved":0,"allocated":569344,"data_stored":487296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27112,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830488,"statfs":{"total":107374182400,"available":107346419712,"internally_reserved":0,"allocated":94208,"data_stored":23474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T02:52:37.003 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-12T02:52:37.320 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:37.323 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-04-12T02:52:37.334 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-12T02:52:35.435053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82180,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490620,"statfs":{"total":322122547200,"available":322038394880,"internally_reserved":0,"allocated":761856,"data_stored":534930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.051627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.558061+0000","last_change":"2024-04-12T02:52:33.434637+0000","last_active":"2024-04-12T02:52:34.558061+0000","last_peered":"2024-04-12T02:52:34.558061+0000","last_clean":"2024-04-12T02:52:34.558061+0000","last_became_active":"2024-04-12T02:52:33.434372+0000","last_became_peered":"2024-04-12T02:52:33.434372+0000","last_unstale":"2024-04-12T02:52:34.558061+0000","last_undegraded":"2024-04-12T02:52:34.558061+0000","last_fullsized":"2024-04-12T02:52:34.558061+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:31:27.075420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557905+0000","last_change":"2024-04-12T02:52:33.435203+0000","last_active":"2024-04-12T02:52:34.557905+0000","last_peered":"2024-04-12T02:52:34.557905+0000","last_clean":"2024-04-12T02:52:34.557905+0000","last_became_active":"2024-04-12T02:52:33.434922+0000","last_became_peered":"2024-04-12T02:52:33.434922+0000","last_unstale":"2024-04-12T02:52:34.557905+0000","last_undegraded":"2024-04-12T02:52:34.557905+0000","last_fullsized":"2024-04-12T02:52:34.557905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:51:36.573433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.434590+0000","last_change":"2024-04-12T02:52:33.436119+0000","last_active":"2024-04-12T02:52:34.434590+0000","last_peered":"2024-04-12T02:52:34.434590+0000","last_clean":"2024-04-12T02:52:34.434590+0000","last_became_active":"2024-04-12T02:52:33.435864+0000","last_became_peered":"2024-04-12T02:52:33.435864+0000","last_unstale":"2024-04-12T02:52:34.434590+0000","last_undegraded":"2024-04-12T02:52:34.434590+0000","last_fullsized":"2024-04-12T02:52:34.434590+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:48:19.092189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557988+0000","last_change":"2024-04-12T02:52:33.435075+0000","last_active":"2024-04-12T02:52:34.557988+0000","last_peered":"2024-04-12T02:52:34.557988+0000","last_clean":"2024-04-12T02:52:34.557988+0000","last_became_active":"2024-04-12T02:52:33.434818+0000","last_became_peered":"2024-04-12T02:52:33.434818+0000","last_unstale":"2024-04-12T02:52:34.557988+0000","last_undegraded":"2024-04-12T02:52:34.557988+0000","last_fullsized":"2024-04-12T02:52:34.557988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:04:48.636664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.756928+0000","last_change":"2024-04-12T02:52:33.434462+0000","last_active":"2024-04-12T02:52:34.756928+0000","last_peered":"2024-04-12T02:52:34.756928+0000","last_clean":"2024-04-12T02:52:34.756928+0000","last_became_active":"2024-04-12T02:52:33.434194+0000","last_became_peered":"2024-04-12T02:52:33.434194+0000","last_unstale":"2024-04-12T02:52:34.756928+0000","last_undegraded":"2024-04-12T02:52:34.756928+0000","last_fullsized":"2024-04-12T02:52:34.756928+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:19:37.172281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'64","reported_seq":97,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.662631+0000","last_change":"2024-04-12T02:52:34.559769+0000","last_active":"2024-04-12T02:52:34.662631+0000","last_peered":"2024-04-12T02:52:34.662631+0000","last_clean":"2024-04-12T02:52:34.662631+0000","last_became_active":"2024-04-12T02:52:34.558838+0000","last_became_peered":"2024-04-12T02:52:34.558838+0000","last_unstale":"2024-04-12T02:52:34.662631+0000","last_undegraded":"2024-04-12T02:52:34.662631+0000","last_fullsized":"2024-04-12T02:52:34.662631+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:33.423839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:33.423839+0000","last_clean_scrub_stamp":"2024-04-12T02:52:33.423839+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:23:45.857988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557905+0000","last_change":"2024-04-12T02:52:33.437561+0000","last_active":"2024-04-12T02:52:34.557905+0000","last_peered":"2024-04-12T02:52:34.557905+0000","last_clean":"2024-04-12T02:52:34.557905+0000","last_became_active":"2024-04-12T02:52:33.437298+0000","last_became_peered":"2024-04-12T02:52:33.437298+0000","last_unstale":"2024-04-12T02:52:34.557905+0000","last_undegraded":"2024-04-12T02:52:34.557905+0000","last_fullsized":"2024-04-12T02:52:34.557905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:41:19.186988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.557973+0000","last_change":"2024-04-12T02:52:33.435542+0000","last_active":"2024-04-12T02:52:34.557973+0000","last_peered":"2024-04-12T02:52:34.557973+0000","last_clean":"2024-04-12T02:52:34.557973+0000","last_became_active":"2024-04-12T02:52:33.435298+0000","last_became_peered":"2024-04-12T02:52:33.435298+0000","last_unstale":"2024-04-12T02:52:34.557973+0000","last_undegraded":"2024-04-12T02:52:34.557973+0000","last_fullsized":"2024-04-12T02:52:34.557973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:16:30.240255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:52:34.434613+0000","last_change":"2024-04-12T02:52:33.436053+0000","last_active":"2024-04-12T02:52:34.434613+0000","last_peered":"2024-04-12T02:52:34.434613+0000","last_clean":"2024-04-12T02:52:34.434613+0000","last_became_active":"2024-04-12T02:52:33.435792+0000","last_became_peered":"2024-04-12T02:52:33.435792+0000","last_unstale":"2024-04-12T02:52:34.434613+0000","last_undegraded":"2024-04-12T02:52:34.434613+0000","last_fullsized":"2024-04-12T02:52:34.434613+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:32.416251+0000","last_clean_scrub_stamp":"2024-04-12T02:52:32.416251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:43:15.515772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056768,"data_stored":1049648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27944,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829656,"statfs":{"total":107374182400,"available":107345567744,"internally_reserved":0,"allocated":569344,"data_stored":487296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27112,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830488,"statfs":{"total":107374182400,"available":107346419712,"internally_reserved":0,"allocated":94208,"data_stored":23474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T02:52:37.335 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-12T02:52:37.335 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-12T02:52:37.335 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-12T02:52:37.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-12T02:52:37.704 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:37.704 INFO:teuthology.orchestra.run.smithi099.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T02:52:37.718 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-12T02:52:37.719 INFO:teuthology.run_tasks:Running task ceph... 2024-04-12T02:52:37.729 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-12T02:52:37.729 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-12T02:52:37.729 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7652588', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi099.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTwpORqxF9iz3chL4tw4yrbtah//4i57koBpG6bhfbnaBL2JrJPBf4di8jSm+YAH/YD8q536515s1peMJjqEzg=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/9ghsepfvA6W427iDW4ZktMrfXGxKfbxzC0J3HL+ULgMBV5nXlZzI6yDzw85DCviyvol6Y+ZF+JXcH/Mlg7oY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-11_20:30:01', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-12T02:52:37.730 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-12T02:52:37.732 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-12T02:52:37.752 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T02:52:37.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T02:52:37.824 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:37.824 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T02:52:37.884 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T02:52:37.884 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 181373 Links: 1 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-12 02:51:50.765722087 +0000 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-12 02:45:40.674501711 +0000 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-12 02:45:40.674501711 +0000 2024-04-12T02:52:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-12T02:52:37.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T02:52:38.011 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-12T02:52:38.011 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-12T02:52:38.011 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000232836 s, 2.2 MB/s 2024-04-12T02:52:38.012 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T02:52:38.072 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 172845 Links: 1 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-12 02:51:50.766722068 +0000 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-12 02:45:41.038494741 +0000 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-12 02:45:41.038494741 +0000 2024-04-12T02:52:38.130 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-12T02:52:38.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T02:52:38.196 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-12T02:52:38.196 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-12T02:52:38.196 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000223122 s, 2.3 MB/s 2024-04-12T02:52:38.197 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T02:52:38.256 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-04-12T02:52:38.314 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 182352 Links: 1 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-12 02:51:50.766722068 +0000 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-12 02:45:41.392487962 +0000 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-12 02:45:41.392487962 +0000 2024-04-12T02:52:38.315 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-12T02:52:38.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T02:52:38.381 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-12T02:52:38.381 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-12T02:52:38.381 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000264775 s, 1.9 MB/s 2024-04-12T02:52:38.383 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T02:52:38.441 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 182424 Links: 1 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-12 02:51:50.766722068 +0000 2024-04-12T02:52:38.499 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-12 02:45:41.741481278 +0000 2024-04-12T02:52:38.500 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-12 02:45:41.741481278 +0000 2024-04-12T02:52:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-12T02:52:38.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T02:52:38.565 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-12T02:52:38.565 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-12T02:52:38.565 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000233904 s, 2.2 MB/s 2024-04-12T02:52:38.566 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T02:52:38.624 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:38.624 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T02:52:38.624 INFO:tasks.ceph:Generating config... 2024-04-12T02:52:38.625 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client] setuser = ceph 2024-04-12T02:52:38.626 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-12T02:52:38.626 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-12T02:52:38.626 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-12T02:52:38.627 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-12T02:52:38.627 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-12T02:52:38.627 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:38.627 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:38.627 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-12T02:52:38.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-12T02:52:38.712 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/c2.keyring 2024-04-12T02:52:38.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-12T02:52:38.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T02:52:38.785 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.181')] 2024-04-12T02:52:38.786 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.181'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-12T02:52:38.786 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:38.786 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T02:52:38.842 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.181 --print /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:38.923 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:52:38.923 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: generated fsid c965940f-cb9a-447f-ba03-6d1d6a79fd28 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:setting min_mon_release = pacific 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 0 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:fsid c965940f-cb9a-447f-ba03-6d1d6a79fd28 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:last_changed 2024-04-12T02:52:38.924209+0000 2024-04-12T02:52:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-04-12T02:52:38.924209+0000 2024-04-12T02:52:38.925 INFO:teuthology.orchestra.run.smithi181.stdout:min_mon_release 16 (pacific) 2024-04-12T02:52:38.925 INFO:teuthology.orchestra.run.smithi181.stdout:election_strategy: 1 2024-04-12T02:52:38.925 INFO:teuthology.orchestra.run.smithi181.stdout:0: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.a 2024-04-12T02:52:38.925 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-12T02:52:38.926 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T02:52:38.982 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID c965940f-cb9a-447f-ba03-6d1d6a79fd28... 2024-04-12T02:52:38.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-12T02:52:38.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-12T02:52:39.038 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-04-12T02:52:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2024-04-12T02:52:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T02:52:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T02:52:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T02:52:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.181 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = c965940f-cb9a-447f-ba03-6d1d6a79fd28 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2024-04-12T02:52:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2024-04-12T02:52:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2024-04-12T02:52:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2024-04-12T02:52:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-04-12T02:52:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2024-04-12T02:52:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.068 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-04-12T02:52:39.069 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-04-12T02:52:39.070 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = 172.21.15.181 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c965940f-cb9a-447f-ba03-6d1d6a79fd28 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-04-12T02:52:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.072 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: rocksdb delete range threshold = 0 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-04-12T02:52:39.073 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-12T02:52:39.074 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T02:52:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw = 20 2024-04-12T02:52:39.076 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw sync = 20 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt require ssl = False 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt s3 kms backend = testing 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw curl low speed time = 300 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw data log num shards = 4 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw md log max shards = 4 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync data inject err probability = 0.1 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync log trim interval = 0 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: rgw sync obj etag verify = True 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: setgroup = ceph 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout: setuser = ceph 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-04-12T02:52:39.077 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout: rgw data notify interval msec = 0 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-04-12T02:52:39.078 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T02:52:39.079 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-12T02:52:39.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-12T02:52:39.177 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-12T02:52:39.177 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:39.177 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-12T02:52:39.194 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:39.194 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-12T02:52:39.250 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T02:52:39.250 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:39.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c2.keyring 2024-04-12T02:52:39.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T02:52:39.293 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-04-12T02:52:39.293 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:39.351 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:52:39.351 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:39.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/c2.keyring 2024-04-12T02:52:39.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T02:52:39.387 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:39.387 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:39.444 INFO:tasks.ceph:Setting up mon nodes... 2024-04-12T02:52:39.444 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-12T02:52:39.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-12T02:52:39.538 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-12T02:52:39.540 INFO:tasks.ceph:Setting up mds nodes... 2024-04-12T02:52:39.541 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-12T02:52:39.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-12T02:52:39.584 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-12T02:52:39.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-12T02:52:39.681 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-12T02:52:39.694 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-12T02:52:39.694 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T02:52:39.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-12T02:52:39.761 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:39.761 INFO:tasks.ceph:role: c2.osd.0 2024-04-12T02:52:39.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:52:39.761 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-12T02:52:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:40.273 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-12T02:52:40.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-12T02:52:40.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-12T02:52:40.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-12T02:52:40.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-12T02:52:40.553 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:40.553 INFO:tasks.ceph:role: c2.osd.1 2024-04-12T02:52:40.553 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:52:40.553 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-12T02:52:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:40.627 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:40.627 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:40.627 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:40.628 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:40.628 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:41.049 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-12T02:52:41.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-12T02:52:41.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-12T02:52:41.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-12T02:52:41.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-12T02:52:41.254 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T02:52:41.254 INFO:tasks.ceph:role: c2.osd.2 2024-04-12T02:52:41.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T02:52:41.254 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T02:52:41.328 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T02:52:41.778 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-12T02:52:41.779 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-12T02:52:41.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-12T02:52:41.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-12T02:52:42.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:42.108 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.106+0000 7f6dffc5d7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-12T02:52:42.112 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.110+0000 7f6dffc5d7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-12T02:52:42.112 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.110+0000 7f6dffc5d7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T02:52:42.112 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.110+0000 7f6dffc5d7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T02:52:42.112 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.110+0000 7f6dffc5d7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T02:52:42.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:42.115+0000 7f6dffc5d7c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-12T02:52:44.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-12T02:52:44.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:44.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.318+0000 7f885b16b7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-12T02:52:44.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.318+0000 7f885b16b7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-12T02:52:44.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.318+0000 7f885b16b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T02:52:44.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.318+0000 7f885b16b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T02:52:44.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.318+0000 7f885b16b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T02:52:44.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:44.319+0000 7f885b16b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-12T02:52:46.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-12T02:52:46.435 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.524+0000 7fb1fa2377c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-12T02:52:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.524+0000 7fb1fa2377c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-12T02:52:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.524+0000 7fb1fa2377c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T02:52:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.524+0000 7fb1fa2377c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T02:52:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.524+0000 7fb1fa2377c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T02:52:46.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:52:46.525+0000 7fb1fa2377c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-12T02:52:48.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-12T02:52:48.641 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-12T02:52:48.642 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-12T02:52:48.712 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-12T02:52:48.782 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-12T02:52:48.851 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-12T02:52:48.920 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.921 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-12T02:52:48.978 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:48.979 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-12T02:52:49.036 INFO:tasks.ceph:Adding keys to all mons... 2024-04-12T02:52:49.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/c2.keyring 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD3oRhmttAVIBAAulQnslzhjX6HEwZrG+KdkQ== 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD6oRhm5A53BhAAlfA8hONEIhxG4ZE+rWj/TQ== 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD8oRhmm9IXExAA726VEA5PP0nSYc/dbkRedg== 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-04-12T02:52:49.102 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD+oRhmXxdYHxAA9ms8CZmQ7EQ0TE4lkpohsA== 2024-04-12T02:52:49.103 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-04-12T02:52:49.103 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD3oRhmAPPRIhAA2L9TB/zerTMf2VFgxpJzvA== 2024-04-12T02:52:49.103 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-04-12T02:52:49.103 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD3oRhm4KefKBAAnnLVGGcIcGf+i27JVabqCA== 2024-04-12T02:52:49.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-12T02:52:49.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:49.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:49.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T02:52:49.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T02:52:49.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T02:52:49.471 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-12T02:52:49.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-12T02:52:49.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-12T02:52:49.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-12T02:52:49.623 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-12T02:52:49.680 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-12T02:52:49.680 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-12T02:52:49.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-12T02:52:49.723 INFO:tasks.ceph.c2.mon.a:Started 2024-04-12T02:52:49.723 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-12T02:52:49.723 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-12T02:52:49.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-12T02:52:49.725 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-12T02:52:49.725 INFO:tasks.ceph:Setting crush tunables to default 2024-04-12T02:52:49.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd crush tunables default 2024-04-12T02:52:49.959 INFO:teuthology.orchestra.run.smithi181.stderr:adjusted tunables profile to default 2024-04-12T02:52:49.970 INFO:tasks.ceph:check_enable_crimson: False 2024-04-12T02:52:49.970 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-12T02:52:49.970 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:49.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-12T02:52:49.996 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:49.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-12T02:52:50.063 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-12T02:52:50.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-12T02:52:50.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd new d6f60fc5-cb83-437a-ab67-4d1532e0c3de 0 2024-04-12T02:52:50.385 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-04-12T02:52:50.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd new 25dc5010-9964-4131-9707-67f6f324e10e 1 2024-04-12T02:52:50.433 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:50.432+0000 7ff725f52200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T02:52:50.518 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:50.517+0000 7ff725f52200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T02:52:50.595 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:50.594+0000 7ff725f52200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T02:52:50.611 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-04-12T02:52:50.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd new c8d63640-693b-4c77-8b1c-6e5681a08d98 2 2024-04-12T02:52:50.669 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:50.669+0000 7ff725f52200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T02:52:50.854 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:50.853+0000 7ff725f52200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T02:52:50.880 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-04-12T02:52:50.893 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-12T02:52:50.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-12T02:52:50.936 INFO:tasks.ceph.c2.osd.0:Started 2024-04-12T02:52:50.936 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-12T02:52:50.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-12T02:52:50.938 INFO:tasks.ceph.c2.osd.1:Started 2024-04-12T02:52:50.938 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-12T02:52:50.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-12T02:52:50.940 INFO:tasks.ceph.c2.osd.2:Started 2024-04-12T02:52:50.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T02:52:51.063 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.062+0000 7ff725f52200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T02:52:51.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:52:51.170 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":5,"fsid":"c965940f-cb9a-447f-ba03-6d1d6a79fd28","created":"2024-04-12T02:52:49.861500+0000","modified":"2024-04-12T02:52:50.877820+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T02:52:50.877820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6f60fc5-cb83-437a-ab67-4d1532e0c3de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25dc5010-9964-4131-9707-67f6f324e10e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c8d63640-693b-4c77-8b1c-6e5681a08d98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:51.183 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-12T02:52:51.183 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-12T02:52:51.189 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.188+0000 7ff725f52200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T02:52:51.285 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.284+0000 7ff725f52200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T02:52:51.363 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.362+0000 7ff725f52200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T02:52:51.485 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-12T02:52:51.546 INFO:tasks.ceph.c2.osd.2.smithi181.stderr:2024-04-12T02:52:51.546+0000 7f2425cde7c0 -1 Falling back to public interface 2024-04-12T02:52:51.547 INFO:tasks.ceph.c2.osd.1.smithi181.stderr:2024-04-12T02:52:51.546+0000 7f4c3412d7c0 -1 Falling back to public interface 2024-04-12T02:52:51.547 INFO:tasks.ceph.c2.osd.2.smithi181.stderr:2024-04-12T02:52:51.546+0000 7f241d09c700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-04-12T02:52:51.567 INFO:tasks.ceph.c2.osd.0.smithi181.stderr:2024-04-12T02:52:51.567+0000 7fd2196887c0 -1 Falling back to public interface 2024-04-12T02:52:51.602 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.601+0000 7ff725f52200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T02:52:51.679 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:51.678+0000 7ff725f52200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T02:52:51.699 INFO:teuthology.misc.health.smithi181.stdout: 2024-04-12T02:52:51.699 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":5,"fsid":"c965940f-cb9a-447f-ba03-6d1d6a79fd28","created":"2024-04-12T02:52:49.861500+0000","modified":"2024-04-12T02:52:50.877820+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T02:52:50.877820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6f60fc5-cb83-437a-ab67-4d1532e0c3de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25dc5010-9964-4131-9707-67f6f324e10e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c8d63640-693b-4c77-8b1c-6e5681a08d98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:51.711 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-12T02:52:52.063 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:52.062+0000 7ff725f52200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T02:52:52.386 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:52.385+0000 7ff725f52200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T02:52:52.545 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:52.544+0000 7ff725f52200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T02:52:52.624 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:52.623+0000 7ff725f52200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T02:52:53.328 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:53.327+0000 7ff725f52200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T02:52:53.410 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:53.409+0000 7ff725f52200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T02:52:53.542 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:53.541+0000 7ff725f52200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T02:52:54.056 INFO:tasks.ceph.c2.osd.1.smithi181.stderr:2024-04-12T02:52:54.055+0000 7f4c3412d7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T02:52:54.100 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:54.099+0000 7ff725f52200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T02:52:54.271 INFO:tasks.ceph.c2.osd.2.smithi181.stderr:2024-04-12T02:52:54.270+0000 7f2425cde7c0 -1 osd.2 0 log_to_monitors true 2024-04-12T02:52:54.287 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:54.286+0000 7ff725f52200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T02:52:54.317 INFO:tasks.ceph.c2.osd.0.smithi181.stderr:2024-04-12T02:52:54.316+0000 7fd2196887c0 -1 osd.0 0 log_to_monitors true 2024-04-12T02:52:54.455 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:54.454+0000 7ff725f52200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T02:52:55.289 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:55.289+0000 7ff725f52200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T02:52:55.544 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:55.544+0000 7ff725f52200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T02:52:55.626 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:55.625+0000 7ff725f52200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T02:52:55.631 INFO:tasks.ceph.c2.osd.1.smithi181.stderr:2024-04-12T02:52:55.631+0000 7f4c28ce6700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T02:52:55.644 INFO:tasks.ceph.c2.osd.1.smithi181.stderr:2024-04-12T02:52:55.644+0000 7f4c200bf700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:56.005 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:56.004+0000 7ff725f52200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T02:52:56.926 INFO:tasks.ceph.c2.osd.0.smithi181.stderr:2024-04-12T02:52:56.925+0000 7fd20da40700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T02:52:56.935 INFO:tasks.ceph.c2.osd.0.smithi181.stderr:2024-04-12T02:52:56.935+0000 7fd20561a700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:56.972 INFO:tasks.ceph.c2.osd.2.smithi181.stderr:2024-04-12T02:52:56.971+0000 7f241a096700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T02:52:56.980 INFO:tasks.ceph.c2.osd.2.smithi181.stderr:2024-04-12T02:52:56.979+0000 7f2411c70700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T02:52:57.013 INFO:tasks.ceph.c2.mgr.x.smithi181.stderr:2024-04-12T02:52:57.012+0000 7ff6e11cc700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-12T02:52:58.014 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-12T02:52:58.341 INFO:teuthology.misc.health.smithi181.stdout: 2024-04-12T02:52:58.342 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":9,"fsid":"c965940f-cb9a-447f-ba03-6d1d6a79fd28","created":"2024-04-12T02:52:49.861500+0000","modified":"2024-04-12T02:52:57.926901+0000","last_up_change":"2024-04-12T02:52:57.926901+0000","last_in_change":"2024-04-12T02:52:50.877820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6f60fc5-cb83-437a-ab67-4d1532e0c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":40849},{"type":"v1","addr":"172.21.15.181:6817","nonce":40849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":40849},{"type":"v1","addr":"172.21.15.181:6819","nonce":40849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":40849},{"type":"v1","addr":"172.21.15.181:6823","nonce":40849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":40849},{"type":"v1","addr":"172.21.15.181:6821","nonce":40849}]},"public_addr":"172.21.15.181:6817/40849","cluster_addr":"172.21.15.181:6819/40849","heartbeat_back_addr":"172.21.15.181:6823/40849","heartbeat_front_addr":"172.21.15.181:6821/40849","state":["exists","up"]},{"osd":1,"uuid":"25dc5010-9964-4131-9707-67f6f324e10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":40847},{"type":"v1","addr":"172.21.15.181:6803","nonce":40847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":40847},{"type":"v1","addr":"172.21.15.181:6807","nonce":40847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":40847},{"type":"v1","addr":"172.21.15.181:6815","nonce":40847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":40847},{"type":"v1","addr":"172.21.15.181:6811","nonce":40847}]},"public_addr":"172.21.15.181:6803/40847","cluster_addr":"172.21.15.181:6807/40847","heartbeat_back_addr":"172.21.15.181:6815/40847","heartbeat_front_addr":"172.21.15.181:6811/40847","state":["exists","up"]},{"osd":2,"uuid":"c8d63640-693b-4c77-8b1c-6e5681a08d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":40848},{"type":"v1","addr":"172.21.15.181:6802","nonce":40848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":40848},{"type":"v1","addr":"172.21.15.181:6806","nonce":40848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":40848},{"type":"v1","addr":"172.21.15.181:6814","nonce":40848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":40848},{"type":"v1","addr":"172.21.15.181:6810","nonce":40848}]},"public_addr":"172.21.15.181:6802/40848","cluster_addr":"172.21.15.181:6806/40848","heartbeat_back_addr":"172.21.15.181:6814/40848","heartbeat_front_addr":"172.21.15.181:6810/40848","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.009675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:52:58.354 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-12T02:52:58.354 INFO:tasks.ceph:Creating RBD pool 2024-04-12T02:52:58.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-12T02:52:58.942 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'rbd' created 2024-04-12T02:52:58.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-12T02:52:59.944 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-12T02:52:59.959 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-12T02:52:59.960 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-12T02:52:59.960 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-12T02:52:59.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-12T02:52:59.960 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-12T02:53:00.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:00.423 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":40465},{"type":"v1","addr":"172.21.15.181:6825","nonce":40465}]},"active_addr":"172.21.15.181:6825/40465","active_change":"2024-04-12T02:52:56.006676+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2720874597}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3054796334}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":1408383839}]}]} 2024-04-12T02:53:00.427 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-12T02:53:00.427 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-12T02:53:00.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T02:53:00.752 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:00.752 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":11,"fsid":"c965940f-cb9a-447f-ba03-6d1d6a79fd28","created":"2024-04-12T02:52:49.861500+0000","modified":"2024-04-12T02:52:59.933681+0000","last_up_change":"2024-04-12T02:52:57.926901+0000","last_in_change":"2024-04-12T02:52:50.877820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T02:52:58.051635+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-12T02:52:58.694969+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6f60fc5-cb83-437a-ab67-4d1532e0c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":40849},{"type":"v1","addr":"172.21.15.181:6817","nonce":40849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":40849},{"type":"v1","addr":"172.21.15.181:6819","nonce":40849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":40849},{"type":"v1","addr":"172.21.15.181:6823","nonce":40849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":40849},{"type":"v1","addr":"172.21.15.181:6821","nonce":40849}]},"public_addr":"172.21.15.181:6817/40849","cluster_addr":"172.21.15.181:6819/40849","heartbeat_back_addr":"172.21.15.181:6823/40849","heartbeat_front_addr":"172.21.15.181:6821/40849","state":["exists","up"]},{"osd":1,"uuid":"25dc5010-9964-4131-9707-67f6f324e10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":40847},{"type":"v1","addr":"172.21.15.181:6803","nonce":40847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":40847},{"type":"v1","addr":"172.21.15.181:6807","nonce":40847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":40847},{"type":"v1","addr":"172.21.15.181:6815","nonce":40847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":40847},{"type":"v1","addr":"172.21.15.181:6811","nonce":40847}]},"public_addr":"172.21.15.181:6803/40847","cluster_addr":"172.21.15.181:6807/40847","heartbeat_back_addr":"172.21.15.181:6815/40847","heartbeat_front_addr":"172.21.15.181:6811/40847","state":["exists","up"]},{"osd":2,"uuid":"c8d63640-693b-4c77-8b1c-6e5681a08d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":40848},{"type":"v1","addr":"172.21.15.181:6802","nonce":40848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":40848},{"type":"v1","addr":"172.21.15.181:6806","nonce":40848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":40848},{"type":"v1","addr":"172.21.15.181:6814","nonce":40848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":40848},{"type":"v1","addr":"172.21.15.181:6810","nonce":40848}]},"public_addr":"172.21.15.181:6802/40848","cluster_addr":"172.21.15.181:6806/40848","heartbeat_back_addr":"172.21.15.181:6814/40848","heartbeat_front_addr":"172.21.15.181:6810/40848","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.286403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.009675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.227867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:53:00.766 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-12T02:53:00.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T02:53:01.095 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:01.095 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":12,"fsid":"c965940f-cb9a-447f-ba03-6d1d6a79fd28","created":"2024-04-12T02:52:49.861500+0000","modified":"2024-04-12T02:53:00.939597+0000","last_up_change":"2024-04-12T02:52:57.926901+0000","last_in_change":"2024-04-12T02:52:50.877820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T02:52:58.051635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-12T02:52:58.694969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6f60fc5-cb83-437a-ab67-4d1532e0c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":40849},{"type":"v1","addr":"172.21.15.181:6817","nonce":40849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":40849},{"type":"v1","addr":"172.21.15.181:6819","nonce":40849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":40849},{"type":"v1","addr":"172.21.15.181:6823","nonce":40849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":40849},{"type":"v1","addr":"172.21.15.181:6821","nonce":40849}]},"public_addr":"172.21.15.181:6817/40849","cluster_addr":"172.21.15.181:6819/40849","heartbeat_back_addr":"172.21.15.181:6823/40849","heartbeat_front_addr":"172.21.15.181:6821/40849","state":["exists","up"]},{"osd":1,"uuid":"25dc5010-9964-4131-9707-67f6f324e10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":40847},{"type":"v1","addr":"172.21.15.181:6803","nonce":40847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":40847},{"type":"v1","addr":"172.21.15.181:6807","nonce":40847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":40847},{"type":"v1","addr":"172.21.15.181:6815","nonce":40847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":40847},{"type":"v1","addr":"172.21.15.181:6811","nonce":40847}]},"public_addr":"172.21.15.181:6803/40847","cluster_addr":"172.21.15.181:6807/40847","heartbeat_back_addr":"172.21.15.181:6815/40847","heartbeat_front_addr":"172.21.15.181:6811/40847","state":["exists","up"]},{"osd":2,"uuid":"c8d63640-693b-4c77-8b1c-6e5681a08d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":40848},{"type":"v1","addr":"172.21.15.181:6802","nonce":40848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":40848},{"type":"v1","addr":"172.21.15.181:6806","nonce":40848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":40848},{"type":"v1","addr":"172.21.15.181:6814","nonce":40848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":40848},{"type":"v1","addr":"172.21.15.181:6810","nonce":40848}]},"public_addr":"172.21.15.181:6802/40848","cluster_addr":"172.21.15.181:6806/40848","heartbeat_back_addr":"172.21.15.181:6814/40848","heartbeat_front_addr":"172.21.15.181:6810/40848","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.286403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.009675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T02:52:55.227867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T02:53:01.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-12T02:53:01.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-12T02:53:01.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-12T02:53:01.276 INFO:teuthology.orchestra.run.smithi181.stdout:38654705666 2024-04-12T02:53:01.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-12T02:53:01.277 INFO:teuthology.orchestra.run.smithi181.stdout:34359738370 2024-04-12T02:53:01.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-12T02:53:01.278 INFO:teuthology.orchestra.run.smithi181.stdout:38654705666 2024-04-12T02:53:01.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-12T02:53:01.628 INFO:teuthology.orchestra.run.smithi181.stdout:38654705665 2024-04-12T02:53:01.643 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.2 2024-04-12T02:53:01.662 INFO:teuthology.orchestra.run.smithi181.stdout:34359738369 2024-04-12T02:53:01.670 INFO:teuthology.orchestra.run.smithi181.stdout:38654705665 2024-04-12T02:53:01.677 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-12T02:53:01.685 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.0 2024-04-12T02:53:02.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-12T02:53:02.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-12T02:53:02.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-12T02:53:03.003 INFO:teuthology.orchestra.run.smithi181.stdout:38654705666 2024-04-12T02:53:03.017 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.2 2024-04-12T02:53:03.017 DEBUG:teuthology.parallel:result is None 2024-04-12T02:53:03.037 INFO:teuthology.orchestra.run.smithi181.stdout:34359738370 2024-04-12T02:53:03.041 INFO:teuthology.orchestra.run.smithi181.stdout:38654705666 2024-04-12T02:53:03.053 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2024-04-12T02:53:03.054 DEBUG:teuthology.parallel:result is None 2024-04-12T02:53:03.056 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.0 2024-04-12T02:53:03.056 DEBUG:teuthology.parallel:result is None 2024-04-12T02:53:03.057 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-12T02:53:03.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-12T02:53:03.419 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:03.421 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-04-12T02:53:03.433 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-12T02:53:02.011509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82824,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489976,"statfs":{"total":322122547200,"available":322037735424,"internally_reserved":0,"allocated":1482752,"data_stored":1255484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956652+0000","last_change":"2024-04-12T02:52:59.951363+0000","last_active":"2024-04-12T02:52:59.956652+0000","last_peered":"2024-04-12T02:52:59.956652+0000","last_clean":"2024-04-12T02:52:59.956652+0000","last_became_active":"2024-04-12T02:52:59.951065+0000","last_became_peered":"2024-04-12T02:52:59.951065+0000","last_unstale":"2024-04-12T02:52:59.956652+0000","last_undegraded":"2024-04-12T02:52:59.956652+0000","last_fullsized":"2024-04-12T02:52:59.956652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:25:25.958362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956640+0000","last_change":"2024-04-12T02:52:59.948491+0000","last_active":"2024-04-12T02:52:59.956640+0000","last_peered":"2024-04-12T02:52:59.956640+0000","last_clean":"2024-04-12T02:52:59.956640+0000","last_became_active":"2024-04-12T02:52:59.948219+0000","last_became_peered":"2024-04-12T02:52:59.948219+0000","last_unstale":"2024-04-12T02:52:59.956640+0000","last_undegraded":"2024-04-12T02:52:59.956640+0000","last_fullsized":"2024-04-12T02:52:59.956640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:24:26.264015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956603+0000","last_change":"2024-04-12T02:52:59.946321+0000","last_active":"2024-04-12T02:52:59.956603+0000","last_peered":"2024-04-12T02:52:59.956603+0000","last_clean":"2024-04-12T02:52:59.956603+0000","last_became_active":"2024-04-12T02:52:59.945956+0000","last_became_peered":"2024-04-12T02:52:59.945956+0000","last_unstale":"2024-04-12T02:52:59.956603+0000","last_undegraded":"2024-04-12T02:52:59.956603+0000","last_fullsized":"2024-04-12T02:52:59.956603+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:51:17.249912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.943018+0000","last_change":"2024-04-12T02:52:59.943701+0000","last_active":"2024-04-12T02:53:00.943018+0000","last_peered":"2024-04-12T02:53:00.943018+0000","last_clean":"2024-04-12T02:53:00.943018+0000","last_became_active":"2024-04-12T02:52:59.943421+0000","last_became_peered":"2024-04-12T02:52:59.943421+0000","last_unstale":"2024-04-12T02:53:00.943018+0000","last_undegraded":"2024-04-12T02:53:00.943018+0000","last_fullsized":"2024-04-12T02:53:00.943018+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:04:27.789094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.417761+0000","last_change":"2024-04-12T02:52:59.946165+0000","last_active":"2024-04-12T02:53:00.417761+0000","last_peered":"2024-04-12T02:53:00.417761+0000","last_clean":"2024-04-12T02:53:00.417761+0000","last_became_active":"2024-04-12T02:52:59.945878+0000","last_became_peered":"2024-04-12T02:52:59.945878+0000","last_unstale":"2024-04-12T02:53:00.417761+0000","last_undegraded":"2024-04-12T02:53:00.417761+0000","last_fullsized":"2024-04-12T02:53:00.417761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:25:20.361886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956222+0000","last_change":"2024-04-12T02:52:59.942161+0000","last_active":"2024-04-12T02:52:59.956222+0000","last_peered":"2024-04-12T02:52:59.956222+0000","last_clean":"2024-04-12T02:52:59.956222+0000","last_became_active":"2024-04-12T02:52:59.941894+0000","last_became_peered":"2024-04-12T02:52:59.941894+0000","last_unstale":"2024-04-12T02:52:59.956222+0000","last_undegraded":"2024-04-12T02:52:59.956222+0000","last_fullsized":"2024-04-12T02:52:59.956222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:59:45.633047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.943033+0000","last_change":"2024-04-12T02:52:59.943733+0000","last_active":"2024-04-12T02:53:00.943033+0000","last_peered":"2024-04-12T02:53:00.943033+0000","last_clean":"2024-04-12T02:53:00.943033+0000","last_became_active":"2024-04-12T02:52:59.943443+0000","last_became_peered":"2024-04-12T02:52:59.943443+0000","last_unstale":"2024-04-12T02:53:00.943033+0000","last_undegraded":"2024-04-12T02:53:00.943033+0000","last_fullsized":"2024-04-12T02:53:00.943033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:10:59.436010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:01.220796+0000","last_change":"2024-04-12T02:52:59.956932+0000","last_active":"2024-04-12T02:53:01.220796+0000","last_peered":"2024-04-12T02:53:01.220796+0000","last_clean":"2024-04-12T02:53:01.220796+0000","last_became_active":"2024-04-12T02:52:59.956627+0000","last_became_peered":"2024-04-12T02:52:59.956627+0000","last_unstale":"2024-04-12T02:53:01.220796+0000","last_undegraded":"2024-04-12T02:53:01.220796+0000","last_fullsized":"2024-04-12T02:53:01.220796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T05:53:58.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956598+0000","last_change":"2024-04-12T02:52:59.946202+0000","last_active":"2024-04-12T02:52:59.956598+0000","last_peered":"2024-04-12T02:52:59.956598+0000","last_clean":"2024-04-12T02:52:59.956598+0000","last_became_active":"2024-04-12T02:52:59.945950+0000","last_became_peered":"2024-04-12T02:52:59.945950+0000","last_unstale":"2024-04-12T02:52:59.956598+0000","last_undegraded":"2024-04-12T02:52:59.956598+0000","last_fullsized":"2024-04-12T02:52:59.956598+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:23:53.300003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28008,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829592,"statfs":{"total":107374182400,"available":107345502208,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T02:53:03.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-12T02:53:03.756 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:03.759 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-04-12T02:53:03.770 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-12T02:53:02.011509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82824,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489976,"statfs":{"total":322122547200,"available":322037735424,"internally_reserved":0,"allocated":1482752,"data_stored":1255484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956652+0000","last_change":"2024-04-12T02:52:59.951363+0000","last_active":"2024-04-12T02:52:59.956652+0000","last_peered":"2024-04-12T02:52:59.956652+0000","last_clean":"2024-04-12T02:52:59.956652+0000","last_became_active":"2024-04-12T02:52:59.951065+0000","last_became_peered":"2024-04-12T02:52:59.951065+0000","last_unstale":"2024-04-12T02:52:59.956652+0000","last_undegraded":"2024-04-12T02:52:59.956652+0000","last_fullsized":"2024-04-12T02:52:59.956652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:25:25.958362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956640+0000","last_change":"2024-04-12T02:52:59.948491+0000","last_active":"2024-04-12T02:52:59.956640+0000","last_peered":"2024-04-12T02:52:59.956640+0000","last_clean":"2024-04-12T02:52:59.956640+0000","last_became_active":"2024-04-12T02:52:59.948219+0000","last_became_peered":"2024-04-12T02:52:59.948219+0000","last_unstale":"2024-04-12T02:52:59.956640+0000","last_undegraded":"2024-04-12T02:52:59.956640+0000","last_fullsized":"2024-04-12T02:52:59.956640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:24:26.264015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956603+0000","last_change":"2024-04-12T02:52:59.946321+0000","last_active":"2024-04-12T02:52:59.956603+0000","last_peered":"2024-04-12T02:52:59.956603+0000","last_clean":"2024-04-12T02:52:59.956603+0000","last_became_active":"2024-04-12T02:52:59.945956+0000","last_became_peered":"2024-04-12T02:52:59.945956+0000","last_unstale":"2024-04-12T02:52:59.956603+0000","last_undegraded":"2024-04-12T02:52:59.956603+0000","last_fullsized":"2024-04-12T02:52:59.956603+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:51:17.249912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.943018+0000","last_change":"2024-04-12T02:52:59.943701+0000","last_active":"2024-04-12T02:53:00.943018+0000","last_peered":"2024-04-12T02:53:00.943018+0000","last_clean":"2024-04-12T02:53:00.943018+0000","last_became_active":"2024-04-12T02:52:59.943421+0000","last_became_peered":"2024-04-12T02:52:59.943421+0000","last_unstale":"2024-04-12T02:53:00.943018+0000","last_undegraded":"2024-04-12T02:53:00.943018+0000","last_fullsized":"2024-04-12T02:53:00.943018+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:04:27.789094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.417761+0000","last_change":"2024-04-12T02:52:59.946165+0000","last_active":"2024-04-12T02:53:00.417761+0000","last_peered":"2024-04-12T02:53:00.417761+0000","last_clean":"2024-04-12T02:53:00.417761+0000","last_became_active":"2024-04-12T02:52:59.945878+0000","last_became_peered":"2024-04-12T02:52:59.945878+0000","last_unstale":"2024-04-12T02:53:00.417761+0000","last_undegraded":"2024-04-12T02:53:00.417761+0000","last_fullsized":"2024-04-12T02:53:00.417761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:25:20.361886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956222+0000","last_change":"2024-04-12T02:52:59.942161+0000","last_active":"2024-04-12T02:52:59.956222+0000","last_peered":"2024-04-12T02:52:59.956222+0000","last_clean":"2024-04-12T02:52:59.956222+0000","last_became_active":"2024-04-12T02:52:59.941894+0000","last_became_peered":"2024-04-12T02:52:59.941894+0000","last_unstale":"2024-04-12T02:52:59.956222+0000","last_undegraded":"2024-04-12T02:52:59.956222+0000","last_fullsized":"2024-04-12T02:52:59.956222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:59:45.633047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:00.943033+0000","last_change":"2024-04-12T02:52:59.943733+0000","last_active":"2024-04-12T02:53:00.943033+0000","last_peered":"2024-04-12T02:53:00.943033+0000","last_clean":"2024-04-12T02:53:00.943033+0000","last_became_active":"2024-04-12T02:52:59.943443+0000","last_became_peered":"2024-04-12T02:52:59.943443+0000","last_unstale":"2024-04-12T02:53:00.943033+0000","last_undegraded":"2024-04-12T02:53:00.943033+0000","last_fullsized":"2024-04-12T02:53:00.943033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:10:59.436010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T02:53:01.220796+0000","last_change":"2024-04-12T02:52:59.956932+0000","last_active":"2024-04-12T02:53:01.220796+0000","last_peered":"2024-04-12T02:53:01.220796+0000","last_clean":"2024-04-12T02:53:01.220796+0000","last_became_active":"2024-04-12T02:52:59.956627+0000","last_became_peered":"2024-04-12T02:52:59.956627+0000","last_unstale":"2024-04-12T02:53:01.220796+0000","last_undegraded":"2024-04-12T02:53:01.220796+0000","last_fullsized":"2024-04-12T02:53:01.220796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T05:53:58.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T02:52:59.956598+0000","last_change":"2024-04-12T02:52:59.946202+0000","last_active":"2024-04-12T02:52:59.956598+0000","last_peered":"2024-04-12T02:52:59.956598+0000","last_clean":"2024-04-12T02:52:59.956598+0000","last_became_active":"2024-04-12T02:52:59.945950+0000","last_became_peered":"2024-04-12T02:52:59.945950+0000","last_unstale":"2024-04-12T02:52:59.956598+0000","last_undegraded":"2024-04-12T02:52:59.956598+0000","last_fullsized":"2024-04-12T02:52:59.956598+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T02:52:58.930665+0000","last_clean_scrub_stamp":"2024-04-12T02:52:58.930665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:23:53.300003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28008,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829592,"statfs":{"total":107374182400,"available":107345502208,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T02:53:03.771 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-12T02:53:03.771 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-12T02:53:03.771 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-12T02:53:03.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-12T02:53:04.140 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-12T02:53:04.140 INFO:teuthology.orchestra.run.smithi181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T02:53:04.154 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-12T02:53:04.155 INFO:teuthology.run_tasks:Running task rgw... 2024-04-12T02:53:04.173 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-12T02:53:04.173 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-12T02:53:04.173 INFO:tasks.rgw:Creating data pools 2024-04-12T02:53:04.173 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-12T02:53:04.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-12T02:53:04.589 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' created 2024-04-12T02:53:04.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-12T02:53:05.605 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T02:53:05.623 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-12T02:53:06.610 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' created 2024-04-12T02:53:06.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-12T02:53:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T02:53:07.644 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-12T02:53:07.644 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-12T02:53:07.971 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-12T02:53:07.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-12T02:53:08.610 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T02:53:08.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-12T02:53:08.946 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-12T02:53:08.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-12T02:53:09.614 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T02:53:09.626 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-12T02:53:09.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-12T02:53:10.971 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'default.rgw.buckets.data' created 2024-04-12T02:53:10.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-12T02:53:11.985 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T02:53:12.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-12T02:53:12.978 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'default.rgw.buckets.index' created 2024-04-12T02:53:12.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-12T02:53:13.992 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T02:53:14.010 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-12T02:53:14.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-12T02:53:14.343 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-12T02:53:14.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-12T02:53:14.982 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T02:53:14.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-12T02:53:15.323 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-12T02:53:15.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-12T02:53:15.983 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T02:53:15.994 DEBUG:tasks.rgw:Pools created 2024-04-12T02:53:15.994 INFO:tasks.rgw:Configuring compression type = random 2024-04-12T02:53:15.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-12T02:53:15.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-12T02:53:15.995 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-12T02:53:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:16.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:16.086+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.495+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.495+0000 7f4a72c5b780 20 realm 2024-04-12T02:53:17.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.495+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.496+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.496+0000 7f4a72c5b780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T02:53:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.496+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.498+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.498+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.499+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.499+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.500+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.500+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.500+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.500+0000 7f4a72c5b780 10 Creating default zonegroup 2024-04-12T02:53:17.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.500+0000 7f4a72c5b780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-12T02:53:17.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.512+0000 7f4a72c5b780 10 WARNING: driver->list_zones() returned r=-2 2024-04-12T02:53:17.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.512+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.512+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.514+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.515+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.515+0000 7f4a72c5b780 10 could not read realm id: (2) No such file or directory 2024-04-12T02:53:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.515+0000 7f4a72c5b780 10 WARNING: failed to set zone as default, r=-22 2024-04-12T02:53:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.515+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.515+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.517+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.518+0000 7f4a72c5b780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T02:53:17.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.518+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T02:53:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 10 Using default name default 2024-04-12T02:53:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 zone default found 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 4 Realm: () 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 4 ZoneGroup: default (4bef5cd5-aef8-4ab8-a78d-d7911cafdd2b) 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 4 Zone: default (c783c65c-3f51-4451-80ba-7fe573ec0744) 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 zonegroup default 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.520+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.521+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:17.521+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:19.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:19.513+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:19.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:19.513+0000 7f4a72c5b780 20 rados->read ofs=0 len=0 2024-04-12T02:53:19.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:19.514+0000 7f4a72c5b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:19.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:19.514+0000 7f4a72c5b780 20 started sync module instance, tier type = 2024-04-12T02:53:19.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:19.514+0000 7f4a72c5b780 20 started zone id=c783c65c-3f51-4451-80ba-7fe573ec0744 (name=default) with tier type = 2024-04-12T02:53:21.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=0 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=1 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=2 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=3 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=4 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=5 2024-04-12T02:53:21.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.549+0000 7f4a72c5b780 20 add_watcher() i=6 2024-04-12T02:53:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.550+0000 7f4a72c5b780 20 add_watcher() i=7 2024-04-12T02:53:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.550+0000 7f4a72c5b780 2 all 8 watchers are set, enabling cache 2024-04-12T02:53:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-12T02:53:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-12T02:53:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.551+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.552+0000 7f4a72c5b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-12T02:53:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.552+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-12T02:53:21.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.553+0000 7f4a72c5b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.559+0000 7f4a57bd6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.559+0000 7f4a72c5b780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:21.559+0000 7f4a72c5b780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:24.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.519+0000 7f4a72c5b780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-12T02:53:24.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.519+0000 7f4a72c5b780 5 note: GC not initialized 2024-04-12T02:53:24.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.520+0000 7f4a72c5b780 20 init_complete bucket index max shards: 11 2024-04-12T02:53:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.520+0000 7f4a72c5b780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T02:53:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.520+0000 7f4a513c9700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 02:53:54 2024 2024-04-12T02:53:24.522 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-04-12T02:53:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.521+0000 7f4a50ba7700 20 reqs_thread_entry: start 2024-04-12T02:53:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.521+0000 7f4a533cd700 20 reqs_thread_entry: start 2024-04-12T02:53:24.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.543+0000 7f4a57bd6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:24.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.545+0000 7f4a72c5b780 20 remove_watcher() i=0 2024-04-12T02:53:24.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.545+0000 7f4a72c5b780 2 removed watcher, disabling cache 2024-04-12T02:53:24.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.546+0000 7f4a72c5b780 20 remove_watcher() i=1 2024-04-12T02:53:24.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.547+0000 7f4a72c5b780 20 remove_watcher() i=2 2024-04-12T02:53:24.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.548+0000 7f4a72c5b780 20 remove_watcher() i=3 2024-04-12T02:53:24.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.550+0000 7f4a72c5b780 20 remove_watcher() i=4 2024-04-12T02:53:24.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.551+0000 7f4a72c5b780 20 remove_watcher() i=5 2024-04-12T02:53:24.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.553+0000 7f4a72c5b780 20 remove_watcher() i=6 2024-04-12T02:53:24.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.554+0000 7f4a72c5b780 20 remove_watcher() i=7 2024-04-12T02:53:24.558 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2024-04-12T02:53:24.560 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T02:53:24.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:24.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:24.560 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T02:53:24.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:24.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:24.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.621+0000 7fafc5556780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:24.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.621+0000 7fafc5556780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:24.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.625+0000 7fafa6cca700 20 reqs_thread_entry: start 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "c783c65c-3f51-4451-80ba-7fe573ec0744", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T02:53:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2024-04-12T02:53:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T02:53:24.635 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:24.636 DEBUG:tasks.util.rgw: json result: {'id': 'c783c65c-3f51-4451-80ba-7fe573ec0744', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T02:53:24.636 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-12T02:53:24.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-12T02:53:24.636 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-12T02:53:24.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:24.680 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:24.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.687+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 realm 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.688+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.689+0000 7f134bb7c780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.689+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.689+0000 7f134bb7c780 20 rados_obj.operate() r=0 bl.length=890 2024-04-12T02:53:24.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.689+0000 7f134bb7c780 20 searching for the correct realm 2024-04-12T02:53:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 RGWRados::pool_iterate: got zonegroup_info.4bef5cd5-aef8-4ab8-a78d-d7911cafdd2b 2024-04-12T02:53:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 RGWRados::pool_iterate: got zone_info.c783c65c-3f51-4451-80ba-7fe573ec0744 2024-04-12T02:53:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-12T02:53:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-12T02:53:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 20 zone default found 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 4 Realm: () 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 4 ZoneGroup: default (4bef5cd5-aef8-4ab8-a78d-d7911cafdd2b) 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 4 Zone: default (c783c65c-3f51-4451-80ba-7fe573ec0744) 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.696+0000 7f134bb7c780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 zonegroup default 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados->read ofs=0 len=0 2024-04-12T02:53:24.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:24.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 started sync module instance, tier type = 2024-04-12T02:53:24.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.697+0000 7f134bb7c780 20 started zone id=c783c65c-3f51-4451-80ba-7fe573ec0744 (name=default) with tier type = 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=0 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=1 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=2 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=3 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=4 2024-04-12T02:53:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=5 2024-04-12T02:53:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.728+0000 7f134bb7c780 20 add_watcher() i=6 2024-04-12T02:53:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.731+0000 7f134bb7c780 20 add_watcher() i=7 2024-04-12T02:53:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.731+0000 7f134bb7c780 2 all 8 watchers are set, enabling cache 2024-04-12T02:53:24.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.734+0000 7f1330af7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:24.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.734+0000 7f134bb7c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:24.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.734+0000 7f134bb7c780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.737+0000 7f134bb7c780 5 note: GC not initialized 2024-04-12T02:53:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.738+0000 7f132c2ee700 20 reqs_thread_entry: start 2024-04-12T02:53:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.738+0000 7f134bb7c780 20 init_complete bucket index max shards: 11 2024-04-12T02:53:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.738+0000 7f134bb7c780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T02:53:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.738+0000 7f132a2ea700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 02:53:55 2024 2024-04-12T02:53:24.740 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-04-12T02:53:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.739+0000 7f1329ac8700 20 reqs_thread_entry: start 2024-04-12T02:53:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.745+0000 7f1330af7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:24.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.749+0000 7f134bb7c780 20 remove_watcher() i=0 2024-04-12T02:53:24.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.749+0000 7f134bb7c780 2 removed watcher, disabling cache 2024-04-12T02:53:24.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.752+0000 7f134bb7c780 20 remove_watcher() i=1 2024-04-12T02:53:24.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.752+0000 7f134bb7c780 20 remove_watcher() i=2 2024-04-12T02:53:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.755+0000 7f134bb7c780 20 remove_watcher() i=3 2024-04-12T02:53:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.756+0000 7f134bb7c780 20 remove_watcher() i=4 2024-04-12T02:53:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.760+0000 7f134bb7c780 20 remove_watcher() i=5 2024-04-12T02:53:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.761+0000 7f134bb7c780 20 remove_watcher() i=6 2024-04-12T02:53:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.762+0000 7f134bb7c780 20 remove_watcher() i=7 2024-04-12T02:53:24.765 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2024-04-12T02:53:24.767 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T02:53:24.767 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:24.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:24.767 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T02:53:24.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:24.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.823+0000 7fb53ea4c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.823+0000 7fb53ea4c780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:24.826+0000 7fb5201c0700 20 reqs_thread_entry: start 2024-04-12T02:53:24.832 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:24.832 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "c783c65c-3f51-4451-80ba-7fe573ec0744", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T02:53:24.833 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T02:53:24.834 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2024-04-12T02:53:24.835 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T02:53:24.836 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:24.836 DEBUG:tasks.util.rgw: json result: {'id': 'c783c65c-3f51-4451-80ba-7fe573ec0744', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T02:53:24.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-12T02:53:24.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-12T02:53:24.837 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-12T02:53:24.919 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:24.920 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:24.930 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:24.928+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 realm 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T02:53:26.013 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.011+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.012+0000 7fd5b1639780 10 Creating default zonegroup 2024-04-12T02:53:26.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.013+0000 7fd5b1639780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-12T02:53:26.025 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.023+0000 7fd5b1639780 10 WARNING: driver->list_zones() returned r=-2 2024-04-12T02:53:26.025 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.023+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.025 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.023+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 10 could not read realm id: (2) No such file or directory 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 10 WARNING: failed to set zone as default, r=-22 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.027+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.031 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.030+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.030+0000 7fd5b1639780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T02:53:26.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.030+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T02:53:26.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 10 Using default name default 2024-04-12T02:53:26.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:26.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 zone default found 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 4 Realm: () 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 4 ZoneGroup: default (eb33fa49-c4a8-439c-9a88-273b4a069351) 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 4 Zone: default (e0019de8-2ad9-4d49-95d8-6081c7d39ac7) 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 zonegroup default 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.031+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.032+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:26.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:26.032+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:28.019 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:28.017+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:28.051 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:28.017+0000 7fd5b1639780 20 rados->read ofs=0 len=0 2024-04-12T02:53:28.051 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:28.024+0000 7fd5b1639780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:28.051 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:28.024+0000 7fd5b1639780 20 started sync module instance, tier type = 2024-04-12T02:53:28.051 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:28.024+0000 7fd5b1639780 20 started zone id=e0019de8-2ad9-4d49-95d8-6081c7d39ac7 (name=default) with tier type = 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=0 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=1 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=2 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=3 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=4 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=5 2024-04-12T02:53:30.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.054+0000 7fd5b1639780 20 add_watcher() i=6 2024-04-12T02:53:30.057 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 add_watcher() i=7 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 2 all 8 watchers are set, enabling cache 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.055+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.056+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.056+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-12T02:53:30.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.056+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:30.059 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.056+0000 7fd5b1639780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-12T02:53:30.059 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.056+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-12T02:53:30.059 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.057+0000 7fd5b1639780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T02:53:30.061 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.059+0000 7fd5965b4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:30.061 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.059+0000 7fd5b1639780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:30.061 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:30.059+0000 7fd5b1639780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:33.031 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.029+0000 7fd5b1639780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-12T02:53:33.031 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.029+0000 7fd5b1639780 5 note: GC not initialized 2024-04-12T02:53:33.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.030+0000 7fd591dab700 20 reqs_thread_entry: start 2024-04-12T02:53:33.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.030+0000 7fd5b1639780 20 init_complete bucket index max shards: 11 2024-04-12T02:53:33.032 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.031+0000 7fd5b1639780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T02:53:33.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.031+0000 7fd58fda7700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 02:54:03 2024 2024-04-12T02:53:33.033 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-04-12T02:53:33.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.031+0000 7fd58f585700 20 reqs_thread_entry: start 2024-04-12T02:53:33.053 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.051+0000 7fd5965b4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:33.055 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.053+0000 7fd5b1639780 20 remove_watcher() i=0 2024-04-12T02:53:33.055 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.053+0000 7fd5b1639780 2 removed watcher, disabling cache 2024-04-12T02:53:33.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.056+0000 7fd5b1639780 20 remove_watcher() i=1 2024-04-12T02:53:33.059 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.057+0000 7fd5b1639780 20 remove_watcher() i=2 2024-04-12T02:53:33.061 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.060+0000 7fd5b1639780 20 remove_watcher() i=3 2024-04-12T02:53:33.064 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.062+0000 7fd5b1639780 20 remove_watcher() i=4 2024-04-12T02:53:33.065 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.063+0000 7fd5b1639780 20 remove_watcher() i=5 2024-04-12T02:53:33.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.064+0000 7fd5b1639780 20 remove_watcher() i=6 2024-04-12T02:53:33.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.065+0000 7fd5b1639780 20 remove_watcher() i=7 2024-04-12T02:53:33.069 INFO:teuthology.orchestra.run.smithi181.stdout:[] 2024-04-12T02:53:33.071 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T02:53:33.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:33.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:33.072 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T02:53:33.124 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:33.124 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:33.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.132+0000 7f1da8fc8780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:33.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.132+0000 7f1da8fc8780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:33.137 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.136+0000 7f1d8a73c700 20 reqs_thread_entry: start 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "e0019de8-2ad9-4d49-95d8-6081c7d39ac7", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "default.rgw.control", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T02:53:33.144 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "default.rgw.log", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "", 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "" 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:53:33.145 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "compression_type": "random" 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T02:53:33.146 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "", 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T02:53:33.147 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:53:33.148 DEBUG:tasks.util.rgw: json result: {'id': 'e0019de8-2ad9-4d49-95d8-6081c7d39ac7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T02:53:33.148 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-12T02:53:33.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-12T02:53:33.148 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-12T02:53:33.191 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:33.191 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:33.200 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.198+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 20 realm 2024-04-12T02:53:33.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.199+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:33.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 rados_obj.operate() r=0 bl.length=890 2024-04-12T02:53:33.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.200+0000 7fd955797780 20 searching for the correct realm 2024-04-12T02:53:33.210 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 RGWRados::pool_iterate: got zonegroup_info.eb33fa49-c4a8-439c-9a88-273b4a069351 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 RGWRados::pool_iterate: got zone_info.e0019de8-2ad9-4d49-95d8-6081c7d39ac7 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.207+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 zone default found 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 4 Realm: () 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 4 ZoneGroup: default (eb33fa49-c4a8-439c-9a88-273b4a069351) 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 4 Zone: default (e0019de8-2ad9-4d49-95d8-6081c7d39ac7) 2024-04-12T02:53:33.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 zonegroup default 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.208+0000 7fd955797780 20 rados->read ofs=0 len=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.209+0000 7fd955797780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.209+0000 7fd955797780 20 started sync module instance, tier type = 2024-04-12T02:53:33.212 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.209+0000 7fd955797780 20 started zone id=e0019de8-2ad9-4d49-95d8-6081c7d39ac7 (name=default) with tier type = 2024-04-12T02:53:33.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=0 2024-04-12T02:53:33.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=1 2024-04-12T02:53:33.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=2 2024-04-12T02:53:33.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=3 2024-04-12T02:53:33.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=4 2024-04-12T02:53:33.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=5 2024-04-12T02:53:33.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.237+0000 7fd955797780 20 add_watcher() i=6 2024-04-12T02:53:33.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.238+0000 7fd955797780 20 add_watcher() i=7 2024-04-12T02:53:33.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.238+0000 7fd955797780 2 all 8 watchers are set, enabling cache 2024-04-12T02:53:33.241 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.239+0000 7fd93a712700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:33.241 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.240+0000 7fd955797780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:33.241 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.240+0000 7fd955797780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:33.245 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.243+0000 7fd955797780 5 note: GC not initialized 2024-04-12T02:53:33.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.244+0000 7fd935f09700 20 reqs_thread_entry: start 2024-04-12T02:53:33.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.244+0000 7fd955797780 20 init_complete bucket index max shards: 11 2024-04-12T02:53:33.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.244+0000 7fd955797780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T02:53:33.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.244+0000 7fd933f05700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 02:54:03 2024 2024-04-12T02:53:33.247 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-04-12T02:53:33.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.244+0000 7fd9336e3700 20 reqs_thread_entry: start 2024-04-12T02:53:33.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.252+0000 7fd93a712700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T02:53:33.255 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.253+0000 7fd955797780 20 remove_watcher() i=0 2024-04-12T02:53:33.255 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.253+0000 7fd955797780 2 removed watcher, disabling cache 2024-04-12T02:53:33.256 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.254+0000 7fd955797780 20 remove_watcher() i=1 2024-04-12T02:53:33.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.255+0000 7fd955797780 20 remove_watcher() i=2 2024-04-12T02:53:33.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.256+0000 7fd955797780 20 remove_watcher() i=3 2024-04-12T02:53:33.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.257+0000 7fd955797780 20 remove_watcher() i=4 2024-04-12T02:53:33.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.258+0000 7fd955797780 20 remove_watcher() i=5 2024-04-12T02:53:33.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.259+0000 7fd955797780 20 remove_watcher() i=6 2024-04-12T02:53:33.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.260+0000 7fd955797780 20 remove_watcher() i=7 2024-04-12T02:53:33.263 INFO:teuthology.orchestra.run.smithi181.stdout:[] 2024-04-12T02:53:33.265 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T02:53:33.265 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:33.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T02:53:33.266 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T02:53:33.354 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:33.354 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:33.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.360+0000 7f471e892780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T02:53:33.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.360+0000 7f471e892780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T02:53:33.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:53:33.364+0000 7f4700006700 20 reqs_thread_entry: start 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "e0019de8-2ad9-4d49-95d8-6081c7d39ac7", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "default.rgw.control", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "default.rgw.log", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T02:53:33.372 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "", 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "" 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:53:33.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "compression_type": "random" 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:53:33.374 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.375 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:53:33.375 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:53:33.375 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "", 2024-04-12T02:53:33.375 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T02:53:33.375 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:53:33.376 DEBUG:tasks.util.rgw: json result: {'id': 'e0019de8-2ad9-4d49-95d8-6081c7d39ac7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T02:53:33.376 INFO:tasks.rgw:Starting rgw... 2024-04-12T02:53:33.376 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-12T02:53:33.376 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T02:53:33.376 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-12T02:53:33.377 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-12T02:53:33.381 INFO:tasks.rgw.c1.client.0:Started 2024-04-12T02:53:33.381 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-12T02:53:33.381 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T02:53:33.381 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T02:53:33.382 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T02:53:33.384 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T02:53:33.384 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-12T02:53:33.384 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T02:53:33.384 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-12T02:53:33.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-12T02:53:33.386 INFO:tasks.rgw.c2.client.0:Started 2024-04-12T02:53:33.387 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-12T02:53:33.387 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T02:53:33.387 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-12T02:53:33.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-12T02:53:33.389 INFO:tasks.rgw.c2.client.1:Started 2024-04-12T02:53:33.389 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8000/ 2024-04-12T02:53:33.389 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2024-04-12T02:53:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T02:53:33.433 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:33.433 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:33.433 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T02:53:34.434 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2024-04-12T02:53:34.455 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:34.455 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:34.456 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:53:34.457 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T02:53:34.457 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8001/ 2024-04-12T02:53:34.457 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001/ 2024-04-12T02:53:34.518 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:34.518 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:34.518 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:53:34.519 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T02:53:34.519 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi181.front.sepia.ceph.com:8000/ 2024-04-12T02:53:34.519 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8000/ 2024-04-12T02:53:34.557 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:34.557 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:34.558 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:53:34.559 INFO:teuthology.orchestra.run.smithi181.stdout:anonymous 2024-04-12T02:53:34.559 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi181.front.sepia.ceph.com:8001/ 2024-04-12T02:53:34.559 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8001/ 2024-04-12T02:53:34.619 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:34.619 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:34.620 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:53:34.621 INFO:teuthology.orchestra.run.smithi181.stdout:anonymous 2024-04-12T02:53:34.621 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-12T02:53:37.525 INFO:tasks.rgw_multisite:creating realm.. 2024-04-12T02:53:37.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:37.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:37.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-realm", 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout: "current_period": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2024-04-12T02:53:37.605 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:37.607 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-12T02:53:37.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:37.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:37.666+0000 7f3b976c1780 -1 old format 2024-04-12T02:53:37.676 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:37.676 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T02:53:37.676 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:37.677 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [] 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:53:37.678 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:37.679 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-12T02:53:37.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:37.680 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --endpoints http://smithi099.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:37.774 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:37.774 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:37.788 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=ff072c13-de01-44da-babe-80b386c4a5cf 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a1.rgw.control", 2024-04-12T02:53:37.808 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a1.rgw.log", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a1.rgw.otp", 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:37.809 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a1.rgw.buckets.data" 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:37.810 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-12T02:53:37.811 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:37.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2024-04-12T02:53:38.504 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'a1.rgw.buckets.data' created 2024-04-12T02:53:38.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2024-04-12T02:53:39.508 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2024-04-12T02:53:39.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:39.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:39.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:39.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:39.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a1.rgw.control", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a1.rgw.log", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-12T02:53:39.601 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a1.rgw.otp", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:39.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a1.rgw.buckets.data", 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:39.603 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-12T02:53:39.604 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:39.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:39.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:39.605 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:39.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:39.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:39.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:39.663+0000 7f8b2e6d0780 0 period (49749ba5-837f-416d-b464-cc0e38ff4676 does not have zone b7d2416c-9d6f-4780-9e14-1f66614caba8 configured 2024-04-12T02:53:46.576 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:46.576 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1, 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:53:46.577 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:53:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:46.579 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.580 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T02:53:46.581 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:53:46.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:53:46.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:53:46.584 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:46.585 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-12T02:53:46.586 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T02:53:46.586 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T02:53:46.585+0000 7fa0fc2b0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 45867) UID: 0 2024-04-12T02:53:46.586 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T02:53:46.585+0000 7fa1014857c0 -1 shutting down 2024-04-12T02:53:48.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-12T02:53:52.688 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-12T02:53:52.689 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-12T02:53:52.689 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-12T02:53:52.692 INFO:tasks.rgw.c1.client.0:Started 2024-04-12T02:53:52.692 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2024-04-12T02:53:52.716 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:52.716 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T02:53:52.717 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T02:53:53.718 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2024-04-12T02:53:53.739 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:53:53.739 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:53:53.739 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:53:53.740 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T02:53:53.740 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-12T02:53:53.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:53.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:53.741 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:53.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:53.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:53.869 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "realm-admin", 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Realm Admin", 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "realm-admin", 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:53:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [], 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "system": true, 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:53:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:53:53.881 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T02:53:53.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:53.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:53.882 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --endpoints http://smithi099.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:53.934 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:53.934 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:53.948 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=ff072c13-de01-44da-babe-80b386c4a5cf 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T02:53:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:53:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data" 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T02:53:53.965 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:53.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2024-04-12T02:53:54.502 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'a2.rgw.buckets.data' created 2024-04-12T02:53:54.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2024-04-12T02:53:54.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-12T02:53:55.529 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2024-04-12T02:53:55.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:55.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:55.546 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:55.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:55.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T02:53:55.617 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:53:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T02:53:55.619 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T02:53:55.620 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:53:55.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:55.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:53:55.621 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:53:55.668 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:53:55.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:53:55.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:53:55.679+0000 7f6bdf87e780 0 period (3ee804bf-b01f-4af0-895f-822e923b43dd does not have zone d800b3e1-d3eb-4250-a3c6-cce8e78e25b6 configured 2024-04-12T02:54:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T02:54:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2, 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T02:54:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T02:54:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T02:54:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:54:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:54:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:54:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:54:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:54:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:02.609 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T02:54:02.609 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:02.609 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:02.610 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T02:54:02.610 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T02:54:02.611 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T02:54:02.610+0000 7fe2cd7a8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 45869) UID: 0 2024-04-12T02:54:02.611 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T02:54:02.610+0000 7fe2d297d7c0 -1 shutting down 2024-04-12T02:54:05.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T02:54:08.712 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T02:54:08.712 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T02:54:08.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T02:54:08.715 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T02:54:08.715 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T02:54:08.739 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:08.739 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:08.739 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T02:54:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T02:54:09.740 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T02:54:09.761 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:09.761 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:09.762 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:54:09.763 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T02:54:09.763 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-12T02:54:09.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.764 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:09.823 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:09.823 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:09.863 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:09.863 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:09.864 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "test-realm", 2024-04-12T02:54:09.864 INFO:teuthology.orchestra.run.smithi181.stdout: "current_period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:09.864 INFO:teuthology.orchestra.run.smithi181.stdout: "epoch": 2 2024-04-12T02:54:09.864 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:09.865 INFO:tasks.rgw_multisite:creating zonegroup.. 2024-04-12T02:54:09.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.865 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:09.913 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:09.913 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:09.928 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:54:09.927+0000 7fd57a9e7780 -1 old format 2024-04-12T02:54:09.940 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:09.940 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "b", 2024-04-12T02:54:09.940 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b", 2024-04-12T02:54:09.940 INFO:teuthology.orchestra.run.smithi181.stdout: "api_name": "", 2024-04-12T02:54:09.940 INFO:teuthology.orchestra.run.smithi181.stdout: "is_master": false, 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames": [], 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames_s3website": [], 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "", 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "zones": [], 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_targets": [ 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default-placement", 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "tags": [], 2024-04-12T02:54:09.941 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": [] 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "default_placement": "default-placement", 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_policy": { 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "groups": [] 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled_features": [ 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:09.942 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:09.943 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T02:54:09.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi181.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi181.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:09.944 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --endpoints http://smithi181.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:10.040 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:10.040 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:10.054 INFO:teuthology.orchestra.run.smithi181.stderr:NOTICE: set zone's realm_id=ff072c13-de01-44da-babe-80b386c4a5cf 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b1", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "b1.rgw.control", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "b1.rgw.log", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-12T02:54:10.077 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "b1.rgw.otp", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:10.078 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "b1.rgw.buckets.data" 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:10.079 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:10.080 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:10.080 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:10.080 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-12T02:54:10.080 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:10.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2024-04-12T02:54:11.091 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'b1.rgw.buckets.data' created 2024-04-12T02:54:11.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2024-04-12T02:54:11.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T02:54:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2024-04-12T02:54:12.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:12.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:12.127 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:12.178 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:12.178 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:12.196 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b1", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "b1.rgw.control", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "b1.rgw.log", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-12T02:54:12.197 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "b1.rgw.otp", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:54:12.198 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "b1.rgw.buckets.data", 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "compression_type": "random" 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-12T02:54:12.199 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:12.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:12.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:12.201 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:12.247 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:12.248 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:12.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:54:12.259+0000 7faf5e88a780 0 period (3ee804bf-b01f-4af0-895f-822e923b43dd does not have zone 79aa477c-57c9-443d-814c-33bacc0229a0 configured 2024-04-12T02:54:19.154 INFO:teuthology.orchestra.run.smithi181.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T02:54:19.193 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:19.193 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:19.193 INFO:teuthology.orchestra.run.smithi181.stdout: "epoch": 3, 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": [], 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "period_map": { 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "zonegroups": [ 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "a", 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a", 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "api_name": "", 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "is_master": true, 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:19.194 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames": [], 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames_s3website": [], 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "zones": [ 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a1", 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.195 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a2", 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:19.197 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_targets": [ 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.198 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default-placement", 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "tags": [], 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": [ 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD" 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "default_placement": "default-placement", 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_policy": { 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "groups": [] 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled_features": [ 2024-04-12T02:54:19.199 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "b", 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b", 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "api_name": "", 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "is_master": false, 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames": [], 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames_s3website": [], 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:19.200 INFO:teuthology.orchestra.run.smithi181.stdout: "zones": [ 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b1", 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": false, 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:19.201 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:19.202 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:19.202 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:19.202 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:19.219 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_targets": [ 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default-placement", 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "tags": [], 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": [ 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD" 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.220 INFO:teuthology.orchestra.run.smithi181.stdout: "default_placement": "default-placement", 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_policy": { 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "groups": [] 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled_features": [ 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "short_zone_ids": [ 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2914034528 2024-04-12T02:54:19.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2928306118 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2226213169 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zonegroup": "a", 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:19.222 INFO:teuthology.orchestra.run.smithi181.stdout: "period_config": { 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_quota": { 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "check_on_raw": false, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size": -1, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size_kb": 0, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_objects": -1 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "user_quota": { 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "check_on_raw": false, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size": -1, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size_kb": 0, 2024-04-12T02:54:19.223 INFO:teuthology.orchestra.run.smithi181.stdout: "max_objects": -1 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "user_ratelimit": { 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_ratelimit": { 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:19.224 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "anonymous_ratelimit": { 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_name": "test-realm", 2024-04-12T02:54:19.225 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:54:19.226 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:19.227 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-12T02:54:19.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T02:54:19.228 INFO:tasks.rgw.c2.client.0.smithi181.stdout:2024-04-12T02:54:19.226+0000 7f652d530700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 46320) UID: 0 2024-04-12T02:54:19.228 INFO:tasks.rgw.c2.client.0.smithi181.stdout:2024-04-12T02:54:19.226+0000 7f65327057c0 -1 shutting down 2024-04-12T02:54:24.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-12T02:54:25.329 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-12T02:54:25.329 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-12T02:54:25.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-12T02:54:25.332 INFO:tasks.rgw.c2.client.0:Started 2024-04-12T02:54:25.332 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8000 2024-04-12T02:54:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T02:54:25.357 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:25.357 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:25.357 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi181.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T02:54:26.358 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8000 2024-04-12T02:54:26.381 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:26.381 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:26.382 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:54:26.383 INFO:teuthology.orchestra.run.smithi181.stdout:anonymous 2024-04-12T02:54:26.383 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-12T02:54:26.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:26.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:26.384 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --url http://smithi099.front.sepia.ceph.com:8000 --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:26.483 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:26.483 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "test-realm", 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout: "current_period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout: "epoch": 2 2024-04-12T02:54:26.515 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:26.516 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T02:54:26.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi181.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:26.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--endpoints', 'http://smithi181.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:26.517 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --endpoints http://smithi181.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:26.569 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:26.569 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:26.583 INFO:teuthology.orchestra.run.smithi181.stderr:NOTICE: set zone's realm_id=ff072c13-de01-44da-babe-80b386c4a5cf 2024-04-12T02:54:26.597 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T02:54:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b2", 2024-04-12T02:54:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-12T02:54:26.597 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "b2.rgw.control", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "b2.rgw.log", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-12T02:54:26.598 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "b2.rgw.otp", 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "b2.rgw.buckets.data" 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:26.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-12T02:54:26.600 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:26.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2024-04-12T02:54:27.152 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'b2.rgw.buckets.data' created 2024-04-12T02:54:27.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2024-04-12T02:54:28.173 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2024-04-12T02:54:28.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:28.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:28.190 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:28.240 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:28.240 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b2", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "control_pool": "b2.rgw.control", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-12T02:54:28.258 INFO:teuthology.orchestra.run.smithi181.stdout: "log_pool": "b2.rgw.log", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "otp_pool": "b2.rgw.otp", 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "system_key": { 2024-04-12T02:54:28.259 INFO:teuthology.orchestra.run.smithi181.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_pools": [ 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "default-placement", 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": { 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD": { 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "data_pool": "b2.rgw.buckets.data", 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: "compression_type": "random" 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: "index_type": 0, 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: "inline_data": true 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-12T02:54:28.261 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:28.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:28.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:28.263 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:28.309 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:28.310 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:28.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:54:28.319+0000 7f4dc6d38780 0 period (3ee804bf-b01f-4af0-895f-822e923b43dd does not have zone 79a72dd1-625f-4b14-843f-f3ed404d65dc configured 2024-04-12T02:54:30.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-12T02:54:35.216 INFO:teuthology.orchestra.run.smithi181.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "epoch": 4, 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": [], 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "period_map": { 2024-04-12T02:54:35.250 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "zonegroups": [ 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "a", 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a", 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "api_name": "", 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "is_master": true, 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames": [], 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames_s3website": [], 2024-04-12T02:54:35.251 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "zones": [ 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a1", 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:35.252 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "a2", 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:35.253 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_targets": [ 2024-04-12T02:54:35.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default-placement", 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "tags": [], 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": [ 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD" 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "default_placement": "default-placement", 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_policy": { 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "groups": [] 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled_features": [ 2024-04-12T02:54:35.255 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "b", 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b", 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "api_name": "", 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "is_master": false, 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames": [], 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "hostnames_s3website": [], 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:35.256 INFO:teuthology.orchestra.run.smithi181.stdout: "zones": [ 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b2", 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:35.258 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:35.258 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "b1", 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "endpoints": [ 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.265 INFO:teuthology.orchestra.run.smithi181.stdout: "log_meta": false, 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "log_data": true, 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_index_max_shards": 11, 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "read_only": false, 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "tier_type": "", 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from_all": true, 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_from": [], 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "redirect_zone": "", 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "supported_features": [ 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "compress-encrypted", 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.266 INFO:teuthology.orchestra.run.smithi181.stdout: "placement_targets": [ 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "default-placement", 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "tags": [], 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "storage_classes": [ 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "STANDARD" 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "default_placement": "default-placement", 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_policy": { 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: "groups": [] 2024-04-12T02:54:35.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled_features": [ 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "resharding" 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "short_zone_ids": [ 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2532217463 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2914034528 2024-04-12T02:54:35.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2928306118 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 2226213169 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zonegroup": "a", 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T02:54:35.269 INFO:teuthology.orchestra.run.smithi181.stdout: "period_config": { 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_quota": { 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "check_on_raw": false, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size": -1, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size_kb": 0, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_objects": -1 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "user_quota": { 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "check_on_raw": false, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size": -1, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_size_kb": 0, 2024-04-12T02:54:35.270 INFO:teuthology.orchestra.run.smithi181.stdout: "max_objects": -1 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "user_ratelimit": { 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "bucket_ratelimit": { 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:35.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "anonymous_ratelimit": { 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_ops": 0, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_ops": 0, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "max_read_bytes": 0, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "max_write_bytes": 0, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "enabled": false 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_name": "test-realm", 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:54:35.272 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:54:35.273 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-12T02:54:35.274 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T02:54:35.274 INFO:tasks.rgw.c2.client.1.smithi181.stdout:2024-04-12T02:54:35.274+0000 7f7a3834d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 46321) UID: 0 2024-04-12T02:54:35.274 INFO:tasks.rgw.c2.client.1.smithi181.stdout:2024-04-12T02:54:35.274+0000 7f7a3d5227c0 -1 shutting down 2024-04-12T02:54:36.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-12T02:54:41.374 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-12T02:54:41.375 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-12T02:54:41.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-12T02:54:41.378 INFO:tasks.rgw.c2.client.1:Started 2024-04-12T02:54:41.378 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8001 2024-04-12T02:54:41.401 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:41.401 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T02:54:41.403 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi181.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T02:54:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-12T02:54:42.404 DEBUG:teuthology.orchestra.run.smithi181:> curl http://smithi181.front.sepia.ceph.com:8001 2024-04-12T02:54:42.425 INFO:teuthology.orchestra.run.smithi181.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T02:54:42.426 INFO:teuthology.orchestra.run.smithi181.stderr: Dload Upload Total Spent Left Speed 2024-04-12T02:54:42.426 INFO:teuthology.orchestra.run.smithi181.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T02:54:42.427 INFO:teuthology.orchestra.run.smithi181.stdout:anonymous 2024-04-12T02:54:42.428 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-12T02:54:42.428 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-12T02:54:42.442 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi 2024-04-12T02:54:42.503 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-12T02:54:42.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:42.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T02:54:42.504 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T02:54:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:42.605 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "rgw-multisite-test-user", 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Multisite Test User", 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "rgw-multisite-test-user", 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "JB2ZC967UHS0RDZ3J3O2", 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "a1qAtUK8zEMfL6A1C4V4VRgEAoNIfD6Lg5UP4v0S" 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [ 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "roles", 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "perm": "*" 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T02:54:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2024-04-12T02:54:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2024-04-12T02:54:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2024-04-12T02:54:42.609 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T02:54:42.618 INFO:rgw_multi.tests:meta checkpoint 2024-04-12T02:54:42.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:42.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:42.618 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:54:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:42.694 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:54:42.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.706 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.706 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:54:42.706 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.706 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:54:42.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:54:42.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:42.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:42.707 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:54:42.754 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:42.754 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:42.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:42.781+0000 7fe247efc780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:42.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:42.782+0000 7fe247efc780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:42.794 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:42.794 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:54:42.794 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:54:42.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:54:42.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:42.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:42.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:54:42Z" 2024-04-12T02:54:42.799 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:42.800 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:54:47.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:47.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:47.801 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:54:47.857 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:47.857 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:47.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:47.887+0000 7fc19bae2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:47.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:47.888+0000 7fc19bae2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:54:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:54:47Z" 2024-04-12T02:54:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:47.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:54:52.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:52.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:52.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:54:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:52.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:52.993+0000 7f9ffc315780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:52.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:52.995+0000 7f9ffc315780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:53.007 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:53.007 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:54:53.007 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:54:53.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:54:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:53.014 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:53.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:53.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:53.014 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:53.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:54:52Z" 2024-04-12T02:54:53.015 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:53.016 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:54:58.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:58.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:54:58.018 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:54:58.073 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:54:58.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:54:58.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:58.102+0000 7fcd3c02b780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:58.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:54:58.103+0000 7fcd3c02b780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:54:58.116 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:58.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:58.118 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:58.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:54:58.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:54:58Z" 2024-04-12T02:54:58.121 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:54:58.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:03.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:03.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:03.123 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:03.182 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:03.182 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:03.208+0000 7fb514213780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:03.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:03.209+0000 7fb514213780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:03.222 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:03Z" 2024-04-12T02:55:03.227 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:03.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:08.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:08.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:08.230 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:08.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:08.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:08.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:08.318+0000 7f8d437f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:08.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:08.319+0000 7f8d437f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:08Z" 2024-04-12T02:55:08.337 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:08.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:13.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:13.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:13.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:13.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:13.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:13.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:13.423+0000 7f28ee865780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:13.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:13.424+0000 7f28ee865780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:13.436 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:13Z" 2024-04-12T02:55:13.441 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:13.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:18.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:18.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:18.444 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:18.500 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:18.500 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:18.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:18.530+0000 7f1cd6748780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:18.531+0000 7f1cd6748780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:18.544 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:18.544 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:18.544 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:18.544 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:18.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:18.546 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:18.547 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:18.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:18Z" 2024-04-12T02:55:18.550 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:18.551 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:23.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:23.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:23.553 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:23.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:23.635+0000 7f701fbb0780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:23.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:23.636+0000 7f701fbb0780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:23.648 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:23.649 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:23.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:23.651 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:23.652 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:23Z" 2024-04-12T02:55:23.653 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:23.654 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:28.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:28.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:28.656 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:28.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:28.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:28.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:28.742+0000 7f0d7bac8780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:28.743+0000 7f0d7bac8780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:28.757 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:28.758 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:28.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:28.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:28.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:28Z" 2024-04-12T02:55:28.762 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:28.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:33.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:33.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:33.765 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:33.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:33.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:33.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:33.850+0000 7f826c565780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:33.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:33.851+0000 7f826c565780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:33.863 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:33.864 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:33.865 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:33.867 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:33Z" 2024-04-12T02:55:33.868 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:33.869 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:38.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:38.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:38.871 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:38.926 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:38.926 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:38.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:38.957+0000 7f26fe284780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:38.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:38.958+0000 7f26fe284780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:38Z" 2024-04-12T02:55:38.976 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:38.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:43.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:43.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:43.978 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:44.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:44.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:44.063+0000 7fcad1825780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:44.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:44.064+0000 7fcad1825780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:44.077 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:44.078 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:44.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:44.080 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:44.081 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:44Z" 2024-04-12T02:55:44.082 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:44.083 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:49.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:49.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:49.084 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:49.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:49.171+0000 7f109b4ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:49.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:49.172+0000 7f109b4ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:49.185 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:49.185 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:49Z" 2024-04-12T02:55:49.190 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:49.191 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:54.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:54.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:54.193 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:54.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:54.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:54.279+0000 7f1c94e84780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:54.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:54.280+0000 7f1c94e84780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:54Z" 2024-04-12T02:55:54.298 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:54.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:55:59.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:59.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:55:59.300 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:55:59.356 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:55:59.356 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:55:59.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:59.385+0000 7ff63a2f1780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:55:59.386+0000 7ff63a2f1780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:55:59.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:59.401 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:55:59.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:55:59.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:55:59.404 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:55:59.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:55:59.404 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:55:59Z" 2024-04-12T02:55:59.404 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:55:59.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:04.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:04.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:04.407 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:04.462 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:04.462 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:04.495+0000 7ff5a1c8d780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:04.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:04.496+0000 7ff5a1c8d780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:04.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:04.511 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:04.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:04Z" 2024-04-12T02:56:04.513 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:04.514 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:09.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:09.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:09.516 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:09.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:09.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:09.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:09.605+0000 7f994d0a7780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:09.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:09.606+0000 7f994d0a7780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:09.620 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:09.620 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:09.620 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:09.620 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:09.621 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:09.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:09.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:09.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:09Z" 2024-04-12T02:56:09.625 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:09.626 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:14.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:14.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:14.627 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:14.683 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:14.683 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:14.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:14.716+0000 7fcd41b03780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:14.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:14.717+0000 7fcd41b03780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:14.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:14.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:14.734 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:14.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:14.736 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:14Z" 2024-04-12T02:56:14.737 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:14.738 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:19.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:19.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:19.739 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:19.794 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:19.795 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:19.824+0000 7f32cda88780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:19.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:19.825+0000 7f32cda88780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:19.838 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:19.838 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:19.838 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:19.838 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:19Z" 2024-04-12T02:56:19.843 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:19.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:24.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:24.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:24.845 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:24.901 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:24.901 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:24.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:24.930+0000 7efdeb84b780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:24.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:24.931+0000 7efdeb84b780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:24Z" 2024-04-12T02:56:24.950 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:24.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:29.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:29.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:29.952 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:30.038+0000 7f1c52680780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:30.039+0000 7f1c52680780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.053 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:30.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:30.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:30Z" 2024-04-12T02:56:30.057 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:30.058 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:35.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:35.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:35.060 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:35.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:35.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:35.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:35.146+0000 7f87649b5780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:35.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:35.147+0000 7f87649b5780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.160 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:35.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:35.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:35.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:35.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:35.165 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:35Z" 2024-04-12T02:56:35.165 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:35.166 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712890482.603296_102.1 target= 2024-04-12T02:56:40.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.167 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:40.253+0000 7f975144e780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:40.254+0000 7f975144e780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.266 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:40.266 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:40.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.271 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.273 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:40.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:40.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-12T02:56:40.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.275 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.329 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.330 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.361+0000 7f45d1c1d780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.362+0000 7f45d1c1d780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": { 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "info": { 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "sync", 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "num_shards": 4, 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "markers": [ 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 0, 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 1, 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 1, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 2, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 3, 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.377 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "full_sync": { 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "total": 1, 2024-04-12T02:56:40.378 INFO:teuthology.orchestra.run.smithi181.stdout: "complete": 1 2024-04-12T02:56:40.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.379 INFO:teuthology.orchestra.run.smithi181.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.379 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:56:40.380 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-12T02:56:40.380 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-12T02:56:40.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.380 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.429 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.429 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.460+0000 7f8617ea6780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.461+0000 7f8617ea6780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.473 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:56:40.473 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": { 2024-04-12T02:56:40.473 INFO:teuthology.orchestra.run.smithi181.stdout: "info": { 2024-04-12T02:56:40.473 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "sync", 2024-04-12T02:56:40.473 INFO:teuthology.orchestra.run.smithi181.stdout: "num_shards": 4, 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "markers": [ 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 0, 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.474 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 1, 2024-04-12T02:56:40.475 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.475 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.475 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.475 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 1, 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.477 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 2, 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.478 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 3, 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.479 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: "full_sync": { 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: "total": 1, 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: "complete": 1 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.480 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:56:40.481 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-12T02:56:40.481 INFO:rgw_multi.tests:meta checkpoint 2024-04-12T02:56:40.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.565 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:40.576 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:40.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.576 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.576 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.577 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:40.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.578 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:40.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:40.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.579 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:40.653+0000 7fac67cf8780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:40.654+0000 7fac67cf8780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.666 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:40.666 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:40.666 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:40.666 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:40.666 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.669 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:40.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.671 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:40.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:40.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-12T02:56:40.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.673 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.721 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.721 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.750 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.749+0000 7f5d82480780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.751 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.750+0000 7f5d82480780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.762 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": { 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "info": { 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "sync", 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "num_shards": 4, 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "markers": [ 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 0, 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.763 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 1, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 1, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.764 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 2, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.765 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 3, 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:56:40.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout: "full_sync": { 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout: "total": 1, 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout: "complete": 1 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.767 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:56:40.768 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-12T02:56:40.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-12T02:56:40.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:40.768 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:40.816 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:40.816 INFO:teuthology.orchestra.run.smithi181.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:40.843 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.843+0000 7ff7d45b2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.844 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-12T02:56:40.843+0000 7ff7d45b2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "sync_status": { 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "info": { 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "sync", 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "num_shards": 4, 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: "markers": [ 2024-04-12T02:56:40.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 0, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 1, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 1, 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.856 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 2, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.857 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "key": 3, 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "val": { 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "state": 1, 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "marker": "", 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "next_step_marker": "", 2024-04-12T02:56:40.858 INFO:teuthology.orchestra.run.smithi181.stdout: "total_entries": 0, 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "pos": 0, 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "timestamp": "0.000000", 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "realm_epoch": 2 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "full_sync": { 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "total": 1, 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "complete": 1 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout: "current_time": "2024-04-12T02:56:40Z" 2024-04-12T02:56:40.859 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-12T02:56:40.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-12T02:56:40.860 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-12T02:56:41.241 DEBUG:boto:Using access key provided by client. 2024-04-12T02:56:41.241 DEBUG:boto:Using secret key provided by client. 2024-04-12T02:56:41.242 DEBUG:boto:Using access key provided by client. 2024-04-12T02:56:41.242 DEBUG:boto:Using secret key provided by client. 2024-04-12T02:56:41.242 DEBUG:boto:Using access key provided by client. 2024-04-12T02:56:41.243 DEBUG:boto:Using secret key provided by client. 2024-04-12T02:56:41.243 DEBUG:boto:Using access key provided by client. 2024-04-12T02:56:41.243 DEBUG:boto:Using secret key provided by client. 2024-04-12T02:56:41.243 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-1 2024-04-12T02:56:41.243 DEBUG:boto:path=/dgbpxg-1/ 2024-04-12T02:56:41.244 DEBUG:boto:auth_path=/dgbpxg-1/ 2024-04-12T02:56:41.244 DEBUG:boto:Method: PUT 2024-04-12T02:56:41.244 DEBUG:boto:Path: /dgbpxg-1/ 2024-04-12T02:56:41.244 DEBUG:boto:Data: 2024-04-12T02:56:41.244 DEBUG:boto:Headers: {} 2024-04-12T02:56:41.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:41.244 DEBUG:boto:Port: 8000 2024-04-12T02:56:41.244 DEBUG:boto:Params: {} 2024-04-12T02:56:41.244 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T02:56:41.244 DEBUG:boto:Token: None 2024-04-12T02:56:41.245 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:41 GMT /dgbpxg-1/ 2024-04-12T02:56:41.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NDTuQHEX+fUW76k3CPjy+OQOVAg= 2024-04-12T02:56:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NDTuQHEX+fUW76k3CPjy+OQOVAg=', 'Content-Length': '0'} 2024-04-12T02:56:43.611 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8d06b33531dc645-006618a2e9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:43.612 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-2 2024-04-12T02:56:43.612 DEBUG:boto:path=/dgbpxg-2/ 2024-04-12T02:56:43.612 DEBUG:boto:auth_path=/dgbpxg-2/ 2024-04-12T02:56:43.612 DEBUG:boto:Method: PUT 2024-04-12T02:56:43.612 DEBUG:boto:Path: /dgbpxg-2/ 2024-04-12T02:56:43.612 DEBUG:boto:Data: 2024-04-12T02:56:43.612 DEBUG:boto:Headers: {} 2024-04-12T02:56:43.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:43.612 DEBUG:boto:Port: 8001 2024-04-12T02:56:43.613 DEBUG:boto:Params: {} 2024-04-12T02:56:43.613 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T02:56:43.613 DEBUG:boto:Token: None 2024-04-12T02:56:43.613 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:43 GMT /dgbpxg-2/ 2024-04-12T02:56:43.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:449X6MJXcQAVK1mSN/HOQgLar9U= 2024-04-12T02:56:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:449X6MJXcQAVK1mSN/HOQgLar9U=', 'Content-Length': '0'} 2024-04-12T02:56:46.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d50a52215d064d32-006618a2eb-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:46.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:46.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:46.620 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:46.700 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:46.711 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:46.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.711 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890603.638509_68.1", 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:43.638509Z" 2024-04-12T02:56:46.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.713 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:46.714 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890603.638509_68.1', 3: ''} 2024-04-12T02:56:46.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:46.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:46.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:46.715 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:46.762 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:46.762 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:46.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:46.793+0000 7f4eae4c2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:46.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:46.794+0000 7f4eae4c2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:46.806 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:46.807 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:46.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890603.582320_61.1", 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:46.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:43.582320Z", 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:46.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:46.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:46.812 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:46Z" 2024-04-12T02:56:46.812 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:46.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890603.582320_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:43.582320Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:46Z" } 2024-04-12T02:56:46.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890603.582320_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:43.582320Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:46.813 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890603.638509_68.1', 3: ''} 2024-04-12T02:56:46.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890603.582320_61.1', 3: ''} 2024-04-12T02:56:46.813 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712890603.638509_68.1 target=1_1712890603.582320_61.1 2024-04-12T02:56:51.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:51.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:51.815 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:51.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:51.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:51.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:51.902+0000 7f369cfc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:51.903+0000 7f369cfc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:51.916 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:51.917 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:51.918 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:51.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890603.638509_68.1", 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:43.638509Z", 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:51.920 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:51.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:51Z" 2024-04-12T02:56:51.922 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:51.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890603.638509_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:43.638509Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:51Z" } 2024-04-12T02:56:51.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890603.638509_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:43.638509Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:51.923 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890603.638509_68.1', 3: ''} 2024-04-12T02:56:51.923 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890603.638509_68.1', 3: ''} 2024-04-12T02:56:51.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:51.924 DEBUG:boto:path=/dgbpxg-1/ 2024-04-12T02:56:51.924 DEBUG:boto:auth_path=/dgbpxg-1/ 2024-04-12T02:56:51.924 DEBUG:boto:Method: HEAD 2024-04-12T02:56:51.924 DEBUG:boto:Path: /dgbpxg-1/ 2024-04-12T02:56:51.924 DEBUG:boto:Data: 2024-04-12T02:56:51.924 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:51.924 DEBUG:boto:Port: 8000 2024-04-12T02:56:51.924 DEBUG:boto:Params: {} 2024-04-12T02:56:51.925 DEBUG:boto:Token: None 2024-04-12T02:56:51.925 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-1/ 2024-04-12T02:56:51.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h1hyMDiYoTc5BXxhSeMd8bM9xa0= 2024-04-12T02:56:51.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h1hyMDiYoTc5BXxhSeMd8bM9xa0=', 'Content-Length': '0'} 2024-04-12T02:56:51.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002510990335196d2e-006618a2f3-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.927 DEBUG:boto:path=/dgbpxg-2/ 2024-04-12T02:56:51.927 DEBUG:boto:auth_path=/dgbpxg-2/ 2024-04-12T02:56:51.928 DEBUG:boto:Method: HEAD 2024-04-12T02:56:51.928 DEBUG:boto:Path: /dgbpxg-2/ 2024-04-12T02:56:51.928 DEBUG:boto:Data: 2024-04-12T02:56:51.928 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:51.928 DEBUG:boto:Port: 8000 2024-04-12T02:56:51.928 DEBUG:boto:Params: {} 2024-04-12T02:56:51.928 DEBUG:boto:Token: None 2024-04-12T02:56:51.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-2/ 2024-04-12T02:56:51.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tnI2ygxOYpANRQMKbQpSIVeePrg= 2024-04-12T02:56:51.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tnI2ygxOYpANRQMKbQpSIVeePrg=', 'Content-Length': '0'} 2024-04-12T02:56:51.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b93de972ba5e1f71-006618a2f3-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.930 DEBUG:boto:path=/dgbpxg-1/ 2024-04-12T02:56:51.930 DEBUG:boto:auth_path=/dgbpxg-1/ 2024-04-12T02:56:51.931 DEBUG:boto:Method: HEAD 2024-04-12T02:56:51.931 DEBUG:boto:Path: /dgbpxg-1/ 2024-04-12T02:56:51.931 DEBUG:boto:Data: 2024-04-12T02:56:51.931 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:51.931 DEBUG:boto:Port: 8001 2024-04-12T02:56:51.931 DEBUG:boto:Params: {} 2024-04-12T02:56:51.931 DEBUG:boto:Token: None 2024-04-12T02:56:51.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-1/ 2024-04-12T02:56:51.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h1hyMDiYoTc5BXxhSeMd8bM9xa0= 2024-04-12T02:56:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h1hyMDiYoTc5BXxhSeMd8bM9xa0=', 'Content-Length': '0'} 2024-04-12T02:56:51.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d283841627e9a7-006618a2f3-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.933 DEBUG:boto:path=/dgbpxg-2/ 2024-04-12T02:56:51.934 DEBUG:boto:auth_path=/dgbpxg-2/ 2024-04-12T02:56:51.934 DEBUG:boto:Method: HEAD 2024-04-12T02:56:51.934 DEBUG:boto:Path: /dgbpxg-2/ 2024-04-12T02:56:51.934 DEBUG:boto:Data: 2024-04-12T02:56:51.934 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:51.934 DEBUG:boto:Port: 8001 2024-04-12T02:56:51.934 DEBUG:boto:Params: {} 2024-04-12T02:56:51.934 DEBUG:boto:Token: None 2024-04-12T02:56:51.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-2/ 2024-04-12T02:56:51.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tnI2ygxOYpANRQMKbQpSIVeePrg= 2024-04-12T02:56:51.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tnI2ygxOYpANRQMKbQpSIVeePrg=', 'Content-Length': '0'} 2024-04-12T02:56:51.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022827e75406f1383-006618a2f3-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-12T02:56:51.937 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-3 2024-04-12T02:56:51.937 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:51.938 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:51.938 DEBUG:boto:Method: PUT 2024-04-12T02:56:51.938 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:51.938 DEBUG:boto:Data: 2024-04-12T02:56:51.938 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:51.938 DEBUG:boto:Port: 8000 2024-04-12T02:56:51.938 DEBUG:boto:Params: {} 2024-04-12T02:56:51.938 DEBUG:boto:Token: None 2024-04-12T02:56:51.938 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-3/ 2024-04-12T02:56:51.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ia8wup0sC/m1j7tPyZsGCoo8Ano= 2024-04-12T02:56:51.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ia8wup0sC/m1j7tPyZsGCoo8Ano=', 'Content-Length': '0'} 2024-04-12T02:56:51.958 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb549a1651518fe1-006618a2f3-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.959 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-4 2024-04-12T02:56:51.959 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:51.959 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:51.959 DEBUG:boto:Method: PUT 2024-04-12T02:56:51.959 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:51.959 DEBUG:boto:Data: 2024-04-12T02:56:51.959 DEBUG:boto:Headers: {} 2024-04-12T02:56:51.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:51.959 DEBUG:boto:Port: 8001 2024-04-12T02:56:51.959 DEBUG:boto:Params: {} 2024-04-12T02:56:51.960 DEBUG:boto:Token: None 2024-04-12T02:56:51.960 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:51 GMT /dgbpxg-4/ 2024-04-12T02:56:51.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bqxia8o3rReauOynJBPtitwcARo= 2024-04-12T02:56:51.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bqxia8o3rReauOynJBPtitwcARo=', 'Content-Length': '0'} 2024-04-12T02:56:51.990 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079b27b7402c5d09a-006618a2f3-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:51.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:51.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:51.991 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.073 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890611.978296_76.1", 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:51.978296Z" 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.086 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.087 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:52.088 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.088 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:52.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.088 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.165+0000 7f4c25004780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.166+0000 7f4c25004780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890611.978296_76.1", 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:51.978296Z", 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:52Z" 2024-04-12T02:56:52.183 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:52.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890611.978296_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:51.978296Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:52Z" } 2024-04-12T02:56:52.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890611.978296_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:51.978296Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:52.185 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:52.185 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.185 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.185 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.185 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.185 DEBUG:boto:Data: 2024-04-12T02:56:52.186 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.186 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.186 DEBUG:boto:Params: {} 2024-04-12T02:56:52.186 DEBUG:boto:Token: None 2024-04-12T02:56:52.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004594b52535af99a9-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.189 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.189 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.189 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.189 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.189 DEBUG:boto:Data: 2024-04-12T02:56:52.190 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.190 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.190 DEBUG:boto:Params: {} 2024-04-12T02:56:52.190 DEBUG:boto:Token: None 2024-04-12T02:56:52.190 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e550128842ad270-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.193 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.193 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.193 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.193 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.193 DEBUG:boto:Data: 2024-04-12T02:56:52.193 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.193 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.194 DEBUG:boto:Params: {} 2024-04-12T02:56:52.194 DEBUG:boto:Token: None 2024-04-12T02:56:52.194 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c506eee8004521a6-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.196 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.196 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.196 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.196 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.197 DEBUG:boto:Data: 2024-04-12T02:56:52.197 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.197 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.197 DEBUG:boto:Params: {} 2024-04-12T02:56:52.197 DEBUG:boto:Token: None 2024-04-12T02:56:52.197 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcd59dd06585253f-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.199 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.199 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.200 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.200 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.200 DEBUG:boto:Data: 2024-04-12T02:56:52.200 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.200 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.200 DEBUG:boto:Params: {} 2024-04-12T02:56:52.200 DEBUG:boto:Token: None 2024-04-12T02:56:52.200 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jaigRu/0cEzbv++hYcztR1pmPx4= 2024-04-12T02:56:52.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jaigRu/0cEzbv++hYcztR1pmPx4=', 'Content-Length': '0'} 2024-04-12T02:56:52.203 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec76e0e8af77c164-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.203 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.203 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.203 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.203 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.203 DEBUG:boto:Data: 2024-04-12T02:56:52.203 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.203 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.204 DEBUG:boto:Params: {} 2024-04-12T02:56:52.204 DEBUG:boto:Token: None 2024-04-12T02:56:52.204 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GwcntMIawj2Fz0E+trjQS2wQxO0= 2024-04-12T02:56:52.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GwcntMIawj2Fz0E+trjQS2wQxO0=', 'Content-Length': '0'} 2024-04-12T02:56:52.206 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cb7e27e2e0e411d6-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.206 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.207 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.207 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.207 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.207 DEBUG:boto:Data: 2024-04-12T02:56:52.207 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.207 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.207 DEBUG:boto:Params: {} 2024-04-12T02:56:52.207 DEBUG:boto:Token: None 2024-04-12T02:56:52.207 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jaigRu/0cEzbv++hYcztR1pmPx4= 2024-04-12T02:56:52.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jaigRu/0cEzbv++hYcztR1pmPx4=', 'Content-Length': '0'} 2024-04-12T02:56:52.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009dbf02d057a9ef1e-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.212 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.212 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.212 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.212 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.212 DEBUG:boto:Data: 2024-04-12T02:56:52.213 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.213 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.213 DEBUG:boto:Params: {} 2024-04-12T02:56:52.213 DEBUG:boto:Token: None 2024-04-12T02:56:52.213 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GwcntMIawj2Fz0E+trjQS2wQxO0= 2024-04-12T02:56:52.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GwcntMIawj2Fz0E+trjQS2wQxO0=', 'Content-Length': '0'} 2024-04-12T02:56:52.257 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b830bb722e1be6b-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.258 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.258 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.258 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.258 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.258 DEBUG:boto:Data: 2024-04-12T02:56:52.258 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.258 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.258 DEBUG:boto:Params: {} 2024-04-12T02:56:52.259 DEBUG:boto:Token: None 2024-04-12T02:56:52.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eeb10d739928cc0-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.261 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.261 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.261 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.262 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.262 DEBUG:boto:Data: 2024-04-12T02:56:52.262 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.262 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.262 DEBUG:boto:Params: {} 2024-04-12T02:56:52.262 DEBUG:boto:Token: None 2024-04-12T02:56:52.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbc8e2c7142071fa-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.265 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.265 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.265 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.265 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.265 DEBUG:boto:Data: 2024-04-12T02:56:52.265 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.266 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.266 DEBUG:boto:Params: {} 2024-04-12T02:56:52.266 DEBUG:boto:Token: None 2024-04-12T02:56:52.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002435d6fc0493c148-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.268 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.268 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.268 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.268 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.268 DEBUG:boto:Data: 2024-04-12T02:56:52.269 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.269 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.269 DEBUG:boto:Params: {} 2024-04-12T02:56:52.269 DEBUG:boto:Token: None 2024-04-12T02:56:52.269 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bef88c84e3c37e85-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.271 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890611.978296_76.1", 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:51.978296Z" 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.365 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:52.366 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.366 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:52.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.366 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.446+0000 7fe1bc357780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.447+0000 7fe1bc357780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.459 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:52.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.462 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890611.978296_76.1", 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.463 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:51.978296Z", 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.464 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:52.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.466 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:52Z" 2024-04-12T02:56:52.466 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:52.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890611.978296_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:51.978296Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:52Z" } 2024-04-12T02:56:52.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890611.978296_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:51.978296Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:52.467 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890611.978296_76.1', 3: ''} 2024-04-12T02:56:52.467 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:52.467 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.468 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.468 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.468 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.468 DEBUG:boto:Data: 2024-04-12T02:56:52.468 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.468 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.468 DEBUG:boto:Params: {} 2024-04-12T02:56:52.468 DEBUG:boto:Token: None 2024-04-12T02:56:52.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001805c86b673b4bdd-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.471 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.471 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.471 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.472 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.472 DEBUG:boto:Data: 2024-04-12T02:56:52.472 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.472 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.472 DEBUG:boto:Params: {} 2024-04-12T02:56:52.472 DEBUG:boto:Token: None 2024-04-12T02:56:52.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a07c717815d74a-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.474 DEBUG:boto:path=/dgbpxg-3/ 2024-04-12T02:56:52.475 DEBUG:boto:auth_path=/dgbpxg-3/ 2024-04-12T02:56:52.475 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.475 DEBUG:boto:Path: /dgbpxg-3/ 2024-04-12T02:56:52.475 DEBUG:boto:Data: 2024-04-12T02:56:52.475 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.475 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.475 DEBUG:boto:Params: {} 2024-04-12T02:56:52.475 DEBUG:boto:Token: None 2024-04-12T02:56:52.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-3/ 2024-04-12T02:56:52.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM= 2024-04-12T02:56:52.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yp9Y3dqAqHa/AftkhelPiUVUSwM=', 'Content-Length': '0'} 2024-04-12T02:56:52.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c17b4a230496ae11-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.478 DEBUG:boto:path=/dgbpxg-4/ 2024-04-12T02:56:52.478 DEBUG:boto:auth_path=/dgbpxg-4/ 2024-04-12T02:56:52.478 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.478 DEBUG:boto:Path: /dgbpxg-4/ 2024-04-12T02:56:52.479 DEBUG:boto:Data: 2024-04-12T02:56:52.479 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.479 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.479 DEBUG:boto:Params: {} 2024-04-12T02:56:52.479 DEBUG:boto:Token: None 2024-04-12T02:56:52.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-4/ 2024-04-12T02:56:52.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M= 2024-04-12T02:56:52.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8wZLLj09LCtYKfy1rlvNQOKg+4M=', 'Content-Length': '0'} 2024-04-12T02:56:52.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000364788accb9ec3fb-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-12T02:56:52.482 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-5 2024-04-12T02:56:52.482 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:52.482 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:52.482 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.482 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:52.482 DEBUG:boto:Data: 2024-04-12T02:56:52.483 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.483 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.483 DEBUG:boto:Params: {} 2024-04-12T02:56:52.483 DEBUG:boto:Token: None 2024-04-12T02:56:52.483 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-5/ 2024-04-12T02:56:52.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fao/j/BrKOF9JHzox9z738RkEZw= 2024-04-12T02:56:52.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fao/j/BrKOF9JHzox9z738RkEZw=', 'Content-Length': '0'} 2024-04-12T02:56:52.501 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ac6c25c12eaf85c-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.501 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-6 2024-04-12T02:56:52.501 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:52.501 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:52.501 DEBUG:boto:Method: PUT 2024-04-12T02:56:52.501 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:52.501 DEBUG:boto:Data: 2024-04-12T02:56:52.501 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.502 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.502 DEBUG:boto:Params: {} 2024-04-12T02:56:52.502 DEBUG:boto:Token: None 2024-04-12T02:56:52.502 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-6/ 2024-04-12T02:56:52.502 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UuyK2vUrSwDt4eMNjtCDNko2IVA= 2024-04-12T02:56:52.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UuyK2vUrSwDt4eMNjtCDNko2IVA=', 'Content-Length': '0'} 2024-04-12T02:56:52.528 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009181dce20393e01f-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.606 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890612.515464_84.1", 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:52.515464Z" 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.624 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:52.625 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.515464_84.1', 3: ''} 2024-04-12T02:56:52.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:52.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.707+0000 7ff3fe5e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.708+0000 7ff3fe5e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890612.515464_84.1", 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:52.515464Z", 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:52Z" 2024-04-12T02:56:52.726 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:52.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890612.515464_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:52.515464Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:52Z" } 2024-04-12T02:56:52.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890612.515464_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:52.515464Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:52.727 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.515464_84.1', 3: ''} 2024-04-12T02:56:52.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.515464_84.1', 3: ''} 2024-04-12T02:56:52.728 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:52.728 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:52.728 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:52.728 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.728 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:52.728 DEBUG:boto:Data: 2024-04-12T02:56:52.728 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.729 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.729 DEBUG:boto:Params: {} 2024-04-12T02:56:52.729 DEBUG:boto:Token: None 2024-04-12T02:56:52.729 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-5/ 2024-04-12T02:56:52.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xoqz2f98gqb+eieZMw/cu5A9G0s= 2024-04-12T02:56:52.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xoqz2f98gqb+eieZMw/cu5A9G0s=', 'Content-Length': '0'} 2024-04-12T02:56:52.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009130bab867a4c34d-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.731 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:52.731 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:52.732 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.732 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:52.732 DEBUG:boto:Data: 2024-04-12T02:56:52.732 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.732 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.732 DEBUG:boto:Params: {} 2024-04-12T02:56:52.732 DEBUG:boto:Token: None 2024-04-12T02:56:52.732 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-6/ 2024-04-12T02:56:52.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xCOstcvln5FnF0zJ/eO1nmfLKss= 2024-04-12T02:56:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCOstcvln5FnF0zJ/eO1nmfLKss=', 'Content-Length': '0'} 2024-04-12T02:56:52.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000428da5d238cdaf90-006618a2f4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.734 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:52.734 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:52.735 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.735 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:52.735 DEBUG:boto:Data: 2024-04-12T02:56:52.735 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.735 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.735 DEBUG:boto:Params: {} 2024-04-12T02:56:52.735 DEBUG:boto:Token: None 2024-04-12T02:56:52.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-5/ 2024-04-12T02:56:52.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xoqz2f98gqb+eieZMw/cu5A9G0s= 2024-04-12T02:56:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xoqz2f98gqb+eieZMw/cu5A9G0s=', 'Content-Length': '0'} 2024-04-12T02:56:52.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de1087d3264bf1e8-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.738 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:52.738 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:52.738 DEBUG:boto:Method: HEAD 2024-04-12T02:56:52.738 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:52.738 DEBUG:boto:Data: 2024-04-12T02:56:52.738 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.738 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.738 DEBUG:boto:Params: {} 2024-04-12T02:56:52.739 DEBUG:boto:Token: None 2024-04-12T02:56:52.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-6/ 2024-04-12T02:56:52.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xCOstcvln5FnF0zJ/eO1nmfLKss= 2024-04-12T02:56:52.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCOstcvln5FnF0zJ/eO1nmfLKss=', 'Content-Length': '0'} 2024-04-12T02:56:52.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd803ef4874cb374-006618a2f4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.741 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:52.741 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:52.741 DEBUG:boto:Method: DELETE 2024-04-12T02:56:52.741 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:52.741 DEBUG:boto:Data: 2024-04-12T02:56:52.742 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:52.742 DEBUG:boto:Port: 8000 2024-04-12T02:56:52.742 DEBUG:boto:Params: {} 2024-04-12T02:56:52.742 DEBUG:boto:Token: None 2024-04-12T02:56:52.742 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-5/ 2024-04-12T02:56:52.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mBhPwB2r7H+5UckH+vbm8uWRXlU= 2024-04-12T02:56:52.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mBhPwB2r7H+5UckH+vbm8uWRXlU=', 'Content-Length': '0'} 2024-04-12T02:56:52.767 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e69e636136a71bba-006618a2f4-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.768 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:52.768 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:52.768 DEBUG:boto:Method: DELETE 2024-04-12T02:56:52.768 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:52.768 DEBUG:boto:Data: 2024-04-12T02:56:52.768 DEBUG:boto:Headers: {} 2024-04-12T02:56:52.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:52.768 DEBUG:boto:Port: 8001 2024-04-12T02:56:52.768 DEBUG:boto:Params: {} 2024-04-12T02:56:52.769 DEBUG:boto:Token: None 2024-04-12T02:56:52.769 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 02:56:52 GMT /dgbpxg-6/ 2024-04-12T02:56:52.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GDp4XUigBn9uW6fLMGS2OjZbERI= 2024-04-12T02:56:52.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GDp4XUigBn9uW6fLMGS2OjZbERI=', 'Content-Length': '0'} 2024-04-12T02:56:52.825 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086fa4ec39aab2501-006618a2f4-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:56:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:52.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.826 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890612.796596_92.1", 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:52.796596Z" 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:52.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:52.918 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:52.919 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.796596_92.1', 3: ''} 2024-04-12T02:56:52.919 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:52.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:52.919 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:52.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.995+0000 7fd93f912780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:52.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:52.996+0000 7fd93f912780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:53.007 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890612.796596_92.1", 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:52.796596Z", 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:53.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:52Z" 2024-04-12T02:56:53.013 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:53.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890612.796596_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:52.796596Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:52Z" } 2024-04-12T02:56:53.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890612.796596_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:52.796596Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:53.014 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.796596_92.1', 3: ''} 2024-04-12T02:56:53.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890612.796596_92.1', 3: ''} 2024-04-12T02:56:53.014 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:53.015 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:53.015 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:53.015 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.015 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:53.015 DEBUG:boto:Data: 2024-04-12T02:56:53.015 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.015 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.015 DEBUG:boto:Params: {} 2024-04-12T02:56:53.015 DEBUG:boto:Token: None 2024-04-12T02:56:53.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-5/ 2024-04-12T02:56:53.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IgSjMqO0+3cwbFNnzdESmBAVWjo= 2024-04-12T02:56:53.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IgSjMqO0+3cwbFNnzdESmBAVWjo=', 'Content-Length': '0'} 2024-04-12T02:56:53.017 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000008ec09da5d45f376-006618a2f5-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.017 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:53.017 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:53.018 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.018 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:53.018 DEBUG:boto:Data: 2024-04-12T02:56:53.018 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.018 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.018 DEBUG:boto:Params: {} 2024-04-12T02:56:53.018 DEBUG:boto:Token: None 2024-04-12T02:56:53.018 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-6/ 2024-04-12T02:56:53.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iCEWr7kScgtN+jYX4mquUu2JBd0= 2024-04-12T02:56:53.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iCEWr7kScgtN+jYX4mquUu2JBd0=', 'Content-Length': '0'} 2024-04-12T02:56:53.020 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000828d0d6deaa73b9a-006618a2f5-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.020 DEBUG:boto:path=/dgbpxg-5/ 2024-04-12T02:56:53.020 DEBUG:boto:auth_path=/dgbpxg-5/ 2024-04-12T02:56:53.020 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.020 DEBUG:boto:Path: /dgbpxg-5/ 2024-04-12T02:56:53.020 DEBUG:boto:Data: 2024-04-12T02:56:53.020 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.020 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.021 DEBUG:boto:Params: {} 2024-04-12T02:56:53.021 DEBUG:boto:Token: None 2024-04-12T02:56:53.021 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-5/ 2024-04-12T02:56:53.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IgSjMqO0+3cwbFNnzdESmBAVWjo= 2024-04-12T02:56:53.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IgSjMqO0+3cwbFNnzdESmBAVWjo=', 'Content-Length': '0'} 2024-04-12T02:56:53.022 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000009f5d0d1c2e9dcebf-006618a2f5-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.023 DEBUG:boto:path=/dgbpxg-6/ 2024-04-12T02:56:53.023 DEBUG:boto:auth_path=/dgbpxg-6/ 2024-04-12T02:56:53.023 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.023 DEBUG:boto:Path: /dgbpxg-6/ 2024-04-12T02:56:53.023 DEBUG:boto:Data: 2024-04-12T02:56:53.023 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.023 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.024 DEBUG:boto:Params: {} 2024-04-12T02:56:53.024 DEBUG:boto:Token: None 2024-04-12T02:56:53.024 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-6/ 2024-04-12T02:56:53.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iCEWr7kScgtN+jYX4mquUu2JBd0= 2024-04-12T02:56:53.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iCEWr7kScgtN+jYX4mquUu2JBd0=', 'Content-Length': '0'} 2024-04-12T02:56:53.025 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000080d8128bcff50e86-006618a2f5-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-12T02:56:53.026 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-7 2024-04-12T02:56:53.026 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:56:53.026 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:56:53.026 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.026 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:56:53.026 DEBUG:boto:Data: 2024-04-12T02:56:53.026 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.027 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.027 DEBUG:boto:Params: {} 2024-04-12T02:56:53.027 DEBUG:boto:Token: None 2024-04-12T02:56:53.027 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/ 2024-04-12T02:56:53.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bd8MtgYAkzNLat66/vwslj40RPM= 2024-04-12T02:56:53.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bd8MtgYAkzNLat66/vwslj40RPM=', 'Content-Length': '0'} 2024-04-12T02:56:53.046 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000502fd497a7a1a89-006618a2f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.046 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-8 2024-04-12T02:56:53.046 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:56:53.046 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:56:53.046 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.046 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:56:53.047 DEBUG:boto:Data: 2024-04-12T02:56:53.047 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.047 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.047 DEBUG:boto:Params: {} 2024-04-12T02:56:53.047 DEBUG:boto:Token: None 2024-04-12T02:56:53.047 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/ 2024-04-12T02:56:53.047 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C394iKvvqeVCZcOYT7aJ+6dhk+A= 2024-04-12T02:56:53.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C394iKvvqeVCZcOYT7aJ+6dhk+A=', 'Content-Length': '0'} 2024-04-12T02:56:53.080 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f25a2d1fb32bf454-006618a2f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.080 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:56:53.081 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:56:53.081 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.081 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:56:53.081 DEBUG:boto:Data: 2024-04-12T02:56:53.081 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.081 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.081 DEBUG:boto:Params: {} 2024-04-12T02:56:53.082 DEBUG:boto:Token: None 2024-04-12T02:56:53.082 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/ 2024-04-12T02:56:53.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY= 2024-04-12T02:56:53.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY=', 'Content-Length': '0'} 2024-04-12T02:56:53.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036de11437f074396-006618a2f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.084 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:56:53.084 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:56:53.084 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.085 DEBUG:boto:Path: /dgbpxg-7/myobj 2024-04-12T02:56:53.085 DEBUG:boto:Data: 2024-04-12T02:56:53.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.085 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.085 DEBUG:boto:Params: {} 2024-04-12T02:56:53.085 DEBUG:boto:Token: None 2024-04-12T02:56:53.085 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/myobj 2024-04-12T02:56:53.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KKZ280dHa2o4zilJGP8LTfuviY= 2024-04-12T02:56:53.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KKZ280dHa2o4zilJGP8LTfuviY='} 2024-04-12T02:56:53.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007ba8aa6b0df9134-006618a2f5-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.100 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:56:53.100 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:56:53.100 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.100 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:56:53.100 DEBUG:boto:Data: 2024-04-12T02:56:53.100 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.101 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.101 DEBUG:boto:Params: {} 2024-04-12T02:56:53.101 DEBUG:boto:Token: None 2024-04-12T02:56:53.101 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/ 2024-04-12T02:56:53.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY= 2024-04-12T02:56:53.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY=', 'Content-Length': '0'} 2024-04-12T02:56:53.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037f5a3d8dfda246c-006618a2f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.103 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:56:53.103 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:56:53.103 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.103 DEBUG:boto:Path: /dgbpxg-7/_myobj 2024-04-12T02:56:53.104 DEBUG:boto:Data: 2024-04-12T02:56:53.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.104 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.104 DEBUG:boto:Params: {} 2024-04-12T02:56:53.104 DEBUG:boto:Token: None 2024-04-12T02:56:53.104 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/_myobj 2024-04-12T02:56:53.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X4Hy5SGmGBqJmu1tk5B4IrYQm2k= 2024-04-12T02:56:53.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X4Hy5SGmGBqJmu1tk5B4IrYQm2k='} 2024-04-12T02:56:53.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025ce2ec8ef8b0b2d-006618a2f5-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.147 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:56:53.147 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:56:53.147 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.147 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:56:53.147 DEBUG:boto:Data: 2024-04-12T02:56:53.147 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.147 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.147 DEBUG:boto:Params: {} 2024-04-12T02:56:53.148 DEBUG:boto:Token: None 2024-04-12T02:56:53.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/ 2024-04-12T02:56:53.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY= 2024-04-12T02:56:53.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY=', 'Content-Length': '0'} 2024-04-12T02:56:53.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065ef0d7acb0145b5-006618a2f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.150 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:56:53.150 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:56:53.151 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.151 DEBUG:boto:Path: /dgbpxg-7/%3A 2024-04-12T02:56:53.151 DEBUG:boto:Data: 2024-04-12T02:56:53.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.151 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.151 DEBUG:boto:Params: {} 2024-04-12T02:56:53.151 DEBUG:boto:Token: None 2024-04-12T02:56:53.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/%3A 2024-04-12T02:56:53.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XCZ5XHJEVr2FI0MHrjgBIGpT8BM= 2024-04-12T02:56:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XCZ5XHJEVr2FI0MHrjgBIGpT8BM='} 2024-04-12T02:56:53.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e503d5c42982d3b3-006618a2f5-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.159 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:56:53.159 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:56:53.159 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.159 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:56:53.160 DEBUG:boto:Data: 2024-04-12T02:56:53.160 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.160 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.160 DEBUG:boto:Params: {} 2024-04-12T02:56:53.160 DEBUG:boto:Token: None 2024-04-12T02:56:53.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/ 2024-04-12T02:56:53.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY= 2024-04-12T02:56:53.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y2P2W83MpiRSUwcHBFynfKNu8uY=', 'Content-Length': '0'} 2024-04-12T02:56:53.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a251165169de9a18-006618a2f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.163 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:56:53.163 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:56:53.163 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.163 DEBUG:boto:Path: /dgbpxg-7/%26 2024-04-12T02:56:53.163 DEBUG:boto:Data: 2024-04-12T02:56:53.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:56:53.163 DEBUG:boto:Port: 8000 2024-04-12T02:56:53.163 DEBUG:boto:Params: {} 2024-04-12T02:56:53.163 DEBUG:boto:Token: None 2024-04-12T02:56:53.163 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-7/%26 2024-04-12T02:56:53.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2HwinrvHdsFldyvFTEp1o9I9FPw= 2024-04-12T02:56:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2HwinrvHdsFldyvFTEp1o9I9FPw='} 2024-04-12T02:56:53.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b2cc7bac527d12c-006618a2f5-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.207 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:56:53.207 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:56:53.207 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.207 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:56:53.207 DEBUG:boto:Data: 2024-04-12T02:56:53.207 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.207 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.207 DEBUG:boto:Params: {} 2024-04-12T02:56:53.207 DEBUG:boto:Token: None 2024-04-12T02:56:53.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/ 2024-04-12T02:56:53.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ= 2024-04-12T02:56:53.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ=', 'Content-Length': '0'} 2024-04-12T02:56:53.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3afa8f1943dd3ee-006618a2f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.211 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:56:53.211 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:56:53.211 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.211 DEBUG:boto:Path: /dgbpxg-8/myobj 2024-04-12T02:56:53.211 DEBUG:boto:Data: 2024-04-12T02:56:53.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.211 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.211 DEBUG:boto:Params: {} 2024-04-12T02:56:53.211 DEBUG:boto:Token: None 2024-04-12T02:56:53.212 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/myobj 2024-04-12T02:56:53.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XucFsyHwL0+cFDZWBb9rUSNJ250= 2024-04-12T02:56:53.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XucFsyHwL0+cFDZWBb9rUSNJ250='} 2024-04-12T02:56:53.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d956c194b826a37e-006618a2f5-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.228 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:56:53.228 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:56:53.228 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.228 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:56:53.228 DEBUG:boto:Data: 2024-04-12T02:56:53.229 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.229 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.229 DEBUG:boto:Params: {} 2024-04-12T02:56:53.229 DEBUG:boto:Token: None 2024-04-12T02:56:53.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/ 2024-04-12T02:56:53.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ= 2024-04-12T02:56:53.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ=', 'Content-Length': '0'} 2024-04-12T02:56:53.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa29cba8934ed3f0-006618a2f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.231 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:56:53.231 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:56:53.231 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.232 DEBUG:boto:Path: /dgbpxg-8/_myobj 2024-04-12T02:56:53.232 DEBUG:boto:Data: 2024-04-12T02:56:53.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.232 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.232 DEBUG:boto:Params: {} 2024-04-12T02:56:53.232 DEBUG:boto:Token: None 2024-04-12T02:56:53.232 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/_myobj 2024-04-12T02:56:53.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+qOrcn0YKXlW/VN/fEHWzC/qWc= 2024-04-12T02:56:53.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+qOrcn0YKXlW/VN/fEHWzC/qWc='} 2024-04-12T02:56:53.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fcc7161ef1ea0a7-006618a2f5-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.279 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:56:53.279 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:56:53.279 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.279 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:56:53.279 DEBUG:boto:Data: 2024-04-12T02:56:53.279 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.279 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.279 DEBUG:boto:Params: {} 2024-04-12T02:56:53.279 DEBUG:boto:Token: None 2024-04-12T02:56:53.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/ 2024-04-12T02:56:53.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ= 2024-04-12T02:56:53.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ=', 'Content-Length': '0'} 2024-04-12T02:56:53.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002be4ecebd569f08a-006618a2f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.282 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:56:53.282 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:56:53.282 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.282 DEBUG:boto:Path: /dgbpxg-8/%3A 2024-04-12T02:56:53.282 DEBUG:boto:Data: 2024-04-12T02:56:53.282 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.282 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.283 DEBUG:boto:Params: {} 2024-04-12T02:56:53.283 DEBUG:boto:Token: None 2024-04-12T02:56:53.283 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/%3A 2024-04-12T02:56:53.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jx6BgVbF01dsZxkNhKs8MYMKY20= 2024-04-12T02:56:53.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jx6BgVbF01dsZxkNhKs8MYMKY20='} 2024-04-12T02:56:53.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e7c3fb20483326e-006618a2f5-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.292 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:56:53.292 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:56:53.292 DEBUG:boto:Method: HEAD 2024-04-12T02:56:53.292 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:56:53.292 DEBUG:boto:Data: 2024-04-12T02:56:53.293 DEBUG:boto:Headers: {} 2024-04-12T02:56:53.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.293 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.293 DEBUG:boto:Params: {} 2024-04-12T02:56:53.293 DEBUG:boto:Token: None 2024-04-12T02:56:53.293 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/ 2024-04-12T02:56:53.293 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ= 2024-04-12T02:56:53.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CN3xviad89c77PekxLn1ZfMp9iQ=', 'Content-Length': '0'} 2024-04-12T02:56:53.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dde775e99648f29-006618a2f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.295 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:56:53.296 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:56:53.296 DEBUG:boto:Method: PUT 2024-04-12T02:56:53.296 DEBUG:boto:Path: /dgbpxg-8/%26 2024-04-12T02:56:53.296 DEBUG:boto:Data: 2024-04-12T02:56:53.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:56:53.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:56:53.296 DEBUG:boto:Port: 8001 2024-04-12T02:56:53.296 DEBUG:boto:Params: {} 2024-04-12T02:56:53.296 DEBUG:boto:Token: None 2024-04-12T02:56:53.297 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:56:53 GMT /dgbpxg-8/%26 2024-04-12T02:56:53.297 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L4XxKQzV7WSPav8kcdphYa96pUo= 2024-04-12T02:56:53.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:56:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L4XxKQzV7WSPav8kcdphYa96pUo='} 2024-04-12T02:56:53.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa0ec775585a9b18-006618a2f5-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:56:53.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:53.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:53.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:53.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:53.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:53.416 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890613.066612_100.1", 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:56:53.066612Z" 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.429 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:56:53.430 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890613.066612_100.1', 3: ''} 2024-04-12T02:56:53.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:56:53.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:53.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:56:53.431 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:56:53.478 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:53.479 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:53.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:53.507+0000 7f79ddb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:53.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:53.508+0000 7f79ddb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:56:53.519 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:56:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890613.066612_100.1", 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:56:53.066612Z", 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:56:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:56:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:56:53Z" 2024-04-12T02:56:53.527 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:56:53.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890613.066612_100.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:56:53.066612Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:56:53Z" } 2024-04-12T02:56:53.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890613.066612_100.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:56:53.066612Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T02:56:53.528 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890613.066612_100.1', 3: ''} 2024-04-12T02:56:53.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890613.066612_100.1', 3: ''} 2024-04-12T02:56:53.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:56:53.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:56:53.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:56:53.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T02:56:53.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:56:53.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:56:53.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:53.617+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:56:56.613 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T02:56:56.612+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T02:56:58.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:56:58.618+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:03.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:03.618+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:08.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:08.618+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:13.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:13.618+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:18.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:18.620+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:23.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:23.621+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:28.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:28.622+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:33.622+0000 7f9e26d30780 1 waiting to reach incremental sync.. 2024-04-12T02:57:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:38.624+0000 7f9e26d30780 1 bucket sync caught up with source: 2024-04-12T02:57:38.625 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.4.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.5.6, , ] 2024-04-12T02:57:38.625 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.4.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.5.6, , ] 2024-04-12T02:57:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:38.624+0000 7f9e26d30780 0 bucket checkpoint complete 2024-04-12T02:57:38.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-7 zones={a2, a1} 2024-04-12T02:57:38.640 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.641 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.641 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.641 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:57:38.641 DEBUG:boto:Data: 2024-04-12T02:57:38.641 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.641 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.641 DEBUG:boto:Params: {} 2024-04-12T02:57:38.642 DEBUG:boto:Token: None 2024-04-12T02:57:38.642 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iFvTJ9POfBsjhM5RI6TJ1mY+iSk= 2024-04-12T02:57:38.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iFvTJ9POfBsjhM5RI6TJ1mY+iSk=', 'Content-Length': '0'} 2024-04-12T02:57:38.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018751b85feefe178-006618a322-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.645 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.645 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.645 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.646 DEBUG:boto:Path: /dgbpxg-7/ 2024-04-12T02:57:38.646 DEBUG:boto:Data: 2024-04-12T02:57:38.646 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.646 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.646 DEBUG:boto:Params: {} 2024-04-12T02:57:38.646 DEBUG:boto:Token: None 2024-04-12T02:57:38.647 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iFvTJ9POfBsjhM5RI6TJ1mY+iSk= 2024-04-12T02:57:38.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iFvTJ9POfBsjhM5RI6TJ1mY+iSk=', 'Content-Length': '0'} 2024-04-12T02:57:38.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000081e50f95c331cf3-006618a322-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T02:57:38.649 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.650 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.650 DEBUG:boto:path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.650 DEBUG:boto:auth_path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.650 DEBUG:boto:Method: GET 2024-04-12T02:57:38.650 DEBUG:boto:Path: /dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.650 DEBUG:boto:Data: 2024-04-12T02:57:38.650 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.650 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.650 DEBUG:boto:Params: {} 2024-04-12T02:57:38.650 DEBUG:boto:Token: None 2024-04-12T02:57:38.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/?versions 2024-04-12T02:57:38.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M= 2024-04-12T02:57:38.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M=', 'Content-Length': '0'} 2024-04-12T02:57:38.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a96b903f0775f2-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.654 DEBUG:boto:b'dgbpxg-7999false&nulltrue2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.701 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T02:57:38.701 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T02:57:38.701 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T02:57:38.701 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:57:38.701 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T02:57:38.701 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.702 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.702 DEBUG:boto:path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.702 DEBUG:boto:auth_path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.702 DEBUG:boto:Method: GET 2024-04-12T02:57:38.702 DEBUG:boto:Path: /dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.702 DEBUG:boto:Data: 2024-04-12T02:57:38.702 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.702 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.702 DEBUG:boto:Params: {} 2024-04-12T02:57:38.703 DEBUG:boto:Token: None 2024-04-12T02:57:38.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/?versions 2024-04-12T02:57:38.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M= 2024-04-12T02:57:38.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M=', 'Content-Length': '0'} 2024-04-12T02:57:38.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107cc6ba1dc2f253-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.706 DEBUG:boto:b'dgbpxg-7999false&nulltrue2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.706 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T02:57:38.706 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T02:57:38.706 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T02:57:38.706 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:57:38.706 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.707 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.707 DEBUG:boto:path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.707 DEBUG:boto:auth_path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.707 DEBUG:boto:Method: GET 2024-04-12T02:57:38.707 DEBUG:boto:Path: /dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.707 DEBUG:boto:Data: 2024-04-12T02:57:38.707 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.707 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.707 DEBUG:boto:Params: {} 2024-04-12T02:57:38.707 DEBUG:boto:Token: None 2024-04-12T02:57:38.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/?versions 2024-04-12T02:57:38.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M= 2024-04-12T02:57:38.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M=', 'Content-Length': '0'} 2024-04-12T02:57:38.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f715e48fa04d4f8f-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.710 DEBUG:boto:b'dgbpxg-7999false&nulltrue2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.711 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.711 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.711 DEBUG:boto:path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.711 DEBUG:boto:auth_path=/dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.711 DEBUG:boto:Method: GET 2024-04-12T02:57:38.711 DEBUG:boto:Path: /dgbpxg-7/?versions&max-keys=999 2024-04-12T02:57:38.711 DEBUG:boto:Data: 2024-04-12T02:57:38.711 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.711 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.712 DEBUG:boto:Params: {} 2024-04-12T02:57:38.712 DEBUG:boto:Token: None 2024-04-12T02:57:38.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/?versions 2024-04-12T02:57:38.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M= 2024-04-12T02:57:38.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sJ2bt4DVgO7HVnoap+G3oiO2U8M=', 'Content-Length': '0'} 2024-04-12T02:57:38.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093072ac7aaf8d685-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.714 DEBUG:boto:b'dgbpxg-7999false&nulltrue2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:38.715 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.715 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.715 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.715 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.715 DEBUG:boto:Method: GET 2024-04-12T02:57:38.715 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.715 DEBUG:boto:Data: 2024-04-12T02:57:38.715 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.716 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.716 DEBUG:boto:Params: {} 2024-04-12T02:57:38.716 DEBUG:boto:Token: None 2024-04-12T02:57:38.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE= 2024-04-12T02:57:38.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE=', 'Content-Length': '0'} 2024-04-12T02:57:38.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a14da61f5c76949a-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.718 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.718 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.718 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.718 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.719 DEBUG:boto:Method: GET 2024-04-12T02:57:38.719 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.719 DEBUG:boto:Data: 2024-04-12T02:57:38.719 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.719 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.719 DEBUG:boto:Params: {} 2024-04-12T02:57:38.719 DEBUG:boto:Token: None 2024-04-12T02:57:38.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE= 2024-04-12T02:57:38.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE=', 'Content-Length': '0'} 2024-04-12T02:57:38.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058fe5386de439be7-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.721 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.721 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.721 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.722 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.722 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.722 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.722 DEBUG:boto:Data: 2024-04-12T02:57:38.722 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.722 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.722 DEBUG:boto:Params: {} 2024-04-12T02:57:38.722 DEBUG:boto:Token: None 2024-04-12T02:57:38.722 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.723 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4oL5C5EhZfDBrtiV29fMvIwxvP4= 2024-04-12T02:57:38.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4oL5C5EhZfDBrtiV29fMvIwxvP4=', 'Content-Length': '0'} 2024-04-12T02:57:38.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095de16be77d873d5-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.725 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.725 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.725 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.725 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.725 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.725 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.725 DEBUG:boto:Data: 2024-04-12T02:57:38.725 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.725 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.725 DEBUG:boto:Params: {} 2024-04-12T02:57:38.726 DEBUG:boto:Token: None 2024-04-12T02:57:38.726 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4oL5C5EhZfDBrtiV29fMvIwxvP4= 2024-04-12T02:57:38.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4oL5C5EhZfDBrtiV29fMvIwxvP4=', 'Content-Length': '0'} 2024-04-12T02:57:38.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd07e943b0ab0f56-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:38.728 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.728 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.728 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.728 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.728 DEBUG:boto:Method: GET 2024-04-12T02:57:38.728 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.728 DEBUG:boto:Data: 2024-04-12T02:57:38.728 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.729 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.729 DEBUG:boto:Params: {} 2024-04-12T02:57:38.729 DEBUG:boto:Token: None 2024-04-12T02:57:38.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE= 2024-04-12T02:57:38.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE=', 'Content-Length': '0'} 2024-04-12T02:57:38.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bce4b3f5a250617-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.731 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.731 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.731 DEBUG:boto:path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.731 DEBUG:boto:auth_path=/dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.732 DEBUG:boto:Method: GET 2024-04-12T02:57:38.732 DEBUG:boto:Path: /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.732 DEBUG:boto:Data: 2024-04-12T02:57:38.732 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.732 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.732 DEBUG:boto:Params: {} 2024-04-12T02:57:38.732 DEBUG:boto:Token: None 2024-04-12T02:57:38.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26?versionId=null 2024-04-12T02:57:38.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE= 2024-04-12T02:57:38.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GS9LVttRorN+jVXYG/fHPKpemrE=', 'Content-Length': '0'} 2024-04-12T02:57:38.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b4916d849d1c1b8-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.734 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.734 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.734 DEBUG:boto:path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.735 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.735 DEBUG:boto:Method: GET 2024-04-12T02:57:38.735 DEBUG:boto:Path: /dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.735 DEBUG:boto:Data: 2024-04-12T02:57:38.735 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.735 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.735 DEBUG:boto:Params: {} 2024-04-12T02:57:38.735 DEBUG:boto:Token: None 2024-04-12T02:57:38.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9aacc5372119a8-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.738 DEBUG:boto:b'dgbpxg-7&1000false&2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.738 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.738 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.738 DEBUG:boto:path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.739 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.739 DEBUG:boto:Method: GET 2024-04-12T02:57:38.739 DEBUG:boto:Path: /dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.739 DEBUG:boto:Data: 2024-04-12T02:57:38.739 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.739 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.739 DEBUG:boto:Params: {} 2024-04-12T02:57:38.739 DEBUG:boto:Token: None 2024-04-12T02:57:38.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007edbd1e927f6f45d-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.742 DEBUG:boto:b'dgbpxg-7&1000false&2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.742 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.743 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.743 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.743 DEBUG:boto:Path: /dgbpxg-7/%26 2024-04-12T02:57:38.743 DEBUG:boto:Data: 2024-04-12T02:57:38.743 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.743 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.743 DEBUG:boto:Params: {} 2024-04-12T02:57:38.743 DEBUG:boto:Token: None 2024-04-12T02:57:38.743 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26 2024-04-12T02:57:38.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uFz0Jd/VZRlDEISQk1pIDCl6YFE= 2024-04-12T02:57:38.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uFz0Jd/VZRlDEISQk1pIDCl6YFE=', 'Content-Length': '0'} 2024-04-12T02:57:38.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf0088b5fc1f77b7-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.745 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.745 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.746 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.746 DEBUG:boto:Path: /dgbpxg-7/%26 2024-04-12T02:57:38.746 DEBUG:boto:Data: 2024-04-12T02:57:38.746 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.746 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.746 DEBUG:boto:Params: {} 2024-04-12T02:57:38.746 DEBUG:boto:Token: None 2024-04-12T02:57:38.746 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26 2024-04-12T02:57:38.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uFz0Jd/VZRlDEISQk1pIDCl6YFE= 2024-04-12T02:57:38.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uFz0Jd/VZRlDEISQk1pIDCl6YFE=', 'Content-Length': '0'} 2024-04-12T02:57:38.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc1ba03922fcdd53-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:38.748 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.748 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.749 DEBUG:boto:Method: GET 2024-04-12T02:57:38.749 DEBUG:boto:Path: /dgbpxg-7/%26 2024-04-12T02:57:38.749 DEBUG:boto:Data: 2024-04-12T02:57:38.749 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.749 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.749 DEBUG:boto:Params: {} 2024-04-12T02:57:38.749 DEBUG:boto:Token: None 2024-04-12T02:57:38.749 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26 2024-04-12T02:57:38.749 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R9WtlPr8cWw0cpRoJEbMxw34cmY= 2024-04-12T02:57:38.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R9WtlPr8cWw0cpRoJEbMxw34cmY=', 'Content-Length': '0'} 2024-04-12T02:57:38.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0e7a8ee8e6b92ad-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.751 DEBUG:boto:path=/dgbpxg-7/%26 2024-04-12T02:57:38.752 DEBUG:boto:auth_path=/dgbpxg-7/%26 2024-04-12T02:57:38.752 DEBUG:boto:Method: GET 2024-04-12T02:57:38.752 DEBUG:boto:Path: /dgbpxg-7/%26 2024-04-12T02:57:38.752 DEBUG:boto:Data: 2024-04-12T02:57:38.752 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.752 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.752 DEBUG:boto:Params: {} 2024-04-12T02:57:38.752 DEBUG:boto:Token: None 2024-04-12T02:57:38.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%26 2024-04-12T02:57:38.753 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R9WtlPr8cWw0cpRoJEbMxw34cmY= 2024-04-12T02:57:38.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R9WtlPr8cWw0cpRoJEbMxw34cmY=', 'Content-Length': '0'} 2024-04-12T02:57:38.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c5d11c4d3015de0-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.754 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.754 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.755 DEBUG:boto:path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.755 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.755 DEBUG:boto:Method: GET 2024-04-12T02:57:38.755 DEBUG:boto:Path: /dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.755 DEBUG:boto:Data: 2024-04-12T02:57:38.755 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.755 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.755 DEBUG:boto:Params: {} 2024-04-12T02:57:38.755 DEBUG:boto:Token: None 2024-04-12T02:57:38.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c61f4ecd4a778dd-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.758 DEBUG:boto:b'dgbpxg-7&1000false&2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.758 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.758 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.759 DEBUG:boto:path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.759 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.759 DEBUG:boto:Method: GET 2024-04-12T02:57:38.759 DEBUG:boto:Path: /dgbpxg-7/?prefix=%26 2024-04-12T02:57:38.759 DEBUG:boto:Data: 2024-04-12T02:57:38.759 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.759 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.759 DEBUG:boto:Params: {} 2024-04-12T02:57:38.759 DEBUG:boto:Token: None 2024-04-12T02:57:38.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1c6c7b73babb726-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.762 DEBUG:boto:b'dgbpxg-7&1000false&2024-04-12T02:56:53.165Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:38.762 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.762 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.762 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.763 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.763 DEBUG:boto:Method: GET 2024-04-12T02:57:38.763 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.763 DEBUG:boto:Data: 2024-04-12T02:57:38.763 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.763 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.763 DEBUG:boto:Params: {} 2024-04-12T02:57:38.763 DEBUG:boto:Token: None 2024-04-12T02:57:38.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA= 2024-04-12T02:57:38.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA=', 'Content-Length': '0'} 2024-04-12T02:57:38.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b4569f5e74ae753-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.765 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.765 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.765 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.766 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.766 DEBUG:boto:Method: GET 2024-04-12T02:57:38.766 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.766 DEBUG:boto:Data: 2024-04-12T02:57:38.766 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.766 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.766 DEBUG:boto:Params: {} 2024-04-12T02:57:38.766 DEBUG:boto:Token: None 2024-04-12T02:57:38.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA= 2024-04-12T02:57:38.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA=', 'Content-Length': '0'} 2024-04-12T02:57:38.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a7d5bae4e2935f3-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.768 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.768 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.768 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.769 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.769 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.769 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.769 DEBUG:boto:Data: 2024-04-12T02:57:38.769 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.769 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.769 DEBUG:boto:Params: {} 2024-04-12T02:57:38.769 DEBUG:boto:Token: None 2024-04-12T02:57:38.769 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:STxune3TUkYx+tCrfMzAxcYDDLk= 2024-04-12T02:57:38.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:STxune3TUkYx+tCrfMzAxcYDDLk=', 'Content-Length': '0'} 2024-04-12T02:57:38.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61e95314b334366-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.771 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.772 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.772 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.772 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.772 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.772 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.772 DEBUG:boto:Data: 2024-04-12T02:57:38.772 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.772 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.772 DEBUG:boto:Params: {} 2024-04-12T02:57:38.772 DEBUG:boto:Token: None 2024-04-12T02:57:38.773 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:STxune3TUkYx+tCrfMzAxcYDDLk= 2024-04-12T02:57:38.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:STxune3TUkYx+tCrfMzAxcYDDLk=', 'Content-Length': '0'} 2024-04-12T02:57:38.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbfe4fe91a6ab615-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:38.774 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.775 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.775 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.775 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.775 DEBUG:boto:Method: GET 2024-04-12T02:57:38.775 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.775 DEBUG:boto:Data: 2024-04-12T02:57:38.775 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.775 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.775 DEBUG:boto:Params: {} 2024-04-12T02:57:38.776 DEBUG:boto:Token: None 2024-04-12T02:57:38.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA= 2024-04-12T02:57:38.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA=', 'Content-Length': '0'} 2024-04-12T02:57:38.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b63a3156742890d-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.818 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.819 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.819 DEBUG:boto:path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.819 DEBUG:boto:auth_path=/dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.819 DEBUG:boto:Method: GET 2024-04-12T02:57:38.819 DEBUG:boto:Path: /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.819 DEBUG:boto:Data: 2024-04-12T02:57:38.819 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.819 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.819 DEBUG:boto:Params: {} 2024-04-12T02:57:38.820 DEBUG:boto:Token: None 2024-04-12T02:57:38.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A?versionId=null 2024-04-12T02:57:38.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA= 2024-04-12T02:57:38.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S0SRdTLXD7XOrb1ZeCtQleu6kpA=', 'Content-Length': '0'} 2024-04-12T02:57:38.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c771210a8ac57f3-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.822 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.823 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.823 DEBUG:boto:path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.823 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.823 DEBUG:boto:Method: GET 2024-04-12T02:57:38.823 DEBUG:boto:Path: /dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.823 DEBUG:boto:Data: 2024-04-12T02:57:38.823 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.823 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.823 DEBUG:boto:Params: {} 2024-04-12T02:57:38.824 DEBUG:boto:Token: None 2024-04-12T02:57:38.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040abd0651d4f3520-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.826 DEBUG:boto:b'dgbpxg-7:1000false:2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.827 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.827 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.827 DEBUG:boto:path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.827 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.827 DEBUG:boto:Method: GET 2024-04-12T02:57:38.827 DEBUG:boto:Path: /dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.827 DEBUG:boto:Data: 2024-04-12T02:57:38.827 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.828 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.828 DEBUG:boto:Params: {} 2024-04-12T02:57:38.828 DEBUG:boto:Token: None 2024-04-12T02:57:38.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080318f0eea8a14b9-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.830 DEBUG:boto:b'dgbpxg-7:1000false:2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.831 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.831 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.831 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.831 DEBUG:boto:Path: /dgbpxg-7/%3A 2024-04-12T02:57:38.831 DEBUG:boto:Data: 2024-04-12T02:57:38.831 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.831 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.831 DEBUG:boto:Params: {} 2024-04-12T02:57:38.832 DEBUG:boto:Token: None 2024-04-12T02:57:38.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A 2024-04-12T02:57:38.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y7SijoKPz2T1G2aKY/ZLFmgMiiQ= 2024-04-12T02:57:38.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y7SijoKPz2T1G2aKY/ZLFmgMiiQ=', 'Content-Length': '0'} 2024-04-12T02:57:38.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006778662e85190751-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.834 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.834 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.834 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.834 DEBUG:boto:Path: /dgbpxg-7/%3A 2024-04-12T02:57:38.834 DEBUG:boto:Data: 2024-04-12T02:57:38.834 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.834 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.834 DEBUG:boto:Params: {} 2024-04-12T02:57:38.835 DEBUG:boto:Token: None 2024-04-12T02:57:38.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A 2024-04-12T02:57:38.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y7SijoKPz2T1G2aKY/ZLFmgMiiQ= 2024-04-12T02:57:38.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y7SijoKPz2T1G2aKY/ZLFmgMiiQ=', 'Content-Length': '0'} 2024-04-12T02:57:38.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d96ee589998725c3-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:38.837 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.837 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.837 DEBUG:boto:Method: GET 2024-04-12T02:57:38.837 DEBUG:boto:Path: /dgbpxg-7/%3A 2024-04-12T02:57:38.837 DEBUG:boto:Data: 2024-04-12T02:57:38.837 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.837 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.838 DEBUG:boto:Params: {} 2024-04-12T02:57:38.838 DEBUG:boto:Token: None 2024-04-12T02:57:38.838 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A 2024-04-12T02:57:38.838 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcVpPqxiUK7shLMz4ig5AtJkUmo= 2024-04-12T02:57:38.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcVpPqxiUK7shLMz4ig5AtJkUmo=', 'Content-Length': '0'} 2024-04-12T02:57:38.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056eae323d2227428-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.840 DEBUG:boto:path=/dgbpxg-7/%3A 2024-04-12T02:57:38.840 DEBUG:boto:auth_path=/dgbpxg-7/%3A 2024-04-12T02:57:38.840 DEBUG:boto:Method: GET 2024-04-12T02:57:38.840 DEBUG:boto:Path: /dgbpxg-7/%3A 2024-04-12T02:57:38.840 DEBUG:boto:Data: 2024-04-12T02:57:38.840 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.840 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.840 DEBUG:boto:Params: {} 2024-04-12T02:57:38.841 DEBUG:boto:Token: None 2024-04-12T02:57:38.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/%3A 2024-04-12T02:57:38.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcVpPqxiUK7shLMz4ig5AtJkUmo= 2024-04-12T02:57:38.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcVpPqxiUK7shLMz4ig5AtJkUmo=', 'Content-Length': '0'} 2024-04-12T02:57:38.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c16d8d7af913ebcd-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.843 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.843 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.843 DEBUG:boto:path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.843 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.843 DEBUG:boto:Method: GET 2024-04-12T02:57:38.843 DEBUG:boto:Path: /dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.843 DEBUG:boto:Data: 2024-04-12T02:57:38.843 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.843 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.844 DEBUG:boto:Params: {} 2024-04-12T02:57:38.844 DEBUG:boto:Token: None 2024-04-12T02:57:38.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea4a3f19b7ea839-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.846 DEBUG:boto:b'dgbpxg-7:1000false:2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.847 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.847 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.847 DEBUG:boto:path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.847 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.847 DEBUG:boto:Method: GET 2024-04-12T02:57:38.847 DEBUG:boto:Path: /dgbpxg-7/?prefix=%3A 2024-04-12T02:57:38.847 DEBUG:boto:Data: 2024-04-12T02:57:38.847 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.847 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.847 DEBUG:boto:Params: {} 2024-04-12T02:57:38.848 DEBUG:boto:Token: None 2024-04-12T02:57:38.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e2ae6498c3f8485-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.850 DEBUG:boto:b'dgbpxg-7:1000false:2024-04-12T02:56:53.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:38.850 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.850 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.851 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.851 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.851 DEBUG:boto:Method: GET 2024-04-12T02:57:38.851 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.851 DEBUG:boto:Data: 2024-04-12T02:57:38.851 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.851 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.851 DEBUG:boto:Params: {} 2024-04-12T02:57:38.851 DEBUG:boto:Token: None 2024-04-12T02:57:38.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE= 2024-04-12T02:57:38.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE=', 'Content-Length': '0'} 2024-04-12T02:57:38.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd859cca47de13b0-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.894 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.895 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.895 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.895 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.895 DEBUG:boto:Method: GET 2024-04-12T02:57:38.896 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.896 DEBUG:boto:Data: 2024-04-12T02:57:38.896 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.896 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.896 DEBUG:boto:Params: {} 2024-04-12T02:57:38.896 DEBUG:boto:Token: None 2024-04-12T02:57:38.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE= 2024-04-12T02:57:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE=', 'Content-Length': '0'} 2024-04-12T02:57:38.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfee079408b2a49c-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.899 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.899 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.899 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.899 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.899 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.899 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.899 DEBUG:boto:Data: 2024-04-12T02:57:38.899 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.899 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.899 DEBUG:boto:Params: {} 2024-04-12T02:57:38.900 DEBUG:boto:Token: None 2024-04-12T02:57:38.900 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XglLD8miO4VMxRdebCPc2rqsJis= 2024-04-12T02:57:38.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XglLD8miO4VMxRdebCPc2rqsJis=', 'Content-Length': '0'} 2024-04-12T02:57:38.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7d1ec70aeaf4778-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.902 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.902 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.902 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.902 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.903 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.903 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.903 DEBUG:boto:Data: 2024-04-12T02:57:38.903 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.903 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.903 DEBUG:boto:Params: {} 2024-04-12T02:57:38.903 DEBUG:boto:Token: None 2024-04-12T02:57:38.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XglLD8miO4VMxRdebCPc2rqsJis= 2024-04-12T02:57:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XglLD8miO4VMxRdebCPc2rqsJis=', 'Content-Length': '0'} 2024-04-12T02:57:38.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009497adc011c3dd75-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:38.906 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.906 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.906 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.906 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.907 DEBUG:boto:Method: GET 2024-04-12T02:57:38.907 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.907 DEBUG:boto:Data: 2024-04-12T02:57:38.907 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.907 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.907 DEBUG:boto:Params: {} 2024-04-12T02:57:38.907 DEBUG:boto:Token: None 2024-04-12T02:57:38.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE= 2024-04-12T02:57:38.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE=', 'Content-Length': '0'} 2024-04-12T02:57:38.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fa8984bccab3bae-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.909 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.909 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.910 DEBUG:boto:path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.910 DEBUG:boto:auth_path=/dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.910 DEBUG:boto:Method: GET 2024-04-12T02:57:38.910 DEBUG:boto:Path: /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.910 DEBUG:boto:Data: 2024-04-12T02:57:38.910 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.910 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.910 DEBUG:boto:Params: {} 2024-04-12T02:57:38.910 DEBUG:boto:Token: None 2024-04-12T02:57:38.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj?versionId=null 2024-04-12T02:57:38.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE= 2024-04-12T02:57:38.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+rUrvrjtOQeUYacor3BgXdG6mE=', 'Content-Length': '0'} 2024-04-12T02:57:38.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050119ecca95be556-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.954 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.955 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.955 DEBUG:boto:path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.955 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.955 DEBUG:boto:Method: GET 2024-04-12T02:57:38.955 DEBUG:boto:Path: /dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.955 DEBUG:boto:Data: 2024-04-12T02:57:38.955 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.955 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.955 DEBUG:boto:Params: {} 2024-04-12T02:57:38.956 DEBUG:boto:Token: None 2024-04-12T02:57:38.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008501d75f63c0326e-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.959 DEBUG:boto:b'dgbpxg-7_myobj1000false_myobj2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.959 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.959 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.959 DEBUG:boto:path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.959 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.960 DEBUG:boto:Method: GET 2024-04-12T02:57:38.960 DEBUG:boto:Path: /dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.960 DEBUG:boto:Data: 2024-04-12T02:57:38.960 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.960 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.960 DEBUG:boto:Params: {} 2024-04-12T02:57:38.960 DEBUG:boto:Token: None 2024-04-12T02:57:38.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbcb1f5d39e1a46-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.963 DEBUG:boto:b'dgbpxg-7_myobj1000false_myobj2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.963 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.963 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.963 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.963 DEBUG:boto:Path: /dgbpxg-7/_myobj 2024-04-12T02:57:38.963 DEBUG:boto:Data: 2024-04-12T02:57:38.963 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.964 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.964 DEBUG:boto:Params: {} 2024-04-12T02:57:38.964 DEBUG:boto:Token: None 2024-04-12T02:57:38.964 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj 2024-04-12T02:57:38.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Ck4BuO7IwiF2j9qaKsSAUk26S8= 2024-04-12T02:57:38.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Ck4BuO7IwiF2j9qaKsSAUk26S8=', 'Content-Length': '0'} 2024-04-12T02:57:38.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb6e1f5267e008f9-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.966 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.966 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.966 DEBUG:boto:Method: HEAD 2024-04-12T02:57:38.967 DEBUG:boto:Path: /dgbpxg-7/_myobj 2024-04-12T02:57:38.967 DEBUG:boto:Data: 2024-04-12T02:57:38.967 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.967 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.967 DEBUG:boto:Params: {} 2024-04-12T02:57:38.967 DEBUG:boto:Token: None 2024-04-12T02:57:38.967 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj 2024-04-12T02:57:38.967 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Ck4BuO7IwiF2j9qaKsSAUk26S8= 2024-04-12T02:57:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Ck4BuO7IwiF2j9qaKsSAUk26S8=', 'Content-Length': '0'} 2024-04-12T02:57:38.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005664efc69fe14217-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:38.969 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.970 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.970 DEBUG:boto:Method: GET 2024-04-12T02:57:38.970 DEBUG:boto:Path: /dgbpxg-7/_myobj 2024-04-12T02:57:38.970 DEBUG:boto:Data: 2024-04-12T02:57:38.970 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.970 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.970 DEBUG:boto:Params: {} 2024-04-12T02:57:38.970 DEBUG:boto:Token: None 2024-04-12T02:57:38.970 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj 2024-04-12T02:57:38.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aERQRWhNENfaIpn2L7rXfkeB2qA= 2024-04-12T02:57:38.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aERQRWhNENfaIpn2L7rXfkeB2qA=', 'Content-Length': '0'} 2024-04-12T02:57:38.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b21599be8801f2a-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.972 DEBUG:boto:path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.973 DEBUG:boto:auth_path=/dgbpxg-7/_myobj 2024-04-12T02:57:38.973 DEBUG:boto:Method: GET 2024-04-12T02:57:38.973 DEBUG:boto:Path: /dgbpxg-7/_myobj 2024-04-12T02:57:38.973 DEBUG:boto:Data: 2024-04-12T02:57:38.973 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.973 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.973 DEBUG:boto:Params: {} 2024-04-12T02:57:38.973 DEBUG:boto:Token: None 2024-04-12T02:57:38.973 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/_myobj 2024-04-12T02:57:38.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aERQRWhNENfaIpn2L7rXfkeB2qA= 2024-04-12T02:57:38.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aERQRWhNENfaIpn2L7rXfkeB2qA=', 'Content-Length': '0'} 2024-04-12T02:57:38.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b14c2e693bc82a0c-006618a322-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.976 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.976 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.976 DEBUG:boto:path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.976 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.976 DEBUG:boto:Method: GET 2024-04-12T02:57:38.976 DEBUG:boto:Path: /dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.976 DEBUG:boto:Data: 2024-04-12T02:57:38.976 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.976 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.977 DEBUG:boto:Params: {} 2024-04-12T02:57:38.977 DEBUG:boto:Token: None 2024-04-12T02:57:38.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002314717499f81074-006618a322-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.979 DEBUG:boto:b'dgbpxg-7_myobj1000false_myobj2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.979 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:38.980 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:38.980 DEBUG:boto:path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.980 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.980 DEBUG:boto:Method: GET 2024-04-12T02:57:38.980 DEBUG:boto:Path: /dgbpxg-7/?prefix=_myobj 2024-04-12T02:57:38.980 DEBUG:boto:Data: 2024-04-12T02:57:38.980 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:38.980 DEBUG:boto:Port: 8000 2024-04-12T02:57:38.980 DEBUG:boto:Params: {} 2024-04-12T02:57:38.981 DEBUG:boto:Token: None 2024-04-12T02:57:38.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/ 2024-04-12T02:57:38.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE= 2024-04-12T02:57:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJX88lywEdQykfieqNi/J9QanGE=', 'Content-Length': '0'} 2024-04-12T02:57:38.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88a075279a9266b-006618a322-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:38.983 DEBUG:boto:b'dgbpxg-7_myobj1000false_myobj2024-04-12T02:56:53.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:38.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:38.984 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:38.984 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:38.984 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:38.984 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:38.984 DEBUG:boto:Method: GET 2024-04-12T02:57:38.984 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:38.984 DEBUG:boto:Data: 2024-04-12T02:57:38.984 DEBUG:boto:Headers: {} 2024-04-12T02:57:38.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:38.984 DEBUG:boto:Port: 8001 2024-04-12T02:57:38.984 DEBUG:boto:Params: {} 2024-04-12T02:57:38.985 DEBUG:boto:Token: None 2024-04-12T02:57:38.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:38 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:38.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsDGFGC7xAHm7s0aDPrXlD2bZw8= 2024-04-12T02:57:38.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsDGFGC7xAHm7s0aDPrXlD2bZw8=', 'Content-Length': '0'} 2024-04-12T02:57:38.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000981ae2c439d281dd-006618a322-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.030 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.030 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.031 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.031 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.031 DEBUG:boto:Method: GET 2024-04-12T02:57:39.031 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.031 DEBUG:boto:Data: 2024-04-12T02:57:39.031 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.031 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.031 DEBUG:boto:Params: {} 2024-04-12T02:57:39.031 DEBUG:boto:Token: None 2024-04-12T02:57:39.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY= 2024-04-12T02:57:39.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY=', 'Content-Length': '0'} 2024-04-12T02:57:39.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe6d050c551cb239-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.034 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.034 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.034 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.034 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.034 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.034 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.034 DEBUG:boto:Data: 2024-04-12T02:57:39.035 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.035 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.035 DEBUG:boto:Params: {} 2024-04-12T02:57:39.035 DEBUG:boto:Token: None 2024-04-12T02:57:39.035 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0LXgtm4WJkvlwHJI06pNgAYjgZQ= 2024-04-12T02:57:39.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0LXgtm4WJkvlwHJI06pNgAYjgZQ=', 'Content-Length': '0'} 2024-04-12T02:57:39.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a88ae4f64b52cb2d-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.037 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.037 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.037 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.037 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.038 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.038 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.038 DEBUG:boto:Data: 2024-04-12T02:57:39.038 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.038 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.038 DEBUG:boto:Params: {} 2024-04-12T02:57:39.038 DEBUG:boto:Token: None 2024-04-12T02:57:39.038 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.038 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0LXgtm4WJkvlwHJI06pNgAYjgZQ= 2024-04-12T02:57:39.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0LXgtm4WJkvlwHJI06pNgAYjgZQ=', 'Content-Length': '0'} 2024-04-12T02:57:39.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da81bd448d5e33eb-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:39.040 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.040 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.041 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.041 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.041 DEBUG:boto:Method: GET 2024-04-12T02:57:39.041 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.041 DEBUG:boto:Data: 2024-04-12T02:57:39.041 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.041 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.041 DEBUG:boto:Params: {} 2024-04-12T02:57:39.041 DEBUG:boto:Token: None 2024-04-12T02:57:39.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY= 2024-04-12T02:57:39.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY=', 'Content-Length': '0'} 2024-04-12T02:57:39.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000580d46cbbf123be2-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.044 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.044 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.044 DEBUG:boto:path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.044 DEBUG:boto:auth_path=/dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.044 DEBUG:boto:Method: GET 2024-04-12T02:57:39.044 DEBUG:boto:Path: /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.044 DEBUG:boto:Data: 2024-04-12T02:57:39.044 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.044 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.045 DEBUG:boto:Params: {} 2024-04-12T02:57:39.045 DEBUG:boto:Token: None 2024-04-12T02:57:39.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj?versionId=null 2024-04-12T02:57:39.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY= 2024-04-12T02:57:39.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OOxc0TeavanrsIKG868cYODFZY=', 'Content-Length': '0'} 2024-04-12T02:57:39.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001571511f2f44e94-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.047 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:39.047 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:39.047 DEBUG:boto:path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.047 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.047 DEBUG:boto:Method: GET 2024-04-12T02:57:39.047 DEBUG:boto:Path: /dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.047 DEBUG:boto:Data: 2024-04-12T02:57:39.047 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.048 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.048 DEBUG:boto:Params: {} 2024-04-12T02:57:39.048 DEBUG:boto:Token: None 2024-04-12T02:57:39.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/ 2024-04-12T02:57:39.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k= 2024-04-12T02:57:39.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k=', 'Content-Length': '0'} 2024-04-12T02:57:39.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf4047608d50cc6-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.050 DEBUG:boto:b'dgbpxg-7myobj1000falsemyobj2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.051 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:39.051 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:39.051 DEBUG:boto:path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.051 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.051 DEBUG:boto:Method: GET 2024-04-12T02:57:39.051 DEBUG:boto:Path: /dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.051 DEBUG:boto:Data: 2024-04-12T02:57:39.051 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.052 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.052 DEBUG:boto:Params: {} 2024-04-12T02:57:39.052 DEBUG:boto:Token: None 2024-04-12T02:57:39.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/ 2024-04-12T02:57:39.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k= 2024-04-12T02:57:39.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k=', 'Content-Length': '0'} 2024-04-12T02:57:39.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca0d71bf493dde2-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.094 DEBUG:boto:b'dgbpxg-7myobj1000falsemyobj2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.095 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.095 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.095 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.095 DEBUG:boto:Path: /dgbpxg-7/myobj 2024-04-12T02:57:39.095 DEBUG:boto:Data: 2024-04-12T02:57:39.095 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.096 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.096 DEBUG:boto:Params: {} 2024-04-12T02:57:39.096 DEBUG:boto:Token: None 2024-04-12T02:57:39.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj 2024-04-12T02:57:39.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2/HNdZl/CrLrmruRWMISJW6ZlMs= 2024-04-12T02:57:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2/HNdZl/CrLrmruRWMISJW6ZlMs=', 'Content-Length': '0'} 2024-04-12T02:57:39.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6002a19319b0657-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.098 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.099 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.099 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.099 DEBUG:boto:Path: /dgbpxg-7/myobj 2024-04-12T02:57:39.099 DEBUG:boto:Data: 2024-04-12T02:57:39.099 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.099 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.099 DEBUG:boto:Params: {} 2024-04-12T02:57:39.099 DEBUG:boto:Token: None 2024-04-12T02:57:39.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj 2024-04-12T02:57:39.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2/HNdZl/CrLrmruRWMISJW6ZlMs= 2024-04-12T02:57:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2/HNdZl/CrLrmruRWMISJW6ZlMs=', 'Content-Length': '0'} 2024-04-12T02:57:39.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025ffdcb9e9e9a60a-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:39.101 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.101 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.102 DEBUG:boto:Method: GET 2024-04-12T02:57:39.102 DEBUG:boto:Path: /dgbpxg-7/myobj 2024-04-12T02:57:39.102 DEBUG:boto:Data: 2024-04-12T02:57:39.102 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.102 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.102 DEBUG:boto:Params: {} 2024-04-12T02:57:39.102 DEBUG:boto:Token: None 2024-04-12T02:57:39.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj 2024-04-12T02:57:39.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a5JUxUHRgjLJQD9PPGXPSWFa0Fk= 2024-04-12T02:57:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a5JUxUHRgjLJQD9PPGXPSWFa0Fk=', 'Content-Length': '0'} 2024-04-12T02:57:39.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bab6b0d2f54de2b3-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.105 DEBUG:boto:path=/dgbpxg-7/myobj 2024-04-12T02:57:39.105 DEBUG:boto:auth_path=/dgbpxg-7/myobj 2024-04-12T02:57:39.105 DEBUG:boto:Method: GET 2024-04-12T02:57:39.105 DEBUG:boto:Path: /dgbpxg-7/myobj 2024-04-12T02:57:39.105 DEBUG:boto:Data: 2024-04-12T02:57:39.105 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.105 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.105 DEBUG:boto:Params: {} 2024-04-12T02:57:39.106 DEBUG:boto:Token: None 2024-04-12T02:57:39.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/myobj 2024-04-12T02:57:39.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a5JUxUHRgjLJQD9PPGXPSWFa0Fk= 2024-04-12T02:57:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a5JUxUHRgjLJQD9PPGXPSWFa0Fk=', 'Content-Length': '0'} 2024-04-12T02:57:39.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000813156cf97b1968a-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.108 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:39.108 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:39.108 DEBUG:boto:path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.108 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.108 DEBUG:boto:Method: GET 2024-04-12T02:57:39.108 DEBUG:boto:Path: /dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.109 DEBUG:boto:Data: 2024-04-12T02:57:39.109 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.109 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.109 DEBUG:boto:Params: {} 2024-04-12T02:57:39.109 DEBUG:boto:Token: None 2024-04-12T02:57:39.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/ 2024-04-12T02:57:39.109 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k= 2024-04-12T02:57:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k=', 'Content-Length': '0'} 2024-04-12T02:57:39.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60e8ee1f355aad9-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.112 DEBUG:boto:b'dgbpxg-7myobj1000falsemyobj2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.112 DEBUG:boto:path=/dgbpxg-7/ 2024-04-12T02:57:39.112 DEBUG:boto:auth_path=/dgbpxg-7/ 2024-04-12T02:57:39.112 DEBUG:boto:path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.112 DEBUG:boto:auth_path=/dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.112 DEBUG:boto:Method: GET 2024-04-12T02:57:39.112 DEBUG:boto:Path: /dgbpxg-7/?prefix=myobj 2024-04-12T02:57:39.113 DEBUG:boto:Data: 2024-04-12T02:57:39.113 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.113 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.113 DEBUG:boto:Params: {} 2024-04-12T02:57:39.113 DEBUG:boto:Token: None 2024-04-12T02:57:39.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-7/ 2024-04-12T02:57:39.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k= 2024-04-12T02:57:39.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KZFFkdx67CffdTz9aZPWVTN24k=', 'Content-Length': '0'} 2024-04-12T02:57:39.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe377b19e042598-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.115 DEBUG:boto:b'dgbpxg-7myobj1000falsemyobj2024-04-12T02:56:53.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.116 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-7 zones={a2, a1} 2024-04-12T02:57:39.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:57:39.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:57:39.117 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T02:57:39.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:57:39.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:57:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:39.212+0000 7f66d8300780 1 bucket sync caught up with source: 2024-04-12T02:57:39.213 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.11.6, , 00000000001.7.6, , , , 00000000001.5.6, 00000000001.14.6, , ] 2024-04-12T02:57:39.213 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.11.6, , 00000000001.7.6, , , , 00000000001.5.6, 00000000001.14.6, , ] 2024-04-12T02:57:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:39.212+0000 7f66d8300780 0 bucket checkpoint complete 2024-04-12T02:57:39.226 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-8 zones={a1, a2} 2024-04-12T02:57:39.226 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.226 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.226 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.226 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:57:39.226 DEBUG:boto:Data: 2024-04-12T02:57:39.226 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.227 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.227 DEBUG:boto:Params: {} 2024-04-12T02:57:39.227 DEBUG:boto:Token: None 2024-04-12T02:57:39.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2ZA1B0oBm6jyZvbocb+ekFAeLE= 2024-04-12T02:57:39.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2ZA1B0oBm6jyZvbocb+ekFAeLE=', 'Content-Length': '0'} 2024-04-12T02:57:39.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000006d249af947e63e-006618a323-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.229 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.230 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.230 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.230 DEBUG:boto:Path: /dgbpxg-8/ 2024-04-12T02:57:39.230 DEBUG:boto:Data: 2024-04-12T02:57:39.230 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.230 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.230 DEBUG:boto:Params: {} 2024-04-12T02:57:39.230 DEBUG:boto:Token: None 2024-04-12T02:57:39.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2ZA1B0oBm6jyZvbocb+ekFAeLE= 2024-04-12T02:57:39.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2ZA1B0oBm6jyZvbocb+ekFAeLE=', 'Content-Length': '0'} 2024-04-12T02:57:39.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f8a7e589cb189bc-006618a323-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.233 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T02:57:39.233 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.233 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.233 DEBUG:boto:path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.233 DEBUG:boto:auth_path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.234 DEBUG:boto:Method: GET 2024-04-12T02:57:39.234 DEBUG:boto:Path: /dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.234 DEBUG:boto:Data: 2024-04-12T02:57:39.234 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.234 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.234 DEBUG:boto:Params: {} 2024-04-12T02:57:39.234 DEBUG:boto:Token: None 2024-04-12T02:57:39.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/?versions 2024-04-12T02:57:39.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E= 2024-04-12T02:57:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E=', 'Content-Length': '0'} 2024-04-12T02:57:39.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acd7eb8c73f5ba98-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.237 DEBUG:boto:b'dgbpxg-8999false&nulltrue2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T02:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T02:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T02:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T02:57:39.238 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.238 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.239 DEBUG:boto:path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.239 DEBUG:boto:auth_path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.239 DEBUG:boto:Method: GET 2024-04-12T02:57:39.239 DEBUG:boto:Path: /dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.239 DEBUG:boto:Data: 2024-04-12T02:57:39.239 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.239 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.239 DEBUG:boto:Params: {} 2024-04-12T02:57:39.239 DEBUG:boto:Token: None 2024-04-12T02:57:39.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/?versions 2024-04-12T02:57:39.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E= 2024-04-12T02:57:39.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E=', 'Content-Length': '0'} 2024-04-12T02:57:39.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34cf35494c5489e-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.242 DEBUG:boto:b'dgbpxg-8999false&nulltrue2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.242 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T02:57:39.242 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T02:57:39.242 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T02:57:39.243 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:57:39.243 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.243 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.243 DEBUG:boto:path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.243 DEBUG:boto:auth_path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.243 DEBUG:boto:Method: GET 2024-04-12T02:57:39.243 DEBUG:boto:Path: /dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.243 DEBUG:boto:Data: 2024-04-12T02:57:39.243 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.243 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.244 DEBUG:boto:Params: {} 2024-04-12T02:57:39.244 DEBUG:boto:Token: None 2024-04-12T02:57:39.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/?versions 2024-04-12T02:57:39.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E= 2024-04-12T02:57:39.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E=', 'Content-Length': '0'} 2024-04-12T02:57:39.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eead61f1b740ab43-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.246 DEBUG:boto:b'dgbpxg-8999false&nulltrue2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.247 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.247 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.247 DEBUG:boto:path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.247 DEBUG:boto:auth_path=/dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.247 DEBUG:boto:Method: GET 2024-04-12T02:57:39.247 DEBUG:boto:Path: /dgbpxg-8/?versions&max-keys=999 2024-04-12T02:57:39.247 DEBUG:boto:Data: 2024-04-12T02:57:39.247 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.248 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.248 DEBUG:boto:Params: {} 2024-04-12T02:57:39.248 DEBUG:boto:Token: None 2024-04-12T02:57:39.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/?versions 2024-04-12T02:57:39.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E= 2024-04-12T02:57:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5jCamsK6MAmP4APiBlRIYPre1E=', 'Content-Length': '0'} 2024-04-12T02:57:39.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84f9324df50381f-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.251 DEBUG:boto:b'dgbpxg-8999false&nulltrue2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:39.251 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.251 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.252 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.252 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.252 DEBUG:boto:Method: GET 2024-04-12T02:57:39.252 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.252 DEBUG:boto:Data: 2024-04-12T02:57:39.252 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.252 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.252 DEBUG:boto:Params: {} 2024-04-12T02:57:39.252 DEBUG:boto:Token: None 2024-04-12T02:57:39.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw= 2024-04-12T02:57:39.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw=', 'Content-Length': '0'} 2024-04-12T02:57:39.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fde4d16d7b6662b-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.255 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.255 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.255 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.255 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.255 DEBUG:boto:Method: GET 2024-04-12T02:57:39.255 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.255 DEBUG:boto:Data: 2024-04-12T02:57:39.255 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.256 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.256 DEBUG:boto:Params: {} 2024-04-12T02:57:39.256 DEBUG:boto:Token: None 2024-04-12T02:57:39.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw= 2024-04-12T02:57:39.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw=', 'Content-Length': '0'} 2024-04-12T02:57:39.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de1148d00945320b-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.258 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.258 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.258 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.258 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.258 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.258 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.258 DEBUG:boto:Data: 2024-04-12T02:57:39.259 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.259 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.259 DEBUG:boto:Params: {} 2024-04-12T02:57:39.259 DEBUG:boto:Token: None 2024-04-12T02:57:39.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8kP40JmO7ZzRI+zoI99YTcfqANE= 2024-04-12T02:57:39.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8kP40JmO7ZzRI+zoI99YTcfqANE=', 'Content-Length': '0'} 2024-04-12T02:57:39.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ca331fa49c000ef-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.261 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.261 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.261 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.261 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.261 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.261 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.261 DEBUG:boto:Data: 2024-04-12T02:57:39.262 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.262 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.262 DEBUG:boto:Params: {} 2024-04-12T02:57:39.262 DEBUG:boto:Token: None 2024-04-12T02:57:39.262 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.262 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8kP40JmO7ZzRI+zoI99YTcfqANE= 2024-04-12T02:57:39.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8kP40JmO7ZzRI+zoI99YTcfqANE=', 'Content-Length': '0'} 2024-04-12T02:57:39.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee01de024128f8d8-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:39.264 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.264 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.265 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.265 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.265 DEBUG:boto:Method: GET 2024-04-12T02:57:39.265 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.265 DEBUG:boto:Data: 2024-04-12T02:57:39.265 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.265 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.265 DEBUG:boto:Params: {} 2024-04-12T02:57:39.265 DEBUG:boto:Token: None 2024-04-12T02:57:39.265 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw= 2024-04-12T02:57:39.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw=', 'Content-Length': '0'} 2024-04-12T02:57:39.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6b2686ffb06388d-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.268 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.268 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.268 DEBUG:boto:path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.268 DEBUG:boto:auth_path=/dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.268 DEBUG:boto:Method: GET 2024-04-12T02:57:39.268 DEBUG:boto:Path: /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.268 DEBUG:boto:Data: 2024-04-12T02:57:39.268 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.268 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.269 DEBUG:boto:Params: {} 2024-04-12T02:57:39.269 DEBUG:boto:Token: None 2024-04-12T02:57:39.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26?versionId=null 2024-04-12T02:57:39.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw= 2024-04-12T02:57:39.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zFL4UYk+SnMF4+I0eUb7/qArdDw=', 'Content-Length': '0'} 2024-04-12T02:57:39.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ae0d6c612e72d84-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.271 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.271 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.271 DEBUG:boto:path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.271 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.271 DEBUG:boto:Method: GET 2024-04-12T02:57:39.271 DEBUG:boto:Path: /dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.271 DEBUG:boto:Data: 2024-04-12T02:57:39.271 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.272 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.272 DEBUG:boto:Params: {} 2024-04-12T02:57:39.272 DEBUG:boto:Token: None 2024-04-12T02:57:39.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d60fe17180f7ab5e-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.275 DEBUG:boto:b'dgbpxg-8&1000false&2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.275 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.275 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.275 DEBUG:boto:path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.275 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.275 DEBUG:boto:Method: GET 2024-04-12T02:57:39.276 DEBUG:boto:Path: /dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.276 DEBUG:boto:Data: 2024-04-12T02:57:39.276 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.276 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.276 DEBUG:boto:Params: {} 2024-04-12T02:57:39.276 DEBUG:boto:Token: None 2024-04-12T02:57:39.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e30c1d4cd4f227-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.322 DEBUG:boto:b'dgbpxg-8&1000false&2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.323 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.323 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.323 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.323 DEBUG:boto:Path: /dgbpxg-8/%26 2024-04-12T02:57:39.323 DEBUG:boto:Data: 2024-04-12T02:57:39.324 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.324 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.324 DEBUG:boto:Params: {} 2024-04-12T02:57:39.324 DEBUG:boto:Token: None 2024-04-12T02:57:39.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26 2024-04-12T02:57:39.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6dn/ojWdjnCEwGBONT4GEivQB6Q= 2024-04-12T02:57:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6dn/ojWdjnCEwGBONT4GEivQB6Q=', 'Content-Length': '0'} 2024-04-12T02:57:39.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc493709a4577956-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.327 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.327 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.327 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.327 DEBUG:boto:Path: /dgbpxg-8/%26 2024-04-12T02:57:39.327 DEBUG:boto:Data: 2024-04-12T02:57:39.327 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.327 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.327 DEBUG:boto:Params: {} 2024-04-12T02:57:39.328 DEBUG:boto:Token: None 2024-04-12T02:57:39.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26 2024-04-12T02:57:39.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6dn/ojWdjnCEwGBONT4GEivQB6Q= 2024-04-12T02:57:39.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6dn/ojWdjnCEwGBONT4GEivQB6Q=', 'Content-Length': '0'} 2024-04-12T02:57:39.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028a01f5aae550ca0-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T02:57:39.330 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.330 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.330 DEBUG:boto:Method: GET 2024-04-12T02:57:39.330 DEBUG:boto:Path: /dgbpxg-8/%26 2024-04-12T02:57:39.330 DEBUG:boto:Data: 2024-04-12T02:57:39.330 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.330 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.330 DEBUG:boto:Params: {} 2024-04-12T02:57:39.330 DEBUG:boto:Token: None 2024-04-12T02:57:39.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26 2024-04-12T02:57:39.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2FURPYOhfRd1IwXjJv70Isyz8hQ= 2024-04-12T02:57:39.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2FURPYOhfRd1IwXjJv70Isyz8hQ=', 'Content-Length': '0'} 2024-04-12T02:57:39.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006411fc602159c2a3-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.374 DEBUG:boto:path=/dgbpxg-8/%26 2024-04-12T02:57:39.375 DEBUG:boto:auth_path=/dgbpxg-8/%26 2024-04-12T02:57:39.375 DEBUG:boto:Method: GET 2024-04-12T02:57:39.375 DEBUG:boto:Path: /dgbpxg-8/%26 2024-04-12T02:57:39.375 DEBUG:boto:Data: 2024-04-12T02:57:39.375 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.376 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.376 DEBUG:boto:Params: {} 2024-04-12T02:57:39.376 DEBUG:boto:Token: None 2024-04-12T02:57:39.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%26 2024-04-12T02:57:39.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2FURPYOhfRd1IwXjJv70Isyz8hQ= 2024-04-12T02:57:39.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2FURPYOhfRd1IwXjJv70Isyz8hQ=', 'Content-Length': '0'} 2024-04-12T02:57:39.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd0f36df6f90c92-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.378 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.378 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.378 DEBUG:boto:path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.379 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.379 DEBUG:boto:Method: GET 2024-04-12T02:57:39.379 DEBUG:boto:Path: /dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.379 DEBUG:boto:Data: 2024-04-12T02:57:39.379 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.379 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.379 DEBUG:boto:Params: {} 2024-04-12T02:57:39.379 DEBUG:boto:Token: None 2024-04-12T02:57:39.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a906710127c5e1d-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.382 DEBUG:boto:b'dgbpxg-8&1000false&2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.382 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.382 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.383 DEBUG:boto:path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.383 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.383 DEBUG:boto:Method: GET 2024-04-12T02:57:39.383 DEBUG:boto:Path: /dgbpxg-8/?prefix=%26 2024-04-12T02:57:39.383 DEBUG:boto:Data: 2024-04-12T02:57:39.383 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.383 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.384 DEBUG:boto:Params: {} 2024-04-12T02:57:39.384 DEBUG:boto:Token: None 2024-04-12T02:57:39.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dd9f46769125f28-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.386 DEBUG:boto:b'dgbpxg-8&1000false&2024-04-12T02:56:53.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:39.387 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.387 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.387 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.387 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.387 DEBUG:boto:Method: GET 2024-04-12T02:57:39.387 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.387 DEBUG:boto:Data: 2024-04-12T02:57:39.387 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.387 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.388 DEBUG:boto:Params: {} 2024-04-12T02:57:39.388 DEBUG:boto:Token: None 2024-04-12T02:57:39.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk= 2024-04-12T02:57:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk=', 'Content-Length': '0'} 2024-04-12T02:57:39.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d056f280b75f5c1-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.390 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.390 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.390 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.390 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.390 DEBUG:boto:Method: GET 2024-04-12T02:57:39.390 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.390 DEBUG:boto:Data: 2024-04-12T02:57:39.391 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.391 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.391 DEBUG:boto:Params: {} 2024-04-12T02:57:39.391 DEBUG:boto:Token: None 2024-04-12T02:57:39.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk= 2024-04-12T02:57:39.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk=', 'Content-Length': '0'} 2024-04-12T02:57:39.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eee8aa6f9e7c19ec-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.393 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.393 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.393 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.394 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.394 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.394 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.394 DEBUG:boto:Data: 2024-04-12T02:57:39.394 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.394 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.394 DEBUG:boto:Params: {} 2024-04-12T02:57:39.394 DEBUG:boto:Token: None 2024-04-12T02:57:39.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t0lLNqNMYN8yx8enJOme9KKYPys= 2024-04-12T02:57:39.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t0lLNqNMYN8yx8enJOme9KKYPys=', 'Content-Length': '0'} 2024-04-12T02:57:39.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e314622d1227f53-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.396 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.397 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.397 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.397 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.397 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.397 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.397 DEBUG:boto:Data: 2024-04-12T02:57:39.397 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.397 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.397 DEBUG:boto:Params: {} 2024-04-12T02:57:39.397 DEBUG:boto:Token: None 2024-04-12T02:57:39.398 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t0lLNqNMYN8yx8enJOme9KKYPys= 2024-04-12T02:57:39.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t0lLNqNMYN8yx8enJOme9KKYPys=', 'Content-Length': '0'} 2024-04-12T02:57:39.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c1e008dbaf59402-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:39.399 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.400 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.400 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.400 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.400 DEBUG:boto:Method: GET 2024-04-12T02:57:39.400 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.400 DEBUG:boto:Data: 2024-04-12T02:57:39.400 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.400 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.400 DEBUG:boto:Params: {} 2024-04-12T02:57:39.400 DEBUG:boto:Token: None 2024-04-12T02:57:39.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk= 2024-04-12T02:57:39.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk=', 'Content-Length': '0'} 2024-04-12T02:57:39.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088cd244984e0719c-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.403 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.403 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.403 DEBUG:boto:path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.403 DEBUG:boto:auth_path=/dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.403 DEBUG:boto:Method: GET 2024-04-12T02:57:39.403 DEBUG:boto:Path: /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.403 DEBUG:boto:Data: 2024-04-12T02:57:39.403 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.403 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.403 DEBUG:boto:Params: {} 2024-04-12T02:57:39.404 DEBUG:boto:Token: None 2024-04-12T02:57:39.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A?versionId=null 2024-04-12T02:57:39.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk= 2024-04-12T02:57:39.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CUX4LdSdehKUCmH2yqde+PrjNjk=', 'Content-Length': '0'} 2024-04-12T02:57:39.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e41e21a95c0a47f-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.406 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.406 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.406 DEBUG:boto:path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.406 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.406 DEBUG:boto:Method: GET 2024-04-12T02:57:39.407 DEBUG:boto:Path: /dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.407 DEBUG:boto:Data: 2024-04-12T02:57:39.407 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.407 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.407 DEBUG:boto:Params: {} 2024-04-12T02:57:39.407 DEBUG:boto:Token: None 2024-04-12T02:57:39.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038972885356ec46f-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.409 DEBUG:boto:b'dgbpxg-8:1000false:2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.410 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.410 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.410 DEBUG:boto:path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.410 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.410 DEBUG:boto:Method: GET 2024-04-12T02:57:39.410 DEBUG:boto:Path: /dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.411 DEBUG:boto:Data: 2024-04-12T02:57:39.411 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.411 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.411 DEBUG:boto:Params: {} 2024-04-12T02:57:39.411 DEBUG:boto:Token: None 2024-04-12T02:57:39.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0fe6b2c3a6637a3-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.413 DEBUG:boto:b'dgbpxg-8:1000false:2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.414 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.414 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.414 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.414 DEBUG:boto:Path: /dgbpxg-8/%3A 2024-04-12T02:57:39.414 DEBUG:boto:Data: 2024-04-12T02:57:39.414 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.414 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.414 DEBUG:boto:Params: {} 2024-04-12T02:57:39.418 DEBUG:boto:Token: None 2024-04-12T02:57:39.418 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A 2024-04-12T02:57:39.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H09aDBDgZsm1W+QTIMJIBIv34lE= 2024-04-12T02:57:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H09aDBDgZsm1W+QTIMJIBIv34lE=', 'Content-Length': '0'} 2024-04-12T02:57:39.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076f60437c747a274-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.420 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.420 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.420 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.420 DEBUG:boto:Path: /dgbpxg-8/%3A 2024-04-12T02:57:39.420 DEBUG:boto:Data: 2024-04-12T02:57:39.420 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.420 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.421 DEBUG:boto:Params: {} 2024-04-12T02:57:39.421 DEBUG:boto:Token: None 2024-04-12T02:57:39.421 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A 2024-04-12T02:57:39.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H09aDBDgZsm1W+QTIMJIBIv34lE= 2024-04-12T02:57:39.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H09aDBDgZsm1W+QTIMJIBIv34lE=', 'Content-Length': '0'} 2024-04-12T02:57:39.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091585c1c3aaf965e-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T02:57:39.423 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.423 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.423 DEBUG:boto:Method: GET 2024-04-12T02:57:39.423 DEBUG:boto:Path: /dgbpxg-8/%3A 2024-04-12T02:57:39.423 DEBUG:boto:Data: 2024-04-12T02:57:39.423 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.424 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.424 DEBUG:boto:Params: {} 2024-04-12T02:57:39.424 DEBUG:boto:Token: None 2024-04-12T02:57:39.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A 2024-04-12T02:57:39.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:npXkmpmxXVLPDN4pwsZ3+xjFaOk= 2024-04-12T02:57:39.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:npXkmpmxXVLPDN4pwsZ3+xjFaOk=', 'Content-Length': '0'} 2024-04-12T02:57:39.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009391ccae43e7eca-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.426 DEBUG:boto:path=/dgbpxg-8/%3A 2024-04-12T02:57:39.426 DEBUG:boto:auth_path=/dgbpxg-8/%3A 2024-04-12T02:57:39.426 DEBUG:boto:Method: GET 2024-04-12T02:57:39.426 DEBUG:boto:Path: /dgbpxg-8/%3A 2024-04-12T02:57:39.427 DEBUG:boto:Data: 2024-04-12T02:57:39.427 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.427 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.427 DEBUG:boto:Params: {} 2024-04-12T02:57:39.427 DEBUG:boto:Token: None 2024-04-12T02:57:39.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/%3A 2024-04-12T02:57:39.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:npXkmpmxXVLPDN4pwsZ3+xjFaOk= 2024-04-12T02:57:39.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:npXkmpmxXVLPDN4pwsZ3+xjFaOk=', 'Content-Length': '0'} 2024-04-12T02:57:39.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001382256fbcc77809-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.470 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.471 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.471 DEBUG:boto:path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.471 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.471 DEBUG:boto:Method: GET 2024-04-12T02:57:39.471 DEBUG:boto:Path: /dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.471 DEBUG:boto:Data: 2024-04-12T02:57:39.471 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.471 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.471 DEBUG:boto:Params: {} 2024-04-12T02:57:39.471 DEBUG:boto:Token: None 2024-04-12T02:57:39.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8b204d43bf191b0-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.474 DEBUG:boto:b'dgbpxg-8:1000false:2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.475 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.475 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.475 DEBUG:boto:path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.475 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.475 DEBUG:boto:Method: GET 2024-04-12T02:57:39.475 DEBUG:boto:Path: /dgbpxg-8/?prefix=%3A 2024-04-12T02:57:39.475 DEBUG:boto:Data: 2024-04-12T02:57:39.475 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.475 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.476 DEBUG:boto:Params: {} 2024-04-12T02:57:39.476 DEBUG:boto:Token: None 2024-04-12T02:57:39.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b759f5c788c8eb6d-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.478 DEBUG:boto:b'dgbpxg-8:1000false:2024-04-12T02:56:53.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:39.479 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.479 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.479 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.479 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.479 DEBUG:boto:Method: GET 2024-04-12T02:57:39.479 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.479 DEBUG:boto:Data: 2024-04-12T02:57:39.479 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.479 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.480 DEBUG:boto:Params: {} 2024-04-12T02:57:39.480 DEBUG:boto:Token: None 2024-04-12T02:57:39.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE= 2024-04-12T02:57:39.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE=', 'Content-Length': '0'} 2024-04-12T02:57:39.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb0c63456865163c-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.482 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.482 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.482 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.482 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.482 DEBUG:boto:Method: GET 2024-04-12T02:57:39.482 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.482 DEBUG:boto:Data: 2024-04-12T02:57:39.482 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.483 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.483 DEBUG:boto:Params: {} 2024-04-12T02:57:39.483 DEBUG:boto:Token: None 2024-04-12T02:57:39.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE= 2024-04-12T02:57:39.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE=', 'Content-Length': '0'} 2024-04-12T02:57:39.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004229fc411c397840-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.485 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.485 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.485 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.485 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.485 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.485 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.486 DEBUG:boto:Data: 2024-04-12T02:57:39.486 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.486 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.486 DEBUG:boto:Params: {} 2024-04-12T02:57:39.486 DEBUG:boto:Token: None 2024-04-12T02:57:39.486 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.486 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GsNsS+VkgFtBCnRA89Qd3+uI78k= 2024-04-12T02:57:39.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GsNsS+VkgFtBCnRA89Qd3+uI78k=', 'Content-Length': '0'} 2024-04-12T02:57:39.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4aa339d919feea7-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.488 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.488 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.488 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.488 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.489 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.489 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.489 DEBUG:boto:Data: 2024-04-12T02:57:39.489 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.489 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.489 DEBUG:boto:Params: {} 2024-04-12T02:57:39.489 DEBUG:boto:Token: None 2024-04-12T02:57:39.489 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.489 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GsNsS+VkgFtBCnRA89Qd3+uI78k= 2024-04-12T02:57:39.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GsNsS+VkgFtBCnRA89Qd3+uI78k=', 'Content-Length': '0'} 2024-04-12T02:57:39.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000205c8863b48afbc0-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:39.492 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.492 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.492 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.492 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.492 DEBUG:boto:Method: GET 2024-04-12T02:57:39.492 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.492 DEBUG:boto:Data: 2024-04-12T02:57:39.492 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.492 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.492 DEBUG:boto:Params: {} 2024-04-12T02:57:39.493 DEBUG:boto:Token: None 2024-04-12T02:57:39.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE= 2024-04-12T02:57:39.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE=', 'Content-Length': '0'} 2024-04-12T02:57:39.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6f42a6094470e5e-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.495 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.495 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.495 DEBUG:boto:path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.495 DEBUG:boto:auth_path=/dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.495 DEBUG:boto:Method: GET 2024-04-12T02:57:39.495 DEBUG:boto:Path: /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.495 DEBUG:boto:Data: 2024-04-12T02:57:39.495 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.495 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.495 DEBUG:boto:Params: {} 2024-04-12T02:57:39.496 DEBUG:boto:Token: None 2024-04-12T02:57:39.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj?versionId=null 2024-04-12T02:57:39.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE= 2024-04-12T02:57:39.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMw1JYVzTRU+wLx6KFjghUqS6PE=', 'Content-Length': '0'} 2024-04-12T02:57:39.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1be630ccdbb0c44-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.538 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.539 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.539 DEBUG:boto:path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.539 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.539 DEBUG:boto:Method: GET 2024-04-12T02:57:39.539 DEBUG:boto:Path: /dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.539 DEBUG:boto:Data: 2024-04-12T02:57:39.539 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.539 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.539 DEBUG:boto:Params: {} 2024-04-12T02:57:39.540 DEBUG:boto:Token: None 2024-04-12T02:57:39.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039140e28c30bd0b2-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.543 DEBUG:boto:b'dgbpxg-8_myobj1000false_myobj2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.543 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.543 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.543 DEBUG:boto:path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.543 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.544 DEBUG:boto:Method: GET 2024-04-12T02:57:39.544 DEBUG:boto:Path: /dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.544 DEBUG:boto:Data: 2024-04-12T02:57:39.544 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.544 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.544 DEBUG:boto:Params: {} 2024-04-12T02:57:39.544 DEBUG:boto:Token: None 2024-04-12T02:57:39.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0d7d3298f740914-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.547 DEBUG:boto:b'dgbpxg-8_myobj1000false_myobj2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.547 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.548 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.548 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.548 DEBUG:boto:Path: /dgbpxg-8/_myobj 2024-04-12T02:57:39.548 DEBUG:boto:Data: 2024-04-12T02:57:39.548 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.548 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.549 DEBUG:boto:Params: {} 2024-04-12T02:57:39.549 DEBUG:boto:Token: None 2024-04-12T02:57:39.549 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj 2024-04-12T02:57:39.550 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FKyb5mupo5YcwNoCxe9dBrkLVrk= 2024-04-12T02:57:39.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FKyb5mupo5YcwNoCxe9dBrkLVrk=', 'Content-Length': '0'} 2024-04-12T02:57:39.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012cac1edafd8ec13-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.552 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.552 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.552 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.552 DEBUG:boto:Path: /dgbpxg-8/_myobj 2024-04-12T02:57:39.552 DEBUG:boto:Data: 2024-04-12T02:57:39.552 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.552 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.552 DEBUG:boto:Params: {} 2024-04-12T02:57:39.552 DEBUG:boto:Token: None 2024-04-12T02:57:39.553 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj 2024-04-12T02:57:39.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FKyb5mupo5YcwNoCxe9dBrkLVrk= 2024-04-12T02:57:39.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FKyb5mupo5YcwNoCxe9dBrkLVrk=', 'Content-Length': '0'} 2024-04-12T02:57:39.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005928dfbfb0514f8d-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T02:57:39.555 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.555 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.555 DEBUG:boto:Method: GET 2024-04-12T02:57:39.555 DEBUG:boto:Path: /dgbpxg-8/_myobj 2024-04-12T02:57:39.555 DEBUG:boto:Data: 2024-04-12T02:57:39.555 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.555 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.555 DEBUG:boto:Params: {} 2024-04-12T02:57:39.556 DEBUG:boto:Token: None 2024-04-12T02:57:39.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj 2024-04-12T02:57:39.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BR8vqqKMA+wq6EKqHfMkK5oGlXc= 2024-04-12T02:57:39.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BR8vqqKMA+wq6EKqHfMkK5oGlXc=', 'Content-Length': '0'} 2024-04-12T02:57:39.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e47f08000dd40c83-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.598 DEBUG:boto:path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.599 DEBUG:boto:auth_path=/dgbpxg-8/_myobj 2024-04-12T02:57:39.599 DEBUG:boto:Method: GET 2024-04-12T02:57:39.599 DEBUG:boto:Path: /dgbpxg-8/_myobj 2024-04-12T02:57:39.599 DEBUG:boto:Data: 2024-04-12T02:57:39.599 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.599 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.599 DEBUG:boto:Params: {} 2024-04-12T02:57:39.599 DEBUG:boto:Token: None 2024-04-12T02:57:39.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/_myobj 2024-04-12T02:57:39.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BR8vqqKMA+wq6EKqHfMkK5oGlXc= 2024-04-12T02:57:39.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BR8vqqKMA+wq6EKqHfMkK5oGlXc=', 'Content-Length': '0'} 2024-04-12T02:57:39.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005491dc2795d33742-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.601 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.602 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.602 DEBUG:boto:path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.602 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.602 DEBUG:boto:Method: GET 2024-04-12T02:57:39.602 DEBUG:boto:Path: /dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.602 DEBUG:boto:Data: 2024-04-12T02:57:39.602 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.602 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.602 DEBUG:boto:Params: {} 2024-04-12T02:57:39.603 DEBUG:boto:Token: None 2024-04-12T02:57:39.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000498156aeeebfed94-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.605 DEBUG:boto:b'dgbpxg-8_myobj1000false_myobj2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.605 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.606 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.606 DEBUG:boto:path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.606 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.606 DEBUG:boto:Method: GET 2024-04-12T02:57:39.606 DEBUG:boto:Path: /dgbpxg-8/?prefix=_myobj 2024-04-12T02:57:39.606 DEBUG:boto:Data: 2024-04-12T02:57:39.606 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.606 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.606 DEBUG:boto:Params: {} 2024-04-12T02:57:39.607 DEBUG:boto:Token: None 2024-04-12T02:57:39.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aad844620d8bd4e-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.609 DEBUG:boto:b'dgbpxg-8_myobj1000false_myobj2024-04-12T02:56:53.234Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:39.609 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.610 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.610 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.610 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.610 DEBUG:boto:Method: GET 2024-04-12T02:57:39.610 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.610 DEBUG:boto:Data: 2024-04-12T02:57:39.610 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.610 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.610 DEBUG:boto:Params: {} 2024-04-12T02:57:39.611 DEBUG:boto:Token: None 2024-04-12T02:57:39.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc= 2024-04-12T02:57:39.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc=', 'Content-Length': '0'} 2024-04-12T02:57:39.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d6450dd3460c644-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.613 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.613 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.613 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.613 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.613 DEBUG:boto:Method: GET 2024-04-12T02:57:39.613 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.613 DEBUG:boto:Data: 2024-04-12T02:57:39.613 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.613 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.614 DEBUG:boto:Params: {} 2024-04-12T02:57:39.614 DEBUG:boto:Token: None 2024-04-12T02:57:39.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc= 2024-04-12T02:57:39.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc=', 'Content-Length': '0'} 2024-04-12T02:57:39.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7dfab295b41246a-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.616 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.616 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.616 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.616 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.616 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.616 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.616 DEBUG:boto:Data: 2024-04-12T02:57:39.616 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.616 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.616 DEBUG:boto:Params: {} 2024-04-12T02:57:39.617 DEBUG:boto:Token: None 2024-04-12T02:57:39.617 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJHwmJ/VdGuNNgXe9jpcpDixXhg= 2024-04-12T02:57:39.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJHwmJ/VdGuNNgXe9jpcpDixXhg=', 'Content-Length': '0'} 2024-04-12T02:57:39.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000891af6026db0bc66-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.619 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.619 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.619 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.619 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.619 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.619 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.619 DEBUG:boto:Data: 2024-04-12T02:57:39.619 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.620 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.620 DEBUG:boto:Params: {} 2024-04-12T02:57:39.620 DEBUG:boto:Token: None 2024-04-12T02:57:39.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJHwmJ/VdGuNNgXe9jpcpDixXhg= 2024-04-12T02:57:39.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJHwmJ/VdGuNNgXe9jpcpDixXhg=', 'Content-Length': '0'} 2024-04-12T02:57:39.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004011d7742669ebd8-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:39.622 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.622 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.622 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.622 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.622 DEBUG:boto:Method: GET 2024-04-12T02:57:39.622 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.622 DEBUG:boto:Data: 2024-04-12T02:57:39.622 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.623 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.623 DEBUG:boto:Params: {} 2024-04-12T02:57:39.623 DEBUG:boto:Token: None 2024-04-12T02:57:39.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc= 2024-04-12T02:57:39.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc=', 'Content-Length': '0'} 2024-04-12T02:57:39.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000312de172b1a17856-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.625 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.625 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.625 DEBUG:boto:path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.625 DEBUG:boto:auth_path=/dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.625 DEBUG:boto:Method: GET 2024-04-12T02:57:39.625 DEBUG:boto:Path: /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.626 DEBUG:boto:Data: 2024-04-12T02:57:39.626 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.626 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.626 DEBUG:boto:Params: {} 2024-04-12T02:57:39.626 DEBUG:boto:Token: None 2024-04-12T02:57:39.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj?versionId=null 2024-04-12T02:57:39.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc= 2024-04-12T02:57:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDVJdfJuWt9mZsY3QY824/TGZJc=', 'Content-Length': '0'} 2024-04-12T02:57:39.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6cba8cf2e44153c-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.628 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.628 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.628 DEBUG:boto:path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.628 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.628 DEBUG:boto:Method: GET 2024-04-12T02:57:39.629 DEBUG:boto:Path: /dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.629 DEBUG:boto:Data: 2024-04-12T02:57:39.629 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.629 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.629 DEBUG:boto:Params: {} 2024-04-12T02:57:39.629 DEBUG:boto:Token: None 2024-04-12T02:57:39.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5233d9764db43b-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.631 DEBUG:boto:b'dgbpxg-8myobj1000falsemyobj2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.632 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.632 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.632 DEBUG:boto:path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.632 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.632 DEBUG:boto:Method: GET 2024-04-12T02:57:39.632 DEBUG:boto:Path: /dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.632 DEBUG:boto:Data: 2024-04-12T02:57:39.632 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.633 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.633 DEBUG:boto:Params: {} 2024-04-12T02:57:39.633 DEBUG:boto:Token: None 2024-04-12T02:57:39.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7804d4032c920b-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.635 DEBUG:boto:b'dgbpxg-8myobj1000falsemyobj2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.635 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.635 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.635 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.636 DEBUG:boto:Path: /dgbpxg-8/myobj 2024-04-12T02:57:39.636 DEBUG:boto:Data: 2024-04-12T02:57:39.636 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.636 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.636 DEBUG:boto:Params: {} 2024-04-12T02:57:39.636 DEBUG:boto:Token: None 2024-04-12T02:57:39.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj 2024-04-12T02:57:39.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:erm24MyIoCd4jybA9nj0lRz8R8I= 2024-04-12T02:57:39.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:erm24MyIoCd4jybA9nj0lRz8R8I=', 'Content-Length': '0'} 2024-04-12T02:57:39.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e9037c06238629d-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.638 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.638 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.638 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.638 DEBUG:boto:Path: /dgbpxg-8/myobj 2024-04-12T02:57:39.638 DEBUG:boto:Data: 2024-04-12T02:57:39.639 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.639 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.639 DEBUG:boto:Params: {} 2024-04-12T02:57:39.639 DEBUG:boto:Token: None 2024-04-12T02:57:39.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj 2024-04-12T02:57:39.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:erm24MyIoCd4jybA9nj0lRz8R8I= 2024-04-12T02:57:39.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:erm24MyIoCd4jybA9nj0lRz8R8I=', 'Content-Length': '0'} 2024-04-12T02:57:39.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2dfa5a5866d4d30-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:57:39.641 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.641 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.641 DEBUG:boto:Method: GET 2024-04-12T02:57:39.641 DEBUG:boto:Path: /dgbpxg-8/myobj 2024-04-12T02:57:39.641 DEBUG:boto:Data: 2024-04-12T02:57:39.641 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.642 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.642 DEBUG:boto:Params: {} 2024-04-12T02:57:39.642 DEBUG:boto:Token: None 2024-04-12T02:57:39.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj 2024-04-12T02:57:39.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QjDM5pxGRTeP2wN4j8pVg+elJVI= 2024-04-12T02:57:39.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QjDM5pxGRTeP2wN4j8pVg+elJVI=', 'Content-Length': '0'} 2024-04-12T02:57:39.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022aec3d0f4b8f2a3-006618a323-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.686 DEBUG:boto:path=/dgbpxg-8/myobj 2024-04-12T02:57:39.686 DEBUG:boto:auth_path=/dgbpxg-8/myobj 2024-04-12T02:57:39.687 DEBUG:boto:Method: GET 2024-04-12T02:57:39.687 DEBUG:boto:Path: /dgbpxg-8/myobj 2024-04-12T02:57:39.687 DEBUG:boto:Data: 2024-04-12T02:57:39.687 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.688 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.688 DEBUG:boto:Params: {} 2024-04-12T02:57:39.688 DEBUG:boto:Token: None 2024-04-12T02:57:39.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/myobj 2024-04-12T02:57:39.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QjDM5pxGRTeP2wN4j8pVg+elJVI= 2024-04-12T02:57:39.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QjDM5pxGRTeP2wN4j8pVg+elJVI=', 'Content-Length': '0'} 2024-04-12T02:57:39.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:56:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003226a99547f53c74-006618a323-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.690 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.690 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.690 DEBUG:boto:path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.690 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.691 DEBUG:boto:Method: GET 2024-04-12T02:57:39.691 DEBUG:boto:Path: /dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.691 DEBUG:boto:Data: 2024-04-12T02:57:39.691 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.691 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.691 DEBUG:boto:Params: {} 2024-04-12T02:57:39.691 DEBUG:boto:Token: None 2024-04-12T02:57:39.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025cded834e115af0-006618a323-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.694 DEBUG:boto:b'dgbpxg-8myobj1000falsemyobj2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.694 DEBUG:boto:path=/dgbpxg-8/ 2024-04-12T02:57:39.694 DEBUG:boto:auth_path=/dgbpxg-8/ 2024-04-12T02:57:39.694 DEBUG:boto:path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.695 DEBUG:boto:auth_path=/dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.695 DEBUG:boto:Method: GET 2024-04-12T02:57:39.695 DEBUG:boto:Path: /dgbpxg-8/?prefix=myobj 2024-04-12T02:57:39.695 DEBUG:boto:Data: 2024-04-12T02:57:39.695 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.695 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.695 DEBUG:boto:Params: {} 2024-04-12T02:57:39.695 DEBUG:boto:Token: None 2024-04-12T02:57:39.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-8/ 2024-04-12T02:57:39.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc= 2024-04-12T02:57:39.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P2gmSWm+mAzd+JcllXZa66aNHrc=', 'Content-Length': '0'} 2024-04-12T02:57:39.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33e75872635c940-006618a323-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.697 DEBUG:boto:b'dgbpxg-8myobj1000falsemyobj2024-04-12T02:56:53.215Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:57:39.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-8 zones={a1, a2} 2024-04-12T02:57:39.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-12T02:57:39.699 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-9 2024-04-12T02:57:39.699 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:57:39.699 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:57:39.699 DEBUG:boto:Method: PUT 2024-04-12T02:57:39.700 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T02:57:39.700 DEBUG:boto:Data: 2024-04-12T02:57:39.700 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.700 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.700 DEBUG:boto:Params: {} 2024-04-12T02:57:39.700 DEBUG:boto:Token: None 2024-04-12T02:57:39.700 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-9/ 2024-04-12T02:57:39.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SfZmJ7SzIHGNdCoF1FFV9JyhU9c= 2024-04-12T02:57:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SfZmJ7SzIHGNdCoF1FFV9JyhU9c=', 'Content-Length': '0'} 2024-04-12T02:57:39.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005459208a7d6ba4a6-006618a323-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.713 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-10 2024-04-12T02:57:39.713 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:57:39.713 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:57:39.713 DEBUG:boto:Method: PUT 2024-04-12T02:57:39.714 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T02:57:39.714 DEBUG:boto:Data: 2024-04-12T02:57:39.714 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.714 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.714 DEBUG:boto:Params: {} 2024-04-12T02:57:39.714 DEBUG:boto:Token: None 2024-04-12T02:57:39.714 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-10/ 2024-04-12T02:57:39.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BTrccrAOBh3dyb6dsSl726mo69s= 2024-04-12T02:57:39.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BTrccrAOBh3dyb6dsSl726mo69s=', 'Content-Length': '0'} 2024-04-12T02:57:39.736 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097da246a05b32bf1-006618a323-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.736 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:57:39.736 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:57:39.736 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.736 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T02:57:39.736 DEBUG:boto:Data: 2024-04-12T02:57:39.736 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.737 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.737 DEBUG:boto:Params: {} 2024-04-12T02:57:39.737 DEBUG:boto:Token: None 2024-04-12T02:57:39.737 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-9/ 2024-04-12T02:57:39.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lFOtM6KeJhUhZqAi2uHDz1M/Z3I= 2024-04-12T02:57:39.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lFOtM6KeJhUhZqAi2uHDz1M/Z3I=', 'Content-Length': '0'} 2024-04-12T02:57:39.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d1ed9be7463b8b-006618a323-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.740 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:57:39.740 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:57:39.740 DEBUG:boto:Method: PUT 2024-04-12T02:57:39.740 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:57:39.740 DEBUG:boto:Data: 2024-04-12T02:57:39.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:57:39.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:57:39.740 DEBUG:boto:Port: 8000 2024-04-12T02:57:39.740 DEBUG:boto:Params: {} 2024-04-12T02:57:39.740 DEBUG:boto:Token: None 2024-04-12T02:57:39.741 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-9/myobj 2024-04-12T02:57:39.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zpcgh+qIetHrjPKTdnC5f1m1Oc4= 2024-04-12T02:57:39.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zpcgh+qIetHrjPKTdnC5f1m1Oc4='} 2024-04-12T02:57:39.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff9f59896ee51695-006618a323-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.783 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:57:39.783 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:57:39.783 DEBUG:boto:Method: HEAD 2024-04-12T02:57:39.783 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T02:57:39.783 DEBUG:boto:Data: 2024-04-12T02:57:39.783 DEBUG:boto:Headers: {} 2024-04-12T02:57:39.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.783 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.783 DEBUG:boto:Params: {} 2024-04-12T02:57:39.784 DEBUG:boto:Token: None 2024-04-12T02:57:39.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-10/ 2024-04-12T02:57:39.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P8rCi6rREkEWMTJsZ3TaHU9OlxE= 2024-04-12T02:57:39.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P8rCi6rREkEWMTJsZ3TaHU9OlxE=', 'Content-Length': '0'} 2024-04-12T02:57:39.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a918fb5029d54fc3-006618a323-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.787 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:57:39.787 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:57:39.787 DEBUG:boto:Method: PUT 2024-04-12T02:57:39.787 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T02:57:39.787 DEBUG:boto:Data: 2024-04-12T02:57:39.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T02:57:39.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:57:39.788 DEBUG:boto:Port: 8001 2024-04-12T02:57:39.788 DEBUG:boto:Params: {} 2024-04-12T02:57:39.788 DEBUG:boto:Token: None 2024-04-12T02:57:39.788 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 02:57:39 GMT /dgbpxg-10/myobj 2024-04-12T02:57:39.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L47AN8+gZlNu+S1ObymrktUTGng= 2024-04-12T02:57:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 02:57:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L47AN8+gZlNu+S1ObymrktUTGng='} 2024-04-12T02:57:39.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087e68cfa42cb0534-006618a323-1112-a2'), ('Date', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:57:39.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:57:39.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:57:39.794 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:57:39.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:57:39.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:57:39.875 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.727226_185.1", 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.727226Z" 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:39.903 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T02:57:39.904 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890659.727226_185.1'} 2024-04-12T02:57:39.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T02:57:39.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:57:39.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T02:57:39.905 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T02:57:39.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:57:39.953 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:57:39.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:39.980+0000 7fed03ec7780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:57:39.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:39.981+0000 7fed03ec7780 20 RGW-SYNC:meta: read sync status 2024-04-12T02:57:39.993 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T02:57:39.993 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T02:57:39.993 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T02:57:39.993 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T02:57:39.993 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T02:57:39.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.996 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T02:57:39.997 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:57:39.997 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:57:39.997 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T02:57:39.997 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:57:39.997 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:39.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:57:39.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:40.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.727226_185.1", 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T02:57:40.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T02:57:40.002 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T02:57:40.002 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.727226Z", 2024-04-12T02:57:40.002 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T02:57:40.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:40.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T02:57:40.003 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T02:57:39Z" 2024-04-12T02:57:40.004 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T02:57:40.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712890659.727226_185.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.727226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T02:57:39Z" } 2024-04-12T02:57:40.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712890659.727226_185.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.727226Z', 'realm_epoch': 2}}] 2024-04-12T02:57:40.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890659.727226_185.1'} 2024-04-12T02:57:40.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890659.727226_185.1'} 2024-04-12T02:57:40.006 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T02:57:40.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:57:40.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:57:40.007 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T02:57:40.061 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:57:40.062 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:57:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:40.093+0000 7f51413cf780 1 waiting to reach incremental sync.. 2024-04-12T02:57:45.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:45.093+0000 7f51413cf780 1 waiting to reach incremental sync.. 2024-04-12T02:57:50.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:50.093+0000 7f51413cf780 1 waiting to reach incremental sync.. 2024-04-12T02:57:55.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:57:55.094+0000 7f51413cf780 1 waiting to reach incremental sync.. 2024-04-12T02:58:00.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:00.095+0000 7f51413cf780 1 bucket sync caught up with source: 2024-04-12T02:58:00.098 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:58:00.098 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:58:00.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:00.095+0000 7f51413cf780 0 bucket checkpoint complete 2024-04-12T02:58:00.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-9 zones={a2, a1} 2024-04-12T02:58:00.113 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.113 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.113 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.113 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T02:58:00.113 DEBUG:boto:Data: 2024-04-12T02:58:00.113 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.113 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.113 DEBUG:boto:Params: {} 2024-04-12T02:58:00.114 DEBUG:boto:Token: None 2024-04-12T02:58:00.114 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RFwNUhWaQxaUBTv9uP696ibQ/AM= 2024-04-12T02:58:00.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RFwNUhWaQxaUBTv9uP696ibQ/AM=', 'Content-Length': '0'} 2024-04-12T02:58:00.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b474904b4f8b1e9d-006618a338-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.117 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.117 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.117 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.117 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T02:58:00.117 DEBUG:boto:Data: 2024-04-12T02:58:00.117 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.117 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.117 DEBUG:boto:Params: {} 2024-04-12T02:58:00.117 DEBUG:boto:Token: None 2024-04-12T02:58:00.118 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.118 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RFwNUhWaQxaUBTv9uP696ibQ/AM= 2024-04-12T02:58:00.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RFwNUhWaQxaUBTv9uP696ibQ/AM=', 'Content-Length': '0'} 2024-04-12T02:58:00.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046c88358112d4ea3-006618a338-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.120 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T02:58:00.120 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.120 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.120 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.120 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.121 DEBUG:boto:Method: GET 2024-04-12T02:58:00.121 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.121 DEBUG:boto:Data: 2024-04-12T02:58:00.121 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.121 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.121 DEBUG:boto:Params: {} 2024-04-12T02:58:00.121 DEBUG:boto:Token: None 2024-04-12T02:58:00.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/?versions 2024-04-12T02:58:00.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM= 2024-04-12T02:58:00.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM=', 'Content-Length': '0'} 2024-04-12T02:58:00.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3bef5ba8cf53327-006618a338-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.124 DEBUG:boto:b'dgbpxg-9999falsemyobjnulltrue2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.124 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:58:00.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T02:58:00.125 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.125 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.125 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.125 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.125 DEBUG:boto:Method: GET 2024-04-12T02:58:00.125 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.125 DEBUG:boto:Data: 2024-04-12T02:58:00.125 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.125 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.125 DEBUG:boto:Params: {} 2024-04-12T02:58:00.125 DEBUG:boto:Token: None 2024-04-12T02:58:00.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/?versions 2024-04-12T02:58:00.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM= 2024-04-12T02:58:00.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM=', 'Content-Length': '0'} 2024-04-12T02:58:00.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a09c9c9291af7e-006618a338-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.128 DEBUG:boto:b'dgbpxg-9999falsemyobjnulltrue2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.129 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:58:00.129 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.129 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.129 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.129 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.129 DEBUG:boto:Method: GET 2024-04-12T02:58:00.129 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.129 DEBUG:boto:Data: 2024-04-12T02:58:00.129 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.129 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.130 DEBUG:boto:Params: {} 2024-04-12T02:58:00.130 DEBUG:boto:Token: None 2024-04-12T02:58:00.130 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/?versions 2024-04-12T02:58:00.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM= 2024-04-12T02:58:00.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM=', 'Content-Length': '0'} 2024-04-12T02:58:00.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002681efc71f42cf51-006618a338-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.132 DEBUG:boto:b'dgbpxg-9999falsemyobjnulltrue2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.133 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.133 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.133 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.133 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.133 DEBUG:boto:Method: GET 2024-04-12T02:58:00.133 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T02:58:00.133 DEBUG:boto:Data: 2024-04-12T02:58:00.133 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.133 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.134 DEBUG:boto:Params: {} 2024-04-12T02:58:00.134 DEBUG:boto:Token: None 2024-04-12T02:58:00.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/?versions 2024-04-12T02:58:00.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM= 2024-04-12T02:58:00.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oI4JbErvfm9P/oKvlzcBrvPGgRM=', 'Content-Length': '0'} 2024-04-12T02:58:00.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000182d2389963bfcb1-006618a338-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.136 DEBUG:boto:b'dgbpxg-9999falsemyobjnulltrue2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:58:00.137 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.137 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.137 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.137 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.137 DEBUG:boto:Method: GET 2024-04-12T02:58:00.137 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.137 DEBUG:boto:Data: 2024-04-12T02:58:00.137 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.137 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.137 DEBUG:boto:Params: {} 2024-04-12T02:58:00.138 DEBUG:boto:Token: None 2024-04-12T02:58:00.138 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs= 2024-04-12T02:58:00.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs=', 'Content-Length': '0'} 2024-04-12T02:58:00.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b70dc512f5d67972-006618a338-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.140 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.140 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.140 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.140 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.140 DEBUG:boto:Method: GET 2024-04-12T02:58:00.140 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.141 DEBUG:boto:Data: 2024-04-12T02:58:00.141 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.141 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.141 DEBUG:boto:Params: {} 2024-04-12T02:58:00.141 DEBUG:boto:Token: None 2024-04-12T02:58:00.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs= 2024-04-12T02:58:00.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs=', 'Content-Length': '0'} 2024-04-12T02:58:00.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e19e004511d5eea-006618a338-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.143 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.144 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.144 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.144 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.144 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.144 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.144 DEBUG:boto:Data: 2024-04-12T02:58:00.144 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.144 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.144 DEBUG:boto:Params: {} 2024-04-12T02:58:00.144 DEBUG:boto:Token: None 2024-04-12T02:58:00.145 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.145 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rrnxBJMu5hrod7RFcgZbU3ieYik= 2024-04-12T02:58:00.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rrnxBJMu5hrod7RFcgZbU3ieYik=', 'Content-Length': '0'} 2024-04-12T02:58:00.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d07b207445366583-006618a338-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.147 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.147 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.147 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.147 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.147 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.147 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.147 DEBUG:boto:Data: 2024-04-12T02:58:00.147 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.147 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.147 DEBUG:boto:Params: {} 2024-04-12T02:58:00.148 DEBUG:boto:Token: None 2024-04-12T02:58:00.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rrnxBJMu5hrod7RFcgZbU3ieYik= 2024-04-12T02:58:00.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rrnxBJMu5hrod7RFcgZbU3ieYik=', 'Content-Length': '0'} 2024-04-12T02:58:00.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e305995a7d1b5ea9-006618a338-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:58:00.150 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.150 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.150 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.150 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.150 DEBUG:boto:Method: GET 2024-04-12T02:58:00.150 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.150 DEBUG:boto:Data: 2024-04-12T02:58:00.151 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.151 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.151 DEBUG:boto:Params: {} 2024-04-12T02:58:00.151 DEBUG:boto:Token: None 2024-04-12T02:58:00.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs= 2024-04-12T02:58:00.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs=', 'Content-Length': '0'} 2024-04-12T02:58:00.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063cfda6c36dc15d7-006618a338-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.153 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.153 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.153 DEBUG:boto:path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.153 DEBUG:boto:auth_path=/dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.153 DEBUG:boto:Method: GET 2024-04-12T02:58:00.154 DEBUG:boto:Path: /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.154 DEBUG:boto:Data: 2024-04-12T02:58:00.154 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.154 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.154 DEBUG:boto:Params: {} 2024-04-12T02:58:00.154 DEBUG:boto:Token: None 2024-04-12T02:58:00.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj?versionId=null 2024-04-12T02:58:00.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs= 2024-04-12T02:58:00.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXKJbBppZJyfLcbqxkGaJuBgdLs=', 'Content-Length': '0'} 2024-04-12T02:58:00.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000448ca3ccc87a4b85-006618a338-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.156 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.156 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.157 DEBUG:boto:path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.157 DEBUG:boto:auth_path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.157 DEBUG:boto:Method: GET 2024-04-12T02:58:00.157 DEBUG:boto:Path: /dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.157 DEBUG:boto:Data: 2024-04-12T02:58:00.157 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.157 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.157 DEBUG:boto:Params: {} 2024-04-12T02:58:00.157 DEBUG:boto:Token: None 2024-04-12T02:58:00.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA= 2024-04-12T02:58:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA=', 'Content-Length': '0'} 2024-04-12T02:58:00.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffcebe80bd42ecd1-006618a338-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.160 DEBUG:boto:b'dgbpxg-9myobj1000falsemyobj2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.161 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.161 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.161 DEBUG:boto:path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.161 DEBUG:boto:auth_path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.161 DEBUG:boto:Method: GET 2024-04-12T02:58:00.161 DEBUG:boto:Path: /dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.161 DEBUG:boto:Data: 2024-04-12T02:58:00.161 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.162 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.162 DEBUG:boto:Params: {} 2024-04-12T02:58:00.162 DEBUG:boto:Token: None 2024-04-12T02:58:00.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA= 2024-04-12T02:58:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA=', 'Content-Length': '0'} 2024-04-12T02:58:00.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f93c4d6c6710b8a6-006618a338-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.164 DEBUG:boto:b'dgbpxg-9myobj1000falsemyobj2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.165 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.165 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.165 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.165 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:58:00.165 DEBUG:boto:Data: 2024-04-12T02:58:00.165 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.165 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.166 DEBUG:boto:Params: {} 2024-04-12T02:58:00.166 DEBUG:boto:Token: None 2024-04-12T02:58:00.166 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj 2024-04-12T02:58:00.166 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:54yjgxLs14YJQcEZKYfmrs5vUTM= 2024-04-12T02:58:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:54yjgxLs14YJQcEZKYfmrs5vUTM=', 'Content-Length': '0'} 2024-04-12T02:58:00.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000353ed6df8c1f3504-006618a338-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.168 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.168 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.168 DEBUG:boto:Method: HEAD 2024-04-12T02:58:00.168 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:58:00.168 DEBUG:boto:Data: 2024-04-12T02:58:00.168 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.168 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.169 DEBUG:boto:Params: {} 2024-04-12T02:58:00.169 DEBUG:boto:Token: None 2024-04-12T02:58:00.169 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj 2024-04-12T02:58:00.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:54yjgxLs14YJQcEZKYfmrs5vUTM= 2024-04-12T02:58:00.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:54yjgxLs14YJQcEZKYfmrs5vUTM=', 'Content-Length': '0'} 2024-04-12T02:58:00.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005742fe5f01898490-006618a338-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:58:00.171 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.171 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.171 DEBUG:boto:Method: GET 2024-04-12T02:58:00.172 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:58:00.172 DEBUG:boto:Data: 2024-04-12T02:58:00.172 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.172 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.172 DEBUG:boto:Params: {} 2024-04-12T02:58:00.172 DEBUG:boto:Token: None 2024-04-12T02:58:00.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj 2024-04-12T02:58:00.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+SUHeZYWZsqUVbOCC6KmU232j0= 2024-04-12T02:58:00.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+SUHeZYWZsqUVbOCC6KmU232j0=', 'Content-Length': '0'} 2024-04-12T02:58:00.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b34de014af7c1f-006618a338-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.175 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:58:00.175 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:58:00.175 DEBUG:boto:Method: GET 2024-04-12T02:58:00.175 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:58:00.175 DEBUG:boto:Data: 2024-04-12T02:58:00.175 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.175 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.175 DEBUG:boto:Params: {} 2024-04-12T02:58:00.175 DEBUG:boto:Token: None 2024-04-12T02:58:00.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/myobj 2024-04-12T02:58:00.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+SUHeZYWZsqUVbOCC6KmU232j0= 2024-04-12T02:58:00.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+SUHeZYWZsqUVbOCC6KmU232j0=', 'Content-Length': '0'} 2024-04-12T02:58:00.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f4e55c322ab9a4-006618a338-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.178 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.178 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.178 DEBUG:boto:path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.178 DEBUG:boto:auth_path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.178 DEBUG:boto:Method: GET 2024-04-12T02:58:00.178 DEBUG:boto:Path: /dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.178 DEBUG:boto:Data: 2024-04-12T02:58:00.178 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:58:00.178 DEBUG:boto:Port: 8001 2024-04-12T02:58:00.179 DEBUG:boto:Params: {} 2024-04-12T02:58:00.179 DEBUG:boto:Token: None 2024-04-12T02:58:00.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA= 2024-04-12T02:58:00.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA=', 'Content-Length': '0'} 2024-04-12T02:58:00.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd241f2f6893923-006618a338-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.226 DEBUG:boto:b'dgbpxg-9myobj1000falsemyobj2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.227 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:58:00.227 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:58:00.227 DEBUG:boto:path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.227 DEBUG:boto:auth_path=/dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.227 DEBUG:boto:Method: GET 2024-04-12T02:58:00.227 DEBUG:boto:Path: /dgbpxg-9/?prefix=myobj 2024-04-12T02:58:00.227 DEBUG:boto:Data: 2024-04-12T02:58:00.228 DEBUG:boto:Headers: {} 2024-04-12T02:58:00.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:58:00.228 DEBUG:boto:Port: 8000 2024-04-12T02:58:00.228 DEBUG:boto:Params: {} 2024-04-12T02:58:00.228 DEBUG:boto:Token: None 2024-04-12T02:58:00.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:58:00 GMT /dgbpxg-9/ 2024-04-12T02:58:00.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA= 2024-04-12T02:58:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:58:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f2GqK2y9pU1KBloYbwrwxDL2MjA=', 'Content-Length': '0'} 2024-04-12T02:58:00.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008347e98ebf4fc0e4-006618a338-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:58:00.231 DEBUG:boto:b'dgbpxg-9myobj1000falsemyobj2024-04-12T02:57:39.742Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:58:00.231 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-9 zones={a2, a1} 2024-04-12T02:58:00.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:58:00.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:58:00.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T02:58:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:58:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:58:00.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:00.321+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:05.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:05.323+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:10.324+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:15.324+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:20.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:20.325+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:25.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:25.326+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:30.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:30.326+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:35.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:35.326+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:40.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:40.327+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:45.328+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:50.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:50.329+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:58:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:58:55.329+0000 7f3efed6e780 1 waiting to reach incremental sync.. 2024-04-12T02:59:00.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:00.331+0000 7f3efed6e780 1 bucket sync caught up with source: 2024-04-12T02:59:00.332 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T02:59:00.332 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T02:59:00.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:00.331+0000 7f3efed6e780 0 bucket checkpoint complete 2024-04-12T02:59:00.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-10 zones={a1, a2} 2024-04-12T02:59:00.348 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.348 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.348 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.348 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T02:59:00.348 DEBUG:boto:Data: 2024-04-12T02:59:00.349 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.349 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.349 DEBUG:boto:Params: {} 2024-04-12T02:59:00.349 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T02:59:00.350 DEBUG:boto:Token: None 2024-04-12T02:59:00.350 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.350 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPRx2Ku1Y26pHLuOK7iEMybL4Qs= 2024-04-12T02:59:00.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPRx2Ku1Y26pHLuOK7iEMybL4Qs=', 'Content-Length': '0'} 2024-04-12T02:59:00.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e636d9ac9d5b947-006618a374-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.365 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.365 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.365 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.365 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T02:59:00.365 DEBUG:boto:Data: 2024-04-12T02:59:00.365 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.365 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.366 DEBUG:boto:Params: {} 2024-04-12T02:59:00.366 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T02:59:00.366 DEBUG:boto:Token: None 2024-04-12T02:59:00.366 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPRx2Ku1Y26pHLuOK7iEMybL4Qs= 2024-04-12T02:59:00.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPRx2Ku1Y26pHLuOK7iEMybL4Qs=', 'Content-Length': '0'} 2024-04-12T02:59:00.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f76e7c710fb8f19-006618a374-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.378 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T02:59:00.378 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.378 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.378 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.379 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.379 DEBUG:boto:Method: GET 2024-04-12T02:59:00.379 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.379 DEBUG:boto:Data: 2024-04-12T02:59:00.379 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.379 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.379 DEBUG:boto:Params: {} 2024-04-12T02:59:00.379 DEBUG:boto:Token: None 2024-04-12T02:59:00.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/?versions 2024-04-12T02:59:00.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4= 2024-04-12T02:59:00.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4=', 'Content-Length': '0'} 2024-04-12T02:59:00.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083e6556cfe4d89b-006618a374-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.382 DEBUG:boto:b'dgbpxg-10999falsemyobjnulltrue2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.383 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:59:00.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T02:59:00.383 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.383 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.383 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.383 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.384 DEBUG:boto:Method: GET 2024-04-12T02:59:00.384 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.384 DEBUG:boto:Data: 2024-04-12T02:59:00.384 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.384 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.384 DEBUG:boto:Params: {} 2024-04-12T02:59:00.384 DEBUG:boto:Token: None 2024-04-12T02:59:00.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/?versions 2024-04-12T02:59:00.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4= 2024-04-12T02:59:00.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4=', 'Content-Length': '0'} 2024-04-12T02:59:00.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a709dab053cd3ab-006618a374-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.387 DEBUG:boto:b'dgbpxg-10999falsemyobjnulltrue2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.387 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T02:59:00.387 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.387 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.387 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.388 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.388 DEBUG:boto:Method: GET 2024-04-12T02:59:00.388 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.388 DEBUG:boto:Data: 2024-04-12T02:59:00.388 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.388 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.388 DEBUG:boto:Params: {} 2024-04-12T02:59:00.388 DEBUG:boto:Token: None 2024-04-12T02:59:00.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/?versions 2024-04-12T02:59:00.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4= 2024-04-12T02:59:00.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4=', 'Content-Length': '0'} 2024-04-12T02:59:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cff4ea2470df9c-006618a374-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.391 DEBUG:boto:b'dgbpxg-10999falsemyobjnulltrue2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.391 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.391 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.391 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.392 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.392 DEBUG:boto:Method: GET 2024-04-12T02:59:00.392 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T02:59:00.392 DEBUG:boto:Data: 2024-04-12T02:59:00.392 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.392 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.392 DEBUG:boto:Params: {} 2024-04-12T02:59:00.392 DEBUG:boto:Token: None 2024-04-12T02:59:00.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/?versions 2024-04-12T02:59:00.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4= 2024-04-12T02:59:00.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsqG3qXXcqFvHUHxmz1egDp0Nd4=', 'Content-Length': '0'} 2024-04-12T02:59:00.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000514535d6e88fef1b-006618a374-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.395 DEBUG:boto:b'dgbpxg-10999falsemyobjnulltrue2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:59:00.395 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.396 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.396 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.396 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.396 DEBUG:boto:Method: GET 2024-04-12T02:59:00.396 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.396 DEBUG:boto:Data: 2024-04-12T02:59:00.396 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.396 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.396 DEBUG:boto:Params: {} 2024-04-12T02:59:00.396 DEBUG:boto:Token: None 2024-04-12T02:59:00.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o= 2024-04-12T02:59:00.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o=', 'Content-Length': '0'} 2024-04-12T02:59:00.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000758dea1899d1f567-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.399 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.399 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.399 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.399 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.399 DEBUG:boto:Method: GET 2024-04-12T02:59:00.399 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.399 DEBUG:boto:Data: 2024-04-12T02:59:00.399 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.400 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.400 DEBUG:boto:Params: {} 2024-04-12T02:59:00.400 DEBUG:boto:Token: None 2024-04-12T02:59:00.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o= 2024-04-12T02:59:00.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o=', 'Content-Length': '0'} 2024-04-12T02:59:00.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e19cd76cf996c610-006618a374-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.402 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.402 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.402 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.403 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.403 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.403 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.403 DEBUG:boto:Data: 2024-04-12T02:59:00.403 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.403 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.403 DEBUG:boto:Params: {} 2024-04-12T02:59:00.403 DEBUG:boto:Token: None 2024-04-12T02:59:00.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lCQ3/BkVX2WHEN5aJ/gcGdvTYzk= 2024-04-12T02:59:00.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lCQ3/BkVX2WHEN5aJ/gcGdvTYzk=', 'Content-Length': '0'} 2024-04-12T02:59:00.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ce2047d01a3af98-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.405 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.405 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.406 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.406 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.406 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.406 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.406 DEBUG:boto:Data: 2024-04-12T02:59:00.406 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.406 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.406 DEBUG:boto:Params: {} 2024-04-12T02:59:00.406 DEBUG:boto:Token: None 2024-04-12T02:59:00.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lCQ3/BkVX2WHEN5aJ/gcGdvTYzk= 2024-04-12T02:59:00.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lCQ3/BkVX2WHEN5aJ/gcGdvTYzk=', 'Content-Length': '0'} 2024-04-12T02:59:00.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072544b79aefebb22-006618a374-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:59:00.409 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.409 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.409 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.409 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.409 DEBUG:boto:Method: GET 2024-04-12T02:59:00.409 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.409 DEBUG:boto:Data: 2024-04-12T02:59:00.409 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.410 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.410 DEBUG:boto:Params: {} 2024-04-12T02:59:00.410 DEBUG:boto:Token: None 2024-04-12T02:59:00.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o= 2024-04-12T02:59:00.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o=', 'Content-Length': '0'} 2024-04-12T02:59:00.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067bff773391f12e7-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.412 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.412 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.413 DEBUG:boto:path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.413 DEBUG:boto:auth_path=/dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.413 DEBUG:boto:Method: GET 2024-04-12T02:59:00.413 DEBUG:boto:Path: /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.413 DEBUG:boto:Data: 2024-04-12T02:59:00.413 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.413 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.413 DEBUG:boto:Params: {} 2024-04-12T02:59:00.413 DEBUG:boto:Token: None 2024-04-12T02:59:00.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj?versionId=null 2024-04-12T02:59:00.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o= 2024-04-12T02:59:00.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bETuGstX6K0kcWxneK5KyR513o=', 'Content-Length': '0'} 2024-04-12T02:59:00.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dd7cc3e6de5556a-006618a374-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.416 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.416 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.416 DEBUG:boto:path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.416 DEBUG:boto:auth_path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.416 DEBUG:boto:Method: GET 2024-04-12T02:59:00.416 DEBUG:boto:Path: /dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.416 DEBUG:boto:Data: 2024-04-12T02:59:00.416 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.417 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.417 DEBUG:boto:Params: {} 2024-04-12T02:59:00.417 DEBUG:boto:Token: None 2024-04-12T02:59:00.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.417 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8= 2024-04-12T02:59:00.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8=', 'Content-Length': '0'} 2024-04-12T02:59:00.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a96a0c023e24cd-006618a374-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.419 DEBUG:boto:b'dgbpxg-10myobj1000falsemyobj2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.421 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.422 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.422 DEBUG:boto:path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.422 DEBUG:boto:auth_path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.422 DEBUG:boto:Method: GET 2024-04-12T02:59:00.422 DEBUG:boto:Path: /dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.422 DEBUG:boto:Data: 2024-04-12T02:59:00.422 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.422 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.422 DEBUG:boto:Params: {} 2024-04-12T02:59:00.423 DEBUG:boto:Token: None 2024-04-12T02:59:00.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.423 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8= 2024-04-12T02:59:00.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8=', 'Content-Length': '0'} 2024-04-12T02:59:00.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121c6d344819c2e7-006618a374-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.425 DEBUG:boto:b'dgbpxg-10myobj1000falsemyobj2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.426 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.426 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.426 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.426 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T02:59:00.426 DEBUG:boto:Data: 2024-04-12T02:59:00.426 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.426 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.427 DEBUG:boto:Params: {} 2024-04-12T02:59:00.427 DEBUG:boto:Token: None 2024-04-12T02:59:00.427 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj 2024-04-12T02:59:00.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcZVJH/KoccDWbyt9r5euugRqvw= 2024-04-12T02:59:00.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcZVJH/KoccDWbyt9r5euugRqvw=', 'Content-Length': '0'} 2024-04-12T02:59:00.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2e475135df63af2-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.429 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.429 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.429 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.429 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T02:59:00.429 DEBUG:boto:Data: 2024-04-12T02:59:00.429 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.429 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.430 DEBUG:boto:Params: {} 2024-04-12T02:59:00.430 DEBUG:boto:Token: None 2024-04-12T02:59:00.430 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj 2024-04-12T02:59:00.430 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcZVJH/KoccDWbyt9r5euugRqvw= 2024-04-12T02:59:00.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcZVJH/KoccDWbyt9r5euugRqvw=', 'Content-Length': '0'} 2024-04-12T02:59:00.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044d74b9991ecb414-006618a374-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T02:59:00.432 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.432 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.432 DEBUG:boto:Method: GET 2024-04-12T02:59:00.432 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T02:59:00.432 DEBUG:boto:Data: 2024-04-12T02:59:00.433 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.433 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.433 DEBUG:boto:Params: {} 2024-04-12T02:59:00.433 DEBUG:boto:Token: None 2024-04-12T02:59:00.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj 2024-04-12T02:59:00.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCKXv4OhWur+fFEVbri2O9ZOV18= 2024-04-12T02:59:00.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCKXv4OhWur+fFEVbri2O9ZOV18=', 'Content-Length': '0'} 2024-04-12T02:59:00.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007beaa295e30788b5-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.478 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T02:59:00.479 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T02:59:00.479 DEBUG:boto:Method: GET 2024-04-12T02:59:00.479 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T02:59:00.479 DEBUG:boto:Data: 2024-04-12T02:59:00.479 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.479 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.479 DEBUG:boto:Params: {} 2024-04-12T02:59:00.479 DEBUG:boto:Token: None 2024-04-12T02:59:00.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/myobj 2024-04-12T02:59:00.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCKXv4OhWur+fFEVbri2O9ZOV18= 2024-04-12T02:59:00.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCKXv4OhWur+fFEVbri2O9ZOV18=', 'Content-Length': '0'} 2024-04-12T02:59:00.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004decfdf24c75d620-006618a374-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.482 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.482 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.482 DEBUG:boto:path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.482 DEBUG:boto:auth_path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.482 DEBUG:boto:Method: GET 2024-04-12T02:59:00.483 DEBUG:boto:Path: /dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.483 DEBUG:boto:Data: 2024-04-12T02:59:00.483 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.483 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.483 DEBUG:boto:Params: {} 2024-04-12T02:59:00.483 DEBUG:boto:Token: None 2024-04-12T02:59:00.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8= 2024-04-12T02:59:00.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8=', 'Content-Length': '0'} 2024-04-12T02:59:00.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b00c3e3b299897-006618a374-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.486 DEBUG:boto:b'dgbpxg-10myobj1000falsemyobj2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.486 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T02:59:00.486 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T02:59:00.486 DEBUG:boto:path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.486 DEBUG:boto:auth_path=/dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.487 DEBUG:boto:Method: GET 2024-04-12T02:59:00.487 DEBUG:boto:Path: /dgbpxg-10/?prefix=myobj 2024-04-12T02:59:00.487 DEBUG:boto:Data: 2024-04-12T02:59:00.487 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T02:59:00.487 DEBUG:boto:Port: 8001 2024-04-12T02:59:00.487 DEBUG:boto:Params: {} 2024-04-12T02:59:00.487 DEBUG:boto:Token: None 2024-04-12T02:59:00.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-10/ 2024-04-12T02:59:00.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8= 2024-04-12T02:59:00.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPYIsE/Bw7KB3xrMtwf01M5gyH8=', 'Content-Length': '0'} 2024-04-12T02:59:00.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047031b6f9df14c2a-006618a374-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.490 DEBUG:boto:b'dgbpxg-10myobj1000falsemyobj2024-04-12T02:57:39.790Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T02:59:00.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-10 zones={a1, a2} 2024-04-12T02:59:00.490 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T02:59:00.490 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T02:59:00.490 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.491 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T02:59:00.491 DEBUG:boto:Data: 2024-04-12T02:59:00.491 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.491 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.491 DEBUG:boto:Params: {} 2024-04-12T02:59:00.491 DEBUG:boto:Token: None 2024-04-12T02:59:00.491 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-9/ 2024-04-12T02:59:00.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XE0BoSDcFguBpF21l/Aer9xG1lY= 2024-04-12T02:59:00.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XE0BoSDcFguBpF21l/Aer9xG1lY=', 'Content-Length': '0'} 2024-04-12T02:59:00.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000751ee8d2ac15065f-006618a374-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.494 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:59:00.494 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:59:00.494 DEBUG:boto:Method: HEAD 2024-04-12T02:59:00.494 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:59:00.494 DEBUG:boto:Data: 2024-04-12T02:59:00.494 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.494 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.494 DEBUG:boto:Params: {} 2024-04-12T02:59:00.495 DEBUG:boto:Token: None 2024-04-12T02:59:00.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-9/myobj 2024-04-12T02:59:00.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N2cittuWuvOCXqM1u0KUuUxh/Xk= 2024-04-12T02:59:00.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N2cittuWuvOCXqM1u0KUuUxh/Xk=', 'Content-Length': '0'} 2024-04-12T02:59:00.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083bd99ad9a748753-006618a374-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.496 DEBUG:boto:path=/dgbpxg-9/myobj 2024-04-12T02:59:00.497 DEBUG:boto:auth_path=/dgbpxg-9/myobj 2024-04-12T02:59:00.497 DEBUG:boto:Method: DELETE 2024-04-12T02:59:00.497 DEBUG:boto:Path: /dgbpxg-9/myobj 2024-04-12T02:59:00.497 DEBUG:boto:Data: 2024-04-12T02:59:00.497 DEBUG:boto:Headers: {} 2024-04-12T02:59:00.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T02:59:00.497 DEBUG:boto:Port: 8000 2024-04-12T02:59:00.497 DEBUG:boto:Params: {} 2024-04-12T02:59:00.497 DEBUG:boto:Token: None 2024-04-12T02:59:00.497 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 02:59:00 GMT /dgbpxg-9/myobj 2024-04-12T02:59:00.498 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pomHYLZmItIjxfrQbcdL5fte1PM= 2024-04-12T02:59:00.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 02:59:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pomHYLZmItIjxfrQbcdL5fte1PM=', 'Content-Length': '0'} 2024-04-12T02:59:00.503 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fd397aad9c7b0e8-006618a374-110e-a1'), ('Date', 'Fri, 12 Apr 2024 02:59:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T02:59:00.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:59:00.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T02:59:00.504 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T02:59:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T02:59:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:00.602+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:05.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:05.603+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:05.605 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:05.605 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:10.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:10.604+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:10.606 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:10.606 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:15.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:15.605+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:15.607 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:15.607 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:20.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:20.606+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:20.609 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:20.609 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:25.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:25.608+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:25.610 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:25.610 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:30.610+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:30.611 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:30.612 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:35.611+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:35.613 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:35.613 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:40.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:40.613+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:40.614 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:40.614 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:45.614+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:45.615 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:45.615 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:50.615+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:50.617 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:50.617 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T02:59:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T02:59:55.617+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T02:59:55.618 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T02:59:55.618 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:00.617+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:00.619 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T03:00:00.620 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:05.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:05.618+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:05.621 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T03:00:05.621 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:10.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:10.620+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:10.622 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T03:00:10.622 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:15.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:15.622+0000 7fbca49bf780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:15.624 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2024-04-12T03:00:15.624 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:20.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:20.623+0000 7fbca49bf780 1 bucket sync caught up with source: 2024-04-12T03:00:20.625 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:20.625 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-12T03:00:20.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:20.623+0000 7fbca49bf780 0 bucket checkpoint complete 2024-04-12T03:00:20.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-9 zones={a2, a1} 2024-04-12T03:00:20.641 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.641 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.641 DEBUG:boto:Method: HEAD 2024-04-12T03:00:20.641 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T03:00:20.641 DEBUG:boto:Data: 2024-04-12T03:00:20.641 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.641 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.641 DEBUG:boto:Params: {} 2024-04-12T03:00:20.642 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:00:20.642 DEBUG:boto:Token: None 2024-04-12T03:00:20.642 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/ 2024-04-12T03:00:20.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aWiE8zEvXHZz7fI6yH+7WVXtiuE= 2024-04-12T03:00:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aWiE8zEvXHZz7fI6yH+7WVXtiuE=', 'Content-Length': '0'} 2024-04-12T03:00:20.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086a4a701ad106095-006618a3c4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.657 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.657 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.657 DEBUG:boto:Method: HEAD 2024-04-12T03:00:20.657 DEBUG:boto:Path: /dgbpxg-9/ 2024-04-12T03:00:20.657 DEBUG:boto:Data: 2024-04-12T03:00:20.658 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:20.658 DEBUG:boto:Port: 8000 2024-04-12T03:00:20.658 DEBUG:boto:Params: {} 2024-04-12T03:00:20.658 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:00:20.658 DEBUG:boto:Token: None 2024-04-12T03:00:20.658 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/ 2024-04-12T03:00:20.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aWiE8zEvXHZz7fI6yH+7WVXtiuE= 2024-04-12T03:00:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aWiE8zEvXHZz7fI6yH+7WVXtiuE=', 'Content-Length': '0'} 2024-04-12T03:00:20.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018b3493d624dff43-006618a3c4-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:00:20.672 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.672 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.672 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.672 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.672 DEBUG:boto:Method: GET 2024-04-12T03:00:20.672 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.672 DEBUG:boto:Data: 2024-04-12T03:00:20.672 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.673 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.673 DEBUG:boto:Params: {} 2024-04-12T03:00:20.673 DEBUG:boto:Token: None 2024-04-12T03:00:20.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/?versions 2024-04-12T03:00:20.673 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig= 2024-04-12T03:00:20.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig=', 'Content-Length': '0'} 2024-04-12T03:00:20.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e26374e2580314-006618a3c4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.676 DEBUG:boto:b'dgbpxg-9999false' 2024-04-12T03:00:20.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:00:20.676 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.677 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.677 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.677 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.677 DEBUG:boto:Method: GET 2024-04-12T03:00:20.677 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.677 DEBUG:boto:Data: 2024-04-12T03:00:20.677 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:20.677 DEBUG:boto:Port: 8000 2024-04-12T03:00:20.677 DEBUG:boto:Params: {} 2024-04-12T03:00:20.678 DEBUG:boto:Token: None 2024-04-12T03:00:20.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/?versions 2024-04-12T03:00:20.678 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig= 2024-04-12T03:00:20.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig=', 'Content-Length': '0'} 2024-04-12T03:00:20.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084e5b98f785e65d7-006618a3c4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.680 DEBUG:boto:b'dgbpxg-9999false' 2024-04-12T03:00:20.680 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.680 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.681 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.681 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.681 DEBUG:boto:Method: GET 2024-04-12T03:00:20.681 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.681 DEBUG:boto:Data: 2024-04-12T03:00:20.681 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.681 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.681 DEBUG:boto:Params: {} 2024-04-12T03:00:20.681 DEBUG:boto:Token: None 2024-04-12T03:00:20.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/?versions 2024-04-12T03:00:20.682 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig= 2024-04-12T03:00:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig=', 'Content-Length': '0'} 2024-04-12T03:00:20.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09958dfc4d1b735-006618a3c4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.684 DEBUG:boto:b'dgbpxg-9999false' 2024-04-12T03:00:20.684 DEBUG:boto:path=/dgbpxg-9/ 2024-04-12T03:00:20.684 DEBUG:boto:auth_path=/dgbpxg-9/ 2024-04-12T03:00:20.684 DEBUG:boto:path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.684 DEBUG:boto:auth_path=/dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.685 DEBUG:boto:Method: GET 2024-04-12T03:00:20.685 DEBUG:boto:Path: /dgbpxg-9/?versions&max-keys=999 2024-04-12T03:00:20.685 DEBUG:boto:Data: 2024-04-12T03:00:20.685 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:20.685 DEBUG:boto:Port: 8000 2024-04-12T03:00:20.685 DEBUG:boto:Params: {} 2024-04-12T03:00:20.685 DEBUG:boto:Token: None 2024-04-12T03:00:20.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-9/?versions 2024-04-12T03:00:20.685 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig= 2024-04-12T03:00:20.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sMwydLMWQNYZ+ypQ//Yii1TvCig=', 'Content-Length': '0'} 2024-04-12T03:00:20.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f2039f63d15d78-006618a3c4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.688 DEBUG:boto:b'dgbpxg-9999false' 2024-04-12T03:00:20.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-9 zones={a2, a1} 2024-04-12T03:00:20.689 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:20.689 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:20.689 DEBUG:boto:Method: HEAD 2024-04-12T03:00:20.689 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T03:00:20.689 DEBUG:boto:Data: 2024-04-12T03:00:20.689 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.689 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.689 DEBUG:boto:Params: {} 2024-04-12T03:00:20.689 DEBUG:boto:Token: None 2024-04-12T03:00:20.689 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-10/ 2024-04-12T03:00:20.690 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wmr1/jmfqUI602UcCizPatX4Td4= 2024-04-12T03:00:20.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wmr1/jmfqUI602UcCizPatX4Td4=', 'Content-Length': '0'} 2024-04-12T03:00:20.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032ad498252b5be80-006618a3c4-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.692 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T03:00:20.692 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T03:00:20.692 DEBUG:boto:Method: HEAD 2024-04-12T03:00:20.692 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T03:00:20.692 DEBUG:boto:Data: 2024-04-12T03:00:20.692 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.693 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.693 DEBUG:boto:Params: {} 2024-04-12T03:00:20.693 DEBUG:boto:Token: None 2024-04-12T03:00:20.693 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-10/myobj 2024-04-12T03:00:20.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2pU+xyz7tBWl4hzROEhxbDvUAw4= 2024-04-12T03:00:20.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2pU+xyz7tBWl4hzROEhxbDvUAw4=', 'Content-Length': '0'} 2024-04-12T03:00:20.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 02:57:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d038f88757b18d6-006618a3c4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.695 DEBUG:boto:path=/dgbpxg-10/myobj 2024-04-12T03:00:20.695 DEBUG:boto:auth_path=/dgbpxg-10/myobj 2024-04-12T03:00:20.695 DEBUG:boto:Method: DELETE 2024-04-12T03:00:20.695 DEBUG:boto:Path: /dgbpxg-10/myobj 2024-04-12T03:00:20.695 DEBUG:boto:Data: 2024-04-12T03:00:20.695 DEBUG:boto:Headers: {} 2024-04-12T03:00:20.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:20.696 DEBUG:boto:Port: 8001 2024-04-12T03:00:20.696 DEBUG:boto:Params: {} 2024-04-12T03:00:20.696 DEBUG:boto:Token: None 2024-04-12T03:00:20.696 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:00:20 GMT /dgbpxg-10/myobj 2024-04-12T03:00:20.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SjtFaOrZDUbyEMJxw0smGFKoLGM= 2024-04-12T03:00:20.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SjtFaOrZDUbyEMJxw0smGFKoLGM=', 'Content-Length': '0'} 2024-04-12T03:00:20.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000718a5da41febba1c-006618a3c4-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:20.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:00:20.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:00:20.703 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:00:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:00:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:00:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:20.797+0000 7fb148bb9780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:20.798 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T03:00:20.798 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:25.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:25.799+0000 7fb148bb9780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:25.800 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T03:00:25.800 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:30.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:30.800+0000 7fb148bb9780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:30.801 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T03:00:30.801 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:35.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:35.801+0000 7fb148bb9780 1 waiting for incremental sync to catch up: 2024-04-12T03:00:35.802 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-04-12T03:00:35.803 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:40.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:40.802+0000 7fb148bb9780 1 bucket sync caught up with source: 2024-04-12T03:00:40.804 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:40.804 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-12T03:00:40.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:40.802+0000 7fb148bb9780 0 bucket checkpoint complete 2024-04-12T03:00:40.819 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-10 zones={a1, a2} 2024-04-12T03:00:40.819 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.820 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.820 DEBUG:boto:Method: HEAD 2024-04-12T03:00:40.820 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T03:00:40.820 DEBUG:boto:Data: 2024-04-12T03:00:40.820 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.820 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.820 DEBUG:boto:Params: {} 2024-04-12T03:00:40.820 DEBUG:boto:Token: None 2024-04-12T03:00:40.821 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/ 2024-04-12T03:00:40.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jydCxmMlG1/D/RtrGdNqI1WtHXU= 2024-04-12T03:00:40.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jydCxmMlG1/D/RtrGdNqI1WtHXU=', 'Content-Length': '0'} 2024-04-12T03:00:40.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006051eec1c4ddb379-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.823 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.823 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.824 DEBUG:boto:Method: HEAD 2024-04-12T03:00:40.824 DEBUG:boto:Path: /dgbpxg-10/ 2024-04-12T03:00:40.824 DEBUG:boto:Data: 2024-04-12T03:00:40.824 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:40.824 DEBUG:boto:Port: 8001 2024-04-12T03:00:40.824 DEBUG:boto:Params: {} 2024-04-12T03:00:40.824 DEBUG:boto:Token: None 2024-04-12T03:00:40.824 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/ 2024-04-12T03:00:40.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jydCxmMlG1/D/RtrGdNqI1WtHXU= 2024-04-12T03:00:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jydCxmMlG1/D/RtrGdNqI1WtHXU=', 'Content-Length': '0'} 2024-04-12T03:00:40.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001514b91432b354b3-006618a3d8-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.827 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:00:40.827 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.827 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.827 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.827 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.827 DEBUG:boto:Method: GET 2024-04-12T03:00:40.827 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.827 DEBUG:boto:Data: 2024-04-12T03:00:40.827 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.828 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.828 DEBUG:boto:Params: {} 2024-04-12T03:00:40.828 DEBUG:boto:Token: None 2024-04-12T03:00:40.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/?versions 2024-04-12T03:00:40.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY= 2024-04-12T03:00:40.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY=', 'Content-Length': '0'} 2024-04-12T03:00:40.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a789d4fdf5dc5ed-006618a3d8-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.831 DEBUG:boto:b'dgbpxg-10999false' 2024-04-12T03:00:40.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:00:40.831 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.831 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.831 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.831 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.832 DEBUG:boto:Method: GET 2024-04-12T03:00:40.832 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.832 DEBUG:boto:Data: 2024-04-12T03:00:40.832 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:40.832 DEBUG:boto:Port: 8001 2024-04-12T03:00:40.832 DEBUG:boto:Params: {} 2024-04-12T03:00:40.832 DEBUG:boto:Token: None 2024-04-12T03:00:40.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/?versions 2024-04-12T03:00:40.833 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY= 2024-04-12T03:00:40.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY=', 'Content-Length': '0'} 2024-04-12T03:00:40.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c0e93f6b1f95e7-006618a3d8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.878 DEBUG:boto:b'dgbpxg-10999false' 2024-04-12T03:00:40.879 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.879 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.879 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.879 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.879 DEBUG:boto:Method: GET 2024-04-12T03:00:40.879 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.880 DEBUG:boto:Data: 2024-04-12T03:00:40.880 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.880 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.880 DEBUG:boto:Params: {} 2024-04-12T03:00:40.880 DEBUG:boto:Token: None 2024-04-12T03:00:40.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/?versions 2024-04-12T03:00:40.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY= 2024-04-12T03:00:40.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY=', 'Content-Length': '0'} 2024-04-12T03:00:40.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023504fd4ce117871-006618a3d8-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.883 DEBUG:boto:b'dgbpxg-10999false' 2024-04-12T03:00:40.883 DEBUG:boto:path=/dgbpxg-10/ 2024-04-12T03:00:40.883 DEBUG:boto:auth_path=/dgbpxg-10/ 2024-04-12T03:00:40.883 DEBUG:boto:path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.883 DEBUG:boto:auth_path=/dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.884 DEBUG:boto:Method: GET 2024-04-12T03:00:40.884 DEBUG:boto:Path: /dgbpxg-10/?versions&max-keys=999 2024-04-12T03:00:40.884 DEBUG:boto:Data: 2024-04-12T03:00:40.884 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:40.884 DEBUG:boto:Port: 8001 2024-04-12T03:00:40.884 DEBUG:boto:Params: {} 2024-04-12T03:00:40.884 DEBUG:boto:Token: None 2024-04-12T03:00:40.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-10/?versions 2024-04-12T03:00:40.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY= 2024-04-12T03:00:40.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sia/B8+BRETCN+Ruc6jOebWPeyY=', 'Content-Length': '0'} 2024-04-12T03:00:40.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670dbc88d1740ae7-006618a3d8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.887 DEBUG:boto:b'dgbpxg-10999false' 2024-04-12T03:00:40.887 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-10 zones={a1, a2} 2024-04-12T03:00:40.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-12T03:00:40.888 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-11 2024-04-12T03:00:40.888 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:40.889 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:40.889 DEBUG:boto:Method: PUT 2024-04-12T03:00:40.889 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:40.889 DEBUG:boto:Data: 2024-04-12T03:00:40.889 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.889 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.889 DEBUG:boto:Params: {} 2024-04-12T03:00:40.889 DEBUG:boto:Token: None 2024-04-12T03:00:40.889 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-11/ 2024-04-12T03:00:40.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1/9SPqIvpsf+z4unCX8zfdbOYBY= 2024-04-12T03:00:40.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1/9SPqIvpsf+z4unCX8zfdbOYBY=', 'Content-Length': '0'} 2024-04-12T03:00:40.907 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009da5fb3101871c69-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.907 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-12 2024-04-12T03:00:40.907 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:40.907 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:40.908 DEBUG:boto:Method: PUT 2024-04-12T03:00:40.908 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:40.908 DEBUG:boto:Data: 2024-04-12T03:00:40.908 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:40.908 DEBUG:boto:Port: 8001 2024-04-12T03:00:40.908 DEBUG:boto:Params: {} 2024-04-12T03:00:40.908 DEBUG:boto:Token: None 2024-04-12T03:00:40.908 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-12/ 2024-04-12T03:00:40.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0SYL+G8xwa5F3Su8ue2zfoh7BVg= 2024-04-12T03:00:40.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0SYL+G8xwa5F3Su8ue2zfoh7BVg=', 'Content-Length': '0'} 2024-04-12T03:00:40.945 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fcc50dc16703d3d8-006618a3d8-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.946 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:40.946 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:40.946 DEBUG:boto:Method: HEAD 2024-04-12T03:00:40.946 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:40.946 DEBUG:boto:Data: 2024-04-12T03:00:40.946 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.946 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.946 DEBUG:boto:Params: {} 2024-04-12T03:00:40.947 DEBUG:boto:Token: None 2024-04-12T03:00:40.947 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-11/ 2024-04-12T03:00:40.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI= 2024-04-12T03:00:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI=', 'Content-Length': '0'} 2024-04-12T03:00:40.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd32b42ce2de8f63-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.950 DEBUG:boto:path=/dgbpxg-11/obj1 2024-04-12T03:00:40.950 DEBUG:boto:auth_path=/dgbpxg-11/obj1 2024-04-12T03:00:40.950 DEBUG:boto:Method: PUT 2024-04-12T03:00:40.950 DEBUG:boto:Path: /dgbpxg-11/obj1 2024-04-12T03:00:40.950 DEBUG:boto:Data: 2024-04-12T03:00:40.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:40.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.951 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.951 DEBUG:boto:Params: {} 2024-04-12T03:00:40.951 DEBUG:boto:Token: None 2024-04-12T03:00:40.951 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-11/obj1 2024-04-12T03:00:40.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NYdVQn09C1oguohq4tigtpQhd+s= 2024-04-12T03:00:40.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NYdVQn09C1oguohq4tigtpQhd+s='} 2024-04-12T03:00:40.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dea6e3bbc2d0217-006618a3d8-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.995 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:40.995 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:40.995 DEBUG:boto:Method: HEAD 2024-04-12T03:00:40.996 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:40.996 DEBUG:boto:Data: 2024-04-12T03:00:40.996 DEBUG:boto:Headers: {} 2024-04-12T03:00:40.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:40.996 DEBUG:boto:Port: 8000 2024-04-12T03:00:40.996 DEBUG:boto:Params: {} 2024-04-12T03:00:40.996 DEBUG:boto:Token: None 2024-04-12T03:00:40.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:40 GMT /dgbpxg-11/ 2024-04-12T03:00:40.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI= 2024-04-12T03:00:40.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI=', 'Content-Length': '0'} 2024-04-12T03:00:40.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004b257621e44aecb-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:40.999 DEBUG:boto:path=/dgbpxg-11/obj2 2024-04-12T03:00:40.999 DEBUG:boto:auth_path=/dgbpxg-11/obj2 2024-04-12T03:00:40.999 DEBUG:boto:Method: PUT 2024-04-12T03:00:40.999 DEBUG:boto:Path: /dgbpxg-11/obj2 2024-04-12T03:00:41.000 DEBUG:boto:Data: 2024-04-12T03:00:41.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.000 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.000 DEBUG:boto:Params: {} 2024-04-12T03:00:41.000 DEBUG:boto:Token: None 2024-04-12T03:00:41.000 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj2 2024-04-12T03:00:41.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JjE/TLJYFzDPH1JZnZCu8jULgo4= 2024-04-12T03:00:41.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JjE/TLJYFzDPH1JZnZCu8jULgo4='} 2024-04-12T03:00:41.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a69c5cc22dd95a-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.006 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.006 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.006 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.007 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.007 DEBUG:boto:Data: 2024-04-12T03:00:41.007 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.007 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.007 DEBUG:boto:Params: {} 2024-04-12T03:00:41.007 DEBUG:boto:Token: None 2024-04-12T03:00:41.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036c239f237cc9310-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.010 DEBUG:boto:path=/dgbpxg-11/obj3 2024-04-12T03:00:41.010 DEBUG:boto:auth_path=/dgbpxg-11/obj3 2024-04-12T03:00:41.010 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.010 DEBUG:boto:Path: /dgbpxg-11/obj3 2024-04-12T03:00:41.010 DEBUG:boto:Data: 2024-04-12T03:00:41.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.011 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.011 DEBUG:boto:Params: {} 2024-04-12T03:00:41.011 DEBUG:boto:Token: None 2024-04-12T03:00:41.011 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj3 2024-04-12T03:00:41.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOZmRCYMfdj/xFj82+2lUIcFQEU= 2024-04-12T03:00:41.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOZmRCYMfdj/xFj82+2lUIcFQEU='} 2024-04-12T03:00:41.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001570df5ef4eec7c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.055 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.055 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.055 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.055 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.055 DEBUG:boto:Data: 2024-04-12T03:00:41.055 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.055 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.055 DEBUG:boto:Params: {} 2024-04-12T03:00:41.056 DEBUG:boto:Token: None 2024-04-12T03:00:41.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000def3974718d891e-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.058 DEBUG:boto:path=/dgbpxg-11/obj4 2024-04-12T03:00:41.058 DEBUG:boto:auth_path=/dgbpxg-11/obj4 2024-04-12T03:00:41.058 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.059 DEBUG:boto:Path: /dgbpxg-11/obj4 2024-04-12T03:00:41.059 DEBUG:boto:Data: 2024-04-12T03:00:41.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.059 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.059 DEBUG:boto:Params: {} 2024-04-12T03:00:41.059 DEBUG:boto:Token: None 2024-04-12T03:00:41.059 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj4 2024-04-12T03:00:41.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XqtAil5lS2ODXPoXYjtbhYpNr6A= 2024-04-12T03:00:41.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XqtAil5lS2ODXPoXYjtbhYpNr6A='} 2024-04-12T03:00:41.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000938d5a1a7bf9fe0d-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.067 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.067 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.067 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.067 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.067 DEBUG:boto:Data: 2024-04-12T03:00:41.068 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.068 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.068 DEBUG:boto:Params: {} 2024-04-12T03:00:41.068 DEBUG:boto:Token: None 2024-04-12T03:00:41.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ead05634fc59cd50-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.071 DEBUG:boto:path=/dgbpxg-11/obj5 2024-04-12T03:00:41.071 DEBUG:boto:auth_path=/dgbpxg-11/obj5 2024-04-12T03:00:41.071 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.071 DEBUG:boto:Path: /dgbpxg-11/obj5 2024-04-12T03:00:41.071 DEBUG:boto:Data: 2024-04-12T03:00:41.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.072 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.072 DEBUG:boto:Params: {} 2024-04-12T03:00:41.072 DEBUG:boto:Token: None 2024-04-12T03:00:41.072 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj5 2024-04-12T03:00:41.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGUEzqaO7Y6OAhGKD9Nh5bnAJtE= 2024-04-12T03:00:41.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGUEzqaO7Y6OAhGKD9Nh5bnAJtE='} 2024-04-12T03:00:41.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f35e53c31e24cb22-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.119 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.119 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.119 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.119 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.121 DEBUG:boto:Data: 2024-04-12T03:00:41.121 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.122 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.122 DEBUG:boto:Params: {} 2024-04-12T03:00:41.122 DEBUG:boto:Token: None 2024-04-12T03:00:41.122 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000913e1b2143d85cb5-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.125 DEBUG:boto:path=/dgbpxg-11/obj6 2024-04-12T03:00:41.126 DEBUG:boto:auth_path=/dgbpxg-11/obj6 2024-04-12T03:00:41.126 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.126 DEBUG:boto:Path: /dgbpxg-11/obj6 2024-04-12T03:00:41.126 DEBUG:boto:Data: 2024-04-12T03:00:41.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.126 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.126 DEBUG:boto:Params: {} 2024-04-12T03:00:41.126 DEBUG:boto:Token: None 2024-04-12T03:00:41.127 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj6 2024-04-12T03:00:41.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xGlZg+UFoSQT366Kb+wPCvDbc0= 2024-04-12T03:00:41.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xGlZg+UFoSQT366Kb+wPCvDbc0='} 2024-04-12T03:00:41.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d66e0e80cd3fa0ba-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.131 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.131 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.132 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.132 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.132 DEBUG:boto:Data: 2024-04-12T03:00:41.132 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.132 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.132 DEBUG:boto:Params: {} 2024-04-12T03:00:41.132 DEBUG:boto:Token: None 2024-04-12T03:00:41.132 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d657b000d46ccfa3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.135 DEBUG:boto:path=/dgbpxg-11/obj7 2024-04-12T03:00:41.135 DEBUG:boto:auth_path=/dgbpxg-11/obj7 2024-04-12T03:00:41.136 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.136 DEBUG:boto:Path: /dgbpxg-11/obj7 2024-04-12T03:00:41.136 DEBUG:boto:Data: 2024-04-12T03:00:41.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.136 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.136 DEBUG:boto:Params: {} 2024-04-12T03:00:41.136 DEBUG:boto:Token: None 2024-04-12T03:00:41.137 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj7 2024-04-12T03:00:41.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nTnpylJgB0c1y43FVbxx7ByfNOk= 2024-04-12T03:00:41.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nTnpylJgB0c1y43FVbxx7ByfNOk='} 2024-04-12T03:00:41.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055a72bca4dbca67d-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.183 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.183 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.183 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.183 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.183 DEBUG:boto:Data: 2024-04-12T03:00:41.183 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.183 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.183 DEBUG:boto:Params: {} 2024-04-12T03:00:41.184 DEBUG:boto:Token: None 2024-04-12T03:00:41.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef5f0eb856067280-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.186 DEBUG:boto:path=/dgbpxg-11/obj8 2024-04-12T03:00:41.187 DEBUG:boto:auth_path=/dgbpxg-11/obj8 2024-04-12T03:00:41.187 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.187 DEBUG:boto:Path: /dgbpxg-11/obj8 2024-04-12T03:00:41.187 DEBUG:boto:Data: 2024-04-12T03:00:41.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.187 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.187 DEBUG:boto:Params: {} 2024-04-12T03:00:41.187 DEBUG:boto:Token: None 2024-04-12T03:00:41.188 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj8 2024-04-12T03:00:41.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:usS/k8mmRnphNOmaI5U28aQcyG8= 2024-04-12T03:00:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:usS/k8mmRnphNOmaI5U28aQcyG8='} 2024-04-12T03:00:41.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019debf5e76510900-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.192 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.192 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.192 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.192 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.192 DEBUG:boto:Data: 2024-04-12T03:00:41.192 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.192 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.193 DEBUG:boto:Params: {} 2024-04-12T03:00:41.193 DEBUG:boto:Token: None 2024-04-12T03:00:41.193 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006923fb3f4df9914c-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.195 DEBUG:boto:path=/dgbpxg-11/obj9 2024-04-12T03:00:41.195 DEBUG:boto:auth_path=/dgbpxg-11/obj9 2024-04-12T03:00:41.195 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.195 DEBUG:boto:Path: /dgbpxg-11/obj9 2024-04-12T03:00:41.195 DEBUG:boto:Data: 2024-04-12T03:00:41.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.196 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.196 DEBUG:boto:Params: {} 2024-04-12T03:00:41.196 DEBUG:boto:Token: None 2024-04-12T03:00:41.196 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj9 2024-04-12T03:00:41.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0bzCJa9Okbr4fkXsmL04HhcVPCM= 2024-04-12T03:00:41.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bzCJa9Okbr4fkXsmL04HhcVPCM='} 2024-04-12T03:00:41.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b3836d1824d5a37-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.239 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.239 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.239 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.239 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.239 DEBUG:boto:Data: 2024-04-12T03:00:41.240 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.240 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.240 DEBUG:boto:Params: {} 2024-04-12T03:00:41.240 DEBUG:boto:Token: None 2024-04-12T03:00:41.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bba538ccc9d61fec-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.243 DEBUG:boto:path=/dgbpxg-11/obj10 2024-04-12T03:00:41.243 DEBUG:boto:auth_path=/dgbpxg-11/obj10 2024-04-12T03:00:41.243 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.243 DEBUG:boto:Path: /dgbpxg-11/obj10 2024-04-12T03:00:41.244 DEBUG:boto:Data: 2024-04-12T03:00:41.244 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.244 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.244 DEBUG:boto:Params: {} 2024-04-12T03:00:41.244 DEBUG:boto:Token: None 2024-04-12T03:00:41.244 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj10 2024-04-12T03:00:41.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/hDnq7jOldxYgkqMmhYqyTJubT8= 2024-04-12T03:00:41.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/hDnq7jOldxYgkqMmhYqyTJubT8='} 2024-04-12T03:00:41.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a70ad4e4d42a78e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.248 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.248 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.248 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.248 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.249 DEBUG:boto:Data: 2024-04-12T03:00:41.249 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.249 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.249 DEBUG:boto:Params: {} 2024-04-12T03:00:41.249 DEBUG:boto:Token: None 2024-04-12T03:00:41.249 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003123ce0d86cb2e1a-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.251 DEBUG:boto:path=/dgbpxg-11/obj11 2024-04-12T03:00:41.251 DEBUG:boto:auth_path=/dgbpxg-11/obj11 2024-04-12T03:00:41.251 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.252 DEBUG:boto:Path: /dgbpxg-11/obj11 2024-04-12T03:00:41.252 DEBUG:boto:Data: 2024-04-12T03:00:41.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.252 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.252 DEBUG:boto:Params: {} 2024-04-12T03:00:41.252 DEBUG:boto:Token: None 2024-04-12T03:00:41.252 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj11 2024-04-12T03:00:41.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ej16i8dLYw9c2YA9KuNWS03p/1E= 2024-04-12T03:00:41.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ej16i8dLYw9c2YA9KuNWS03p/1E='} 2024-04-12T03:00:41.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a7997b5ffc9b211-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.295 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.295 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.295 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.295 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.295 DEBUG:boto:Data: 2024-04-12T03:00:41.295 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.295 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.295 DEBUG:boto:Params: {} 2024-04-12T03:00:41.296 DEBUG:boto:Token: None 2024-04-12T03:00:41.296 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f61e6b95b5dca9f3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.298 DEBUG:boto:path=/dgbpxg-11/obj12 2024-04-12T03:00:41.299 DEBUG:boto:auth_path=/dgbpxg-11/obj12 2024-04-12T03:00:41.299 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.299 DEBUG:boto:Path: /dgbpxg-11/obj12 2024-04-12T03:00:41.299 DEBUG:boto:Data: 2024-04-12T03:00:41.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.299 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.299 DEBUG:boto:Params: {} 2024-04-12T03:00:41.299 DEBUG:boto:Token: None 2024-04-12T03:00:41.300 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj12 2024-04-12T03:00:41.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KOyfy4FdmGL72mAaDfjYTJ9GLjI= 2024-04-12T03:00:41.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOyfy4FdmGL72mAaDfjYTJ9GLjI='} 2024-04-12T03:00:41.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027cc2fd1fd7ef4d2-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.303 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.304 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.304 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.304 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.304 DEBUG:boto:Data: 2024-04-12T03:00:41.304 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.304 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.304 DEBUG:boto:Params: {} 2024-04-12T03:00:41.305 DEBUG:boto:Token: None 2024-04-12T03:00:41.305 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8382277f9e028d6-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.307 DEBUG:boto:path=/dgbpxg-11/obj13 2024-04-12T03:00:41.307 DEBUG:boto:auth_path=/dgbpxg-11/obj13 2024-04-12T03:00:41.307 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.307 DEBUG:boto:Path: /dgbpxg-11/obj13 2024-04-12T03:00:41.307 DEBUG:boto:Data: 2024-04-12T03:00:41.307 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.307 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.307 DEBUG:boto:Params: {} 2024-04-12T03:00:41.308 DEBUG:boto:Token: None 2024-04-12T03:00:41.308 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj13 2024-04-12T03:00:41.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OiqETF7WW1wuPW1vvFPmNZ1avKo= 2024-04-12T03:00:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OiqETF7WW1wuPW1vvFPmNZ1avKo='} 2024-04-12T03:00:41.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec8b7a32bfdceedd-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.351 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.351 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.351 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.351 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.351 DEBUG:boto:Data: 2024-04-12T03:00:41.351 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.351 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.351 DEBUG:boto:Params: {} 2024-04-12T03:00:41.352 DEBUG:boto:Token: None 2024-04-12T03:00:41.352 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb9bc0f3eec57028-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.354 DEBUG:boto:path=/dgbpxg-11/obj14 2024-04-12T03:00:41.354 DEBUG:boto:auth_path=/dgbpxg-11/obj14 2024-04-12T03:00:41.354 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.355 DEBUG:boto:Path: /dgbpxg-11/obj14 2024-04-12T03:00:41.355 DEBUG:boto:Data: 2024-04-12T03:00:41.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.355 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.355 DEBUG:boto:Params: {} 2024-04-12T03:00:41.355 DEBUG:boto:Token: None 2024-04-12T03:00:41.355 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj14 2024-04-12T03:00:41.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A2+AlL1/od+uEdxiiRumHVMhvMU= 2024-04-12T03:00:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A2+AlL1/od+uEdxiiRumHVMhvMU='} 2024-04-12T03:00:41.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005750028f2e74e19-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.359 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.359 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.359 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.360 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.360 DEBUG:boto:Data: 2024-04-12T03:00:41.360 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.360 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.360 DEBUG:boto:Params: {} 2024-04-12T03:00:41.360 DEBUG:boto:Token: None 2024-04-12T03:00:41.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4b331f3ce5ec103-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.362 DEBUG:boto:path=/dgbpxg-11/obj15 2024-04-12T03:00:41.363 DEBUG:boto:auth_path=/dgbpxg-11/obj15 2024-04-12T03:00:41.363 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.363 DEBUG:boto:Path: /dgbpxg-11/obj15 2024-04-12T03:00:41.363 DEBUG:boto:Data: 2024-04-12T03:00:41.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.363 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.363 DEBUG:boto:Params: {} 2024-04-12T03:00:41.363 DEBUG:boto:Token: None 2024-04-12T03:00:41.363 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj15 2024-04-12T03:00:41.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CBSgIVnrRnH/3Dw22uwzCqFcZc8= 2024-04-12T03:00:41.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CBSgIVnrRnH/3Dw22uwzCqFcZc8='} 2024-04-12T03:00:41.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f3dff57c1984652-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.407 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.407 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.407 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.407 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.407 DEBUG:boto:Data: 2024-04-12T03:00:41.407 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.408 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.408 DEBUG:boto:Params: {} 2024-04-12T03:00:41.408 DEBUG:boto:Token: None 2024-04-12T03:00:41.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b2538dd84dc7814-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.411 DEBUG:boto:path=/dgbpxg-11/obj16 2024-04-12T03:00:41.411 DEBUG:boto:auth_path=/dgbpxg-11/obj16 2024-04-12T03:00:41.411 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.411 DEBUG:boto:Path: /dgbpxg-11/obj16 2024-04-12T03:00:41.411 DEBUG:boto:Data: 2024-04-12T03:00:41.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.411 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.411 DEBUG:boto:Params: {} 2024-04-12T03:00:41.412 DEBUG:boto:Token: None 2024-04-12T03:00:41.412 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj16 2024-04-12T03:00:41.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YiDXr4PDwCK1/aUGMhFZfrS/FSE= 2024-04-12T03:00:41.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YiDXr4PDwCK1/aUGMhFZfrS/FSE='} 2024-04-12T03:00:41.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c39c4d5306c32c20-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.417 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.417 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.417 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.417 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.418 DEBUG:boto:Data: 2024-04-12T03:00:41.418 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.418 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.418 DEBUG:boto:Params: {} 2024-04-12T03:00:41.418 DEBUG:boto:Token: None 2024-04-12T03:00:41.418 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a32de531a7a61c1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.421 DEBUG:boto:path=/dgbpxg-11/obj17 2024-04-12T03:00:41.421 DEBUG:boto:auth_path=/dgbpxg-11/obj17 2024-04-12T03:00:41.421 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.421 DEBUG:boto:Path: /dgbpxg-11/obj17 2024-04-12T03:00:41.421 DEBUG:boto:Data: 2024-04-12T03:00:41.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.422 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.422 DEBUG:boto:Params: {} 2024-04-12T03:00:41.422 DEBUG:boto:Token: None 2024-04-12T03:00:41.422 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj17 2024-04-12T03:00:41.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gxN2Q3xHacYPGMHoV7kiVbF+n1E= 2024-04-12T03:00:41.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gxN2Q3xHacYPGMHoV7kiVbF+n1E='} 2024-04-12T03:00:41.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45bd7bd61f5bf97-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.467 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.467 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.467 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.467 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.467 DEBUG:boto:Data: 2024-04-12T03:00:41.467 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.468 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.468 DEBUG:boto:Params: {} 2024-04-12T03:00:41.468 DEBUG:boto:Token: None 2024-04-12T03:00:41.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf208e3208301566-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.470 DEBUG:boto:path=/dgbpxg-11/obj18 2024-04-12T03:00:41.471 DEBUG:boto:auth_path=/dgbpxg-11/obj18 2024-04-12T03:00:41.471 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.471 DEBUG:boto:Path: /dgbpxg-11/obj18 2024-04-12T03:00:41.471 DEBUG:boto:Data: 2024-04-12T03:00:41.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.471 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.471 DEBUG:boto:Params: {} 2024-04-12T03:00:41.472 DEBUG:boto:Token: None 2024-04-12T03:00:41.472 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj18 2024-04-12T03:00:41.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+XuogUMCCOLEhoXIIECVZ8usW8= 2024-04-12T03:00:41.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+XuogUMCCOLEhoXIIECVZ8usW8='} 2024-04-12T03:00:41.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008595a842326005e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.476 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.476 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.476 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.476 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.476 DEBUG:boto:Data: 2024-04-12T03:00:41.476 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.476 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.476 DEBUG:boto:Params: {} 2024-04-12T03:00:41.476 DEBUG:boto:Token: None 2024-04-12T03:00:41.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbfa7f35c584d9bd-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.479 DEBUG:boto:path=/dgbpxg-11/obj19 2024-04-12T03:00:41.479 DEBUG:boto:auth_path=/dgbpxg-11/obj19 2024-04-12T03:00:41.479 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.479 DEBUG:boto:Path: /dgbpxg-11/obj19 2024-04-12T03:00:41.479 DEBUG:boto:Data: 2024-04-12T03:00:41.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.480 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.480 DEBUG:boto:Params: {} 2024-04-12T03:00:41.480 DEBUG:boto:Token: None 2024-04-12T03:00:41.480 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj19 2024-04-12T03:00:41.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIupO6AyMWF3huzEAFrKZkj3FtU= 2024-04-12T03:00:41.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIupO6AyMWF3huzEAFrKZkj3FtU='} 2024-04-12T03:00:41.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000705298fbbc2aa6a7-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.523 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.523 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.523 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.523 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.523 DEBUG:boto:Data: 2024-04-12T03:00:41.523 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.524 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.524 DEBUG:boto:Params: {} 2024-04-12T03:00:41.524 DEBUG:boto:Token: None 2024-04-12T03:00:41.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd6307206b488058-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.527 DEBUG:boto:path=/dgbpxg-11/obj20 2024-04-12T03:00:41.527 DEBUG:boto:auth_path=/dgbpxg-11/obj20 2024-04-12T03:00:41.527 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.527 DEBUG:boto:Path: /dgbpxg-11/obj20 2024-04-12T03:00:41.527 DEBUG:boto:Data: 2024-04-12T03:00:41.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.528 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.528 DEBUG:boto:Params: {} 2024-04-12T03:00:41.528 DEBUG:boto:Token: None 2024-04-12T03:00:41.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj20 2024-04-12T03:00:41.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i86t7ac4We05US/aVkcjo83N63E= 2024-04-12T03:00:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i86t7ac4We05US/aVkcjo83N63E='} 2024-04-12T03:00:41.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d39c327618fef22-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.533 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.533 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.533 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.533 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.533 DEBUG:boto:Data: 2024-04-12T03:00:41.533 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.533 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.533 DEBUG:boto:Params: {} 2024-04-12T03:00:41.534 DEBUG:boto:Token: None 2024-04-12T03:00:41.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002acd464fea51f253-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.537 DEBUG:boto:path=/dgbpxg-11/obj21 2024-04-12T03:00:41.537 DEBUG:boto:auth_path=/dgbpxg-11/obj21 2024-04-12T03:00:41.537 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.537 DEBUG:boto:Path: /dgbpxg-11/obj21 2024-04-12T03:00:41.537 DEBUG:boto:Data: 2024-04-12T03:00:41.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.537 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.537 DEBUG:boto:Params: {} 2024-04-12T03:00:41.537 DEBUG:boto:Token: None 2024-04-12T03:00:41.538 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj21 2024-04-12T03:00:41.538 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T8H1dzShULjhX4e1WGWws1Ycw1k= 2024-04-12T03:00:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T8H1dzShULjhX4e1WGWws1Ycw1k='} 2024-04-12T03:00:41.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000205d23ae842f406f-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.583 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.583 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.583 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.583 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.583 DEBUG:boto:Data: 2024-04-12T03:00:41.583 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.583 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.583 DEBUG:boto:Params: {} 2024-04-12T03:00:41.584 DEBUG:boto:Token: None 2024-04-12T03:00:41.584 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a821a588d1716c9a-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.586 DEBUG:boto:path=/dgbpxg-11/obj22 2024-04-12T03:00:41.586 DEBUG:boto:auth_path=/dgbpxg-11/obj22 2024-04-12T03:00:41.586 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.586 DEBUG:boto:Path: /dgbpxg-11/obj22 2024-04-12T03:00:41.587 DEBUG:boto:Data: 2024-04-12T03:00:41.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.587 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.587 DEBUG:boto:Params: {} 2024-04-12T03:00:41.587 DEBUG:boto:Token: None 2024-04-12T03:00:41.587 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj22 2024-04-12T03:00:41.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jlHEr8vQF9m5TQa9EXsDY4+DYjM= 2024-04-12T03:00:41.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jlHEr8vQF9m5TQa9EXsDY4+DYjM='} 2024-04-12T03:00:41.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096f052f61bee4379-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.591 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.591 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.591 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.591 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.591 DEBUG:boto:Data: 2024-04-12T03:00:41.591 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.592 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.592 DEBUG:boto:Params: {} 2024-04-12T03:00:41.592 DEBUG:boto:Token: None 2024-04-12T03:00:41.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000baa6b6f143d28b3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.594 DEBUG:boto:path=/dgbpxg-11/obj23 2024-04-12T03:00:41.594 DEBUG:boto:auth_path=/dgbpxg-11/obj23 2024-04-12T03:00:41.595 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.595 DEBUG:boto:Path: /dgbpxg-11/obj23 2024-04-12T03:00:41.595 DEBUG:boto:Data: 2024-04-12T03:00:41.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.595 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.595 DEBUG:boto:Params: {} 2024-04-12T03:00:41.595 DEBUG:boto:Token: None 2024-04-12T03:00:41.595 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj23 2024-04-12T03:00:41.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:64KhN9zPvDOmePW+o4vXzGGzprY= 2024-04-12T03:00:41.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:64KhN9zPvDOmePW+o4vXzGGzprY='} 2024-04-12T03:00:41.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006890ee63e3e3c868-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.599 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.599 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.600 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.600 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.600 DEBUG:boto:Data: 2024-04-12T03:00:41.600 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.600 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.600 DEBUG:boto:Params: {} 2024-04-12T03:00:41.600 DEBUG:boto:Token: None 2024-04-12T03:00:41.600 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002104865adc0fcad2-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.603 DEBUG:boto:path=/dgbpxg-11/obj24 2024-04-12T03:00:41.603 DEBUG:boto:auth_path=/dgbpxg-11/obj24 2024-04-12T03:00:41.603 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.603 DEBUG:boto:Path: /dgbpxg-11/obj24 2024-04-12T03:00:41.603 DEBUG:boto:Data: 2024-04-12T03:00:41.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.604 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.604 DEBUG:boto:Params: {} 2024-04-12T03:00:41.604 DEBUG:boto:Token: None 2024-04-12T03:00:41.604 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj24 2024-04-12T03:00:41.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8PC4VfiXWUtcPUFnwdgYaet6Szk= 2024-04-12T03:00:41.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8PC4VfiXWUtcPUFnwdgYaet6Szk='} 2024-04-12T03:00:41.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adc80db49ada131e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.647 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.647 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.647 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.647 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.647 DEBUG:boto:Data: 2024-04-12T03:00:41.647 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.647 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.648 DEBUG:boto:Params: {} 2024-04-12T03:00:41.648 DEBUG:boto:Token: None 2024-04-12T03:00:41.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff047f7004873fbd-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.650 DEBUG:boto:path=/dgbpxg-11/obj25 2024-04-12T03:00:41.651 DEBUG:boto:auth_path=/dgbpxg-11/obj25 2024-04-12T03:00:41.651 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.651 DEBUG:boto:Path: /dgbpxg-11/obj25 2024-04-12T03:00:41.651 DEBUG:boto:Data: 2024-04-12T03:00:41.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.651 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.651 DEBUG:boto:Params: {} 2024-04-12T03:00:41.651 DEBUG:boto:Token: None 2024-04-12T03:00:41.651 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj25 2024-04-12T03:00:41.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oTuzu8/F8O/mvalUcSSPr55TCJU= 2024-04-12T03:00:41.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oTuzu8/F8O/mvalUcSSPr55TCJU='} 2024-04-12T03:00:41.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022c4a6cbae330048-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.655 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.655 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.655 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.655 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.656 DEBUG:boto:Data: 2024-04-12T03:00:41.656 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.656 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.656 DEBUG:boto:Params: {} 2024-04-12T03:00:41.656 DEBUG:boto:Token: None 2024-04-12T03:00:41.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e3ada157a9dc406-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.659 DEBUG:boto:path=/dgbpxg-11/obj26 2024-04-12T03:00:41.659 DEBUG:boto:auth_path=/dgbpxg-11/obj26 2024-04-12T03:00:41.659 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.659 DEBUG:boto:Path: /dgbpxg-11/obj26 2024-04-12T03:00:41.659 DEBUG:boto:Data: 2024-04-12T03:00:41.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.659 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.659 DEBUG:boto:Params: {} 2024-04-12T03:00:41.659 DEBUG:boto:Token: None 2024-04-12T03:00:41.660 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj26 2024-04-12T03:00:41.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tgwVZ8YHwyRu3dIdIkkRqkYvqk4= 2024-04-12T03:00:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tgwVZ8YHwyRu3dIdIkkRqkYvqk4='} 2024-04-12T03:00:41.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a02435eed21353e4-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.663 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.663 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.663 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.663 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.664 DEBUG:boto:Data: 2024-04-12T03:00:41.664 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.664 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.664 DEBUG:boto:Params: {} 2024-04-12T03:00:41.664 DEBUG:boto:Token: None 2024-04-12T03:00:41.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000db94d1fca07fafa-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.666 DEBUG:boto:path=/dgbpxg-11/obj27 2024-04-12T03:00:41.666 DEBUG:boto:auth_path=/dgbpxg-11/obj27 2024-04-12T03:00:41.667 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.667 DEBUG:boto:Path: /dgbpxg-11/obj27 2024-04-12T03:00:41.667 DEBUG:boto:Data: 2024-04-12T03:00:41.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.667 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.667 DEBUG:boto:Params: {} 2024-04-12T03:00:41.668 DEBUG:boto:Token: None 2024-04-12T03:00:41.668 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj27 2024-04-12T03:00:41.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m7BmwTEVzW65ZFUje6vF5KSOD7Q= 2024-04-12T03:00:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m7BmwTEVzW65ZFUje6vF5KSOD7Q='} 2024-04-12T03:00:41.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4555a998e0c2728-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.715 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.715 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.715 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.715 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.715 DEBUG:boto:Data: 2024-04-12T03:00:41.715 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.715 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.715 DEBUG:boto:Params: {} 2024-04-12T03:00:41.716 DEBUG:boto:Token: None 2024-04-12T03:00:41.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000397fcf3785fc0ad1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.718 DEBUG:boto:path=/dgbpxg-11/obj28 2024-04-12T03:00:41.718 DEBUG:boto:auth_path=/dgbpxg-11/obj28 2024-04-12T03:00:41.718 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.718 DEBUG:boto:Path: /dgbpxg-11/obj28 2024-04-12T03:00:41.719 DEBUG:boto:Data: 2024-04-12T03:00:41.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.719 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.719 DEBUG:boto:Params: {} 2024-04-12T03:00:41.719 DEBUG:boto:Token: None 2024-04-12T03:00:41.719 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj28 2024-04-12T03:00:41.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dk5scZzQNeJCUU5XLPqslXKMc48= 2024-04-12T03:00:41.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dk5scZzQNeJCUU5XLPqslXKMc48='} 2024-04-12T03:00:41.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025684601abf7aa99-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.723 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.723 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.723 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.723 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.723 DEBUG:boto:Data: 2024-04-12T03:00:41.724 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.724 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.724 DEBUG:boto:Params: {} 2024-04-12T03:00:41.724 DEBUG:boto:Token: None 2024-04-12T03:00:41.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043bacdb9ab46b030-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.726 DEBUG:boto:path=/dgbpxg-11/obj29 2024-04-12T03:00:41.726 DEBUG:boto:auth_path=/dgbpxg-11/obj29 2024-04-12T03:00:41.726 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.726 DEBUG:boto:Path: /dgbpxg-11/obj29 2024-04-12T03:00:41.726 DEBUG:boto:Data: 2024-04-12T03:00:41.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.727 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.727 DEBUG:boto:Params: {} 2024-04-12T03:00:41.727 DEBUG:boto:Token: None 2024-04-12T03:00:41.727 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj29 2024-04-12T03:00:41.727 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RzaXEGh9GYS5fIivVsHHufr1gVU= 2024-04-12T03:00:41.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RzaXEGh9GYS5fIivVsHHufr1gVU='} 2024-04-12T03:00:41.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eed8067895c54f75-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.731 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.731 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.731 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.731 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.731 DEBUG:boto:Data: 2024-04-12T03:00:41.731 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.731 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.731 DEBUG:boto:Params: {} 2024-04-12T03:00:41.732 DEBUG:boto:Token: None 2024-04-12T03:00:41.732 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000904553afaddf5ec1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.734 DEBUG:boto:path=/dgbpxg-11/obj30 2024-04-12T03:00:41.734 DEBUG:boto:auth_path=/dgbpxg-11/obj30 2024-04-12T03:00:41.734 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.734 DEBUG:boto:Path: /dgbpxg-11/obj30 2024-04-12T03:00:41.734 DEBUG:boto:Data: 2024-04-12T03:00:41.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.735 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.735 DEBUG:boto:Params: {} 2024-04-12T03:00:41.735 DEBUG:boto:Token: None 2024-04-12T03:00:41.735 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj30 2024-04-12T03:00:41.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sAGWzrTVp0tjOFN1GWKHT5LAsWw= 2024-04-12T03:00:41.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sAGWzrTVp0tjOFN1GWKHT5LAsWw='} 2024-04-12T03:00:41.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4c7b5b1d3b8169c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.779 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.779 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.779 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.779 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.779 DEBUG:boto:Data: 2024-04-12T03:00:41.779 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.779 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.780 DEBUG:boto:Params: {} 2024-04-12T03:00:41.780 DEBUG:boto:Token: None 2024-04-12T03:00:41.780 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.780 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b09894ab2036e10-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.782 DEBUG:boto:path=/dgbpxg-11/obj31 2024-04-12T03:00:41.783 DEBUG:boto:auth_path=/dgbpxg-11/obj31 2024-04-12T03:00:41.783 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.783 DEBUG:boto:Path: /dgbpxg-11/obj31 2024-04-12T03:00:41.783 DEBUG:boto:Data: 2024-04-12T03:00:41.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.783 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.783 DEBUG:boto:Params: {} 2024-04-12T03:00:41.783 DEBUG:boto:Token: None 2024-04-12T03:00:41.783 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj31 2024-04-12T03:00:41.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MheFR3qI96xZYl4H71R/D8GyWS4= 2024-04-12T03:00:41.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MheFR3qI96xZYl4H71R/D8GyWS4='} 2024-04-12T03:00:41.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007964599670dbadc3-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.787 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.787 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.787 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.787 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.788 DEBUG:boto:Data: 2024-04-12T03:00:41.788 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.788 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.788 DEBUG:boto:Params: {} 2024-04-12T03:00:41.788 DEBUG:boto:Token: None 2024-04-12T03:00:41.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000108931aefc153c32-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.790 DEBUG:boto:path=/dgbpxg-11/obj32 2024-04-12T03:00:41.791 DEBUG:boto:auth_path=/dgbpxg-11/obj32 2024-04-12T03:00:41.791 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.791 DEBUG:boto:Path: /dgbpxg-11/obj32 2024-04-12T03:00:41.791 DEBUG:boto:Data: 2024-04-12T03:00:41.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.791 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.791 DEBUG:boto:Params: {} 2024-04-12T03:00:41.791 DEBUG:boto:Token: None 2024-04-12T03:00:41.791 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj32 2024-04-12T03:00:41.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RbLXcLwEzElMw2g+KY3qDde/IdE= 2024-04-12T03:00:41.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RbLXcLwEzElMw2g+KY3qDde/IdE='} 2024-04-12T03:00:41.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edce7301e28b0aa4-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.835 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.835 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.835 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.835 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.835 DEBUG:boto:Data: 2024-04-12T03:00:41.835 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.835 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.835 DEBUG:boto:Params: {} 2024-04-12T03:00:41.836 DEBUG:boto:Token: None 2024-04-12T03:00:41.836 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc3eeb5bbca33cf0-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.838 DEBUG:boto:path=/dgbpxg-11/obj33 2024-04-12T03:00:41.838 DEBUG:boto:auth_path=/dgbpxg-11/obj33 2024-04-12T03:00:41.838 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.838 DEBUG:boto:Path: /dgbpxg-11/obj33 2024-04-12T03:00:41.838 DEBUG:boto:Data: 2024-04-12T03:00:41.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.839 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.839 DEBUG:boto:Params: {} 2024-04-12T03:00:41.839 DEBUG:boto:Token: None 2024-04-12T03:00:41.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj33 2024-04-12T03:00:41.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H/am/ybng9doEmrbDdKpD9RCDmg= 2024-04-12T03:00:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H/am/ybng9doEmrbDdKpD9RCDmg='} 2024-04-12T03:00:41.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044005802d5240491-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.843 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.843 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.843 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.843 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.843 DEBUG:boto:Data: 2024-04-12T03:00:41.843 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.843 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.843 DEBUG:boto:Params: {} 2024-04-12T03:00:41.844 DEBUG:boto:Token: None 2024-04-12T03:00:41.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000059ff258f93fe025-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.846 DEBUG:boto:path=/dgbpxg-11/obj34 2024-04-12T03:00:41.846 DEBUG:boto:auth_path=/dgbpxg-11/obj34 2024-04-12T03:00:41.846 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.846 DEBUG:boto:Path: /dgbpxg-11/obj34 2024-04-12T03:00:41.846 DEBUG:boto:Data: 2024-04-12T03:00:41.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.847 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.847 DEBUG:boto:Params: {} 2024-04-12T03:00:41.847 DEBUG:boto:Token: None 2024-04-12T03:00:41.847 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj34 2024-04-12T03:00:41.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5shWXGHMeA+hOP5oW1InF7POsPc= 2024-04-12T03:00:41.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5shWXGHMeA+hOP5oW1InF7POsPc='} 2024-04-12T03:00:41.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005b688a40eafb2ea-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.851 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.851 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.851 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.851 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.852 DEBUG:boto:Data: 2024-04-12T03:00:41.852 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.852 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.852 DEBUG:boto:Params: {} 2024-04-12T03:00:41.852 DEBUG:boto:Token: None 2024-04-12T03:00:41.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a050f9a9c72c9920-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.854 DEBUG:boto:path=/dgbpxg-11/obj35 2024-04-12T03:00:41.854 DEBUG:boto:auth_path=/dgbpxg-11/obj35 2024-04-12T03:00:41.855 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.855 DEBUG:boto:Path: /dgbpxg-11/obj35 2024-04-12T03:00:41.855 DEBUG:boto:Data: 2024-04-12T03:00:41.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.855 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.855 DEBUG:boto:Params: {} 2024-04-12T03:00:41.855 DEBUG:boto:Token: None 2024-04-12T03:00:41.856 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj35 2024-04-12T03:00:41.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mbTFzLq/OjNoJywlYl0aZntXUow= 2024-04-12T03:00:41.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mbTFzLq/OjNoJywlYl0aZntXUow='} 2024-04-12T03:00:41.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000194423722bd6322a-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.899 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.899 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.899 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.899 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.899 DEBUG:boto:Data: 2024-04-12T03:00:41.899 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.900 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.900 DEBUG:boto:Params: {} 2024-04-12T03:00:41.900 DEBUG:boto:Token: None 2024-04-12T03:00:41.901 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e49b541a10f35a24-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.903 DEBUG:boto:path=/dgbpxg-11/obj36 2024-04-12T03:00:41.903 DEBUG:boto:auth_path=/dgbpxg-11/obj36 2024-04-12T03:00:41.903 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.903 DEBUG:boto:Path: /dgbpxg-11/obj36 2024-04-12T03:00:41.903 DEBUG:boto:Data: 2024-04-12T03:00:41.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.904 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.904 DEBUG:boto:Params: {} 2024-04-12T03:00:41.904 DEBUG:boto:Token: None 2024-04-12T03:00:41.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj36 2024-04-12T03:00:41.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A/+dm21Uo1691jY0FADOLPck2gg= 2024-04-12T03:00:41.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A/+dm21Uo1691jY0FADOLPck2gg='} 2024-04-12T03:00:41.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077c793fdc06d8487-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.908 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.908 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.908 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.908 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.908 DEBUG:boto:Data: 2024-04-12T03:00:41.908 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.908 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.908 DEBUG:boto:Params: {} 2024-04-12T03:00:41.908 DEBUG:boto:Token: None 2024-04-12T03:00:41.909 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.909 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f316dc22f7557e7-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.911 DEBUG:boto:path=/dgbpxg-11/obj37 2024-04-12T03:00:41.911 DEBUG:boto:auth_path=/dgbpxg-11/obj37 2024-04-12T03:00:41.911 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.911 DEBUG:boto:Path: /dgbpxg-11/obj37 2024-04-12T03:00:41.911 DEBUG:boto:Data: 2024-04-12T03:00:41.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.911 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.912 DEBUG:boto:Params: {} 2024-04-12T03:00:41.912 DEBUG:boto:Token: None 2024-04-12T03:00:41.912 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj37 2024-04-12T03:00:41.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fTHnFfOdFx/nx0sJQPDGwbO1RAc= 2024-04-12T03:00:41.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fTHnFfOdFx/nx0sJQPDGwbO1RAc='} 2024-04-12T03:00:41.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c9803888dc28cc5-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.955 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.955 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.955 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.955 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.955 DEBUG:boto:Data: 2024-04-12T03:00:41.956 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.956 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.957 DEBUG:boto:Params: {} 2024-04-12T03:00:41.957 DEBUG:boto:Token: None 2024-04-12T03:00:41.957 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d0be1dfda3bd8e-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.960 DEBUG:boto:path=/dgbpxg-11/obj38 2024-04-12T03:00:41.960 DEBUG:boto:auth_path=/dgbpxg-11/obj38 2024-04-12T03:00:41.961 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.961 DEBUG:boto:Path: /dgbpxg-11/obj38 2024-04-12T03:00:41.961 DEBUG:boto:Data: 2024-04-12T03:00:41.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.962 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.962 DEBUG:boto:Params: {} 2024-04-12T03:00:41.963 DEBUG:boto:Token: None 2024-04-12T03:00:41.963 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj38 2024-04-12T03:00:41.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dJXfNboX/l6dAMUAi83hubtSqiA= 2024-04-12T03:00:41.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dJXfNboX/l6dAMUAi83hubtSqiA='} 2024-04-12T03:00:41.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000258c312d97b12976-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.967 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.967 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.967 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.967 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.967 DEBUG:boto:Data: 2024-04-12T03:00:41.967 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.968 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.968 DEBUG:boto:Params: {} 2024-04-12T03:00:41.968 DEBUG:boto:Token: None 2024-04-12T03:00:41.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009456c076c592eb19-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.970 DEBUG:boto:path=/dgbpxg-11/obj39 2024-04-12T03:00:41.970 DEBUG:boto:auth_path=/dgbpxg-11/obj39 2024-04-12T03:00:41.970 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.970 DEBUG:boto:Path: /dgbpxg-11/obj39 2024-04-12T03:00:41.970 DEBUG:boto:Data: 2024-04-12T03:00:41.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.971 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.971 DEBUG:boto:Params: {} 2024-04-12T03:00:41.971 DEBUG:boto:Token: None 2024-04-12T03:00:41.971 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj39 2024-04-12T03:00:41.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H2knZbAtkrNbKbZ7bLWnnRZbOZk= 2024-04-12T03:00:41.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H2knZbAtkrNbKbZ7bLWnnRZbOZk='} 2024-04-12T03:00:41.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a259bb26d8d0e38-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.975 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:41.975 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:41.975 DEBUG:boto:Method: HEAD 2024-04-12T03:00:41.975 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:41.975 DEBUG:boto:Data: 2024-04-12T03:00:41.975 DEBUG:boto:Headers: {} 2024-04-12T03:00:41.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.975 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.975 DEBUG:boto:Params: {} 2024-04-12T03:00:41.975 DEBUG:boto:Token: None 2024-04-12T03:00:41.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/ 2024-04-12T03:00:41.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T03:00:41.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T03:00:41.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015aef08849a7b8d9-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:41.978 DEBUG:boto:path=/dgbpxg-11/obj40 2024-04-12T03:00:41.978 DEBUG:boto:auth_path=/dgbpxg-11/obj40 2024-04-12T03:00:41.978 DEBUG:boto:Method: PUT 2024-04-12T03:00:41.978 DEBUG:boto:Path: /dgbpxg-11/obj40 2024-04-12T03:00:41.978 DEBUG:boto:Data: 2024-04-12T03:00:41.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:41.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:41.979 DEBUG:boto:Port: 8000 2024-04-12T03:00:41.979 DEBUG:boto:Params: {} 2024-04-12T03:00:41.979 DEBUG:boto:Token: None 2024-04-12T03:00:41.979 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:41 GMT /dgbpxg-11/obj40 2024-04-12T03:00:41.979 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNlWfC/5yWBpfrFbE/5Kk393nxM= 2024-04-12T03:00:41.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNlWfC/5yWBpfrFbE/5Kk393nxM='} 2024-04-12T03:00:42.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004903be498575ef3c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.027 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.027 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.027 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.027 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.027 DEBUG:boto:Data: 2024-04-12T03:00:42.027 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.027 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.028 DEBUG:boto:Params: {} 2024-04-12T03:00:42.028 DEBUG:boto:Token: None 2024-04-12T03:00:42.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a60d6612db44694-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.031 DEBUG:boto:path=/dgbpxg-11/obj41 2024-04-12T03:00:42.031 DEBUG:boto:auth_path=/dgbpxg-11/obj41 2024-04-12T03:00:42.031 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.031 DEBUG:boto:Path: /dgbpxg-11/obj41 2024-04-12T03:00:42.031 DEBUG:boto:Data: 2024-04-12T03:00:42.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.032 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.032 DEBUG:boto:Params: {} 2024-04-12T03:00:42.032 DEBUG:boto:Token: None 2024-04-12T03:00:42.032 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj41 2024-04-12T03:00:42.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FFjBR4rpUm6YMHKSpippxVCGOOI= 2024-04-12T03:00:42.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FFjBR4rpUm6YMHKSpippxVCGOOI='} 2024-04-12T03:00:42.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cccba0c7663827c-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.036 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.036 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.037 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.037 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.037 DEBUG:boto:Data: 2024-04-12T03:00:42.037 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.037 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.037 DEBUG:boto:Params: {} 2024-04-12T03:00:42.037 DEBUG:boto:Token: None 2024-04-12T03:00:42.037 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000361f015c02a93367-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.039 DEBUG:boto:path=/dgbpxg-11/obj42 2024-04-12T03:00:42.039 DEBUG:boto:auth_path=/dgbpxg-11/obj42 2024-04-12T03:00:42.039 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.040 DEBUG:boto:Path: /dgbpxg-11/obj42 2024-04-12T03:00:42.040 DEBUG:boto:Data: 2024-04-12T03:00:42.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.040 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.040 DEBUG:boto:Params: {} 2024-04-12T03:00:42.040 DEBUG:boto:Token: None 2024-04-12T03:00:42.040 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj42 2024-04-12T03:00:42.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vnUEfuKf/Yl2GMD5ZwMFnrfi+ww= 2024-04-12T03:00:42.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vnUEfuKf/Yl2GMD5ZwMFnrfi+ww='} 2024-04-12T03:00:42.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c199d659b1b425-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.083 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.083 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.083 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.083 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.083 DEBUG:boto:Data: 2024-04-12T03:00:42.083 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.083 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.083 DEBUG:boto:Params: {} 2024-04-12T03:00:42.084 DEBUG:boto:Token: None 2024-04-12T03:00:42.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008db3e08c81e92b01-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.086 DEBUG:boto:path=/dgbpxg-11/obj43 2024-04-12T03:00:42.086 DEBUG:boto:auth_path=/dgbpxg-11/obj43 2024-04-12T03:00:42.086 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.086 DEBUG:boto:Path: /dgbpxg-11/obj43 2024-04-12T03:00:42.086 DEBUG:boto:Data: 2024-04-12T03:00:42.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.087 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.087 DEBUG:boto:Params: {} 2024-04-12T03:00:42.087 DEBUG:boto:Token: None 2024-04-12T03:00:42.087 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj43 2024-04-12T03:00:42.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMy3JxK8epAhBgFsgeL2XkmAusQ= 2024-04-12T03:00:42.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMy3JxK8epAhBgFsgeL2XkmAusQ='} 2024-04-12T03:00:42.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c2be66efbc7911-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.090 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.091 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.091 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.091 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.091 DEBUG:boto:Data: 2024-04-12T03:00:42.091 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.091 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.091 DEBUG:boto:Params: {} 2024-04-12T03:00:42.091 DEBUG:boto:Token: None 2024-04-12T03:00:42.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c3472965dcd77c4-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.094 DEBUG:boto:path=/dgbpxg-11/obj44 2024-04-12T03:00:42.094 DEBUG:boto:auth_path=/dgbpxg-11/obj44 2024-04-12T03:00:42.094 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.094 DEBUG:boto:Path: /dgbpxg-11/obj44 2024-04-12T03:00:42.094 DEBUG:boto:Data: 2024-04-12T03:00:42.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.094 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.094 DEBUG:boto:Params: {} 2024-04-12T03:00:42.094 DEBUG:boto:Token: None 2024-04-12T03:00:42.095 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj44 2024-04-12T03:00:42.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i2U7aM3nNezas3lEccCiO8UXZFw= 2024-04-12T03:00:42.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i2U7aM3nNezas3lEccCiO8UXZFw='} 2024-04-12T03:00:42.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e639b960968fb5ae-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.099 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.099 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.099 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.099 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.100 DEBUG:boto:Data: 2024-04-12T03:00:42.100 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.100 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.100 DEBUG:boto:Params: {} 2024-04-12T03:00:42.100 DEBUG:boto:Token: None 2024-04-12T03:00:42.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae8f5431cd86acb2-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.103 DEBUG:boto:path=/dgbpxg-11/obj45 2024-04-12T03:00:42.103 DEBUG:boto:auth_path=/dgbpxg-11/obj45 2024-04-12T03:00:42.103 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.103 DEBUG:boto:Path: /dgbpxg-11/obj45 2024-04-12T03:00:42.103 DEBUG:boto:Data: 2024-04-12T03:00:42.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.103 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.103 DEBUG:boto:Params: {} 2024-04-12T03:00:42.104 DEBUG:boto:Token: None 2024-04-12T03:00:42.104 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj45 2024-04-12T03:00:42.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dA6us5rrTPnF0RdzPEHAuKhGXZ4= 2024-04-12T03:00:42.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dA6us5rrTPnF0RdzPEHAuKhGXZ4='} 2024-04-12T03:00:42.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e87849bbc0840349-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.147 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.149 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.150 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.150 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.150 DEBUG:boto:Data: 2024-04-12T03:00:42.150 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.150 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.150 DEBUG:boto:Params: {} 2024-04-12T03:00:42.150 DEBUG:boto:Token: None 2024-04-12T03:00:42.150 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0fe00dc641c3c2-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.153 DEBUG:boto:path=/dgbpxg-11/obj46 2024-04-12T03:00:42.153 DEBUG:boto:auth_path=/dgbpxg-11/obj46 2024-04-12T03:00:42.153 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.153 DEBUG:boto:Path: /dgbpxg-11/obj46 2024-04-12T03:00:42.153 DEBUG:boto:Data: 2024-04-12T03:00:42.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.154 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.154 DEBUG:boto:Params: {} 2024-04-12T03:00:42.154 DEBUG:boto:Token: None 2024-04-12T03:00:42.154 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj46 2024-04-12T03:00:42.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yTX/yfWlmkH7b3/Yy8/HecZa1zU= 2024-04-12T03:00:42.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yTX/yfWlmkH7b3/Yy8/HecZa1zU='} 2024-04-12T03:00:42.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b1402cef176122-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.158 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.158 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.158 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.158 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.158 DEBUG:boto:Data: 2024-04-12T03:00:42.158 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.159 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.159 DEBUG:boto:Params: {} 2024-04-12T03:00:42.159 DEBUG:boto:Token: None 2024-04-12T03:00:42.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000798e0e34d456ff3f-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.161 DEBUG:boto:path=/dgbpxg-11/obj47 2024-04-12T03:00:42.161 DEBUG:boto:auth_path=/dgbpxg-11/obj47 2024-04-12T03:00:42.162 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.162 DEBUG:boto:Path: /dgbpxg-11/obj47 2024-04-12T03:00:42.162 DEBUG:boto:Data: 2024-04-12T03:00:42.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.162 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.162 DEBUG:boto:Params: {} 2024-04-12T03:00:42.162 DEBUG:boto:Token: None 2024-04-12T03:00:42.162 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj47 2024-04-12T03:00:42.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y8kjnBy2Mxsf4VQhSCH1Gs2gj1c= 2024-04-12T03:00:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y8kjnBy2Mxsf4VQhSCH1Gs2gj1c='} 2024-04-12T03:00:42.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000837aa5c7c44b5492-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.168 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.168 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.168 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.168 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.168 DEBUG:boto:Data: 2024-04-12T03:00:42.169 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.169 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.169 DEBUG:boto:Params: {} 2024-04-12T03:00:42.169 DEBUG:boto:Token: None 2024-04-12T03:00:42.169 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009dd03e9f2e42b68-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.171 DEBUG:boto:path=/dgbpxg-11/obj48 2024-04-12T03:00:42.171 DEBUG:boto:auth_path=/dgbpxg-11/obj48 2024-04-12T03:00:42.172 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.172 DEBUG:boto:Path: /dgbpxg-11/obj48 2024-04-12T03:00:42.172 DEBUG:boto:Data: 2024-04-12T03:00:42.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.172 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.172 DEBUG:boto:Params: {} 2024-04-12T03:00:42.172 DEBUG:boto:Token: None 2024-04-12T03:00:42.172 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj48 2024-04-12T03:00:42.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4UQfxwsarRTD8RgfpJNU4CapI4c= 2024-04-12T03:00:42.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4UQfxwsarRTD8RgfpJNU4CapI4c='} 2024-04-12T03:00:42.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d724e30790d9e3d6-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.215 DEBUG:boto:path=/dgbpxg-11/ 2024-04-12T03:00:42.215 DEBUG:boto:auth_path=/dgbpxg-11/ 2024-04-12T03:00:42.215 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.215 DEBUG:boto:Path: /dgbpxg-11/ 2024-04-12T03:00:42.215 DEBUG:boto:Data: 2024-04-12T03:00:42.215 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.217 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.217 DEBUG:boto:Params: {} 2024-04-12T03:00:42.217 DEBUG:boto:Token: None 2024-04-12T03:00:42.217 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/ 2024-04-12T03:00:42.217 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T03:00:42.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T03:00:42.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1b65a70af018016-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.220 DEBUG:boto:path=/dgbpxg-11/obj49 2024-04-12T03:00:42.220 DEBUG:boto:auth_path=/dgbpxg-11/obj49 2024-04-12T03:00:42.220 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.220 DEBUG:boto:Path: /dgbpxg-11/obj49 2024-04-12T03:00:42.220 DEBUG:boto:Data: 2024-04-12T03:00:42.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:00:42.220 DEBUG:boto:Port: 8000 2024-04-12T03:00:42.220 DEBUG:boto:Params: {} 2024-04-12T03:00:42.221 DEBUG:boto:Token: None 2024-04-12T03:00:42.221 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-11/obj49 2024-04-12T03:00:42.221 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6DPZqCPJOAwFAYD7UvqlU3bZpH0= 2024-04-12T03:00:42.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6DPZqCPJOAwFAYD7UvqlU3bZpH0='} 2024-04-12T03:00:42.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a14c42a8a64ccf6-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.226 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.226 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.226 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.226 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.226 DEBUG:boto:Data: 2024-04-12T03:00:42.227 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.227 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.227 DEBUG:boto:Params: {} 2024-04-12T03:00:42.227 DEBUG:boto:Token: None 2024-04-12T03:00:42.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d47b1d02eb72564e-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.229 DEBUG:boto:path=/dgbpxg-12/obj1 2024-04-12T03:00:42.230 DEBUG:boto:auth_path=/dgbpxg-12/obj1 2024-04-12T03:00:42.230 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.230 DEBUG:boto:Path: /dgbpxg-12/obj1 2024-04-12T03:00:42.230 DEBUG:boto:Data: 2024-04-12T03:00:42.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.230 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.230 DEBUG:boto:Params: {} 2024-04-12T03:00:42.230 DEBUG:boto:Token: None 2024-04-12T03:00:42.230 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj1 2024-04-12T03:00:42.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MKBOZ6r5mT191v4BxQAxG8brfYE= 2024-04-12T03:00:42.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKBOZ6r5mT191v4BxQAxG8brfYE='} 2024-04-12T03:00:42.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a05b3249851ad76b-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.237 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.237 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.237 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.237 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.237 DEBUG:boto:Data: 2024-04-12T03:00:42.238 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.238 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.238 DEBUG:boto:Params: {} 2024-04-12T03:00:42.238 DEBUG:boto:Token: None 2024-04-12T03:00:42.238 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001da415d18898e31c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.240 DEBUG:boto:path=/dgbpxg-12/obj2 2024-04-12T03:00:42.240 DEBUG:boto:auth_path=/dgbpxg-12/obj2 2024-04-12T03:00:42.241 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.241 DEBUG:boto:Path: /dgbpxg-12/obj2 2024-04-12T03:00:42.241 DEBUG:boto:Data: 2024-04-12T03:00:42.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.241 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.241 DEBUG:boto:Params: {} 2024-04-12T03:00:42.241 DEBUG:boto:Token: None 2024-04-12T03:00:42.241 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj2 2024-04-12T03:00:42.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N1/EvSOtbR94vMqQuxRdKuxkcSk= 2024-04-12T03:00:42.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N1/EvSOtbR94vMqQuxRdKuxkcSk='} 2024-04-12T03:00:42.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da9ce9f17d2cfb0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.287 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.288 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.288 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.288 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.288 DEBUG:boto:Data: 2024-04-12T03:00:42.288 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.288 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.288 DEBUG:boto:Params: {} 2024-04-12T03:00:42.288 DEBUG:boto:Token: None 2024-04-12T03:00:42.289 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.289 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c066e87246d1121-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.291 DEBUG:boto:path=/dgbpxg-12/obj3 2024-04-12T03:00:42.291 DEBUG:boto:auth_path=/dgbpxg-12/obj3 2024-04-12T03:00:42.291 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.291 DEBUG:boto:Path: /dgbpxg-12/obj3 2024-04-12T03:00:42.291 DEBUG:boto:Data: 2024-04-12T03:00:42.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.292 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.292 DEBUG:boto:Params: {} 2024-04-12T03:00:42.292 DEBUG:boto:Token: None 2024-04-12T03:00:42.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj3 2024-04-12T03:00:42.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QHcE2sC6FRBkW7NO0xra7F8/KDM= 2024-04-12T03:00:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QHcE2sC6FRBkW7NO0xra7F8/KDM='} 2024-04-12T03:00:42.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a724c332208be2a-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.297 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.297 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.298 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.298 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.298 DEBUG:boto:Data: 2024-04-12T03:00:42.298 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.298 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.298 DEBUG:boto:Params: {} 2024-04-12T03:00:42.299 DEBUG:boto:Token: None 2024-04-12T03:00:42.299 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a52bc89d28f787-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.301 DEBUG:boto:path=/dgbpxg-12/obj4 2024-04-12T03:00:42.301 DEBUG:boto:auth_path=/dgbpxg-12/obj4 2024-04-12T03:00:42.302 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.302 DEBUG:boto:Path: /dgbpxg-12/obj4 2024-04-12T03:00:42.302 DEBUG:boto:Data: 2024-04-12T03:00:42.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.302 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.302 DEBUG:boto:Params: {} 2024-04-12T03:00:42.302 DEBUG:boto:Token: None 2024-04-12T03:00:42.302 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj4 2024-04-12T03:00:42.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4hzvcaH1uINM5XRAwjF11z6HuGM= 2024-04-12T03:00:42.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4hzvcaH1uINM5XRAwjF11z6HuGM='} 2024-04-12T03:00:42.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008eccbe4f8c940f7f-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.347 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.347 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.347 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.347 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.347 DEBUG:boto:Data: 2024-04-12T03:00:42.347 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.347 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.347 DEBUG:boto:Params: {} 2024-04-12T03:00:42.348 DEBUG:boto:Token: None 2024-04-12T03:00:42.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000817190d707f43377-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.350 DEBUG:boto:path=/dgbpxg-12/obj5 2024-04-12T03:00:42.351 DEBUG:boto:auth_path=/dgbpxg-12/obj5 2024-04-12T03:00:42.351 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.351 DEBUG:boto:Path: /dgbpxg-12/obj5 2024-04-12T03:00:42.351 DEBUG:boto:Data: 2024-04-12T03:00:42.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.351 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.351 DEBUG:boto:Params: {} 2024-04-12T03:00:42.352 DEBUG:boto:Token: None 2024-04-12T03:00:42.352 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj5 2024-04-12T03:00:42.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jzSFy+a5okPVXFBBEqpdTulsnzs= 2024-04-12T03:00:42.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzSFy+a5okPVXFBBEqpdTulsnzs='} 2024-04-12T03:00:42.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc828099c58ac911-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.356 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.356 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.356 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.356 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.356 DEBUG:boto:Data: 2024-04-12T03:00:42.356 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.357 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.357 DEBUG:boto:Params: {} 2024-04-12T03:00:42.357 DEBUG:boto:Token: None 2024-04-12T03:00:42.357 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.357 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f4bb21a48077a19-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.360 DEBUG:boto:path=/dgbpxg-12/obj6 2024-04-12T03:00:42.360 DEBUG:boto:auth_path=/dgbpxg-12/obj6 2024-04-12T03:00:42.360 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.360 DEBUG:boto:Path: /dgbpxg-12/obj6 2024-04-12T03:00:42.361 DEBUG:boto:Data: 2024-04-12T03:00:42.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.361 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.361 DEBUG:boto:Params: {} 2024-04-12T03:00:42.361 DEBUG:boto:Token: None 2024-04-12T03:00:42.361 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj6 2024-04-12T03:00:42.361 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MnbZcW5xEgFdTGiuowUJWKxODr8= 2024-04-12T03:00:42.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MnbZcW5xEgFdTGiuowUJWKxODr8='} 2024-04-12T03:00:42.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6fe86d264941d5d-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.365 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.365 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.365 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.365 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.366 DEBUG:boto:Data: 2024-04-12T03:00:42.366 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.366 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.366 DEBUG:boto:Params: {} 2024-04-12T03:00:42.366 DEBUG:boto:Token: None 2024-04-12T03:00:42.366 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf8bd9da6eea9a53-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.368 DEBUG:boto:path=/dgbpxg-12/obj7 2024-04-12T03:00:42.368 DEBUG:boto:auth_path=/dgbpxg-12/obj7 2024-04-12T03:00:42.369 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.369 DEBUG:boto:Path: /dgbpxg-12/obj7 2024-04-12T03:00:42.369 DEBUG:boto:Data: 2024-04-12T03:00:42.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.369 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.369 DEBUG:boto:Params: {} 2024-04-12T03:00:42.369 DEBUG:boto:Token: None 2024-04-12T03:00:42.369 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj7 2024-04-12T03:00:42.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HaWVdKaA5JwFRt6A+BfT2c0R0bY= 2024-04-12T03:00:42.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HaWVdKaA5JwFRt6A+BfT2c0R0bY='} 2024-04-12T03:00:42.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c086d580a6453e6-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.419 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.419 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.419 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.419 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.419 DEBUG:boto:Data: 2024-04-12T03:00:42.420 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.420 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.420 DEBUG:boto:Params: {} 2024-04-12T03:00:42.420 DEBUG:boto:Token: None 2024-04-12T03:00:42.420 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.420 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4ab61516c599c91-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.423 DEBUG:boto:path=/dgbpxg-12/obj8 2024-04-12T03:00:42.423 DEBUG:boto:auth_path=/dgbpxg-12/obj8 2024-04-12T03:00:42.423 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.423 DEBUG:boto:Path: /dgbpxg-12/obj8 2024-04-12T03:00:42.423 DEBUG:boto:Data: 2024-04-12T03:00:42.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.423 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.424 DEBUG:boto:Params: {} 2024-04-12T03:00:42.424 DEBUG:boto:Token: None 2024-04-12T03:00:42.424 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj8 2024-04-12T03:00:42.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Aqpwzjjl68yQf/RO29dsKYstuw= 2024-04-12T03:00:42.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Aqpwzjjl68yQf/RO29dsKYstuw='} 2024-04-12T03:00:42.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0e84387ddb5947f-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.428 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.428 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.428 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.429 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.429 DEBUG:boto:Data: 2024-04-12T03:00:42.429 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.429 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.429 DEBUG:boto:Params: {} 2024-04-12T03:00:42.429 DEBUG:boto:Token: None 2024-04-12T03:00:42.429 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.429 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050db539f9b35799-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.432 DEBUG:boto:path=/dgbpxg-12/obj9 2024-04-12T03:00:42.432 DEBUG:boto:auth_path=/dgbpxg-12/obj9 2024-04-12T03:00:42.432 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.432 DEBUG:boto:Path: /dgbpxg-12/obj9 2024-04-12T03:00:42.432 DEBUG:boto:Data: 2024-04-12T03:00:42.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.432 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.432 DEBUG:boto:Params: {} 2024-04-12T03:00:42.433 DEBUG:boto:Token: None 2024-04-12T03:00:42.433 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj9 2024-04-12T03:00:42.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ji4a+UnS2/AyXGEHWs70oRdjk0U= 2024-04-12T03:00:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ji4a+UnS2/AyXGEHWs70oRdjk0U='} 2024-04-12T03:00:42.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ecd635ae93679502-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.475 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.475 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.475 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.475 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.475 DEBUG:boto:Data: 2024-04-12T03:00:42.475 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.475 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.475 DEBUG:boto:Params: {} 2024-04-12T03:00:42.476 DEBUG:boto:Token: None 2024-04-12T03:00:42.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076a69beb0df414e0-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.478 DEBUG:boto:path=/dgbpxg-12/obj10 2024-04-12T03:00:42.478 DEBUG:boto:auth_path=/dgbpxg-12/obj10 2024-04-12T03:00:42.479 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.479 DEBUG:boto:Path: /dgbpxg-12/obj10 2024-04-12T03:00:42.479 DEBUG:boto:Data: 2024-04-12T03:00:42.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.479 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.479 DEBUG:boto:Params: {} 2024-04-12T03:00:42.479 DEBUG:boto:Token: None 2024-04-12T03:00:42.479 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj10 2024-04-12T03:00:42.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fxQu/IxSWMeguV+6WCU348RPyao= 2024-04-12T03:00:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fxQu/IxSWMeguV+6WCU348RPyao='} 2024-04-12T03:00:42.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0cca18cd1d49c4c-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.483 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.483 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.483 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.483 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.483 DEBUG:boto:Data: 2024-04-12T03:00:42.483 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.484 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.484 DEBUG:boto:Params: {} 2024-04-12T03:00:42.484 DEBUG:boto:Token: None 2024-04-12T03:00:42.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047955ef6a0c6e6c6-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.487 DEBUG:boto:path=/dgbpxg-12/obj11 2024-04-12T03:00:42.487 DEBUG:boto:auth_path=/dgbpxg-12/obj11 2024-04-12T03:00:42.487 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.487 DEBUG:boto:Path: /dgbpxg-12/obj11 2024-04-12T03:00:42.487 DEBUG:boto:Data: 2024-04-12T03:00:42.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.487 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.487 DEBUG:boto:Params: {} 2024-04-12T03:00:42.488 DEBUG:boto:Token: None 2024-04-12T03:00:42.488 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj11 2024-04-12T03:00:42.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uemqwaNovFwpP6aXh1ud9zH8HEc= 2024-04-12T03:00:42.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uemqwaNovFwpP6aXh1ud9zH8HEc='} 2024-04-12T03:00:42.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063fbfa2157d6df56-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.531 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.531 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.531 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.531 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.532 DEBUG:boto:Data: 2024-04-12T03:00:42.532 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.532 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.532 DEBUG:boto:Params: {} 2024-04-12T03:00:42.532 DEBUG:boto:Token: None 2024-04-12T03:00:42.533 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003733e3398246d70a-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.535 DEBUG:boto:path=/dgbpxg-12/obj12 2024-04-12T03:00:42.535 DEBUG:boto:auth_path=/dgbpxg-12/obj12 2024-04-12T03:00:42.536 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.536 DEBUG:boto:Path: /dgbpxg-12/obj12 2024-04-12T03:00:42.536 DEBUG:boto:Data: 2024-04-12T03:00:42.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.536 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.536 DEBUG:boto:Params: {} 2024-04-12T03:00:42.537 DEBUG:boto:Token: None 2024-04-12T03:00:42.537 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj12 2024-04-12T03:00:42.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z3bavVZPlBaqKCOGuHZYge47pdk= 2024-04-12T03:00:42.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z3bavVZPlBaqKCOGuHZYge47pdk='} 2024-04-12T03:00:42.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbcc8e52f5f97168-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.541 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.541 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.542 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.542 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.542 DEBUG:boto:Data: 2024-04-12T03:00:42.542 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.542 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.542 DEBUG:boto:Params: {} 2024-04-12T03:00:42.543 DEBUG:boto:Token: None 2024-04-12T03:00:42.543 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.543 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006fc8b8bc06901a3-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.545 DEBUG:boto:path=/dgbpxg-12/obj13 2024-04-12T03:00:42.545 DEBUG:boto:auth_path=/dgbpxg-12/obj13 2024-04-12T03:00:42.545 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.545 DEBUG:boto:Path: /dgbpxg-12/obj13 2024-04-12T03:00:42.546 DEBUG:boto:Data: 2024-04-12T03:00:42.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.546 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.546 DEBUG:boto:Params: {} 2024-04-12T03:00:42.546 DEBUG:boto:Token: None 2024-04-12T03:00:42.546 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj13 2024-04-12T03:00:42.546 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:83MoXt4qgs69Xy6gtdGnkJVkUlo= 2024-04-12T03:00:42.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:83MoXt4qgs69Xy6gtdGnkJVkUlo='} 2024-04-12T03:00:42.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fcec73b61bb68c0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.591 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.591 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.591 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.591 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.591 DEBUG:boto:Data: 2024-04-12T03:00:42.591 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.592 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.592 DEBUG:boto:Params: {} 2024-04-12T03:00:42.592 DEBUG:boto:Token: None 2024-04-12T03:00:42.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000014ebafaaa707ee6-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.594 DEBUG:boto:path=/dgbpxg-12/obj14 2024-04-12T03:00:42.594 DEBUG:boto:auth_path=/dgbpxg-12/obj14 2024-04-12T03:00:42.595 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.595 DEBUG:boto:Path: /dgbpxg-12/obj14 2024-04-12T03:00:42.595 DEBUG:boto:Data: 2024-04-12T03:00:42.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.595 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.595 DEBUG:boto:Params: {} 2024-04-12T03:00:42.595 DEBUG:boto:Token: None 2024-04-12T03:00:42.595 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj14 2024-04-12T03:00:42.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:22r2tOEHUA+Q6mHqbE3jznGkzcc= 2024-04-12T03:00:42.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:22r2tOEHUA+Q6mHqbE3jznGkzcc='} 2024-04-12T03:00:42.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c64c90327fff8d2-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.600 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.600 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.600 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.600 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.600 DEBUG:boto:Data: 2024-04-12T03:00:42.600 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.600 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.601 DEBUG:boto:Params: {} 2024-04-12T03:00:42.601 DEBUG:boto:Token: None 2024-04-12T03:00:42.601 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e4599acfde9a05-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.603 DEBUG:boto:path=/dgbpxg-12/obj15 2024-04-12T03:00:42.603 DEBUG:boto:auth_path=/dgbpxg-12/obj15 2024-04-12T03:00:42.603 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.603 DEBUG:boto:Path: /dgbpxg-12/obj15 2024-04-12T03:00:42.604 DEBUG:boto:Data: 2024-04-12T03:00:42.604 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.604 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.604 DEBUG:boto:Params: {} 2024-04-12T03:00:42.604 DEBUG:boto:Token: None 2024-04-12T03:00:42.604 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj15 2024-04-12T03:00:42.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mjqTY8nna7KLxlZCj6qw/5yv2gg= 2024-04-12T03:00:42.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mjqTY8nna7KLxlZCj6qw/5yv2gg='} 2024-04-12T03:00:42.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fca26cd2a4a74344-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.647 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.647 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.647 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.647 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.647 DEBUG:boto:Data: 2024-04-12T03:00:42.647 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.647 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.647 DEBUG:boto:Params: {} 2024-04-12T03:00:42.648 DEBUG:boto:Token: None 2024-04-12T03:00:42.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001989a1894f726a17-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.650 DEBUG:boto:path=/dgbpxg-12/obj16 2024-04-12T03:00:42.652 DEBUG:boto:auth_path=/dgbpxg-12/obj16 2024-04-12T03:00:42.652 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.652 DEBUG:boto:Path: /dgbpxg-12/obj16 2024-04-12T03:00:42.652 DEBUG:boto:Data: 2024-04-12T03:00:42.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.652 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.653 DEBUG:boto:Params: {} 2024-04-12T03:00:42.654 DEBUG:boto:Token: None 2024-04-12T03:00:42.654 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj16 2024-04-12T03:00:42.654 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8SIDLvw1vt6C9Lb+Qib4xtJrIec= 2024-04-12T03:00:42.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8SIDLvw1vt6C9Lb+Qib4xtJrIec='} 2024-04-12T03:00:42.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064dddb9d2f848216-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.663 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.663 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.663 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.663 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.663 DEBUG:boto:Data: 2024-04-12T03:00:42.663 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.663 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.663 DEBUG:boto:Params: {} 2024-04-12T03:00:42.664 DEBUG:boto:Token: None 2024-04-12T03:00:42.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4fc182d51421314-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.666 DEBUG:boto:path=/dgbpxg-12/obj17 2024-04-12T03:00:42.666 DEBUG:boto:auth_path=/dgbpxg-12/obj17 2024-04-12T03:00:42.666 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.666 DEBUG:boto:Path: /dgbpxg-12/obj17 2024-04-12T03:00:42.667 DEBUG:boto:Data: 2024-04-12T03:00:42.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.667 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.667 DEBUG:boto:Params: {} 2024-04-12T03:00:42.667 DEBUG:boto:Token: None 2024-04-12T03:00:42.667 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj17 2024-04-12T03:00:42.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VpMB6FlsuF6s/qwlTlRaPmNciIQ= 2024-04-12T03:00:42.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VpMB6FlsuF6s/qwlTlRaPmNciIQ='} 2024-04-12T03:00:42.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b628837d630202d0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.671 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.671 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.671 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.671 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.671 DEBUG:boto:Data: 2024-04-12T03:00:42.671 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.672 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.672 DEBUG:boto:Params: {} 2024-04-12T03:00:42.672 DEBUG:boto:Token: None 2024-04-12T03:00:42.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d23006395520aea-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.674 DEBUG:boto:path=/dgbpxg-12/obj18 2024-04-12T03:00:42.674 DEBUG:boto:auth_path=/dgbpxg-12/obj18 2024-04-12T03:00:42.674 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.674 DEBUG:boto:Path: /dgbpxg-12/obj18 2024-04-12T03:00:42.675 DEBUG:boto:Data: 2024-04-12T03:00:42.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.675 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.675 DEBUG:boto:Params: {} 2024-04-12T03:00:42.675 DEBUG:boto:Token: None 2024-04-12T03:00:42.675 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj18 2024-04-12T03:00:42.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ohR8izBcktUWBew1XJ6Bv1fapY8= 2024-04-12T03:00:42.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ohR8izBcktUWBew1XJ6Bv1fapY8='} 2024-04-12T03:00:42.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005905fb954b799ba0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.719 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.719 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.720 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.721 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.721 DEBUG:boto:Data: 2024-04-12T03:00:42.721 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.721 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.721 DEBUG:boto:Params: {} 2024-04-12T03:00:42.721 DEBUG:boto:Token: None 2024-04-12T03:00:42.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.722 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044aa880b9cea82da-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.724 DEBUG:boto:path=/dgbpxg-12/obj19 2024-04-12T03:00:42.724 DEBUG:boto:auth_path=/dgbpxg-12/obj19 2024-04-12T03:00:42.724 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.725 DEBUG:boto:Path: /dgbpxg-12/obj19 2024-04-12T03:00:42.725 DEBUG:boto:Data: 2024-04-12T03:00:42.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.725 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.725 DEBUG:boto:Params: {} 2024-04-12T03:00:42.725 DEBUG:boto:Token: None 2024-04-12T03:00:42.725 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj19 2024-04-12T03:00:42.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1RPStJq6t4DhUtTY2oOaFFmhlXQ= 2024-04-12T03:00:42.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1RPStJq6t4DhUtTY2oOaFFmhlXQ='} 2024-04-12T03:00:42.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091b18d74f8b823ce-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.729 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.729 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.729 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.729 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.729 DEBUG:boto:Data: 2024-04-12T03:00:42.730 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.730 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.730 DEBUG:boto:Params: {} 2024-04-12T03:00:42.730 DEBUG:boto:Token: None 2024-04-12T03:00:42.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cec405d693a9d814-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.733 DEBUG:boto:path=/dgbpxg-12/obj20 2024-04-12T03:00:42.733 DEBUG:boto:auth_path=/dgbpxg-12/obj20 2024-04-12T03:00:42.733 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.733 DEBUG:boto:Path: /dgbpxg-12/obj20 2024-04-12T03:00:42.733 DEBUG:boto:Data: 2024-04-12T03:00:42.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.733 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.733 DEBUG:boto:Params: {} 2024-04-12T03:00:42.733 DEBUG:boto:Token: None 2024-04-12T03:00:42.734 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj20 2024-04-12T03:00:42.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QQzPMik5J4pboDEyJsn882U8PNA= 2024-04-12T03:00:42.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QQzPMik5J4pboDEyJsn882U8PNA='} 2024-04-12T03:00:42.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0768b962bb5e425-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.738 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.738 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.739 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.739 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.739 DEBUG:boto:Data: 2024-04-12T03:00:42.739 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.739 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.739 DEBUG:boto:Params: {} 2024-04-12T03:00:42.739 DEBUG:boto:Token: None 2024-04-12T03:00:42.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1b485759e7c6785-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.742 DEBUG:boto:path=/dgbpxg-12/obj21 2024-04-12T03:00:42.742 DEBUG:boto:auth_path=/dgbpxg-12/obj21 2024-04-12T03:00:42.742 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.742 DEBUG:boto:Path: /dgbpxg-12/obj21 2024-04-12T03:00:42.742 DEBUG:boto:Data: 2024-04-12T03:00:42.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.743 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.743 DEBUG:boto:Params: {} 2024-04-12T03:00:42.743 DEBUG:boto:Token: None 2024-04-12T03:00:42.743 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj21 2024-04-12T03:00:42.743 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YXzXsblAn/33ad1bSg0tvYKYW94= 2024-04-12T03:00:42.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YXzXsblAn/33ad1bSg0tvYKYW94='} 2024-04-12T03:00:42.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000414dc7b1adacd763-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.751 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.751 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.751 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.751 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.752 DEBUG:boto:Data: 2024-04-12T03:00:42.752 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.752 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.752 DEBUG:boto:Params: {} 2024-04-12T03:00:42.752 DEBUG:boto:Token: None 2024-04-12T03:00:42.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034f65e16615d6fa0-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.755 DEBUG:boto:path=/dgbpxg-12/obj22 2024-04-12T03:00:42.755 DEBUG:boto:auth_path=/dgbpxg-12/obj22 2024-04-12T03:00:42.755 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.755 DEBUG:boto:Path: /dgbpxg-12/obj22 2024-04-12T03:00:42.755 DEBUG:boto:Data: 2024-04-12T03:00:42.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.755 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.755 DEBUG:boto:Params: {} 2024-04-12T03:00:42.755 DEBUG:boto:Token: None 2024-04-12T03:00:42.756 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj22 2024-04-12T03:00:42.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KNm5ZVYT8il+BU4NY2gm2BCfdQ4= 2024-04-12T03:00:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNm5ZVYT8il+BU4NY2gm2BCfdQ4='} 2024-04-12T03:00:42.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009300aebb579ed707-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.799 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.799 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.799 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.799 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.799 DEBUG:boto:Data: 2024-04-12T03:00:42.799 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.799 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.800 DEBUG:boto:Params: {} 2024-04-12T03:00:42.800 DEBUG:boto:Token: None 2024-04-12T03:00:42.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e59b58625d256ffa-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.802 DEBUG:boto:path=/dgbpxg-12/obj23 2024-04-12T03:00:42.802 DEBUG:boto:auth_path=/dgbpxg-12/obj23 2024-04-12T03:00:42.802 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.802 DEBUG:boto:Path: /dgbpxg-12/obj23 2024-04-12T03:00:42.803 DEBUG:boto:Data: 2024-04-12T03:00:42.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.803 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.803 DEBUG:boto:Params: {} 2024-04-12T03:00:42.803 DEBUG:boto:Token: None 2024-04-12T03:00:42.803 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj23 2024-04-12T03:00:42.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yqFctJJoTXuTBnBQBUKFNht1cPc= 2024-04-12T03:00:42.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yqFctJJoTXuTBnBQBUKFNht1cPc='} 2024-04-12T03:00:42.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000684b46e12b095342-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.807 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.807 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.807 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.807 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.807 DEBUG:boto:Data: 2024-04-12T03:00:42.807 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.808 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.808 DEBUG:boto:Params: {} 2024-04-12T03:00:42.808 DEBUG:boto:Token: None 2024-04-12T03:00:42.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa031cecbccc6cd-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.810 DEBUG:boto:path=/dgbpxg-12/obj24 2024-04-12T03:00:42.810 DEBUG:boto:auth_path=/dgbpxg-12/obj24 2024-04-12T03:00:42.810 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.810 DEBUG:boto:Path: /dgbpxg-12/obj24 2024-04-12T03:00:42.810 DEBUG:boto:Data: 2024-04-12T03:00:42.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.811 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.811 DEBUG:boto:Params: {} 2024-04-12T03:00:42.811 DEBUG:boto:Token: None 2024-04-12T03:00:42.811 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj24 2024-04-12T03:00:42.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pLWj9b3kknADciRfG027mRYMnu0= 2024-04-12T03:00:42.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pLWj9b3kknADciRfG027mRYMnu0='} 2024-04-12T03:00:42.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cdf17b1219e2414-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.815 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.815 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.815 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.815 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.815 DEBUG:boto:Data: 2024-04-12T03:00:42.815 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.816 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.816 DEBUG:boto:Params: {} 2024-04-12T03:00:42.816 DEBUG:boto:Token: None 2024-04-12T03:00:42.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dbc5bbb3b643952-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.818 DEBUG:boto:path=/dgbpxg-12/obj25 2024-04-12T03:00:42.818 DEBUG:boto:auth_path=/dgbpxg-12/obj25 2024-04-12T03:00:42.819 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.819 DEBUG:boto:Path: /dgbpxg-12/obj25 2024-04-12T03:00:42.819 DEBUG:boto:Data: 2024-04-12T03:00:42.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.819 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.819 DEBUG:boto:Params: {} 2024-04-12T03:00:42.819 DEBUG:boto:Token: None 2024-04-12T03:00:42.820 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj25 2024-04-12T03:00:42.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:glXa+MR0LbvNb61mD/SLVEmb28U= 2024-04-12T03:00:42.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:glXa+MR0LbvNb61mD/SLVEmb28U='} 2024-04-12T03:00:42.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001acb0ee6a752e0e8-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.863 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.864 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.864 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.864 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.864 DEBUG:boto:Data: 2024-04-12T03:00:42.864 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.864 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.865 DEBUG:boto:Params: {} 2024-04-12T03:00:42.865 DEBUG:boto:Token: None 2024-04-12T03:00:42.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a5ebdcf9a86b803-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.867 DEBUG:boto:path=/dgbpxg-12/obj26 2024-04-12T03:00:42.867 DEBUG:boto:auth_path=/dgbpxg-12/obj26 2024-04-12T03:00:42.868 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.868 DEBUG:boto:Path: /dgbpxg-12/obj26 2024-04-12T03:00:42.868 DEBUG:boto:Data: 2024-04-12T03:00:42.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.868 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.868 DEBUG:boto:Params: {} 2024-04-12T03:00:42.868 DEBUG:boto:Token: None 2024-04-12T03:00:42.868 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj26 2024-04-12T03:00:42.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wWji12qCwLwi4YZUmChn2sPPbdY= 2024-04-12T03:00:42.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wWji12qCwLwi4YZUmChn2sPPbdY='} 2024-04-12T03:00:42.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000794131ca6954c8a7-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.872 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.872 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.872 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.872 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.872 DEBUG:boto:Data: 2024-04-12T03:00:42.872 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.873 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.873 DEBUG:boto:Params: {} 2024-04-12T03:00:42.873 DEBUG:boto:Token: None 2024-04-12T03:00:42.873 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c2084a40574a9c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.875 DEBUG:boto:path=/dgbpxg-12/obj27 2024-04-12T03:00:42.875 DEBUG:boto:auth_path=/dgbpxg-12/obj27 2024-04-12T03:00:42.875 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.875 DEBUG:boto:Path: /dgbpxg-12/obj27 2024-04-12T03:00:42.876 DEBUG:boto:Data: 2024-04-12T03:00:42.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.876 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.876 DEBUG:boto:Params: {} 2024-04-12T03:00:42.876 DEBUG:boto:Token: None 2024-04-12T03:00:42.876 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj27 2024-04-12T03:00:42.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rSGnsnIDXWzBzfc/bj6F1a/X8sE= 2024-04-12T03:00:42.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rSGnsnIDXWzBzfc/bj6F1a/X8sE='} 2024-04-12T03:00:42.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e0eb472369827ac-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.923 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.923 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.923 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.923 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.923 DEBUG:boto:Data: 2024-04-12T03:00:42.923 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.923 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.924 DEBUG:boto:Params: {} 2024-04-12T03:00:42.924 DEBUG:boto:Token: None 2024-04-12T03:00:42.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee3217b27cc4eb8-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.926 DEBUG:boto:path=/dgbpxg-12/obj28 2024-04-12T03:00:42.926 DEBUG:boto:auth_path=/dgbpxg-12/obj28 2024-04-12T03:00:42.926 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.926 DEBUG:boto:Path: /dgbpxg-12/obj28 2024-04-12T03:00:42.926 DEBUG:boto:Data: 2024-04-12T03:00:42.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.927 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.927 DEBUG:boto:Params: {} 2024-04-12T03:00:42.927 DEBUG:boto:Token: None 2024-04-12T03:00:42.927 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj28 2024-04-12T03:00:42.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8UlYIGELU4lJ7qSxmuMqsSke3b4= 2024-04-12T03:00:42.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8UlYIGELU4lJ7qSxmuMqsSke3b4='} 2024-04-12T03:00:42.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a8cdc80618e2d96-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.931 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.931 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.931 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.931 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.931 DEBUG:boto:Data: 2024-04-12T03:00:42.931 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.931 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.932 DEBUG:boto:Params: {} 2024-04-12T03:00:42.932 DEBUG:boto:Token: None 2024-04-12T03:00:42.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034c97c27dcc71501-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.935 DEBUG:boto:path=/dgbpxg-12/obj29 2024-04-12T03:00:42.935 DEBUG:boto:auth_path=/dgbpxg-12/obj29 2024-04-12T03:00:42.935 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.935 DEBUG:boto:Path: /dgbpxg-12/obj29 2024-04-12T03:00:42.935 DEBUG:boto:Data: 2024-04-12T03:00:42.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.936 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.936 DEBUG:boto:Params: {} 2024-04-12T03:00:42.936 DEBUG:boto:Token: None 2024-04-12T03:00:42.936 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj29 2024-04-12T03:00:42.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H4kevjP9aqQvmg1c3HfRmGs7oQw= 2024-04-12T03:00:42.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H4kevjP9aqQvmg1c3HfRmGs7oQw='} 2024-04-12T03:00:42.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5d768e04af300b3-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.983 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.983 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.983 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.983 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.983 DEBUG:boto:Data: 2024-04-12T03:00:42.983 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.983 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.983 DEBUG:boto:Params: {} 2024-04-12T03:00:42.984 DEBUG:boto:Token: None 2024-04-12T03:00:42.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2f674cf99075b5c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.986 DEBUG:boto:path=/dgbpxg-12/obj30 2024-04-12T03:00:42.986 DEBUG:boto:auth_path=/dgbpxg-12/obj30 2024-04-12T03:00:42.986 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.986 DEBUG:boto:Path: /dgbpxg-12/obj30 2024-04-12T03:00:42.986 DEBUG:boto:Data: 2024-04-12T03:00:42.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.987 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.987 DEBUG:boto:Params: {} 2024-04-12T03:00:42.987 DEBUG:boto:Token: None 2024-04-12T03:00:42.987 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj30 2024-04-12T03:00:42.987 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JGBPbglgoUfAMpJf6TCw4Of/VO0= 2024-04-12T03:00:42.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JGBPbglgoUfAMpJf6TCw4Of/VO0='} 2024-04-12T03:00:42.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008c1df1e3d37234d-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.991 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:42.991 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:42.991 DEBUG:boto:Method: HEAD 2024-04-12T03:00:42.991 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:42.991 DEBUG:boto:Data: 2024-04-12T03:00:42.991 DEBUG:boto:Headers: {} 2024-04-12T03:00:42.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.991 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.991 DEBUG:boto:Params: {} 2024-04-12T03:00:42.991 DEBUG:boto:Token: None 2024-04-12T03:00:42.992 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/ 2024-04-12T03:00:42.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T03:00:42.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T03:00:42.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000927e0e9ed9edf8c4-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:42.994 DEBUG:boto:path=/dgbpxg-12/obj31 2024-04-12T03:00:42.994 DEBUG:boto:auth_path=/dgbpxg-12/obj31 2024-04-12T03:00:42.995 DEBUG:boto:Method: PUT 2024-04-12T03:00:42.995 DEBUG:boto:Path: /dgbpxg-12/obj31 2024-04-12T03:00:42.995 DEBUG:boto:Data: 2024-04-12T03:00:42.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:42.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:42.995 DEBUG:boto:Port: 8001 2024-04-12T03:00:42.995 DEBUG:boto:Params: {} 2024-04-12T03:00:42.996 DEBUG:boto:Token: None 2024-04-12T03:00:42.996 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:42 GMT /dgbpxg-12/obj31 2024-04-12T03:00:42.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OboIowbcpQP1ZHCijmXuoRuezkY= 2024-04-12T03:00:42.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OboIowbcpQP1ZHCijmXuoRuezkY='} 2024-04-12T03:00:43.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000987aa84857942667-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.000 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.000 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.000 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.001 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.001 DEBUG:boto:Data: 2024-04-12T03:00:43.001 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.002 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.002 DEBUG:boto:Params: {} 2024-04-12T03:00:43.002 DEBUG:boto:Token: None 2024-04-12T03:00:43.002 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.002 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000519b1dddf7209679-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.005 DEBUG:boto:path=/dgbpxg-12/obj32 2024-04-12T03:00:43.005 DEBUG:boto:auth_path=/dgbpxg-12/obj32 2024-04-12T03:00:43.006 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.006 DEBUG:boto:Path: /dgbpxg-12/obj32 2024-04-12T03:00:43.006 DEBUG:boto:Data: 2024-04-12T03:00:43.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.007 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.007 DEBUG:boto:Params: {} 2024-04-12T03:00:43.007 DEBUG:boto:Token: None 2024-04-12T03:00:43.007 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj32 2024-04-12T03:00:43.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:egPuEagRrUag04ZhF2ySTfHEJGc= 2024-04-12T03:00:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:egPuEagRrUag04ZhF2ySTfHEJGc='} 2024-04-12T03:00:43.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b487cac52c06324f-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.051 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.051 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.051 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.051 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.051 DEBUG:boto:Data: 2024-04-12T03:00:43.051 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.051 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.051 DEBUG:boto:Params: {} 2024-04-12T03:00:43.052 DEBUG:boto:Token: None 2024-04-12T03:00:43.052 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000037e774939991b55-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.054 DEBUG:boto:path=/dgbpxg-12/obj33 2024-04-12T03:00:43.054 DEBUG:boto:auth_path=/dgbpxg-12/obj33 2024-04-12T03:00:43.055 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.055 DEBUG:boto:Path: /dgbpxg-12/obj33 2024-04-12T03:00:43.055 DEBUG:boto:Data: 2024-04-12T03:00:43.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.055 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.055 DEBUG:boto:Params: {} 2024-04-12T03:00:43.055 DEBUG:boto:Token: None 2024-04-12T03:00:43.055 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj33 2024-04-12T03:00:43.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Ndif9qjKO3MMQOtwGyhoxloF/I= 2024-04-12T03:00:43.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Ndif9qjKO3MMQOtwGyhoxloF/I='} 2024-04-12T03:00:43.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043cef1157f30fcdd-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.059 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.060 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.060 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.060 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.060 DEBUG:boto:Data: 2024-04-12T03:00:43.060 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.060 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.060 DEBUG:boto:Params: {} 2024-04-12T03:00:43.060 DEBUG:boto:Token: None 2024-04-12T03:00:43.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa75f57522b68edb-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.063 DEBUG:boto:path=/dgbpxg-12/obj34 2024-04-12T03:00:43.063 DEBUG:boto:auth_path=/dgbpxg-12/obj34 2024-04-12T03:00:43.063 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.063 DEBUG:boto:Path: /dgbpxg-12/obj34 2024-04-12T03:00:43.063 DEBUG:boto:Data: 2024-04-12T03:00:43.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.063 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.063 DEBUG:boto:Params: {} 2024-04-12T03:00:43.064 DEBUG:boto:Token: None 2024-04-12T03:00:43.064 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj34 2024-04-12T03:00:43.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C/Ar3f41rfm7OxIhBhL8dg1L+v0= 2024-04-12T03:00:43.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/Ar3f41rfm7OxIhBhL8dg1L+v0='} 2024-04-12T03:00:43.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b5552b52ffe3da8-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.107 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.107 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.107 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.107 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.107 DEBUG:boto:Data: 2024-04-12T03:00:43.107 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.107 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.107 DEBUG:boto:Params: {} 2024-04-12T03:00:43.108 DEBUG:boto:Token: None 2024-04-12T03:00:43.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003db96a3438f25349-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.110 DEBUG:boto:path=/dgbpxg-12/obj35 2024-04-12T03:00:43.110 DEBUG:boto:auth_path=/dgbpxg-12/obj35 2024-04-12T03:00:43.110 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.110 DEBUG:boto:Path: /dgbpxg-12/obj35 2024-04-12T03:00:43.110 DEBUG:boto:Data: 2024-04-12T03:00:43.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.111 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.111 DEBUG:boto:Params: {} 2024-04-12T03:00:43.111 DEBUG:boto:Token: None 2024-04-12T03:00:43.111 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj35 2024-04-12T03:00:43.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VvcKi5+YVhjRcTj5yh3nbfCAh+I= 2024-04-12T03:00:43.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VvcKi5+YVhjRcTj5yh3nbfCAh+I='} 2024-04-12T03:00:43.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006edfacff82307a53-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.114 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.115 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.115 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.115 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.115 DEBUG:boto:Data: 2024-04-12T03:00:43.115 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.115 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.115 DEBUG:boto:Params: {} 2024-04-12T03:00:43.115 DEBUG:boto:Token: None 2024-04-12T03:00:43.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006704af4a2d0f3b4d-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.118 DEBUG:boto:path=/dgbpxg-12/obj36 2024-04-12T03:00:43.118 DEBUG:boto:auth_path=/dgbpxg-12/obj36 2024-04-12T03:00:43.118 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.118 DEBUG:boto:Path: /dgbpxg-12/obj36 2024-04-12T03:00:43.118 DEBUG:boto:Data: 2024-04-12T03:00:43.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.118 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.118 DEBUG:boto:Params: {} 2024-04-12T03:00:43.119 DEBUG:boto:Token: None 2024-04-12T03:00:43.119 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj36 2024-04-12T03:00:43.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QegDawR8aXocfM3wirIpoaPGK+4= 2024-04-12T03:00:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QegDawR8aXocfM3wirIpoaPGK+4='} 2024-04-12T03:00:43.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005115d1fd9ac4c58b-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.123 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.123 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.123 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.123 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.123 DEBUG:boto:Data: 2024-04-12T03:00:43.123 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.123 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.124 DEBUG:boto:Params: {} 2024-04-12T03:00:43.124 DEBUG:boto:Token: None 2024-04-12T03:00:43.124 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f312a54fcdff8497-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.126 DEBUG:boto:path=/dgbpxg-12/obj37 2024-04-12T03:00:43.126 DEBUG:boto:auth_path=/dgbpxg-12/obj37 2024-04-12T03:00:43.126 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.126 DEBUG:boto:Path: /dgbpxg-12/obj37 2024-04-12T03:00:43.126 DEBUG:boto:Data: 2024-04-12T03:00:43.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.127 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.127 DEBUG:boto:Params: {} 2024-04-12T03:00:43.127 DEBUG:boto:Token: None 2024-04-12T03:00:43.127 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj37 2024-04-12T03:00:43.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aj8Dw5K6QfEdlOAfTGKd3fCewoQ= 2024-04-12T03:00:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aj8Dw5K6QfEdlOAfTGKd3fCewoQ='} 2024-04-12T03:00:43.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c2b7917e38ab4ab-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.173 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.174 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.174 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.174 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.174 DEBUG:boto:Data: 2024-04-12T03:00:43.174 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.174 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.174 DEBUG:boto:Params: {} 2024-04-12T03:00:43.174 DEBUG:boto:Token: None 2024-04-12T03:00:43.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000028af2c7d666886c-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.177 DEBUG:boto:path=/dgbpxg-12/obj38 2024-04-12T03:00:43.177 DEBUG:boto:auth_path=/dgbpxg-12/obj38 2024-04-12T03:00:43.178 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.178 DEBUG:boto:Path: /dgbpxg-12/obj38 2024-04-12T03:00:43.178 DEBUG:boto:Data: 2024-04-12T03:00:43.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.178 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.178 DEBUG:boto:Params: {} 2024-04-12T03:00:43.178 DEBUG:boto:Token: None 2024-04-12T03:00:43.178 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj38 2024-04-12T03:00:43.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MZrTYmzWMYkzv0qH+wxf0diJbPs= 2024-04-12T03:00:43.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MZrTYmzWMYkzv0qH+wxf0diJbPs='} 2024-04-12T03:00:43.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bb711c431051b91-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.182 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.182 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.182 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.183 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.183 DEBUG:boto:Data: 2024-04-12T03:00:43.183 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.183 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.183 DEBUG:boto:Params: {} 2024-04-12T03:00:43.183 DEBUG:boto:Token: None 2024-04-12T03:00:43.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba63ec1902b5d69a-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.185 DEBUG:boto:path=/dgbpxg-12/obj39 2024-04-12T03:00:43.185 DEBUG:boto:auth_path=/dgbpxg-12/obj39 2024-04-12T03:00:43.186 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.186 DEBUG:boto:Path: /dgbpxg-12/obj39 2024-04-12T03:00:43.186 DEBUG:boto:Data: 2024-04-12T03:00:43.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.186 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.186 DEBUG:boto:Params: {} 2024-04-12T03:00:43.186 DEBUG:boto:Token: None 2024-04-12T03:00:43.186 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj39 2024-04-12T03:00:43.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmmKXBSlK98MQ+QvYx8vzyTPy9U= 2024-04-12T03:00:43.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmmKXBSlK98MQ+QvYx8vzyTPy9U='} 2024-04-12T03:00:43.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edab207d453fccc-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.190 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.191 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.191 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.191 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.191 DEBUG:boto:Data: 2024-04-12T03:00:43.191 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.191 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.192 DEBUG:boto:Params: {} 2024-04-12T03:00:43.192 DEBUG:boto:Token: None 2024-04-12T03:00:43.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061f44a40d29e24dc-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.194 DEBUG:boto:path=/dgbpxg-12/obj40 2024-04-12T03:00:43.195 DEBUG:boto:auth_path=/dgbpxg-12/obj40 2024-04-12T03:00:43.195 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.195 DEBUG:boto:Path: /dgbpxg-12/obj40 2024-04-12T03:00:43.195 DEBUG:boto:Data: 2024-04-12T03:00:43.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.196 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.196 DEBUG:boto:Params: {} 2024-04-12T03:00:43.196 DEBUG:boto:Token: None 2024-04-12T03:00:43.196 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj40 2024-04-12T03:00:43.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FkvoPd5sLbpJGyJaanP015SBq0M= 2024-04-12T03:00:43.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FkvoPd5sLbpJGyJaanP015SBq0M='} 2024-04-12T03:00:43.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094f80f6e7f9c65c9-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.200 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.200 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.200 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.200 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.200 DEBUG:boto:Data: 2024-04-12T03:00:43.200 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.201 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.201 DEBUG:boto:Params: {} 2024-04-12T03:00:43.201 DEBUG:boto:Token: None 2024-04-12T03:00:43.201 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d538300762de1d8-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.203 DEBUG:boto:path=/dgbpxg-12/obj41 2024-04-12T03:00:43.203 DEBUG:boto:auth_path=/dgbpxg-12/obj41 2024-04-12T03:00:43.203 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.203 DEBUG:boto:Path: /dgbpxg-12/obj41 2024-04-12T03:00:43.203 DEBUG:boto:Data: 2024-04-12T03:00:43.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.204 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.204 DEBUG:boto:Params: {} 2024-04-12T03:00:43.204 DEBUG:boto:Token: None 2024-04-12T03:00:43.204 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj41 2024-04-12T03:00:43.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PTAm6ibmBjkIFcPfq0ANLGGaKV0= 2024-04-12T03:00:43.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PTAm6ibmBjkIFcPfq0ANLGGaKV0='} 2024-04-12T03:00:43.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02e8bb7bef60b72-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.247 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.247 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.247 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.247 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.247 DEBUG:boto:Data: 2024-04-12T03:00:43.247 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.247 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.248 DEBUG:boto:Params: {} 2024-04-12T03:00:43.248 DEBUG:boto:Token: None 2024-04-12T03:00:43.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d768ad18d62286-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.250 DEBUG:boto:path=/dgbpxg-12/obj42 2024-04-12T03:00:43.251 DEBUG:boto:auth_path=/dgbpxg-12/obj42 2024-04-12T03:00:43.251 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.251 DEBUG:boto:Path: /dgbpxg-12/obj42 2024-04-12T03:00:43.251 DEBUG:boto:Data: 2024-04-12T03:00:43.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.251 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.251 DEBUG:boto:Params: {} 2024-04-12T03:00:43.251 DEBUG:boto:Token: None 2024-04-12T03:00:43.251 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj42 2024-04-12T03:00:43.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hfaijiiVBOhcaPHWdOrrt2uiroo= 2024-04-12T03:00:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hfaijiiVBOhcaPHWdOrrt2uiroo='} 2024-04-12T03:00:43.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e97dd088e018243d-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.255 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.255 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.255 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.255 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.255 DEBUG:boto:Data: 2024-04-12T03:00:43.256 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.256 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.256 DEBUG:boto:Params: {} 2024-04-12T03:00:43.256 DEBUG:boto:Token: None 2024-04-12T03:00:43.256 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002763b4ab6846a7c6-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.258 DEBUG:boto:path=/dgbpxg-12/obj43 2024-04-12T03:00:43.258 DEBUG:boto:auth_path=/dgbpxg-12/obj43 2024-04-12T03:00:43.258 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.258 DEBUG:boto:Path: /dgbpxg-12/obj43 2024-04-12T03:00:43.258 DEBUG:boto:Data: 2024-04-12T03:00:43.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.259 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.259 DEBUG:boto:Params: {} 2024-04-12T03:00:43.259 DEBUG:boto:Token: None 2024-04-12T03:00:43.259 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj43 2024-04-12T03:00:43.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gyN8exwLa9H1DRZ+vdMxOHv1rFQ= 2024-04-12T03:00:43.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gyN8exwLa9H1DRZ+vdMxOHv1rFQ='} 2024-04-12T03:00:43.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c74601776cffefef-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.262 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.263 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.263 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.263 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.263 DEBUG:boto:Data: 2024-04-12T03:00:43.263 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.263 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.263 DEBUG:boto:Params: {} 2024-04-12T03:00:43.263 DEBUG:boto:Token: None 2024-04-12T03:00:43.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b3b6cd167de5f4b-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.265 DEBUG:boto:path=/dgbpxg-12/obj44 2024-04-12T03:00:43.266 DEBUG:boto:auth_path=/dgbpxg-12/obj44 2024-04-12T03:00:43.266 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.266 DEBUG:boto:Path: /dgbpxg-12/obj44 2024-04-12T03:00:43.266 DEBUG:boto:Data: 2024-04-12T03:00:43.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.266 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.266 DEBUG:boto:Params: {} 2024-04-12T03:00:43.266 DEBUG:boto:Token: None 2024-04-12T03:00:43.267 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj44 2024-04-12T03:00:43.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lw4+dxevUFlpnr20Ed3jVaZ+tUM= 2024-04-12T03:00:43.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lw4+dxevUFlpnr20Ed3jVaZ+tUM='} 2024-04-12T03:00:43.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f938e835a94a9362-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.311 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.311 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.311 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.311 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.311 DEBUG:boto:Data: 2024-04-12T03:00:43.311 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.311 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.311 DEBUG:boto:Params: {} 2024-04-12T03:00:43.312 DEBUG:boto:Token: None 2024-04-12T03:00:43.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021578669ed157c7d-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.314 DEBUG:boto:path=/dgbpxg-12/obj45 2024-04-12T03:00:43.315 DEBUG:boto:auth_path=/dgbpxg-12/obj45 2024-04-12T03:00:43.315 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.315 DEBUG:boto:Path: /dgbpxg-12/obj45 2024-04-12T03:00:43.315 DEBUG:boto:Data: 2024-04-12T03:00:43.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.315 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.315 DEBUG:boto:Params: {} 2024-04-12T03:00:43.315 DEBUG:boto:Token: None 2024-04-12T03:00:43.315 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj45 2024-04-12T03:00:43.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UeKOVpzQoDA47P18wqYlQPGR2Oc= 2024-04-12T03:00:43.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UeKOVpzQoDA47P18wqYlQPGR2Oc='} 2024-04-12T03:00:43.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0f74757b0ff2880-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.319 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.319 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.319 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.319 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.319 DEBUG:boto:Data: 2024-04-12T03:00:43.319 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.320 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.320 DEBUG:boto:Params: {} 2024-04-12T03:00:43.320 DEBUG:boto:Token: None 2024-04-12T03:00:43.320 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6e00f13047ee3db-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.322 DEBUG:boto:path=/dgbpxg-12/obj46 2024-04-12T03:00:43.322 DEBUG:boto:auth_path=/dgbpxg-12/obj46 2024-04-12T03:00:43.322 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.322 DEBUG:boto:Path: /dgbpxg-12/obj46 2024-04-12T03:00:43.322 DEBUG:boto:Data: 2024-04-12T03:00:43.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.323 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.323 DEBUG:boto:Params: {} 2024-04-12T03:00:43.323 DEBUG:boto:Token: None 2024-04-12T03:00:43.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj46 2024-04-12T03:00:43.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TD4aXzIltSvZzIV0BkAI+z+ykQw= 2024-04-12T03:00:43.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TD4aXzIltSvZzIV0BkAI+z+ykQw='} 2024-04-12T03:00:43.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d94d8a897e63061-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.327 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.327 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.327 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.327 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.327 DEBUG:boto:Data: 2024-04-12T03:00:43.327 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.327 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.327 DEBUG:boto:Params: {} 2024-04-12T03:00:43.327 DEBUG:boto:Token: None 2024-04-12T03:00:43.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006983d882f7e95357-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.330 DEBUG:boto:path=/dgbpxg-12/obj47 2024-04-12T03:00:43.330 DEBUG:boto:auth_path=/dgbpxg-12/obj47 2024-04-12T03:00:43.330 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.330 DEBUG:boto:Path: /dgbpxg-12/obj47 2024-04-12T03:00:43.330 DEBUG:boto:Data: 2024-04-12T03:00:43.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.331 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.331 DEBUG:boto:Params: {} 2024-04-12T03:00:43.331 DEBUG:boto:Token: None 2024-04-12T03:00:43.331 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj47 2024-04-12T03:00:43.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O5GIfTdTpatRSaIdDc9TBO774NA= 2024-04-12T03:00:43.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O5GIfTdTpatRSaIdDc9TBO774NA='} 2024-04-12T03:00:43.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b2891839c30f9aa-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.379 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.379 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.379 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.379 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.379 DEBUG:boto:Data: 2024-04-12T03:00:43.379 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.379 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.379 DEBUG:boto:Params: {} 2024-04-12T03:00:43.380 DEBUG:boto:Token: None 2024-04-12T03:00:43.380 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bd5416fcc22af94-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.382 DEBUG:boto:path=/dgbpxg-12/obj48 2024-04-12T03:00:43.382 DEBUG:boto:auth_path=/dgbpxg-12/obj48 2024-04-12T03:00:43.382 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.383 DEBUG:boto:Path: /dgbpxg-12/obj48 2024-04-12T03:00:43.383 DEBUG:boto:Data: 2024-04-12T03:00:43.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.383 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.383 DEBUG:boto:Params: {} 2024-04-12T03:00:43.383 DEBUG:boto:Token: None 2024-04-12T03:00:43.383 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj48 2024-04-12T03:00:43.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:op1tJmrKH9mbkMKRPPlpYJ/Ls18= 2024-04-12T03:00:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:op1tJmrKH9mbkMKRPPlpYJ/Ls18='} 2024-04-12T03:00:43.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd1d7031baefda38-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.387 DEBUG:boto:path=/dgbpxg-12/ 2024-04-12T03:00:43.387 DEBUG:boto:auth_path=/dgbpxg-12/ 2024-04-12T03:00:43.387 DEBUG:boto:Method: HEAD 2024-04-12T03:00:43.387 DEBUG:boto:Path: /dgbpxg-12/ 2024-04-12T03:00:43.387 DEBUG:boto:Data: 2024-04-12T03:00:43.388 DEBUG:boto:Headers: {} 2024-04-12T03:00:43.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.388 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.388 DEBUG:boto:Params: {} 2024-04-12T03:00:43.388 DEBUG:boto:Token: None 2024-04-12T03:00:43.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/ 2024-04-12T03:00:43.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T03:00:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T03:00:43.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e592d9c101e47268-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.390 DEBUG:boto:path=/dgbpxg-12/obj49 2024-04-12T03:00:43.390 DEBUG:boto:auth_path=/dgbpxg-12/obj49 2024-04-12T03:00:43.390 DEBUG:boto:Method: PUT 2024-04-12T03:00:43.390 DEBUG:boto:Path: /dgbpxg-12/obj49 2024-04-12T03:00:43.391 DEBUG:boto:Data: 2024-04-12T03:00:43.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:00:43.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:00:43.391 DEBUG:boto:Port: 8001 2024-04-12T03:00:43.391 DEBUG:boto:Params: {} 2024-04-12T03:00:43.391 DEBUG:boto:Token: None 2024-04-12T03:00:43.391 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:00:43 GMT /dgbpxg-12/obj49 2024-04-12T03:00:43.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j50a27Lv8G3A+Op7RcOXia+lw+g= 2024-04-12T03:00:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j50a27Lv8G3A+Op7RcOXia+lw+g='} 2024-04-12T03:00:43.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ff6dcafb04b5451-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:00:43.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:00:43.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:00:43.395 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:00:43.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:00:43.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:00:43.470 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:00:43.481 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:00:43.482 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890840.928479_199.1", 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:00:40.928479Z" 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.483 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:00:43.484 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T03:00:43.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:00:43.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:00:43.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:00:43.484 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:00:43.533 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:00:43.533 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:00:43.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:43.565+0000 7f321ad6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:00:43.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:43.566+0000 7f321ad6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:00:43.580 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:00:43.580 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:00:43.580 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:00:43.580 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:00:43.580 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:00:43.581 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:00:43.581 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:00:43.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:00:43.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:00:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:00:43.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:00:43.585 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890840.928479_199.1", 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:00:40.928479Z", 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.586 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:00:43.587 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:00:43.587 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:00:43.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:00:43.587 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:00:43Z" 2024-04-12T03:00:43.587 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:00:43.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712890840.928479_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:00:40.928479Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:00:43Z" } 2024-04-12T03:00:43.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712890840.928479_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:00:40.928479Z', 'realm_epoch': 2}}] 2024-04-12T03:00:43.588 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T03:00:43.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T03:00:43.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:00:43.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:00:43.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:00:43.589 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:00:43.645 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:00:43.645 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:00:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:43.678+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:00:48.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:48.679+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:00:53.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:53.681+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:00:58.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:00:58.681+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:03.682+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:08.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:08.683+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:13.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:13.683+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:18.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:18.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:23.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:28.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:28.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:33.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:33.685+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:38.686+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:43.687+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:48.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:48.687+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:53.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:53.688+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:01:58.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:01:58.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:03.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:03.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:08.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:08.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:13.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:13.690+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:18.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:18.691+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:20.292 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:02:20.289+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:02:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:23.692+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:28.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:28.693+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:33.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:33.693+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:38.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:38.694+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:43.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:43.695+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:48.696+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:53.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:53.695+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:02:58.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:02:58.696+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:03.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:03.697+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:08.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:08.698+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:13.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:13.699+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:18.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:18.699+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:23.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:23.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:28.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:28.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:33.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:33.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:38.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:38.701+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:43.702+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:48.703+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:53.704+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:03:58.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:03:58.705+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:03.705+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:08.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:08.706+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:13.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:13.707+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:18.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:18.706+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:23.707+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:28.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:28.709+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:33.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:33.710+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:38.710+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:43.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:43.711+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:48.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:48.711+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:53.712+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:04:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:04:58.712+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:03.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:03.713+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:08.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:08.713+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:13.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:13.715+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:18.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:18.715+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:23.716+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:28.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:28.716+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:33.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:33.717+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T03:05:38.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:38.718+0000 7f2b112c0780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T03:05:38.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:38.718+0000 7f2b112c0780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T03:05:38.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:38.718+0000 7f2b112c0780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T03:05:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T03:05:38.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-12T03:05:38.758 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-13 2024-04-12T03:05:38.758 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:05:38.758 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:05:38.758 DEBUG:boto:Method: PUT 2024-04-12T03:05:38.758 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:05:38.759 DEBUG:boto:Data: 2024-04-12T03:05:38.759 DEBUG:boto:Headers: {} 2024-04-12T03:05:38.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:05:38.759 DEBUG:boto:Port: 8000 2024-04-12T03:05:38.759 DEBUG:boto:Params: {} 2024-04-12T03:05:38.759 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:05:38.759 DEBUG:boto:Token: None 2024-04-12T03:05:38.760 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:05:38 GMT /dgbpxg-13/ 2024-04-12T03:05:38.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oVL0XgjgEjo33Tb+p3R1Oa5kplQ= 2024-04-12T03:05:38.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:05:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oVL0XgjgEjo33Tb+p3R1Oa5kplQ=', 'Content-Length': '0'} 2024-04-12T03:05:38.790 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bc74f5a5d385c27b-006618a502-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:05:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:38.790 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-14 2024-04-12T03:05:38.790 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:05:38.790 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:05:38.790 DEBUG:boto:Method: PUT 2024-04-12T03:05:38.790 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:05:38.790 DEBUG:boto:Data: 2024-04-12T03:05:38.790 DEBUG:boto:Headers: {} 2024-04-12T03:05:38.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:05:38.791 DEBUG:boto:Port: 8001 2024-04-12T03:05:38.791 DEBUG:boto:Params: {} 2024-04-12T03:05:38.791 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:05:38.791 DEBUG:boto:Token: None 2024-04-12T03:05:38.791 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:05:38 GMT /dgbpxg-14/ 2024-04-12T03:05:38.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z8Ukgd3lUlAQI8tCdYWWi4VJ6Xk= 2024-04-12T03:05:38.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:05:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z8Ukgd3lUlAQI8tCdYWWi4VJ6Xk=', 'Content-Length': '0'} 2024-04-12T03:05:38.828 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f5ff8026da8c31e-006618a502-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:05:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:38.828 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:05:38.828 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:05:38.828 DEBUG:boto:path=/dgbpxg-13/?versioning 2024-04-12T03:05:38.828 DEBUG:boto:auth_path=/dgbpxg-13/?versioning 2024-04-12T03:05:38.828 DEBUG:boto:Method: PUT 2024-04-12T03:05:38.829 DEBUG:boto:Path: /dgbpxg-13/?versioning 2024-04-12T03:05:38.829 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:05:38.829 DEBUG:boto:Headers: {} 2024-04-12T03:05:38.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:05:38.829 DEBUG:boto:Port: 8000 2024-04-12T03:05:38.829 DEBUG:boto:Params: {} 2024-04-12T03:05:38.829 DEBUG:boto:Token: None 2024-04-12T03:05:38.829 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:05:38 GMT /dgbpxg-13/?versioning 2024-04-12T03:05:38.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bGlPNfzQcoPhJxKaD7SvvLAPmtw= 2024-04-12T03:05:38.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:05:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bGlPNfzQcoPhJxKaD7SvvLAPmtw=', 'Content-Length': '228'} 2024-04-12T03:05:38.835 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003023fc0c5461f5ca-006618a502-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:05:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:38.835 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:05:38.835 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:05:38.835 DEBUG:boto:path=/dgbpxg-14/?versioning 2024-04-12T03:05:38.835 DEBUG:boto:auth_path=/dgbpxg-14/?versioning 2024-04-12T03:05:38.835 DEBUG:boto:Method: PUT 2024-04-12T03:05:38.835 DEBUG:boto:Path: /dgbpxg-14/?versioning 2024-04-12T03:05:38.836 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:05:38.836 DEBUG:boto:Headers: {} 2024-04-12T03:05:38.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:05:38.836 DEBUG:boto:Port: 8001 2024-04-12T03:05:38.836 DEBUG:boto:Params: {} 2024-04-12T03:05:38.836 DEBUG:boto:Token: None 2024-04-12T03:05:38.836 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:05:38 GMT /dgbpxg-14/?versioning 2024-04-12T03:05:38.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7urhDopGFGWxznPnWkdHBZrWWZU= 2024-04-12T03:05:38.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:05:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7urhDopGFGWxznPnWkdHBZrWWZU=', 'Content-Length': '228'} 2024-04-12T03:05:38.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d53c21f6e8c6a19c-006618a502-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:05:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:38.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:38.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:38.844 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:05:38.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:05:38.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:05:38.935 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:05:38.947 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:05:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:05:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:05:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891138.841439_248.1", 2024-04-12T03:05:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:05:38.841439Z" 2024-04-12T03:05:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:38.949 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:05:38.949 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891138.841439_248.1'} 2024-04-12T03:05:38.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:05:38.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:38.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:38.950 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:05:38.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:05:38.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:05:39.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:39.027+0000 7f8ba38df780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:05:39.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:39.028+0000 7f8ba38df780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891138.778590_237.1", 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:05:38.778590Z", 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:05:39Z" 2024-04-12T03:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:05:39.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891138.778590_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:05:38.778590Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:05:39Z" } 2024-04-12T03:05:39.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891138.778590_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:05:38.778590Z', 'realm_epoch': 2}}] 2024-04-12T03:05:39.047 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891138.841439_248.1'} 2024-04-12T03:05:39.047 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891138.778590_237.1'} 2024-04-12T03:05:39.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891138.841439_248.1 target=1_1712891138.778590_237.1 2024-04-12T03:05:44.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:44.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:05:44.048 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:05:44.104 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:05:44.104 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:05:44.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:44.133+0000 7fdcda5f2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:05:44.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:44.134+0000 7fdcda5f2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:05:44.148 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:05:44.148 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:44.149 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:05:44.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:05:44.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:05:44.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891138.841439_248.1", 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:05:38.841439Z", 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:05:44.153 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:05:44Z" 2024-04-12T03:05:44.154 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:05:44.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891138.841439_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:05:38.841439Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:05:44Z" } 2024-04-12T03:05:44.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891138.841439_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:05:38.841439Z', 'realm_epoch': 2}}] 2024-04-12T03:05:44.155 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891138.841439_248.1'} 2024-04-12T03:05:44.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891138.841439_248.1'} 2024-04-12T03:05:44.156 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:05:44.156 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:05:44.156 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:05:44.156 DEBUG:boto:Method: HEAD 2024-04-12T03:05:44.156 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:05:44.156 DEBUG:boto:Data: 2024-04-12T03:05:44.156 DEBUG:boto:Headers: {} 2024-04-12T03:05:44.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:05:44.156 DEBUG:boto:Port: 8000 2024-04-12T03:05:44.156 DEBUG:boto:Params: {} 2024-04-12T03:05:44.157 DEBUG:boto:Token: None 2024-04-12T03:05:44.157 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:05:44 GMT /dgbpxg-13/ 2024-04-12T03:05:44.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fi89JE6iW3QNXDoTMoyZfcbD3iA= 2024-04-12T03:05:44.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:05:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fi89JE6iW3QNXDoTMoyZfcbD3iA=', 'Content-Length': '0'} 2024-04-12T03:05:44.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c65411035904cfc9-006618a508-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:44.160 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:05:44.160 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:05:44.160 DEBUG:boto:Method: PUT 2024-04-12T03:05:44.160 DEBUG:boto:Path: /dgbpxg-13/dummy 2024-04-12T03:05:44.160 DEBUG:boto:Data: 2024-04-12T03:05:44.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:05:44.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:05:44.160 DEBUG:boto:Port: 8000 2024-04-12T03:05:44.160 DEBUG:boto:Params: {} 2024-04-12T03:05:44.161 DEBUG:boto:Token: None 2024-04-12T03:05:44.161 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:05:44 GMT /dgbpxg-13/dummy 2024-04-12T03:05:44.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7w+YrIQNhOWVoEZcWCpVdPCEjOE= 2024-04-12T03:05:44.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:05:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7w+YrIQNhOWVoEZcWCpVdPCEjOE='} 2024-04-12T03:05:44.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-amz-request-id', 'tx0000013ea637b32a94a47-006618a508-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:05:44.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:05:44.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:05:44.172 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:05:44.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:05:44.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:05:44.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:44.258+0000 7fe1b8fd7780 1 waiting to reach incremental sync.. 2024-04-12T03:05:49.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:49.259+0000 7fe1b8fd7780 1 waiting to reach incremental sync.. 2024-04-12T03:05:54.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:54.261+0000 7fe1b8fd7780 1 waiting to reach incremental sync.. 2024-04-12T03:05:59.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:05:59.261+0000 7fe1b8fd7780 1 waiting to reach incremental sync.. 2024-04-12T03:06:04.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:04.263+0000 7fe1b8fd7780 1 bucket sync caught up with source: 2024-04-12T03:06:04.265 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2024-04-12T03:06:04.265 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.5.11, , , , , ] 2024-04-12T03:06:04.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:04.263+0000 7fe1b8fd7780 0 bucket checkpoint complete 2024-04-12T03:06:04.281 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:04.281 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:04.281 DEBUG:boto:Method: HEAD 2024-04-12T03:06:04.281 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:04.281 DEBUG:boto:Data: 2024-04-12T03:06:04.281 DEBUG:boto:Headers: {} 2024-04-12T03:06:04.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:04.281 DEBUG:boto:Port: 8001 2024-04-12T03:06:04.281 DEBUG:boto:Params: {} 2024-04-12T03:06:04.282 DEBUG:boto:Token: None 2024-04-12T03:06:04.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:04 GMT /dgbpxg-14/ 2024-04-12T03:06:04.282 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s7X2fdtAfn2BV888vwJvmrMZt88= 2024-04-12T03:06:04.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s7X2fdtAfn2BV888vwJvmrMZt88=', 'Content-Length': '0'} 2024-04-12T03:06:04.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006caa58e494bc5741-006618a51c-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:04.285 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:04.285 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:04.285 DEBUG:boto:Method: PUT 2024-04-12T03:06:04.286 DEBUG:boto:Path: /dgbpxg-14/dummy 2024-04-12T03:06:04.286 DEBUG:boto:Data: 2024-04-12T03:06:04.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:06:04.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:04.286 DEBUG:boto:Port: 8001 2024-04-12T03:06:04.286 DEBUG:boto:Params: {} 2024-04-12T03:06:04.286 DEBUG:boto:Token: None 2024-04-12T03:06:04.286 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:06:04 GMT /dgbpxg-14/dummy 2024-04-12T03:06:04.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UARE66rGHnkub7BcmWiNb83Tbso= 2024-04-12T03:06:04.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UARE66rGHnkub7BcmWiNb83Tbso='} 2024-04-12T03:06:04.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-amz-request-id', 'tx00000297cecfa6b615dc3-006618a51c-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:04.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:04.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:04.299 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:04.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:04.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:04.393+0000 7f8b99201780 1 waiting to reach incremental sync.. 2024-04-12T03:06:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:09.393+0000 7f8b99201780 1 waiting to reach incremental sync.. 2024-04-12T03:06:14.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:14.394+0000 7f8b99201780 1 waiting to reach incremental sync.. 2024-04-12T03:06:19.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:19.396+0000 7f8b99201780 1 bucket sync caught up with source: 2024-04-12T03:06:19.397 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.34.11, , , , , ] 2024-04-12T03:06:19.397 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.34.11, , , , , ] 2024-04-12T03:06:19.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:19.396+0000 7f8b99201780 0 bucket checkpoint complete 2024-04-12T03:06:19.412 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:19.412 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:19.413 DEBUG:boto:Method: HEAD 2024-04-12T03:06:19.413 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:19.413 DEBUG:boto:Data: 2024-04-12T03:06:19.413 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.413 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.413 DEBUG:boto:Params: {} 2024-04-12T03:06:19.414 DEBUG:boto:Token: None 2024-04-12T03:06:19.414 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/ 2024-04-12T03:06:19.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VYwICv3c+k7+iFfKMBc4drLQxzI= 2024-04-12T03:06:19.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VYwICv3c+k7+iFfKMBc4drLQxzI=', 'Content-Length': '0'} 2024-04-12T03:06:19.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee3601e47ebd2d6-006618a52b-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.417 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.417 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.417 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.417 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:06:19.417 DEBUG:boto:Data: 2024-04-12T03:06:19.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.417 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.417 DEBUG:boto:Params: {} 2024-04-12T03:06:19.417 DEBUG:boto:Token: None 2024-04-12T03:06:19.418 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a1 2024-04-12T03:06:19.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5j9TSbMgOiexHVwTjXglycsytxE= 2024-04-12T03:06:19.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5j9TSbMgOiexHVwTjXglycsytxE='} 2024-04-12T03:06:19.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-amz-request-id', 'tx000008475f483263a0a47-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.431 DEBUG:rgw_multi.tests:version1 id=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:19.432 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.432 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.432 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.432 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:06:19.432 DEBUG:boto:Data: 2024-04-12T03:06:19.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.432 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.432 DEBUG:boto:Params: {} 2024-04-12T03:06:19.433 DEBUG:boto:Token: None 2024-04-12T03:06:19.433 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a1 2024-04-12T03:06:19.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f9YYNgQLDgGsHbyEA+eoPImTfgo= 2024-04-12T03:06:19.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f9YYNgQLDgGsHbyEA+eoPImTfgo='} 2024-04-12T03:06:19.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c'), ('x-amz-request-id', 'tx00000b6d72f007743a061-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.444 DEBUG:rgw_multi.tests:version2 id=E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c 2024-04-12T03:06:19.444 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.444 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.444 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c 2024-04-12T03:06:19.444 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c 2024-04-12T03:06:19.444 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.444 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c 2024-04-12T03:06:19.444 DEBUG:boto:Data: 2024-04-12T03:06:19.444 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.445 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.445 DEBUG:boto:Params: {} 2024-04-12T03:06:19.445 DEBUG:boto:Token: None 2024-04-12T03:06:19.445 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a1?versionId=E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c 2024-04-12T03:06:19.445 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1rV9AZdU+B6pzPmmnFkHXM+oEQ0= 2024-04-12T03:06:19.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1rV9AZdU+B6pzPmmnFkHXM+oEQ0=', 'Content-Length': '0'} 2024-04-12T03:06:19.455 DEBUG:boto:Response headers: [('x-amz-version-id', 'E6BZmwY1oMmZf8RjmjU9P0bOu98hm9c'), ('x-amz-request-id', 'tx000005fec4af3a4a9ef85-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.455 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.455 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.455 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.456 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:06:19.456 DEBUG:boto:Data: 2024-04-12T03:06:19.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.456 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.456 DEBUG:boto:Params: {} 2024-04-12T03:06:19.456 DEBUG:boto:Token: None 2024-04-12T03:06:19.456 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a1 2024-04-12T03:06:19.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KnIEU6GePkfLgN51A8may3R44hs= 2024-04-12T03:06:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KnIEU6GePkfLgN51A8may3R44hs='} 2024-04-12T03:06:19.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BKJhDo9vrdH44wd6L7v0WCwsbUljS-K'), ('x-amz-request-id', 'tx00000655760460187566d-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.498 DEBUG:rgw_multi.tests:version3 id=BKJhDo9vrdH44wd6L7v0WCwsbUljS-K 2024-04-12T03:06:19.499 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.499 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:19.499 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=BKJhDo9vrdH44wd6L7v0WCwsbUljS-K 2024-04-12T03:06:19.499 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=BKJhDo9vrdH44wd6L7v0WCwsbUljS-K 2024-04-12T03:06:19.499 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.499 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=BKJhDo9vrdH44wd6L7v0WCwsbUljS-K 2024-04-12T03:06:19.499 DEBUG:boto:Data: 2024-04-12T03:06:19.499 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.499 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.500 DEBUG:boto:Params: {} 2024-04-12T03:06:19.500 DEBUG:boto:Token: None 2024-04-12T03:06:19.500 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a1?versionId=BKJhDo9vrdH44wd6L7v0WCwsbUljS-K 2024-04-12T03:06:19.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGjDmstDmFm6AfQ6UlsX0Mly+s0= 2024-04-12T03:06:19.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGjDmstDmFm6AfQ6UlsX0Mly+s0=', 'Content-Length': '0'} 2024-04-12T03:06:19.508 DEBUG:boto:Response headers: [('x-amz-version-id', 'BKJhDo9vrdH44wd6L7v0WCwsbUljS-K'), ('x-amz-request-id', 'tx0000012efb3e5a258c5e9-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.508 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:19.508 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:19.508 DEBUG:boto:Method: HEAD 2024-04-12T03:06:19.509 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:19.509 DEBUG:boto:Data: 2024-04-12T03:06:19.509 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.509 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.509 DEBUG:boto:Params: {} 2024-04-12T03:06:19.509 DEBUG:boto:Token: None 2024-04-12T03:06:19.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/ 2024-04-12T03:06:19.509 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VYwICv3c+k7+iFfKMBc4drLQxzI= 2024-04-12T03:06:19.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VYwICv3c+k7+iFfKMBc4drLQxzI=', 'Content-Length': '0'} 2024-04-12T03:06:19.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb4f24af04a9eb16-006618a52b-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.512 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.512 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.512 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.512 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:06:19.512 DEBUG:boto:Data: 2024-04-12T03:06:19.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.513 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.513 DEBUG:boto:Params: {} 2024-04-12T03:06:19.513 DEBUG:boto:Token: None 2024-04-12T03:06:19.513 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a2 2024-04-12T03:06:19.513 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lZ2TkpK5u0/R0GqpC7o3281q+Ho= 2024-04-12T03:06:19.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lZ2TkpK5u0/R0GqpC7o3281q+Ho='} 2024-04-12T03:06:19.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-amz-request-id', 'tx00000b189d9b85c451c21-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.527 DEBUG:rgw_multi.tests:version1 id=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:19.527 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.527 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.527 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.527 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:06:19.527 DEBUG:boto:Data: 2024-04-12T03:06:19.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.527 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.527 DEBUG:boto:Params: {} 2024-04-12T03:06:19.528 DEBUG:boto:Token: None 2024-04-12T03:06:19.528 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a2 2024-04-12T03:06:19.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+8DAAk6cpVzer3m1fkNf1BqRo3E= 2024-04-12T03:06:19.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+8DAAk6cpVzer3m1fkNf1BqRo3E='} 2024-04-12T03:06:19.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb'), ('x-amz-request-id', 'tx00000cc7a58aab6c04303-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.570 DEBUG:rgw_multi.tests:version2 id=oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb 2024-04-12T03:06:19.571 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.571 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.571 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb 2024-04-12T03:06:19.571 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb 2024-04-12T03:06:19.571 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.571 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb 2024-04-12T03:06:19.571 DEBUG:boto:Data: 2024-04-12T03:06:19.571 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.571 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.571 DEBUG:boto:Params: {} 2024-04-12T03:06:19.572 DEBUG:boto:Token: None 2024-04-12T03:06:19.572 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a2?versionId=oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb 2024-04-12T03:06:19.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SGNdASJOB+UHKIUQoik8LJq8SMg= 2024-04-12T03:06:19.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SGNdASJOB+UHKIUQoik8LJq8SMg=', 'Content-Length': '0'} 2024-04-12T03:06:19.580 DEBUG:boto:Response headers: [('x-amz-version-id', 'oJhGFu5.GKPs8fMwRYwuCALt1bSAAUb'), ('x-amz-request-id', 'tx000005a702d418b96fccb-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.581 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.581 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.581 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.581 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:06:19.581 DEBUG:boto:Data: 2024-04-12T03:06:19.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.581 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.581 DEBUG:boto:Params: {} 2024-04-12T03:06:19.582 DEBUG:boto:Token: None 2024-04-12T03:06:19.582 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a2 2024-04-12T03:06:19.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJplPAStus263rU72c+U3y+Ke0w= 2024-04-12T03:06:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJplPAStus263rU72c+U3y+Ke0w='} 2024-04-12T03:06:19.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ'), ('x-amz-request-id', 'tx00000ec9930e8c852e187-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.591 DEBUG:rgw_multi.tests:version3 id=yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ 2024-04-12T03:06:19.591 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.591 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:19.591 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ 2024-04-12T03:06:19.591 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ 2024-04-12T03:06:19.591 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.592 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ 2024-04-12T03:06:19.592 DEBUG:boto:Data: 2024-04-12T03:06:19.592 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.592 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.592 DEBUG:boto:Params: {} 2024-04-12T03:06:19.592 DEBUG:boto:Token: None 2024-04-12T03:06:19.592 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-13/obj-a2?versionId=yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ 2024-04-12T03:06:19.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a56XXpITVNKEnjtiIZmI2gek/SU= 2024-04-12T03:06:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a56XXpITVNKEnjtiIZmI2gek/SU=', 'Content-Length': '0'} 2024-04-12T03:06:19.601 DEBUG:boto:Response headers: [('x-amz-version-id', 'yyBRfR4PAxuCY7FxXLkP3a4kchR83GQ'), ('x-amz-request-id', 'tx000003b4e0dda15a790ca-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.601 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:19.601 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:19.601 DEBUG:boto:Method: HEAD 2024-04-12T03:06:19.601 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:19.601 DEBUG:boto:Data: 2024-04-12T03:06:19.601 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.601 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.601 DEBUG:boto:Params: {} 2024-04-12T03:06:19.601 DEBUG:boto:Token: None 2024-04-12T03:06:19.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/ 2024-04-12T03:06:19.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:osnuu0NQYGBjRZxaTG2TrqGIDe4= 2024-04-12T03:06:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:osnuu0NQYGBjRZxaTG2TrqGIDe4=', 'Content-Length': '0'} 2024-04-12T03:06:19.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f727885aac3caa4-006618a52b-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.604 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.604 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.604 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.604 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:06:19.605 DEBUG:boto:Data: 2024-04-12T03:06:19.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.605 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.605 DEBUG:boto:Params: {} 2024-04-12T03:06:19.605 DEBUG:boto:Token: None 2024-04-12T03:06:19.605 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a1 2024-04-12T03:06:19.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:skpr6E1azkjk8kNPeNuv6JZIOBY= 2024-04-12T03:06:19.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:skpr6E1azkjk8kNPeNuv6JZIOBY='} 2024-04-12T03:06:19.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-amz-request-id', 'tx0000095cf4531de54fba9-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.614 DEBUG:rgw_multi.tests:version1 id=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:19.614 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.615 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.615 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.615 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:06:19.615 DEBUG:boto:Data: 2024-04-12T03:06:19.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.615 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.615 DEBUG:boto:Params: {} 2024-04-12T03:06:19.615 DEBUG:boto:Token: None 2024-04-12T03:06:19.615 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a1 2024-04-12T03:06:19.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bbMg67lrkA9U5agBfqkV88lmoZ8= 2024-04-12T03:06:19.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bbMg67lrkA9U5agBfqkV88lmoZ8='} 2024-04-12T03:06:19.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E6u0K9kuymvrDHema14tqeXiVDqM1rY'), ('x-amz-request-id', 'tx000006286afe066f0240d-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.623 DEBUG:rgw_multi.tests:version2 id=E6u0K9kuymvrDHema14tqeXiVDqM1rY 2024-04-12T03:06:19.623 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.623 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.624 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=E6u0K9kuymvrDHema14tqeXiVDqM1rY 2024-04-12T03:06:19.624 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=E6u0K9kuymvrDHema14tqeXiVDqM1rY 2024-04-12T03:06:19.624 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.624 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=E6u0K9kuymvrDHema14tqeXiVDqM1rY 2024-04-12T03:06:19.624 DEBUG:boto:Data: 2024-04-12T03:06:19.624 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.624 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.624 DEBUG:boto:Params: {} 2024-04-12T03:06:19.624 DEBUG:boto:Token: None 2024-04-12T03:06:19.625 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a1?versionId=E6u0K9kuymvrDHema14tqeXiVDqM1rY 2024-04-12T03:06:19.625 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/2WfwaZhef72rnVb6pTh+NN1big= 2024-04-12T03:06:19.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/2WfwaZhef72rnVb6pTh+NN1big=', 'Content-Length': '0'} 2024-04-12T03:06:19.634 DEBUG:boto:Response headers: [('x-amz-version-id', 'E6u0K9kuymvrDHema14tqeXiVDqM1rY'), ('x-amz-request-id', 'tx00000093634cce6b0755b-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.635 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.635 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.635 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.635 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:06:19.635 DEBUG:boto:Data: 2024-04-12T03:06:19.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.635 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.635 DEBUG:boto:Params: {} 2024-04-12T03:06:19.636 DEBUG:boto:Token: None 2024-04-12T03:06:19.636 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a1 2024-04-12T03:06:19.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2T8L4qVJuS23c6JeUbOGDyATnt4= 2024-04-12T03:06:19.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2T8L4qVJuS23c6JeUbOGDyATnt4='} 2024-04-12T03:06:19.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AjdO7u6siXNSjRkSnahCwA2lv8EiXDR'), ('x-amz-request-id', 'tx00000b5502a09eeb20c0f-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.683 DEBUG:rgw_multi.tests:version3 id=AjdO7u6siXNSjRkSnahCwA2lv8EiXDR 2024-04-12T03:06:19.683 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.683 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:19.683 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=AjdO7u6siXNSjRkSnahCwA2lv8EiXDR 2024-04-12T03:06:19.683 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=AjdO7u6siXNSjRkSnahCwA2lv8EiXDR 2024-04-12T03:06:19.683 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.684 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=AjdO7u6siXNSjRkSnahCwA2lv8EiXDR 2024-04-12T03:06:19.684 DEBUG:boto:Data: 2024-04-12T03:06:19.684 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:19.684 DEBUG:boto:Port: 8000 2024-04-12T03:06:19.684 DEBUG:boto:Params: {} 2024-04-12T03:06:19.684 DEBUG:boto:Token: None 2024-04-12T03:06:19.684 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a1?versionId=AjdO7u6siXNSjRkSnahCwA2lv8EiXDR 2024-04-12T03:06:19.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gBCPqN4eC72K1VPA1ThoDjU8N9Q= 2024-04-12T03:06:19.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gBCPqN4eC72K1VPA1ThoDjU8N9Q=', 'Content-Length': '0'} 2024-04-12T03:06:19.693 DEBUG:boto:Response headers: [('x-amz-version-id', 'AjdO7u6siXNSjRkSnahCwA2lv8EiXDR'), ('x-amz-request-id', 'tx000002f10e5cec07bccc3-006618a52b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.694 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:19.694 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:19.694 DEBUG:boto:Method: HEAD 2024-04-12T03:06:19.694 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:19.694 DEBUG:boto:Data: 2024-04-12T03:06:19.694 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.694 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.694 DEBUG:boto:Params: {} 2024-04-12T03:06:19.694 DEBUG:boto:Token: None 2024-04-12T03:06:19.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/ 2024-04-12T03:06:19.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:osnuu0NQYGBjRZxaTG2TrqGIDe4= 2024-04-12T03:06:19.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:osnuu0NQYGBjRZxaTG2TrqGIDe4=', 'Content-Length': '0'} 2024-04-12T03:06:19.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002492c8002ec678cf-006618a52b-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.697 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.697 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.697 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.697 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:06:19.697 DEBUG:boto:Data: 2024-04-12T03:06:19.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.697 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.698 DEBUG:boto:Params: {} 2024-04-12T03:06:19.698 DEBUG:boto:Token: None 2024-04-12T03:06:19.698 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a2 2024-04-12T03:06:19.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zbBa1vE7DGknTzXhje/I60+Y78M= 2024-04-12T03:06:19.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zbBa1vE7DGknTzXhje/I60+Y78M='} 2024-04-12T03:06:19.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-amz-request-id', 'tx000008b7f41b7f16c81d1-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.707 DEBUG:rgw_multi.tests:version1 id=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:19.707 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.707 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.707 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.707 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:06:19.708 DEBUG:boto:Data: 2024-04-12T03:06:19.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.708 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.708 DEBUG:boto:Params: {} 2024-04-12T03:06:19.708 DEBUG:boto:Token: None 2024-04-12T03:06:19.708 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a2 2024-04-12T03:06:19.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ROtiVcW4Mm8fpe9NGnt5mIAxbD8= 2024-04-12T03:06:19.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ROtiVcW4Mm8fpe9NGnt5mIAxbD8='} 2024-04-12T03:06:19.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf'), ('x-amz-request-id', 'tx0000009e68799d4070aac-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.718 DEBUG:rgw_multi.tests:version2 id=rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf 2024-04-12T03:06:19.718 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.718 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.718 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf 2024-04-12T03:06:19.718 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf 2024-04-12T03:06:19.719 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.719 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf 2024-04-12T03:06:19.719 DEBUG:boto:Data: 2024-04-12T03:06:19.719 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.719 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.719 DEBUG:boto:Params: {} 2024-04-12T03:06:19.719 DEBUG:boto:Token: None 2024-04-12T03:06:19.719 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a2?versionId=rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf 2024-04-12T03:06:19.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CEBmDJmuTVj0WGTbbOZfp8hfUnY= 2024-04-12T03:06:19.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CEBmDJmuTVj0WGTbbOZfp8hfUnY=', 'Content-Length': '0'} 2024-04-12T03:06:19.729 DEBUG:boto:Response headers: [('x-amz-version-id', 'rp.6qJyLIsKJD65j6hbD.rqIWpdLIPf'), ('x-amz-request-id', 'tx00000c8eee50ed4480048-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.729 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.730 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.730 DEBUG:boto:Method: PUT 2024-04-12T03:06:19.730 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:06:19.730 DEBUG:boto:Data: 2024-04-12T03:06:19.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:06:19.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.730 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.730 DEBUG:boto:Params: {} 2024-04-12T03:06:19.730 DEBUG:boto:Token: None 2024-04-12T03:06:19.731 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a2 2024-04-12T03:06:19.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fnWYcrCj/4pqsaL+HEMTdhXCzDk= 2024-04-12T03:06:19.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fnWYcrCj/4pqsaL+HEMTdhXCzDk='} 2024-04-12T03:06:19.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF'), ('x-amz-request-id', 'tx0000084230bf8a143f269-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.775 DEBUG:rgw_multi.tests:version3 id=dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF 2024-04-12T03:06:19.775 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.775 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:19.775 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF 2024-04-12T03:06:19.775 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF 2024-04-12T03:06:19.775 DEBUG:boto:Method: DELETE 2024-04-12T03:06:19.775 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF 2024-04-12T03:06:19.775 DEBUG:boto:Data: 2024-04-12T03:06:19.775 DEBUG:boto:Headers: {} 2024-04-12T03:06:19.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:19.776 DEBUG:boto:Port: 8001 2024-04-12T03:06:19.776 DEBUG:boto:Params: {} 2024-04-12T03:06:19.776 DEBUG:boto:Token: None 2024-04-12T03:06:19.776 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:06:19 GMT /dgbpxg-14/obj-a2?versionId=dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF 2024-04-12T03:06:19.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RCY7P9MzFFGpKYpYVPkBU21rzLM= 2024-04-12T03:06:19.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RCY7P9MzFFGpKYpYVPkBU21rzLM=', 'Content-Length': '0'} 2024-04-12T03:06:19.784 DEBUG:boto:Response headers: [('x-amz-version-id', 'dv9VU6VcKGp1rGZ-oBFTdWXlJiJXyzF'), ('x-amz-request-id', 'tx0000060efc7e3389a9c4a-006618a52b-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:19.785 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-13 2024-04-12T03:06:19.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:19.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:19.785 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:19.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:19.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:19.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:19.875+0000 7f1811b44780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:19.877 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2024-04-12T03:06:19.877 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.27.6, ] 2024-04-12T03:06:24.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:24.876+0000 7f1811b44780 1 bucket sync caught up with source: 2024-04-12T03:06:24.879 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , , 00000000010.27.6, ] 2024-04-12T03:06:24.879 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.27.6, ] 2024-04-12T03:06:24.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:24.876+0000 7f1811b44780 0 bucket checkpoint complete 2024-04-12T03:06:24.894 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-13 2024-04-12T03:06:24.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:24.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:24.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:24.955 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:24.955 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:24.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:24.992+0000 7ffb21557780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:24.994 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2024-04-12T03:06:24.994 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.49.6, 00000000002.16.11, ] 2024-04-12T03:06:29.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:29.993+0000 7ffb21557780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:29.996 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2024-04-12T03:06:29.996 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.49.6, 00000000002.16.11, ] 2024-04-12T03:06:34.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:34.996+0000 7ffb21557780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:34.997 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2024-04-12T03:06:34.997 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.49.6, 00000000002.16.11, ] 2024-04-12T03:06:39.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:39.997+0000 7ffb21557780 1 bucket sync caught up with source: 2024-04-12T03:06:39.998 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.49.6, 00000000002.16.11, ] 2024-04-12T03:06:39.999 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.49.6, 00000000002.16.11, ] 2024-04-12T03:06:39.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:39.997+0000 7ffb21557780 0 bucket checkpoint complete 2024-04-12T03:06:40.014 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-13 zones={a2, a1} 2024-04-12T03:06:40.015 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.015 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.015 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.015 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:40.015 DEBUG:boto:Data: 2024-04-12T03:06:40.015 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.015 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.015 DEBUG:boto:Params: {} 2024-04-12T03:06:40.016 DEBUG:boto:Token: None 2024-04-12T03:06:40.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NPe9c96Ou5uCKr56BVh4dR15CIs= 2024-04-12T03:06:40.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NPe9c96Ou5uCKr56BVh4dR15CIs=', 'Content-Length': '0'} 2024-04-12T03:06:40.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff2f6b0b07b68c7-006618a540-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.019 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.019 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.019 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.019 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:40.019 DEBUG:boto:Data: 2024-04-12T03:06:40.019 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.019 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.019 DEBUG:boto:Params: {} 2024-04-12T03:06:40.020 DEBUG:boto:Token: None 2024-04-12T03:06:40.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NPe9c96Ou5uCKr56BVh4dR15CIs= 2024-04-12T03:06:40.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NPe9c96Ou5uCKr56BVh4dR15CIs=', 'Content-Length': '0'} 2024-04-12T03:06:40.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee670f65a6af2a83-006618a540-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.022 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:06:40.022 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.022 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.022 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.023 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.023 DEBUG:boto:Method: GET 2024-04-12T03:06:40.023 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.023 DEBUG:boto:Data: 2024-04-12T03:06:40.023 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.023 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.023 DEBUG:boto:Params: {} 2024-04-12T03:06:40.023 DEBUG:boto:Token: None 2024-04-12T03:06:40.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/?versions 2024-04-12T03:06:40.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw= 2024-04-12T03:06:40.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw=', 'Content-Length': '0'} 2024-04-12T03:06:40.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e64f26b913bc875-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.026 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.027 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:06:40.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:06:40.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:06:40.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:06:40.028 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.028 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.028 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.028 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.028 DEBUG:boto:Method: GET 2024-04-12T03:06:40.028 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.028 DEBUG:boto:Data: 2024-04-12T03:06:40.028 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.028 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.029 DEBUG:boto:Params: {} 2024-04-12T03:06:40.029 DEBUG:boto:Token: None 2024-04-12T03:06:40.029 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/?versions 2024-04-12T03:06:40.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw= 2024-04-12T03:06:40.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw=', 'Content-Length': '0'} 2024-04-12T03:06:40.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050aa79bb8d937c09-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.032 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.032 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:06:40.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:06:40.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:06:40.032 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.033 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.033 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.033 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.033 DEBUG:boto:Method: GET 2024-04-12T03:06:40.033 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.033 DEBUG:boto:Data: 2024-04-12T03:06:40.033 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.033 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.033 DEBUG:boto:Params: {} 2024-04-12T03:06:40.033 DEBUG:boto:Token: None 2024-04-12T03:06:40.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/?versions 2024-04-12T03:06:40.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw= 2024-04-12T03:06:40.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw=', 'Content-Length': '0'} 2024-04-12T03:06:40.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000781c4e38b4378aec-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.036 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.037 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.037 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.037 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.037 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.037 DEBUG:boto:Method: GET 2024-04-12T03:06:40.037 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:06:40.037 DEBUG:boto:Data: 2024-04-12T03:06:40.037 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.037 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.037 DEBUG:boto:Params: {} 2024-04-12T03:06:40.038 DEBUG:boto:Token: None 2024-04-12T03:06:40.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/?versions 2024-04-12T03:06:40.038 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw= 2024-04-12T03:06:40.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8qym1wJBdWw1Ea66b8Y9xv+egnw=', 'Content-Length': '0'} 2024-04-12T03:06:40.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf0478a9e13ee2f-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.040 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:40.041 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.041 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.041 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.041 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.041 DEBUG:boto:Method: GET 2024-04-12T03:06:40.041 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.041 DEBUG:boto:Data: 2024-04-12T03:06:40.042 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.042 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.042 DEBUG:boto:Params: {} 2024-04-12T03:06:40.042 DEBUG:boto:Token: None 2024-04-12T03:06:40.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5a44a6ef397082e-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.044 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.044 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.044 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.044 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.045 DEBUG:boto:Method: GET 2024-04-12T03:06:40.045 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.045 DEBUG:boto:Data: 2024-04-12T03:06:40.045 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.045 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.045 DEBUG:boto:Params: {} 2024-04-12T03:06:40.045 DEBUG:boto:Token: None 2024-04-12T03:06:40.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002889b8983145d0d0-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.047 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.047 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.047 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.047 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.048 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.048 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.048 DEBUG:boto:Data: 2024-04-12T03:06:40.048 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.048 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.048 DEBUG:boto:Params: {} 2024-04-12T03:06:40.048 DEBUG:boto:Token: None 2024-04-12T03:06:40.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vqrWhrb3zpq+Cqylw39PIB5oexE= 2024-04-12T03:06:40.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vqrWhrb3zpq+Cqylw39PIB5oexE=', 'Content-Length': '0'} 2024-04-12T03:06:40.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082ca3b0fcab052ad-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.050 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.051 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.051 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.051 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.051 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.051 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.051 DEBUG:boto:Data: 2024-04-12T03:06:40.051 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.051 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.051 DEBUG:boto:Params: {} 2024-04-12T03:06:40.052 DEBUG:boto:Token: None 2024-04-12T03:06:40.052 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vqrWhrb3zpq+Cqylw39PIB5oexE= 2024-04-12T03:06:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vqrWhrb3zpq+Cqylw39PIB5oexE=', 'Content-Length': '0'} 2024-04-12T03:06:40.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8f6882a3ef438c3-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:40.054 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.054 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.054 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.054 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.054 DEBUG:boto:Method: GET 2024-04-12T03:06:40.054 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.054 DEBUG:boto:Data: 2024-04-12T03:06:40.054 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.054 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.055 DEBUG:boto:Params: {} 2024-04-12T03:06:40.055 DEBUG:boto:Token: None 2024-04-12T03:06:40.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1171d23e49a492f-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.057 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.057 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.057 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.057 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.057 DEBUG:boto:Method: GET 2024-04-12T03:06:40.057 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.058 DEBUG:boto:Data: 2024-04-12T03:06:40.058 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.058 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.058 DEBUG:boto:Params: {} 2024-04-12T03:06:40.058 DEBUG:boto:Token: None 2024-04-12T03:06:40.058 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006900403afaf27600-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.060 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.060 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.060 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.060 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.060 DEBUG:boto:Method: GET 2024-04-12T03:06:40.061 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.061 DEBUG:boto:Data: 2024-04-12T03:06:40.061 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.061 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.061 DEBUG:boto:Params: {} 2024-04-12T03:06:40.061 DEBUG:boto:Token: None 2024-04-12T03:06:40.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7b091de73e6e4a-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.063 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.064 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.064 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.064 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.064 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.064 DEBUG:boto:Method: GET 2024-04-12T03:06:40.064 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.064 DEBUG:boto:Data: 2024-04-12T03:06:40.065 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.065 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.065 DEBUG:boto:Params: {} 2024-04-12T03:06:40.065 DEBUG:boto:Token: None 2024-04-12T03:06:40.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.065 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308be52d1eb6fb19-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.110 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.111 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.111 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.111 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.111 DEBUG:boto:Path: /dgbpxg-13/dummy 2024-04-12T03:06:40.111 DEBUG:boto:Data: 2024-04-12T03:06:40.111 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.111 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.111 DEBUG:boto:Params: {} 2024-04-12T03:06:40.111 DEBUG:boto:Token: None 2024-04-12T03:06:40.111 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy 2024-04-12T03:06:40.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpXHQa11AjXcTv5FefFaLb/Dnz4= 2024-04-12T03:06:40.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpXHQa11AjXcTv5FefFaLb/Dnz4=', 'Content-Length': '0'} 2024-04-12T03:06:40.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec9c9ece8a8d5807-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.114 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.114 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.114 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.114 DEBUG:boto:Path: /dgbpxg-13/dummy 2024-04-12T03:06:40.114 DEBUG:boto:Data: 2024-04-12T03:06:40.115 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.115 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.115 DEBUG:boto:Params: {} 2024-04-12T03:06:40.115 DEBUG:boto:Token: None 2024-04-12T03:06:40.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy 2024-04-12T03:06:40.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpXHQa11AjXcTv5FefFaLb/Dnz4= 2024-04-12T03:06:40.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpXHQa11AjXcTv5FefFaLb/Dnz4=', 'Content-Length': '0'} 2024-04-12T03:06:40.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082317844f2da2262-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:40.117 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.117 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.117 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.117 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.118 DEBUG:boto:Method: GET 2024-04-12T03:06:40.118 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.118 DEBUG:boto:Data: 2024-04-12T03:06:40.118 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.118 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.118 DEBUG:boto:Params: {} 2024-04-12T03:06:40.118 DEBUG:boto:Token: None 2024-04-12T03:06:40.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfed2cdad885da5c-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.120 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:06:40.121 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:06:40.121 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.121 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.121 DEBUG:boto:Method: GET 2024-04-12T03:06:40.121 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.121 DEBUG:boto:Data: 2024-04-12T03:06:40.121 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.121 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.121 DEBUG:boto:Params: {} 2024-04-12T03:06:40.122 DEBUG:boto:Token: None 2024-04-12T03:06:40.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:06:40.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk= 2024-04-12T03:06:40.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJeNsCJtCCStJAP31qgfIxM6Jpk=', 'Content-Length': '0'} 2024-04-12T03:06:40.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057190195f90b974b-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.124 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.124 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.124 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.124 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.124 DEBUG:boto:Method: GET 2024-04-12T03:06:40.124 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.124 DEBUG:boto:Data: 2024-04-12T03:06:40.124 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.125 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.125 DEBUG:boto:Params: {} 2024-04-12T03:06:40.125 DEBUG:boto:Token: None 2024-04-12T03:06:40.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ec4e26e24f0202-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.170 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.171 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.171 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.171 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.171 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.171 DEBUG:boto:Method: GET 2024-04-12T03:06:40.171 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:06:40.171 DEBUG:boto:Data: 2024-04-12T03:06:40.171 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.171 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.172 DEBUG:boto:Params: {} 2024-04-12T03:06:40.172 DEBUG:boto:Token: None 2024-04-12T03:06:40.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac2ded3345c69cd-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.174 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:40.175 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.175 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.175 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.175 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.175 DEBUG:boto:Method: GET 2024-04-12T03:06:40.175 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.175 DEBUG:boto:Data: 2024-04-12T03:06:40.175 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.176 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.176 DEBUG:boto:Params: {} 2024-04-12T03:06:40.176 DEBUG:boto:Token: None 2024-04-12T03:06:40.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061dc95e6aa79f4c0-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.178 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.178 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.178 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.178 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.178 DEBUG:boto:Method: GET 2024-04-12T03:06:40.179 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.179 DEBUG:boto:Data: 2024-04-12T03:06:40.179 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.179 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.179 DEBUG:boto:Params: {} 2024-04-12T03:06:40.179 DEBUG:boto:Token: None 2024-04-12T03:06:40.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009836e1a995595f2b-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.181 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.181 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.181 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.181 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.181 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.181 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.182 DEBUG:boto:Data: 2024-04-12T03:06:40.182 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.182 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.182 DEBUG:boto:Params: {} 2024-04-12T03:06:40.182 DEBUG:boto:Token: None 2024-04-12T03:06:40.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aqq5Xysmzeg97faU7DQ2cxLxKkc= 2024-04-12T03:06:40.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aqq5Xysmzeg97faU7DQ2cxLxKkc=', 'Content-Length': '0'} 2024-04-12T03:06:40.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3c3ea9697b08ec4-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.184 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.184 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.184 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.185 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.185 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.185 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.185 DEBUG:boto:Data: 2024-04-12T03:06:40.185 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.185 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.185 DEBUG:boto:Params: {} 2024-04-12T03:06:40.185 DEBUG:boto:Token: None 2024-04-12T03:06:40.185 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.185 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aqq5Xysmzeg97faU7DQ2cxLxKkc= 2024-04-12T03:06:40.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aqq5Xysmzeg97faU7DQ2cxLxKkc=', 'Content-Length': '0'} 2024-04-12T03:06:40.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000064c6b146ce2b989-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:40.187 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.187 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.188 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.188 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.188 DEBUG:boto:Method: GET 2024-04-12T03:06:40.188 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.188 DEBUG:boto:Data: 2024-04-12T03:06:40.188 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.188 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.188 DEBUG:boto:Params: {} 2024-04-12T03:06:40.188 DEBUG:boto:Token: None 2024-04-12T03:06:40.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f86dfdf918c5a5ef-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.238 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.238 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.238 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.239 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.239 DEBUG:boto:Method: GET 2024-04-12T03:06:40.239 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.239 DEBUG:boto:Data: 2024-04-12T03:06:40.239 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.239 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.239 DEBUG:boto:Params: {} 2024-04-12T03:06:40.239 DEBUG:boto:Token: None 2024-04-12T03:06:40.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cedca3e433a87094-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.242 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.242 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.242 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.242 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.242 DEBUG:boto:Method: GET 2024-04-12T03:06:40.242 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.242 DEBUG:boto:Data: 2024-04-12T03:06:40.242 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.243 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.243 DEBUG:boto:Params: {} 2024-04-12T03:06:40.243 DEBUG:boto:Token: None 2024-04-12T03:06:40.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd921b8310530e27-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.245 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.245 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.246 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.246 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.246 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.246 DEBUG:boto:Method: GET 2024-04-12T03:06:40.246 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.246 DEBUG:boto:Data: 2024-04-12T03:06:40.246 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.246 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.246 DEBUG:boto:Params: {} 2024-04-12T03:06:40.247 DEBUG:boto:Token: None 2024-04-12T03:06:40.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a9fc17e1248253-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.249 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.249 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.250 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.250 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.250 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:06:40.250 DEBUG:boto:Data: 2024-04-12T03:06:40.250 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.250 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.250 DEBUG:boto:Params: {} 2024-04-12T03:06:40.250 DEBUG:boto:Token: None 2024-04-12T03:06:40.250 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1 2024-04-12T03:06:40.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lmIOn5DvZA97b8+PtQivsasojUs= 2024-04-12T03:06:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lmIOn5DvZA97b8+PtQivsasojUs=', 'Content-Length': '0'} 2024-04-12T03:06:40.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000887aea29d95d4407-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.253 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.253 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.253 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.253 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:06:40.253 DEBUG:boto:Data: 2024-04-12T03:06:40.253 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.254 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.254 DEBUG:boto:Params: {} 2024-04-12T03:06:40.254 DEBUG:boto:Token: None 2024-04-12T03:06:40.254 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1 2024-04-12T03:06:40.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lmIOn5DvZA97b8+PtQivsasojUs= 2024-04-12T03:06:40.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lmIOn5DvZA97b8+PtQivsasojUs=', 'Content-Length': '0'} 2024-04-12T03:06:40.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000059dc5b95bcc6f5-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:40.256 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.256 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.256 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.256 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.256 DEBUG:boto:Method: GET 2024-04-12T03:06:40.256 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.257 DEBUG:boto:Data: 2024-04-12T03:06:40.257 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.257 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.257 DEBUG:boto:Params: {} 2024-04-12T03:06:40.257 DEBUG:boto:Token: None 2024-04-12T03:06:40.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c27e33e94d0e9456-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.302 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.302 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:40.303 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.303 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.303 DEBUG:boto:Method: GET 2024-04-12T03:06:40.303 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.303 DEBUG:boto:Data: 2024-04-12T03:06:40.303 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.303 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.303 DEBUG:boto:Params: {} 2024-04-12T03:06:40.303 DEBUG:boto:Token: None 2024-04-12T03:06:40.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:06:40.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g= 2024-04-12T03:06:40.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DP8ofcNuywIaf75Kia1MdmCIn8g=', 'Content-Length': '0'} 2024-04-12T03:06:40.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000389d960c8156827b-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.305 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.306 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.306 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.306 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.306 DEBUG:boto:Method: GET 2024-04-12T03:06:40.306 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.306 DEBUG:boto:Data: 2024-04-12T03:06:40.306 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.306 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.306 DEBUG:boto:Params: {} 2024-04-12T03:06:40.306 DEBUG:boto:Token: None 2024-04-12T03:06:40.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004375265df2807cf6-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.350 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.351 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.351 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.351 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.351 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.351 DEBUG:boto:Method: GET 2024-04-12T03:06:40.351 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:06:40.351 DEBUG:boto:Data: 2024-04-12T03:06:40.351 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.351 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.351 DEBUG:boto:Params: {} 2024-04-12T03:06:40.352 DEBUG:boto:Token: None 2024-04-12T03:06:40.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9956209b65c9281-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.354 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:40.355 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.355 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.355 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.355 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.355 DEBUG:boto:Method: GET 2024-04-12T03:06:40.355 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.355 DEBUG:boto:Data: 2024-04-12T03:06:40.355 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.355 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.356 DEBUG:boto:Params: {} 2024-04-12T03:06:40.356 DEBUG:boto:Token: None 2024-04-12T03:06:40.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4482860e988c9af-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.358 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.358 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.358 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.358 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.358 DEBUG:boto:Method: GET 2024-04-12T03:06:40.358 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.358 DEBUG:boto:Data: 2024-04-12T03:06:40.358 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.359 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.359 DEBUG:boto:Params: {} 2024-04-12T03:06:40.359 DEBUG:boto:Token: None 2024-04-12T03:06:40.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058aea0737e96dd72-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.402 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.402 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.403 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.403 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.403 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.403 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.403 DEBUG:boto:Data: 2024-04-12T03:06:40.403 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.403 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.403 DEBUG:boto:Params: {} 2024-04-12T03:06:40.403 DEBUG:boto:Token: None 2024-04-12T03:06:40.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oDN8rvDj47NzSPOycsP/OtAUuWk= 2024-04-12T03:06:40.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oDN8rvDj47NzSPOycsP/OtAUuWk=', 'Content-Length': '0'} 2024-04-12T03:06:40.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000237a698c6565ce27-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.406 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.406 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.406 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.406 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.406 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.406 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.406 DEBUG:boto:Data: 2024-04-12T03:06:40.406 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.407 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.407 DEBUG:boto:Params: {} 2024-04-12T03:06:40.407 DEBUG:boto:Token: None 2024-04-12T03:06:40.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oDN8rvDj47NzSPOycsP/OtAUuWk= 2024-04-12T03:06:40.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oDN8rvDj47NzSPOycsP/OtAUuWk=', 'Content-Length': '0'} 2024-04-12T03:06:40.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000565c45b62e59e96f-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:40.409 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.409 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.409 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.409 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.410 DEBUG:boto:Method: GET 2024-04-12T03:06:40.410 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.410 DEBUG:boto:Data: 2024-04-12T03:06:40.410 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.410 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.410 DEBUG:boto:Params: {} 2024-04-12T03:06:40.410 DEBUG:boto:Token: None 2024-04-12T03:06:40.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001603374f823c03b5-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.412 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.412 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.412 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.412 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.413 DEBUG:boto:Method: GET 2024-04-12T03:06:40.413 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.413 DEBUG:boto:Data: 2024-04-12T03:06:40.413 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.413 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.413 DEBUG:boto:Params: {} 2024-04-12T03:06:40.413 DEBUG:boto:Token: None 2024-04-12T03:06:40.413 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d01e7f73c40c997f-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.458 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.458 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.458 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.459 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.459 DEBUG:boto:Method: GET 2024-04-12T03:06:40.459 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.459 DEBUG:boto:Data: 2024-04-12T03:06:40.459 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.459 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.459 DEBUG:boto:Params: {} 2024-04-12T03:06:40.459 DEBUG:boto:Token: None 2024-04-12T03:06:40.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006065a072de7ca3b7-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.462 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.463 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.463 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.463 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.463 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.463 DEBUG:boto:Method: GET 2024-04-12T03:06:40.463 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.463 DEBUG:boto:Data: 2024-04-12T03:06:40.463 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.463 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.464 DEBUG:boto:Params: {} 2024-04-12T03:06:40.464 DEBUG:boto:Token: None 2024-04-12T03:06:40.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d085ebc44e5f1aae-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.467 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.467 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.467 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.467 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.467 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:06:40.467 DEBUG:boto:Data: 2024-04-12T03:06:40.467 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.468 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.468 DEBUG:boto:Params: {} 2024-04-12T03:06:40.468 DEBUG:boto:Token: None 2024-04-12T03:06:40.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2 2024-04-12T03:06:40.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NpOYMMeGYzVb0xcbFVoYhv8xgrw= 2024-04-12T03:06:40.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NpOYMMeGYzVb0xcbFVoYhv8xgrw=', 'Content-Length': '0'} 2024-04-12T03:06:40.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048338acbbeed91fa-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.470 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.470 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.471 DEBUG:boto:Method: HEAD 2024-04-12T03:06:40.471 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:06:40.471 DEBUG:boto:Data: 2024-04-12T03:06:40.471 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.471 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.471 DEBUG:boto:Params: {} 2024-04-12T03:06:40.471 DEBUG:boto:Token: None 2024-04-12T03:06:40.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2 2024-04-12T03:06:40.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NpOYMMeGYzVb0xcbFVoYhv8xgrw= 2024-04-12T03:06:40.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NpOYMMeGYzVb0xcbFVoYhv8xgrw=', 'Content-Length': '0'} 2024-04-12T03:06:40.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000719b66eadc97bdb4-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:40.474 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.474 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.474 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.474 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.474 DEBUG:boto:Method: GET 2024-04-12T03:06:40.474 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.474 DEBUG:boto:Data: 2024-04-12T03:06:40.474 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.475 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.475 DEBUG:boto:Params: {} 2024-04-12T03:06:40.475 DEBUG:boto:Token: None 2024-04-12T03:06:40.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b82fae47a3fd16b-006618a540-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.477 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.477 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:40.477 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.477 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.478 DEBUG:boto:Method: GET 2024-04-12T03:06:40.478 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.478 DEBUG:boto:Data: 2024-04-12T03:06:40.478 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.478 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.478 DEBUG:boto:Params: {} 2024-04-12T03:06:40.478 DEBUG:boto:Token: None 2024-04-12T03:06:40.478 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:06:40.478 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g= 2024-04-12T03:06:40.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pCkarGOUXkTznJqWhcuOCS+nt4g=', 'Content-Length': '0'} 2024-04-12T03:06:40.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b02a5abf98b0fbbe-006618a540-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.522 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.523 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.523 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.523 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.523 DEBUG:boto:Method: GET 2024-04-12T03:06:40.523 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.523 DEBUG:boto:Data: 2024-04-12T03:06:40.523 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:40.523 DEBUG:boto:Port: 8001 2024-04-12T03:06:40.523 DEBUG:boto:Params: {} 2024-04-12T03:06:40.523 DEBUG:boto:Token: None 2024-04-12T03:06:40.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b202107303065e05-006618a540-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.526 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.526 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:40.526 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:40.526 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.527 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.527 DEBUG:boto:Method: GET 2024-04-12T03:06:40.527 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:06:40.527 DEBUG:boto:Data: 2024-04-12T03:06:40.527 DEBUG:boto:Headers: {} 2024-04-12T03:06:40.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:40.527 DEBUG:boto:Port: 8000 2024-04-12T03:06:40.527 DEBUG:boto:Params: {} 2024-04-12T03:06:40.527 DEBUG:boto:Token: None 2024-04-12T03:06:40.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:40 GMT /dgbpxg-13/ 2024-04-12T03:06:40.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc= 2024-04-12T03:06:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MbR3fcf84UK4B2m+2HP+lSDVugc=', 'Content-Length': '0'} 2024-04-12T03:06:40.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e75b77daaec9341b-006618a540-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:40.530 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:40.530 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-13 zones={a2, a1} 2024-04-12T03:06:40.530 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-14 2024-04-12T03:06:40.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:40.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:40.531 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:40.625+0000 7f76c6ad7780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:40.626 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.19.11, , , , 00000000010.37.6, ] 2024-04-12T03:06:40.626 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000002.26.11, 00000000010.37.6, ] 2024-04-12T03:06:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:45.626+0000 7f76c6ad7780 1 bucket sync caught up with source: 2024-04-12T03:06:45.628 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000002.26.11, 00000000010.37.6, ] 2024-04-12T03:06:45.628 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000002.26.11, 00000000010.37.6, ] 2024-04-12T03:06:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:45.626+0000 7f76c6ad7780 0 bucket checkpoint complete 2024-04-12T03:06:45.642 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-14 2024-04-12T03:06:45.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:45.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:45.643 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:45.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:45.736+0000 7f498881e780 1 bucket sync caught up with source: 2024-04-12T03:06:45.737 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000010.55.6, 00000000002.41.11, ] 2024-04-12T03:06:45.737 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000010.55.6, 00000000002.41.11, ] 2024-04-12T03:06:45.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:45.736+0000 7f498881e780 0 bucket checkpoint complete 2024-04-12T03:06:45.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-14 zones={a2, a1} 2024-04-12T03:06:45.749 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.749 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.750 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.750 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:45.750 DEBUG:boto:Data: 2024-04-12T03:06:45.750 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.750 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.750 DEBUG:boto:Params: {} 2024-04-12T03:06:45.750 DEBUG:boto:Token: None 2024-04-12T03:06:45.750 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VI4G3EbvjmXG1imdePb9/gcxNaM= 2024-04-12T03:06:45.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VI4G3EbvjmXG1imdePb9/gcxNaM=', 'Content-Length': '0'} 2024-04-12T03:06:45.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8ba8f79bd5b5fe1-006618a545-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.753 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.753 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.753 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.753 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:45.753 DEBUG:boto:Data: 2024-04-12T03:06:45.753 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.754 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.754 DEBUG:boto:Params: {} 2024-04-12T03:06:45.754 DEBUG:boto:Token: None 2024-04-12T03:06:45.754 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VI4G3EbvjmXG1imdePb9/gcxNaM= 2024-04-12T03:06:45.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VI4G3EbvjmXG1imdePb9/gcxNaM=', 'Content-Length': '0'} 2024-04-12T03:06:45.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb2d7c7de1dd439b-006618a545-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.756 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:06:45.756 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.757 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.757 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.757 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.757 DEBUG:boto:Method: GET 2024-04-12T03:06:45.757 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.757 DEBUG:boto:Data: 2024-04-12T03:06:45.757 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.757 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.757 DEBUG:boto:Params: {} 2024-04-12T03:06:45.757 DEBUG:boto:Token: None 2024-04-12T03:06:45.757 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/?versions 2024-04-12T03:06:45.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E= 2024-04-12T03:06:45.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E=', 'Content-Length': '0'} 2024-04-12T03:06:45.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da91aaa538717506-006618a545-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.760 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.760 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:06:45.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:06:45.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:06:45.760 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:06:45.761 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.761 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.761 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.761 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.761 DEBUG:boto:Method: GET 2024-04-12T03:06:45.761 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.761 DEBUG:boto:Data: 2024-04-12T03:06:45.761 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.761 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.761 DEBUG:boto:Params: {} 2024-04-12T03:06:45.762 DEBUG:boto:Token: None 2024-04-12T03:06:45.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/?versions 2024-04-12T03:06:45.762 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E= 2024-04-12T03:06:45.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E=', 'Content-Length': '0'} 2024-04-12T03:06:45.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006086246f0b6d0056-006618a545-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.764 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.764 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:06:45.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:06:45.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:06:45.764 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.765 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.765 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.765 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.765 DEBUG:boto:Method: GET 2024-04-12T03:06:45.765 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.765 DEBUG:boto:Data: 2024-04-12T03:06:45.765 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.765 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.765 DEBUG:boto:Params: {} 2024-04-12T03:06:45.765 DEBUG:boto:Token: None 2024-04-12T03:06:45.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/?versions 2024-04-12T03:06:45.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E= 2024-04-12T03:06:45.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E=', 'Content-Length': '0'} 2024-04-12T03:06:45.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c13bf2738c2d81-006618a545-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.768 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.768 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.768 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.768 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.768 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.768 DEBUG:boto:Method: GET 2024-04-12T03:06:45.769 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:06:45.769 DEBUG:boto:Data: 2024-04-12T03:06:45.769 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.769 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.769 DEBUG:boto:Params: {} 2024-04-12T03:06:45.769 DEBUG:boto:Token: None 2024-04-12T03:06:45.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/?versions 2024-04-12T03:06:45.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E= 2024-04-12T03:06:45.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aX37CFYWQ5cmPMjCfS4GfJk0E+E=', 'Content-Length': '0'} 2024-04-12T03:06:45.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aec8039cdfad22da-006618a545-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.771 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:45.772 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.772 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.772 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.772 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.772 DEBUG:boto:Method: GET 2024-04-12T03:06:45.772 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.772 DEBUG:boto:Data: 2024-04-12T03:06:45.772 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.773 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.773 DEBUG:boto:Params: {} 2024-04-12T03:06:45.773 DEBUG:boto:Token: None 2024-04-12T03:06:45.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebdc77ba7f02cf7a-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.775 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.775 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.775 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.775 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.775 DEBUG:boto:Method: GET 2024-04-12T03:06:45.775 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.775 DEBUG:boto:Data: 2024-04-12T03:06:45.775 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.776 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.776 DEBUG:boto:Params: {} 2024-04-12T03:06:45.776 DEBUG:boto:Token: None 2024-04-12T03:06:45.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0883d5bfecb36fb-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.778 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.778 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.778 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.778 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.778 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.778 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.778 DEBUG:boto:Data: 2024-04-12T03:06:45.778 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.779 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.779 DEBUG:boto:Params: {} 2024-04-12T03:06:45.779 DEBUG:boto:Token: None 2024-04-12T03:06:45.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:70FRrt5YThJh4T3ZDBfUF5smE0Q= 2024-04-12T03:06:45.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:70FRrt5YThJh4T3ZDBfUF5smE0Q=', 'Content-Length': '0'} 2024-04-12T03:06:45.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ac8a75a8ba5656f-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.781 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.781 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.781 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.781 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.781 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.781 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.781 DEBUG:boto:Data: 2024-04-12T03:06:45.781 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.782 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.782 DEBUG:boto:Params: {} 2024-04-12T03:06:45.782 DEBUG:boto:Token: None 2024-04-12T03:06:45.782 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:70FRrt5YThJh4T3ZDBfUF5smE0Q= 2024-04-12T03:06:45.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:70FRrt5YThJh4T3ZDBfUF5smE0Q=', 'Content-Length': '0'} 2024-04-12T03:06:45.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b40782a36208339d-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:45.784 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.784 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.784 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.784 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.784 DEBUG:boto:Method: GET 2024-04-12T03:06:45.784 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.784 DEBUG:boto:Data: 2024-04-12T03:06:45.785 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.785 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.785 DEBUG:boto:Params: {} 2024-04-12T03:06:45.785 DEBUG:boto:Token: None 2024-04-12T03:06:45.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b57fee5eee0c4c6-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.787 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.787 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.787 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.787 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.787 DEBUG:boto:Method: GET 2024-04-12T03:06:45.787 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.787 DEBUG:boto:Data: 2024-04-12T03:06:45.787 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.787 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.788 DEBUG:boto:Params: {} 2024-04-12T03:06:45.788 DEBUG:boto:Token: None 2024-04-12T03:06:45.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062bf0520cad8ca6c-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.790 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.790 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.790 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.790 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.790 DEBUG:boto:Method: GET 2024-04-12T03:06:45.790 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.790 DEBUG:boto:Data: 2024-04-12T03:06:45.790 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.790 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.790 DEBUG:boto:Params: {} 2024-04-12T03:06:45.791 DEBUG:boto:Token: None 2024-04-12T03:06:45.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c682d31e5400204-006618a545-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.834 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.835 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.835 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.835 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.835 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.835 DEBUG:boto:Method: GET 2024-04-12T03:06:45.835 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.835 DEBUG:boto:Data: 2024-04-12T03:06:45.835 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.835 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.836 DEBUG:boto:Params: {} 2024-04-12T03:06:45.836 DEBUG:boto:Token: None 2024-04-12T03:06:45.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8498224a2dbfc0a-006618a545-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.838 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.838 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.839 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.839 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.839 DEBUG:boto:Path: /dgbpxg-14/dummy 2024-04-12T03:06:45.839 DEBUG:boto:Data: 2024-04-12T03:06:45.839 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.839 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.839 DEBUG:boto:Params: {} 2024-04-12T03:06:45.839 DEBUG:boto:Token: None 2024-04-12T03:06:45.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy 2024-04-12T03:06:45.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KZLWmXm/31J02At8izHJ7HcTDlc= 2024-04-12T03:06:45.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KZLWmXm/31J02At8izHJ7HcTDlc=', 'Content-Length': '0'} 2024-04-12T03:06:45.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045f96d70dd1b8263-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.842 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.842 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.842 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.842 DEBUG:boto:Path: /dgbpxg-14/dummy 2024-04-12T03:06:45.842 DEBUG:boto:Data: 2024-04-12T03:06:45.842 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.842 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.843 DEBUG:boto:Params: {} 2024-04-12T03:06:45.843 DEBUG:boto:Token: None 2024-04-12T03:06:45.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy 2024-04-12T03:06:45.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KZLWmXm/31J02At8izHJ7HcTDlc= 2024-04-12T03:06:45.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KZLWmXm/31J02At8izHJ7HcTDlc=', 'Content-Length': '0'} 2024-04-12T03:06:45.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006237404729de9682-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:06:45.845 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.845 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.845 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.845 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.846 DEBUG:boto:Method: GET 2024-04-12T03:06:45.846 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.846 DEBUG:boto:Data: 2024-04-12T03:06:45.846 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.846 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.846 DEBUG:boto:Params: {} 2024-04-12T03:06:45.846 DEBUG:boto:Token: None 2024-04-12T03:06:45.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c736bbf018018034-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.848 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:06:45.848 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:06:45.848 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.849 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.849 DEBUG:boto:Method: GET 2024-04-12T03:06:45.849 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.849 DEBUG:boto:Data: 2024-04-12T03:06:45.849 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.849 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.849 DEBUG:boto:Params: {} 2024-04-12T03:06:45.849 DEBUG:boto:Token: None 2024-04-12T03:06:45.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:06:45.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is= 2024-04-12T03:06:45.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c3F4XCXp8fHMibywqA9tJtI51Is=', 'Content-Length': '0'} 2024-04-12T03:06:45.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084e6ad45f55b3c19-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.851 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.851 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.852 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.852 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.852 DEBUG:boto:Method: GET 2024-04-12T03:06:45.852 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.852 DEBUG:boto:Data: 2024-04-12T03:06:45.852 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.852 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.852 DEBUG:boto:Params: {} 2024-04-12T03:06:45.852 DEBUG:boto:Token: None 2024-04-12T03:06:45.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13a6669fd4c8b35-006618a545-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.855 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.855 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.855 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.855 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.855 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.855 DEBUG:boto:Method: GET 2024-04-12T03:06:45.855 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:06:45.855 DEBUG:boto:Data: 2024-04-12T03:06:45.856 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.856 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.856 DEBUG:boto:Params: {} 2024-04-12T03:06:45.856 DEBUG:boto:Token: None 2024-04-12T03:06:45.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d36205a4468285-006618a545-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.898 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:45.899 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.899 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.899 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.899 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.899 DEBUG:boto:Method: GET 2024-04-12T03:06:45.899 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.899 DEBUG:boto:Data: 2024-04-12T03:06:45.899 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.899 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.899 DEBUG:boto:Params: {} 2024-04-12T03:06:45.900 DEBUG:boto:Token: None 2024-04-12T03:06:45.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc= 2024-04-12T03:06:45.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc=', 'Content-Length': '0'} 2024-04-12T03:06:45.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d330ff0f1eb1d073-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.902 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.902 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.902 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.902 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.903 DEBUG:boto:Method: GET 2024-04-12T03:06:45.903 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.903 DEBUG:boto:Data: 2024-04-12T03:06:45.903 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.903 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.903 DEBUG:boto:Params: {} 2024-04-12T03:06:45.903 DEBUG:boto:Token: None 2024-04-12T03:06:45.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc= 2024-04-12T03:06:45.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc=', 'Content-Length': '0'} 2024-04-12T03:06:45.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004680606f1275fbff-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.905 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.905 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.905 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.905 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.906 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.906 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.906 DEBUG:boto:Data: 2024-04-12T03:06:45.906 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.906 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.906 DEBUG:boto:Params: {} 2024-04-12T03:06:45.906 DEBUG:boto:Token: None 2024-04-12T03:06:45.906 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.906 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4fwnN1FoL+jt1NfYs7IErU1Je30= 2024-04-12T03:06:45.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4fwnN1FoL+jt1NfYs7IErU1Je30=', 'Content-Length': '0'} 2024-04-12T03:06:45.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000113670f8558c24a3-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.908 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.908 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.909 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.909 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.909 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.909 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.909 DEBUG:boto:Data: 2024-04-12T03:06:45.909 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.909 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.909 DEBUG:boto:Params: {} 2024-04-12T03:06:45.909 DEBUG:boto:Token: None 2024-04-12T03:06:45.909 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.910 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4fwnN1FoL+jt1NfYs7IErU1Je30= 2024-04-12T03:06:45.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4fwnN1FoL+jt1NfYs7IErU1Je30=', 'Content-Length': '0'} 2024-04-12T03:06:45.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000250b6d230e271e69-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:45.912 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.912 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.912 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.912 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.912 DEBUG:boto:Method: GET 2024-04-12T03:06:45.912 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.912 DEBUG:boto:Data: 2024-04-12T03:06:45.912 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.912 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.912 DEBUG:boto:Params: {} 2024-04-12T03:06:45.913 DEBUG:boto:Token: None 2024-04-12T03:06:45.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.913 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc= 2024-04-12T03:06:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc=', 'Content-Length': '0'} 2024-04-12T03:06:45.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000439d7e22231327c7-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.958 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.958 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.958 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.959 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.959 DEBUG:boto:Method: GET 2024-04-12T03:06:45.959 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.959 DEBUG:boto:Data: 2024-04-12T03:06:45.959 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.959 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.959 DEBUG:boto:Params: {} 2024-04-12T03:06:45.959 DEBUG:boto:Token: None 2024-04-12T03:06:45.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc= 2024-04-12T03:06:45.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc=', 'Content-Length': '0'} 2024-04-12T03:06:45.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093a48786deeba502-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.961 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.961 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.962 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.962 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.962 DEBUG:boto:Method: GET 2024-04-12T03:06:45.962 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.962 DEBUG:boto:Data: 2024-04-12T03:06:45.962 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.962 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.962 DEBUG:boto:Params: {} 2024-04-12T03:06:45.962 DEBUG:boto:Token: None 2024-04-12T03:06:45.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdce88588d7ccb34-006618a545-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.965 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.965 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:45.965 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:45.966 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.966 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.966 DEBUG:boto:Method: GET 2024-04-12T03:06:45.966 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:45.966 DEBUG:boto:Data: 2024-04-12T03:06:45.966 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.966 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.966 DEBUG:boto:Params: {} 2024-04-12T03:06:45.966 DEBUG:boto:Token: None 2024-04-12T03:06:45.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/ 2024-04-12T03:06:45.967 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs= 2024-04-12T03:06:45.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOegZYW/xh9oksP3rIuiGagtHUs=', 'Content-Length': '0'} 2024-04-12T03:06:45.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010751e3eb0550c18-006618a545-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.969 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:45.969 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.969 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.969 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.969 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:06:45.970 DEBUG:boto:Data: 2024-04-12T03:06:45.970 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.970 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.970 DEBUG:boto:Params: {} 2024-04-12T03:06:45.970 DEBUG:boto:Token: None 2024-04-12T03:06:45.970 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1 2024-04-12T03:06:45.970 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fPHoakaZde8b22i+NbTuAgSfup8= 2024-04-12T03:06:45.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fPHoakaZde8b22i+NbTuAgSfup8=', 'Content-Length': '0'} 2024-04-12T03:06:45.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003996c8b54e4ba355-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.972 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.973 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.973 DEBUG:boto:Method: HEAD 2024-04-12T03:06:45.973 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:06:45.973 DEBUG:boto:Data: 2024-04-12T03:06:45.973 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:45.973 DEBUG:boto:Port: 8000 2024-04-12T03:06:45.973 DEBUG:boto:Params: {} 2024-04-12T03:06:45.973 DEBUG:boto:Token: None 2024-04-12T03:06:45.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1 2024-04-12T03:06:45.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fPHoakaZde8b22i+NbTuAgSfup8= 2024-04-12T03:06:45.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fPHoakaZde8b22i+NbTuAgSfup8=', 'Content-Length': '0'} 2024-04-12T03:06:45.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c105394077d1a2f9-006618a545-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:45.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:06:45.976 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.976 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:45.976 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.976 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.976 DEBUG:boto:Method: GET 2024-04-12T03:06:45.976 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.976 DEBUG:boto:Data: 2024-04-12T03:06:45.977 DEBUG:boto:Headers: {} 2024-04-12T03:06:45.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:45.977 DEBUG:boto:Port: 8001 2024-04-12T03:06:45.977 DEBUG:boto:Params: {} 2024-04-12T03:06:45.977 DEBUG:boto:Token: None 2024-04-12T03:06:45.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:45 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:45.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc= 2024-04-12T03:06:45.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yideiftWmhxeolraOntiyeV01gc=', 'Content-Length': '0'} 2024-04-12T03:06:45.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd5b3a2f8b28835c-006618a545-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.022 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:46.022 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:46.022 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:46.023 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:46.023 DEBUG:boto:Method: GET 2024-04-12T03:06:46.023 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:46.023 DEBUG:boto:Data: 2024-04-12T03:06:46.023 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.023 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.023 DEBUG:boto:Params: {} 2024-04-12T03:06:46.023 DEBUG:boto:Token: None 2024-04-12T03:06:46.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:06:46.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6Eptmip23ebAi6QS5hJbkS16r0M= 2024-04-12T03:06:46.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6Eptmip23ebAi6QS5hJbkS16r0M=', 'Content-Length': '0'} 2024-04-12T03:06:46.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022dd4a2bd0ec5a21-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.025 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.025 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.026 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.026 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.026 DEBUG:boto:Method: GET 2024-04-12T03:06:46.026 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.026 DEBUG:boto:Data: 2024-04-12T03:06:46.026 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.026 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.026 DEBUG:boto:Params: {} 2024-04-12T03:06:46.026 DEBUG:boto:Token: None 2024-04-12T03:06:46.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7baf6c2085407d9-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.070 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.071 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.071 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.071 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.071 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.071 DEBUG:boto:Method: GET 2024-04-12T03:06:46.071 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:06:46.071 DEBUG:boto:Data: 2024-04-12T03:06:46.071 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.071 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.071 DEBUG:boto:Params: {} 2024-04-12T03:06:46.072 DEBUG:boto:Token: None 2024-04-12T03:06:46.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000581e4f9d7b81fd02-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.074 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:46.075 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.075 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.075 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.075 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.075 DEBUG:boto:Method: GET 2024-04-12T03:06:46.075 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.075 DEBUG:boto:Data: 2024-04-12T03:06:46.075 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.075 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.075 DEBUG:boto:Params: {} 2024-04-12T03:06:46.076 DEBUG:boto:Token: None 2024-04-12T03:06:46.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac3456410f46ef9d-006618a546-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.078 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.078 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.078 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.078 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.078 DEBUG:boto:Method: GET 2024-04-12T03:06:46.079 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.079 DEBUG:boto:Data: 2024-04-12T03:06:46.079 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.079 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.079 DEBUG:boto:Params: {} 2024-04-12T03:06:46.079 DEBUG:boto:Token: None 2024-04-12T03:06:46.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d660a590ecac7b4c-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.081 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.081 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.081 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.082 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.082 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.082 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.082 DEBUG:boto:Data: 2024-04-12T03:06:46.082 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.082 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.082 DEBUG:boto:Params: {} 2024-04-12T03:06:46.082 DEBUG:boto:Token: None 2024-04-12T03:06:46.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q+2Vm6N92lb8fKnlFgAF80isUpY= 2024-04-12T03:06:46.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q+2Vm6N92lb8fKnlFgAF80isUpY=', 'Content-Length': '0'} 2024-04-12T03:06:46.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43e39abd1832a53-006618a546-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.085 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.085 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.085 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.085 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.085 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.085 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.085 DEBUG:boto:Data: 2024-04-12T03:06:46.085 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.085 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.086 DEBUG:boto:Params: {} 2024-04-12T03:06:46.086 DEBUG:boto:Token: None 2024-04-12T03:06:46.086 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q+2Vm6N92lb8fKnlFgAF80isUpY= 2024-04-12T03:06:46.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q+2Vm6N92lb8fKnlFgAF80isUpY=', 'Content-Length': '0'} 2024-04-12T03:06:46.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007054b16fcf4955d7-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:46.088 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.088 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.088 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.088 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.088 DEBUG:boto:Method: GET 2024-04-12T03:06:46.088 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.088 DEBUG:boto:Data: 2024-04-12T03:06:46.088 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.089 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.089 DEBUG:boto:Params: {} 2024-04-12T03:06:46.089 DEBUG:boto:Token: None 2024-04-12T03:06:46.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d579b5eafd6a156-006618a546-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.091 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.091 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.091 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.091 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.091 DEBUG:boto:Method: GET 2024-04-12T03:06:46.091 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.091 DEBUG:boto:Data: 2024-04-12T03:06:46.091 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.092 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.092 DEBUG:boto:Params: {} 2024-04-12T03:06:46.092 DEBUG:boto:Token: None 2024-04-12T03:06:46.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004de55ddc63564dbc-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.134 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.134 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.134 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.135 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.135 DEBUG:boto:Method: GET 2024-04-12T03:06:46.135 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.135 DEBUG:boto:Data: 2024-04-12T03:06:46.135 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.135 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.135 DEBUG:boto:Params: {} 2024-04-12T03:06:46.135 DEBUG:boto:Token: None 2024-04-12T03:06:46.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd37b76c353075f-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.138 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.138 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.138 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.139 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.139 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.139 DEBUG:boto:Method: GET 2024-04-12T03:06:46.139 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.139 DEBUG:boto:Data: 2024-04-12T03:06:46.139 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.139 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.139 DEBUG:boto:Params: {} 2024-04-12T03:06:46.139 DEBUG:boto:Token: None 2024-04-12T03:06:46.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf8f0f9834823a5-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.142 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.142 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.142 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.142 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.142 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:06:46.142 DEBUG:boto:Data: 2024-04-12T03:06:46.142 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.143 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.143 DEBUG:boto:Params: {} 2024-04-12T03:06:46.143 DEBUG:boto:Token: None 2024-04-12T03:06:46.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2 2024-04-12T03:06:46.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m5kGMnlbHE5BlL+UQswnJ4lgMd0= 2024-04-12T03:06:46.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m5kGMnlbHE5BlL+UQswnJ4lgMd0=', 'Content-Length': '0'} 2024-04-12T03:06:46.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000030f075b931c37bc-006618a546-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.145 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.145 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.145 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.145 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:06:46.145 DEBUG:boto:Data: 2024-04-12T03:06:46.145 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.146 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.146 DEBUG:boto:Params: {} 2024-04-12T03:06:46.146 DEBUG:boto:Token: None 2024-04-12T03:06:46.146 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2 2024-04-12T03:06:46.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m5kGMnlbHE5BlL+UQswnJ4lgMd0= 2024-04-12T03:06:46.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m5kGMnlbHE5BlL+UQswnJ4lgMd0=', 'Content-Length': '0'} 2024-04-12T03:06:46.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f67d101b3f68896b-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:06:46.149 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.149 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.149 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.149 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.149 DEBUG:boto:Method: GET 2024-04-12T03:06:46.149 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.149 DEBUG:boto:Data: 2024-04-12T03:06:46.149 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.149 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.149 DEBUG:boto:Params: {} 2024-04-12T03:06:46.149 DEBUG:boto:Token: None 2024-04-12T03:06:46.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfa647db3cd98dd8-006618a546-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.152 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.152 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.152 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.152 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.152 DEBUG:boto:Method: GET 2024-04-12T03:06:46.152 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.152 DEBUG:boto:Data: 2024-04-12T03:06:46.152 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.152 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.152 DEBUG:boto:Params: {} 2024-04-12T03:06:46.153 DEBUG:boto:Token: None 2024-04-12T03:06:46.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:06:46.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg= 2024-04-12T03:06:46.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JavxB8uHn06Vlw5euPnTLB7NZrg=', 'Content-Length': '0'} 2024-04-12T03:06:46.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f257fe311497865-006618a546-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.198 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.198 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.198 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.199 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.199 DEBUG:boto:Method: GET 2024-04-12T03:06:46.199 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.199 DEBUG:boto:Data: 2024-04-12T03:06:46.199 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.199 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.199 DEBUG:boto:Params: {} 2024-04-12T03:06:46.199 DEBUG:boto:Token: None 2024-04-12T03:06:46.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a53a378c9098d7-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.202 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.202 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.202 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.202 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.202 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.203 DEBUG:boto:Method: GET 2024-04-12T03:06:46.203 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:06:46.203 DEBUG:boto:Data: 2024-04-12T03:06:46.203 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.203 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.203 DEBUG:boto:Params: {} 2024-04-12T03:06:46.203 DEBUG:boto:Token: None 2024-04-12T03:06:46.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU= 2024-04-12T03:06:46.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pl9L34t0/+RMHyMMnj+lH5gapwU=', 'Content-Length': '0'} 2024-04-12T03:06:46.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c8e0bb1511542f-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.246 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-14 zones={a2, a1} 2024-04-12T03:06:46.247 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:46.247 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:46.247 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.247 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:46.247 DEBUG:boto:Data: 2024-04-12T03:06:46.247 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.247 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.248 DEBUG:boto:Params: {} 2024-04-12T03:06:46.248 DEBUG:boto:Token: None 2024-04-12T03:06:46.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-13/ 2024-04-12T03:06:46.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6RnGozRmcnYsMoJTupdywPHISxo= 2024-04-12T03:06:46.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6RnGozRmcnYsMoJTupdywPHISxo=', 'Content-Length': '0'} 2024-04-12T03:06:46.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e2e44706667c7d0-006618a546-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.250 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:46.250 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:46.251 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.251 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.251 DEBUG:boto:Method: GET 2024-04-12T03:06:46.251 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.251 DEBUG:boto:Data: 2024-04-12T03:06:46.251 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.251 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.251 DEBUG:boto:Params: {} 2024-04-12T03:06:46.251 DEBUG:boto:Token: None 2024-04-12T03:06:46.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-13/?versions 2024-04-12T03:06:46.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GVTGEnzYn0iCmF9J2bZkc1Jkro= 2024-04-12T03:06:46.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GVTGEnzYn0iCmF9J2bZkc1Jkro=', 'Content-Length': '0'} 2024-04-12T03:06:46.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000258774e0a9f7542f-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.254 DEBUG:boto:b'dgbpxg-13obj-a1999falseobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.254 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:06:46.254 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:06:46.254 DEBUG:boto:path=/dgbpxg-13/obj-a1?acl 2024-04-12T03:06:46.254 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?acl 2024-04-12T03:06:46.254 DEBUG:boto:Method: PUT 2024-04-12T03:06:46.254 DEBUG:boto:Path: /dgbpxg-13/obj-a1?acl 2024-04-12T03:06:46.255 DEBUG:boto:Data: 2024-04-12T03:06:46.255 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:06:46.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.255 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.255 DEBUG:boto:Params: {} 2024-04-12T03:06:46.255 DEBUG:boto:Token: None 2024-04-12T03:06:46.255 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:06:46 GMT x-amz-acl:public-read /dgbpxg-13/obj-a1?acl 2024-04-12T03:06:46.255 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOfGVGK8QkCuOM6kHcifzc9/qSs= 2024-04-12T03:06:46.255 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOfGVGK8QkCuOM6kHcifzc9/qSs=', 'Content-Length': '0'} 2024-04-12T03:06:46.260 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce1fddffe3d38feb-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.260 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:46.260 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:46.260 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.260 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:06:46.260 DEBUG:boto:Data: 2024-04-12T03:06:46.260 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.261 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.261 DEBUG:boto:Params: {} 2024-04-12T03:06:46.261 DEBUG:boto:Token: None 2024-04-12T03:06:46.261 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-13/ 2024-04-12T03:06:46.261 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6RnGozRmcnYsMoJTupdywPHISxo= 2024-04-12T03:06:46.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6RnGozRmcnYsMoJTupdywPHISxo=', 'Content-Length': '0'} 2024-04-12T03:06:46.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fefbb017ffe1069e-006618a546-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.263 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:06:46.263 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:06:46.263 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.263 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.263 DEBUG:boto:Method: GET 2024-04-12T03:06:46.264 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.264 DEBUG:boto:Data: 2024-04-12T03:06:46.264 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.264 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.264 DEBUG:boto:Params: {} 2024-04-12T03:06:46.264 DEBUG:boto:Token: None 2024-04-12T03:06:46.264 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-13/?versions 2024-04-12T03:06:46.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GVTGEnzYn0iCmF9J2bZkc1Jkro= 2024-04-12T03:06:46.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GVTGEnzYn0iCmF9J2bZkc1Jkro=', 'Content-Length': '0'} 2024-04-12T03:06:46.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d504fb1921965a-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.266 DEBUG:boto:b'dgbpxg-13obj-a2999falseobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.267 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:06:46.267 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:06:46.267 DEBUG:boto:path=/dgbpxg-13/obj-a2?acl 2024-04-12T03:06:46.267 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?acl 2024-04-12T03:06:46.267 DEBUG:boto:Method: PUT 2024-04-12T03:06:46.267 DEBUG:boto:Path: /dgbpxg-13/obj-a2?acl 2024-04-12T03:06:46.267 DEBUG:boto:Data: 2024-04-12T03:06:46.267 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:06:46.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.267 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.268 DEBUG:boto:Params: {} 2024-04-12T03:06:46.268 DEBUG:boto:Token: None 2024-04-12T03:06:46.268 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:06:46 GMT x-amz-acl:public-read /dgbpxg-13/obj-a2?acl 2024-04-12T03:06:46.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZGS334ONxfkJThVmdRIAUALVBjg= 2024-04-12T03:06:46.268 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZGS334ONxfkJThVmdRIAUALVBjg=', 'Content-Length': '0'} 2024-04-12T03:06:46.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087ce103eb3e30468-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.272 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.272 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.273 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.273 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:46.273 DEBUG:boto:Data: 2024-04-12T03:06:46.273 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.273 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.273 DEBUG:boto:Params: {} 2024-04-12T03:06:46.273 DEBUG:boto:Token: None 2024-04-12T03:06:46.273 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MSJym9WKOOVfd7XRrAIMK4rcgnA= 2024-04-12T03:06:46.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MSJym9WKOOVfd7XRrAIMK4rcgnA=', 'Content-Length': '0'} 2024-04-12T03:06:46.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcd207bb8eae3f37-006618a546-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.275 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.275 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.276 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.276 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.276 DEBUG:boto:Method: GET 2024-04-12T03:06:46.276 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:06:46.276 DEBUG:boto:Data: 2024-04-12T03:06:46.276 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.276 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.276 DEBUG:boto:Params: {} 2024-04-12T03:06:46.276 DEBUG:boto:Token: None 2024-04-12T03:06:46.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/?versions 2024-04-12T03:06:46.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3isI090TVBHWMAxwIhqxMUxTe9o= 2024-04-12T03:06:46.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3isI090TVBHWMAxwIhqxMUxTe9o=', 'Content-Length': '0'} 2024-04-12T03:06:46.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0ad7ea099f6226e-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.322 DEBUG:boto:b'dgbpxg-14obj-a1999falseobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.323 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:06:46.323 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:06:46.323 DEBUG:boto:path=/dgbpxg-14/obj-a1?acl 2024-04-12T03:06:46.323 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?acl 2024-04-12T03:06:46.323 DEBUG:boto:Method: PUT 2024-04-12T03:06:46.323 DEBUG:boto:Path: /dgbpxg-14/obj-a1?acl 2024-04-12T03:06:46.323 DEBUG:boto:Data: 2024-04-12T03:06:46.323 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:06:46.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:06:46.323 DEBUG:boto:Port: 8000 2024-04-12T03:06:46.323 DEBUG:boto:Params: {} 2024-04-12T03:06:46.323 DEBUG:boto:Token: None 2024-04-12T03:06:46.324 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:06:46 GMT x-amz-acl:public-read /dgbpxg-14/obj-a1?acl 2024-04-12T03:06:46.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UIGh4fmuhNnDYhd1PZs8QPZsBec= 2024-04-12T03:06:46.324 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UIGh4fmuhNnDYhd1PZs8QPZsBec=', 'Content-Length': '0'} 2024-04-12T03:06:46.328 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026c2d5b867afdbe3-006618a546-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.328 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.328 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.328 DEBUG:boto:Method: HEAD 2024-04-12T03:06:46.328 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:06:46.328 DEBUG:boto:Data: 2024-04-12T03:06:46.329 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.329 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.329 DEBUG:boto:Params: {} 2024-04-12T03:06:46.329 DEBUG:boto:Token: None 2024-04-12T03:06:46.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/ 2024-04-12T03:06:46.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MSJym9WKOOVfd7XRrAIMK4rcgnA= 2024-04-12T03:06:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MSJym9WKOOVfd7XRrAIMK4rcgnA=', 'Content-Length': '0'} 2024-04-12T03:06:46.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a7b61dacabebdb6-006618a546-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.331 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:06:46.331 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:06:46.331 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.331 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.331 DEBUG:boto:Method: GET 2024-04-12T03:06:46.332 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:06:46.332 DEBUG:boto:Data: 2024-04-12T03:06:46.332 DEBUG:boto:Headers: {} 2024-04-12T03:06:46.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.332 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.332 DEBUG:boto:Params: {} 2024-04-12T03:06:46.332 DEBUG:boto:Token: None 2024-04-12T03:06:46.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:06:46 GMT /dgbpxg-14/?versions 2024-04-12T03:06:46.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3isI090TVBHWMAxwIhqxMUxTe9o= 2024-04-12T03:06:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3isI090TVBHWMAxwIhqxMUxTe9o=', 'Content-Length': '0'} 2024-04-12T03:06:46.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007793a28a9cdb9e6e-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.334 DEBUG:boto:b'dgbpxg-14obj-a2999falseobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:06:46.335 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.335 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:06:46.335 DEBUG:boto:path=/dgbpxg-14/obj-a2?acl 2024-04-12T03:06:46.335 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?acl 2024-04-12T03:06:46.335 DEBUG:boto:Method: PUT 2024-04-12T03:06:46.335 DEBUG:boto:Path: /dgbpxg-14/obj-a2?acl 2024-04-12T03:06:46.335 DEBUG:boto:Data: 2024-04-12T03:06:46.335 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:06:46.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:06:46.335 DEBUG:boto:Port: 8001 2024-04-12T03:06:46.336 DEBUG:boto:Params: {} 2024-04-12T03:06:46.336 DEBUG:boto:Token: None 2024-04-12T03:06:46.336 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:06:46 GMT x-amz-acl:public-read /dgbpxg-14/obj-a2?acl 2024-04-12T03:06:46.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:03WIt/MFdxra0UkXofXLnheD0/Y= 2024-04-12T03:06:46.336 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:06:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:03WIt/MFdxra0UkXofXLnheD0/Y=', 'Content-Length': '0'} 2024-04-12T03:06:46.340 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ed21bceacf8c12a-006618a546-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:06:46.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-13 2024-04-12T03:06:46.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:46.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:06:46.341 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:06:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:06:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:06:46.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:46.428+0000 7f25d9511780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:46.429 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000010.27.6, ] 2024-04-12T03:06:46.429 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:06:51.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:51.429+0000 7f25d9511780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:51.430 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000010.27.6, ] 2024-04-12T03:06:51.430 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:06:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:06:56.430+0000 7f25d9511780 1 waiting for incremental sync to catch up: 2024-04-12T03:06:56.432 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000010.27.6, ] 2024-04-12T03:06:56.432 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:07:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:01.431+0000 7f25d9511780 1 waiting for incremental sync to catch up: 2024-04-12T03:07:01.433 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000010.27.6, ] 2024-04-12T03:07:01.433 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:07:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:06.432+0000 7f25d9511780 1 bucket sync caught up with source: 2024-04-12T03:07:06.434 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:07:06.434 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.23.11, 00000000011.30.6, ] 2024-04-12T03:07:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:06.432+0000 7f25d9511780 0 bucket checkpoint complete 2024-04-12T03:07:06.450 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-13 2024-04-12T03:07:06.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:06.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:06.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:06.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:06.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:06.545+0000 7fca41d35780 1 bucket sync caught up with source: 2024-04-12T03:07:06.548 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000011.52.6, 00000000002.16.11, ] 2024-04-12T03:07:06.548 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000011.52.6, 00000000002.16.11, ] 2024-04-12T03:07:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:06.545+0000 7fca41d35780 0 bucket checkpoint complete 2024-04-12T03:07:06.560 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-13 zones={a2, a1} 2024-04-12T03:07:06.560 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.560 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.561 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.561 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:07:06.561 DEBUG:boto:Data: 2024-04-12T03:07:06.561 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.561 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.561 DEBUG:boto:Params: {} 2024-04-12T03:07:06.561 DEBUG:boto:Token: None 2024-04-12T03:07:06.561 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.562 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FeygtB6lpaRTggAGQOsNW2vfzkM= 2024-04-12T03:07:06.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FeygtB6lpaRTggAGQOsNW2vfzkM=', 'Content-Length': '0'} 2024-04-12T03:07:06.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e27973b41d544878-006618a55a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.564 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.564 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.564 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.564 DEBUG:boto:Path: /dgbpxg-13/ 2024-04-12T03:07:06.564 DEBUG:boto:Data: 2024-04-12T03:07:06.565 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.565 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.565 DEBUG:boto:Params: {} 2024-04-12T03:07:06.565 DEBUG:boto:Token: None 2024-04-12T03:07:06.565 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FeygtB6lpaRTggAGQOsNW2vfzkM= 2024-04-12T03:07:06.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FeygtB6lpaRTggAGQOsNW2vfzkM=', 'Content-Length': '0'} 2024-04-12T03:07:06.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d26acae778bf7f6-006618a55a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:07:06.567 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.567 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.568 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.568 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.568 DEBUG:boto:Method: GET 2024-04-12T03:07:06.568 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.568 DEBUG:boto:Data: 2024-04-12T03:07:06.568 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.568 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.568 DEBUG:boto:Params: {} 2024-04-12T03:07:06.568 DEBUG:boto:Token: None 2024-04-12T03:07:06.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/?versions 2024-04-12T03:07:06.569 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w= 2024-04-12T03:07:06.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w=', 'Content-Length': '0'} 2024-04-12T03:07:06.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d35718acab074a-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.571 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.571 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:06.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:07:06.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:07:06.572 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:07:06.572 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.572 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.572 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.572 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.572 DEBUG:boto:Method: GET 2024-04-12T03:07:06.572 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.572 DEBUG:boto:Data: 2024-04-12T03:07:06.572 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.572 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.572 DEBUG:boto:Params: {} 2024-04-12T03:07:06.573 DEBUG:boto:Token: None 2024-04-12T03:07:06.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/?versions 2024-04-12T03:07:06.573 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w= 2024-04-12T03:07:06.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w=', 'Content-Length': '0'} 2024-04-12T03:07:06.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013813ca8c5665982-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.575 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.576 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:06.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:07:06.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:07:06.576 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.576 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.576 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.576 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.576 DEBUG:boto:Method: GET 2024-04-12T03:07:06.576 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.576 DEBUG:boto:Data: 2024-04-12T03:07:06.577 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.577 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.577 DEBUG:boto:Params: {} 2024-04-12T03:07:06.577 DEBUG:boto:Token: None 2024-04-12T03:07:06.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/?versions 2024-04-12T03:07:06.577 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w= 2024-04-12T03:07:06.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w=', 'Content-Length': '0'} 2024-04-12T03:07:06.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc351da2b5881a6c-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.579 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.580 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.580 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.580 DEBUG:boto:path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.580 DEBUG:boto:auth_path=/dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.580 DEBUG:boto:Method: GET 2024-04-12T03:07:06.580 DEBUG:boto:Path: /dgbpxg-13/?versions&max-keys=999 2024-04-12T03:07:06.580 DEBUG:boto:Data: 2024-04-12T03:07:06.580 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.580 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.580 DEBUG:boto:Params: {} 2024-04-12T03:07:06.581 DEBUG:boto:Token: None 2024-04-12T03:07:06.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/?versions 2024-04-12T03:07:06.581 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w= 2024-04-12T03:07:06.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7JUmGyWo9D8VTijXfYauMm+6J8w=', 'Content-Length': '0'} 2024-04-12T03:07:06.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fe2c558628b1017-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.583 DEBUG:boto:b'dgbpxg-13999falsedummy9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqctrue2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1-JKHF.VUe8lm81DsrEjUPNkGYL30akptrue2024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0true2024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:06.583 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.583 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.584 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.584 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.584 DEBUG:boto:Method: GET 2024-04-12T03:07:06.584 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.584 DEBUG:boto:Data: 2024-04-12T03:07:06.584 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.584 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.584 DEBUG:boto:Params: {} 2024-04-12T03:07:06.584 DEBUG:boto:Token: None 2024-04-12T03:07:06.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5f67c68baed7fc8-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.586 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.586 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.587 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.587 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.587 DEBUG:boto:Method: GET 2024-04-12T03:07:06.587 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.587 DEBUG:boto:Data: 2024-04-12T03:07:06.587 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.587 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.587 DEBUG:boto:Params: {} 2024-04-12T03:07:06.587 DEBUG:boto:Token: None 2024-04-12T03:07:06.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098dfe03b4077ac0d-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.590 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.590 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.590 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.590 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.590 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.590 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.590 DEBUG:boto:Data: 2024-04-12T03:07:06.590 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.590 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.590 DEBUG:boto:Params: {} 2024-04-12T03:07:06.591 DEBUG:boto:Token: None 2024-04-12T03:07:06.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tUYLeZEwFm9JIw2dWXtJMQbKYhY= 2024-04-12T03:07:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tUYLeZEwFm9JIw2dWXtJMQbKYhY=', 'Content-Length': '0'} 2024-04-12T03:07:06.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd5386e4782fc188-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.593 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.593 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.593 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.593 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.593 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.593 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.593 DEBUG:boto:Data: 2024-04-12T03:07:06.593 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.593 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.593 DEBUG:boto:Params: {} 2024-04-12T03:07:06.594 DEBUG:boto:Token: None 2024-04-12T03:07:06.594 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tUYLeZEwFm9JIw2dWXtJMQbKYhY= 2024-04-12T03:07:06.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tUYLeZEwFm9JIw2dWXtJMQbKYhY=', 'Content-Length': '0'} 2024-04-12T03:07:06.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b0a39178c2be4ad-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:06.596 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.596 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.596 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.596 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.596 DEBUG:boto:Method: GET 2024-04-12T03:07:06.596 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.596 DEBUG:boto:Data: 2024-04-12T03:07:06.596 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.596 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.596 DEBUG:boto:Params: {} 2024-04-12T03:07:06.596 DEBUG:boto:Token: None 2024-04-12T03:07:06.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.597 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e88ff93ee640458-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.599 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.599 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.599 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.599 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.599 DEBUG:boto:Method: GET 2024-04-12T03:07:06.599 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.599 DEBUG:boto:Data: 2024-04-12T03:07:06.599 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.600 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.600 DEBUG:boto:Params: {} 2024-04-12T03:07:06.600 DEBUG:boto:Token: None 2024-04-12T03:07:06.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bf3d8d2bc72bf71-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.602 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.602 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.602 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.602 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.602 DEBUG:boto:Method: GET 2024-04-12T03:07:06.602 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.602 DEBUG:boto:Data: 2024-04-12T03:07:06.602 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.603 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.603 DEBUG:boto:Params: {} 2024-04-12T03:07:06.603 DEBUG:boto:Token: None 2024-04-12T03:07:06.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e6c6b3f8653abf-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.646 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.647 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.647 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.647 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.647 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.647 DEBUG:boto:Method: GET 2024-04-12T03:07:06.647 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.647 DEBUG:boto:Data: 2024-04-12T03:07:06.647 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.647 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.647 DEBUG:boto:Params: {} 2024-04-12T03:07:06.648 DEBUG:boto:Token: None 2024-04-12T03:07:06.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530d56a4d532d67b-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.650 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.650 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.650 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.650 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.650 DEBUG:boto:Path: /dgbpxg-13/dummy 2024-04-12T03:07:06.651 DEBUG:boto:Data: 2024-04-12T03:07:06.651 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.651 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.651 DEBUG:boto:Params: {} 2024-04-12T03:07:06.651 DEBUG:boto:Token: None 2024-04-12T03:07:06.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy 2024-04-12T03:07:06.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MLeAbBwFrAUPd04RkR1a8ddon7Y= 2024-04-12T03:07:06.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MLeAbBwFrAUPd04RkR1a8ddon7Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7e2ffe93095f457-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.653 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.653 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.653 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.653 DEBUG:boto:Path: /dgbpxg-13/dummy 2024-04-12T03:07:06.653 DEBUG:boto:Data: 2024-04-12T03:07:06.654 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.654 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.654 DEBUG:boto:Params: {} 2024-04-12T03:07:06.654 DEBUG:boto:Token: None 2024-04-12T03:07:06.654 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy 2024-04-12T03:07:06.654 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MLeAbBwFrAUPd04RkR1a8ddon7Y= 2024-04-12T03:07:06.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MLeAbBwFrAUPd04RkR1a8ddon7Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a599ec86a289e96e-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:06.657 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.657 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.657 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.657 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.657 DEBUG:boto:Method: GET 2024-04-12T03:07:06.657 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.657 DEBUG:boto:Data: 2024-04-12T03:07:06.657 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.657 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.658 DEBUG:boto:Params: {} 2024-04-12T03:07:06.658 DEBUG:boto:Token: None 2024-04-12T03:07:06.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bdd77fc626d4fe8-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.660 DEBUG:boto:path=/dgbpxg-13/dummy 2024-04-12T03:07:06.660 DEBUG:boto:auth_path=/dgbpxg-13/dummy 2024-04-12T03:07:06.660 DEBUG:boto:path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.660 DEBUG:boto:auth_path=/dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.660 DEBUG:boto:Method: GET 2024-04-12T03:07:06.660 DEBUG:boto:Path: /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.660 DEBUG:boto:Data: 2024-04-12T03:07:06.660 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.660 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.661 DEBUG:boto:Params: {} 2024-04-12T03:07:06.661 DEBUG:boto:Token: None 2024-04-12T03:07:06.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/dummy?versionId=9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc 2024-04-12T03:07:06.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI= 2024-04-12T03:07:06.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4D7nLmJQDKSccAnN1M7tHwz38xI=', 'Content-Length': '0'} 2024-04-12T03:07:06.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:05:44 GMT'), ('x-amz-version-id', '9WI7c2ZoloEkLDtT.GtOwrXMPeGBfqc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b954473632d0448b-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.663 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.663 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.663 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.663 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.663 DEBUG:boto:Method: GET 2024-04-12T03:07:06.663 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.663 DEBUG:boto:Data: 2024-04-12T03:07:06.663 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.664 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.664 DEBUG:boto:Params: {} 2024-04-12T03:07:06.664 DEBUG:boto:Token: None 2024-04-12T03:07:06.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa37e7f0eff9a86-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.666 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.666 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.666 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.667 DEBUG:boto:path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.667 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.667 DEBUG:boto:Method: GET 2024-04-12T03:07:06.667 DEBUG:boto:Path: /dgbpxg-13/?prefix=dummy 2024-04-12T03:07:06.667 DEBUG:boto:Data: 2024-04-12T03:07:06.667 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.667 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.667 DEBUG:boto:Params: {} 2024-04-12T03:07:06.667 DEBUG:boto:Token: None 2024-04-12T03:07:06.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0647eb4e9e60171-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.670 DEBUG:boto:b'dgbpxg-13dummy1000falsedummy2024-04-12T03:05:44.162Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:06.670 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.670 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.670 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.670 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.670 DEBUG:boto:Method: GET 2024-04-12T03:07:06.670 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.671 DEBUG:boto:Data: 2024-04-12T03:07:06.671 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.671 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.671 DEBUG:boto:Params: {} 2024-04-12T03:07:06.671 DEBUG:boto:Token: None 2024-04-12T03:07:06.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4f99ec90c6c8e84-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.673 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.673 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.673 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.673 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.673 DEBUG:boto:Method: GET 2024-04-12T03:07:06.673 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.674 DEBUG:boto:Data: 2024-04-12T03:07:06.674 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.674 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.674 DEBUG:boto:Params: {} 2024-04-12T03:07:06.674 DEBUG:boto:Token: None 2024-04-12T03:07:06.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006056b2e03699ffcf-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.676 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.676 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.677 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.677 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.677 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.677 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.677 DEBUG:boto:Data: 2024-04-12T03:07:06.677 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.677 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.677 DEBUG:boto:Params: {} 2024-04-12T03:07:06.677 DEBUG:boto:Token: None 2024-04-12T03:07:06.677 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.678 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n3R4ZPKE35AQlifQsBCYJsMEia8= 2024-04-12T03:07:06.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n3R4ZPKE35AQlifQsBCYJsMEia8=', 'Content-Length': '0'} 2024-04-12T03:07:06.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017c66c03e8a19f31-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.679 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.680 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.680 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.680 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.680 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.680 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.680 DEBUG:boto:Data: 2024-04-12T03:07:06.680 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.680 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.680 DEBUG:boto:Params: {} 2024-04-12T03:07:06.680 DEBUG:boto:Token: None 2024-04-12T03:07:06.680 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n3R4ZPKE35AQlifQsBCYJsMEia8= 2024-04-12T03:07:06.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n3R4ZPKE35AQlifQsBCYJsMEia8=', 'Content-Length': '0'} 2024-04-12T03:07:06.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c28752277a33410-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:06.683 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.683 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.683 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.683 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.683 DEBUG:boto:Method: GET 2024-04-12T03:07:06.683 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.683 DEBUG:boto:Data: 2024-04-12T03:07:06.683 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.684 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.684 DEBUG:boto:Params: {} 2024-04-12T03:07:06.684 DEBUG:boto:Token: None 2024-04-12T03:07:06.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000090aa0b30d4e57aa-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.726 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.727 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.727 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.727 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.727 DEBUG:boto:Method: GET 2024-04-12T03:07:06.727 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.727 DEBUG:boto:Data: 2024-04-12T03:07:06.727 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.727 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.727 DEBUG:boto:Params: {} 2024-04-12T03:07:06.727 DEBUG:boto:Token: None 2024-04-12T03:07:06.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b78d54d0e14440e1-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.730 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.730 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.730 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.730 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.730 DEBUG:boto:Method: GET 2024-04-12T03:07:06.730 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.730 DEBUG:boto:Data: 2024-04-12T03:07:06.730 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.730 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.730 DEBUG:boto:Params: {} 2024-04-12T03:07:06.731 DEBUG:boto:Token: None 2024-04-12T03:07:06.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e58a9e0c48faf3a-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.733 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.733 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.733 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.734 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.734 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.734 DEBUG:boto:Method: GET 2024-04-12T03:07:06.734 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.734 DEBUG:boto:Data: 2024-04-12T03:07:06.734 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.734 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.734 DEBUG:boto:Params: {} 2024-04-12T03:07:06.734 DEBUG:boto:Token: None 2024-04-12T03:07:06.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd9792b87de83634-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.737 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.737 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.737 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.737 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.738 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:07:06.738 DEBUG:boto:Data: 2024-04-12T03:07:06.738 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.738 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.738 DEBUG:boto:Params: {} 2024-04-12T03:07:06.738 DEBUG:boto:Token: None 2024-04-12T03:07:06.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1 2024-04-12T03:07:06.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QHgwvRp19Ny19U9Dt9QX61UohLg= 2024-04-12T03:07:06.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QHgwvRp19Ny19U9Dt9QX61UohLg=', 'Content-Length': '0'} 2024-04-12T03:07:06.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a942d78fa1a378c9-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.741 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.741 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.741 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.741 DEBUG:boto:Path: /dgbpxg-13/obj-a1 2024-04-12T03:07:06.741 DEBUG:boto:Data: 2024-04-12T03:07:06.741 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.741 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.741 DEBUG:boto:Params: {} 2024-04-12T03:07:06.741 DEBUG:boto:Token: None 2024-04-12T03:07:06.741 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1 2024-04-12T03:07:06.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QHgwvRp19Ny19U9Dt9QX61UohLg= 2024-04-12T03:07:06.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QHgwvRp19Ny19U9Dt9QX61UohLg=', 'Content-Length': '0'} 2024-04-12T03:07:06.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e3c685306d9f7d6-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:06.744 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.744 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.744 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.744 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.744 DEBUG:boto:Method: GET 2024-04-12T03:07:06.744 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.744 DEBUG:boto:Data: 2024-04-12T03:07:06.744 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.745 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.745 DEBUG:boto:Params: {} 2024-04-12T03:07:06.745 DEBUG:boto:Token: None 2024-04-12T03:07:06.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb16685d8be4d09e-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.790 DEBUG:boto:path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.791 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1 2024-04-12T03:07:06.791 DEBUG:boto:path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.791 DEBUG:boto:auth_path=/dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.791 DEBUG:boto:Method: GET 2024-04-12T03:07:06.791 DEBUG:boto:Path: /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.791 DEBUG:boto:Data: 2024-04-12T03:07:06.791 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.791 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.791 DEBUG:boto:Params: {} 2024-04-12T03:07:06.791 DEBUG:boto:Token: None 2024-04-12T03:07:06.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a1?versionId=-JKHF.VUe8lm81DsrEjUPNkGYL30akp 2024-04-12T03:07:06.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI= 2024-04-12T03:07:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mtaQnsq58/1Rj+f2c3GGIthKVsI=', 'Content-Length': '0'} 2024-04-12T03:07:06.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', '-JKHF.VUe8lm81DsrEjUPNkGYL30akp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6704d613413119a-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.793 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.793 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.794 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.794 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.794 DEBUG:boto:Method: GET 2024-04-12T03:07:06.794 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.794 DEBUG:boto:Data: 2024-04-12T03:07:06.794 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.794 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.794 DEBUG:boto:Params: {} 2024-04-12T03:07:06.794 DEBUG:boto:Token: None 2024-04-12T03:07:06.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3985243dc235910-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.838 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.839 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.839 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.839 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.839 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.839 DEBUG:boto:Method: GET 2024-04-12T03:07:06.839 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a1 2024-04-12T03:07:06.839 DEBUG:boto:Data: 2024-04-12T03:07:06.839 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.840 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.840 DEBUG:boto:Params: {} 2024-04-12T03:07:06.840 DEBUG:boto:Token: None 2024-04-12T03:07:06.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002320548fb29e874e-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.842 DEBUG:boto:b'dgbpxg-13obj-a11000falseobj-a12024-04-12T03:06:19.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:06.843 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.843 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.843 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.843 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.843 DEBUG:boto:Method: GET 2024-04-12T03:07:06.843 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.843 DEBUG:boto:Data: 2024-04-12T03:07:06.843 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.843 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.844 DEBUG:boto:Params: {} 2024-04-12T03:07:06.844 DEBUG:boto:Token: None 2024-04-12T03:07:06.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009df8ebdc0aa870d-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.846 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.846 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.846 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.846 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.846 DEBUG:boto:Method: GET 2024-04-12T03:07:06.846 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.846 DEBUG:boto:Data: 2024-04-12T03:07:06.846 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.847 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.847 DEBUG:boto:Params: {} 2024-04-12T03:07:06.847 DEBUG:boto:Token: None 2024-04-12T03:07:06.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019e44a3acdfa0243-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.849 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.850 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.850 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.850 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.850 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.850 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.850 DEBUG:boto:Data: 2024-04-12T03:07:06.850 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.850 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.850 DEBUG:boto:Params: {} 2024-04-12T03:07:06.850 DEBUG:boto:Token: None 2024-04-12T03:07:06.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rLOXDs03IEGx/4VGp6BY21ZmmUU= 2024-04-12T03:07:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rLOXDs03IEGx/4VGp6BY21ZmmUU=', 'Content-Length': '0'} 2024-04-12T03:07:06.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e6eb4b012780464-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.852 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.853 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.853 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.853 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.853 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.853 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.853 DEBUG:boto:Data: 2024-04-12T03:07:06.853 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.853 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.853 DEBUG:boto:Params: {} 2024-04-12T03:07:06.853 DEBUG:boto:Token: None 2024-04-12T03:07:06.854 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rLOXDs03IEGx/4VGp6BY21ZmmUU= 2024-04-12T03:07:06.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rLOXDs03IEGx/4VGp6BY21ZmmUU=', 'Content-Length': '0'} 2024-04-12T03:07:06.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002694b024bd1e42eb-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:06.856 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.856 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.856 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.856 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.856 DEBUG:boto:Method: GET 2024-04-12T03:07:06.856 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.856 DEBUG:boto:Data: 2024-04-12T03:07:06.856 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.857 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.857 DEBUG:boto:Params: {} 2024-04-12T03:07:06.857 DEBUG:boto:Token: None 2024-04-12T03:07:06.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c027d7c5eef1c52d-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.859 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.859 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.860 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.860 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.860 DEBUG:boto:Method: GET 2024-04-12T03:07:06.860 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.860 DEBUG:boto:Data: 2024-04-12T03:07:06.860 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.860 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.860 DEBUG:boto:Params: {} 2024-04-12T03:07:06.860 DEBUG:boto:Token: None 2024-04-12T03:07:06.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc6149cbdce61b19-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.906 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.906 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.907 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.907 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.907 DEBUG:boto:Method: GET 2024-04-12T03:07:06.907 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.907 DEBUG:boto:Data: 2024-04-12T03:07:06.907 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.907 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.907 DEBUG:boto:Params: {} 2024-04-12T03:07:06.907 DEBUG:boto:Token: None 2024-04-12T03:07:06.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd684f3460b23645-006618a55a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.910 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.910 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:06.910 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:06.911 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.911 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.911 DEBUG:boto:Method: GET 2024-04-12T03:07:06.911 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:06.911 DEBUG:boto:Data: 2024-04-12T03:07:06.911 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.911 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.911 DEBUG:boto:Params: {} 2024-04-12T03:07:06.911 DEBUG:boto:Token: None 2024-04-12T03:07:06.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/ 2024-04-12T03:07:06.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y= 2024-04-12T03:07:06.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vckicNAhxRLDHvMJ+SXnpmFkk1Y=', 'Content-Length': '0'} 2024-04-12T03:07:06.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad9aac4254e477e-006618a55a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.954 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:06.955 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.955 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.955 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.955 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:07:06.955 DEBUG:boto:Data: 2024-04-12T03:07:06.955 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.955 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.955 DEBUG:boto:Params: {} 2024-04-12T03:07:06.955 DEBUG:boto:Token: None 2024-04-12T03:07:06.956 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2 2024-04-12T03:07:06.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3fR5Zv6p2N+QoPl2AM1dYQZssMk= 2024-04-12T03:07:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3fR5Zv6p2N+QoPl2AM1dYQZssMk=', 'Content-Length': '0'} 2024-04-12T03:07:06.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ab9443c677fe82-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.958 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.958 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.958 DEBUG:boto:Method: HEAD 2024-04-12T03:07:06.958 DEBUG:boto:Path: /dgbpxg-13/obj-a2 2024-04-12T03:07:06.958 DEBUG:boto:Data: 2024-04-12T03:07:06.958 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.959 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.959 DEBUG:boto:Params: {} 2024-04-12T03:07:06.959 DEBUG:boto:Token: None 2024-04-12T03:07:06.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2 2024-04-12T03:07:06.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3fR5Zv6p2N+QoPl2AM1dYQZssMk= 2024-04-12T03:07:06.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3fR5Zv6p2N+QoPl2AM1dYQZssMk=', 'Content-Length': '0'} 2024-04-12T03:07:06.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ce466531a625bdd-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:06.961 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.961 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.961 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.962 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.962 DEBUG:boto:Method: GET 2024-04-12T03:07:06.962 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.962 DEBUG:boto:Data: 2024-04-12T03:07:06.962 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:06.962 DEBUG:boto:Port: 8001 2024-04-12T03:07:06.962 DEBUG:boto:Params: {} 2024-04-12T03:07:06.962 DEBUG:boto:Token: None 2024-04-12T03:07:06.962 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b24d79272f66be5a-006618a55a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:06.965 DEBUG:boto:path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.965 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2 2024-04-12T03:07:06.965 DEBUG:boto:path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.965 DEBUG:boto:auth_path=/dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.965 DEBUG:boto:Method: GET 2024-04-12T03:07:06.965 DEBUG:boto:Path: /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.965 DEBUG:boto:Data: 2024-04-12T03:07:06.965 DEBUG:boto:Headers: {} 2024-04-12T03:07:06.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:06.965 DEBUG:boto:Port: 8000 2024-04-12T03:07:06.966 DEBUG:boto:Params: {} 2024-04-12T03:07:06.966 DEBUG:boto:Token: None 2024-04-12T03:07:06.966 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:06 GMT /dgbpxg-13/obj-a2?versionId=WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0 2024-04-12T03:07:06.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE= 2024-04-12T03:07:06.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bMksbteLp6ZtzDaJplLx8H3QVuE=', 'Content-Length': '0'} 2024-04-12T03:07:06.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'WbvMl1BWTvK6sDcJc6p5GdNRlH-LZl0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b28c76a17d52fbc-006618a55a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.010 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:07.010 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:07.011 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.011 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.011 DEBUG:boto:Method: GET 2024-04-12T03:07:07.011 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.011 DEBUG:boto:Data: 2024-04-12T03:07:07.011 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.011 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.011 DEBUG:boto:Params: {} 2024-04-12T03:07:07.011 DEBUG:boto:Token: None 2024-04-12T03:07:07.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-13/ 2024-04-12T03:07:07.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yPH6xzeE0lz7dyUqdCjOEWcNsU= 2024-04-12T03:07:07.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yPH6xzeE0lz7dyUqdCjOEWcNsU=', 'Content-Length': '0'} 2024-04-12T03:07:07.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4ff6e224740729-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.014 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.014 DEBUG:boto:path=/dgbpxg-13/ 2024-04-12T03:07:07.015 DEBUG:boto:auth_path=/dgbpxg-13/ 2024-04-12T03:07:07.015 DEBUG:boto:path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.015 DEBUG:boto:auth_path=/dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.015 DEBUG:boto:Method: GET 2024-04-12T03:07:07.015 DEBUG:boto:Path: /dgbpxg-13/?prefix=obj-a2 2024-04-12T03:07:07.015 DEBUG:boto:Data: 2024-04-12T03:07:07.015 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.015 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.015 DEBUG:boto:Params: {} 2024-04-12T03:07:07.015 DEBUG:boto:Token: None 2024-04-12T03:07:07.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-13/ 2024-04-12T03:07:07.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yPH6xzeE0lz7dyUqdCjOEWcNsU= 2024-04-12T03:07:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yPH6xzeE0lz7dyUqdCjOEWcNsU=', 'Content-Length': '0'} 2024-04-12T03:07:07.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18b1154dc13f2d0-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.018 DEBUG:boto:b'dgbpxg-13obj-a21000falseobj-a22024-04-12T03:06:19.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-13 zones={a2, a1} 2024-04-12T03:07:07.018 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-14 2024-04-12T03:07:07.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.019 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:07.075 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:07.075 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:07.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.107+0000 7f37c09f0780 1 bucket sync caught up with source: 2024-04-12T03:07:07.109 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000002.26.11, 00000000011.40.6, ] 2024-04-12T03:07:07.110 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000002.26.11, 00000000011.40.6, ] 2024-04-12T03:07:07.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.107+0000 7f37c09f0780 0 bucket checkpoint complete 2024-04-12T03:07:07.123 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-14 2024-04-12T03:07:07.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.123 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.209+0000 7fefe4120780 1 bucket sync caught up with source: 2024-04-12T03:07:07.211 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000011.58.6, 00000000002.41.11, ] 2024-04-12T03:07:07.211 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000011.58.6, 00000000002.41.11, ] 2024-04-12T03:07:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.209+0000 7fefe4120780 0 bucket checkpoint complete 2024-04-12T03:07:07.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-14 zones={a2, a1} 2024-04-12T03:07:07.223 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.224 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.224 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.224 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:07:07.224 DEBUG:boto:Data: 2024-04-12T03:07:07.224 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.224 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.224 DEBUG:boto:Params: {} 2024-04-12T03:07:07.224 DEBUG:boto:Token: None 2024-04-12T03:07:07.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u8yWdyTtl0IkyTMQ5Ld8rg8uXt8= 2024-04-12T03:07:07.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u8yWdyTtl0IkyTMQ5Ld8rg8uXt8=', 'Content-Length': '0'} 2024-04-12T03:07:07.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebd85fbee58e2e43-006618a55b-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.227 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.227 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.227 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.227 DEBUG:boto:Path: /dgbpxg-14/ 2024-04-12T03:07:07.227 DEBUG:boto:Data: 2024-04-12T03:07:07.227 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.228 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.228 DEBUG:boto:Params: {} 2024-04-12T03:07:07.228 DEBUG:boto:Token: None 2024-04-12T03:07:07.228 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u8yWdyTtl0IkyTMQ5Ld8rg8uXt8= 2024-04-12T03:07:07.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u8yWdyTtl0IkyTMQ5Ld8rg8uXt8=', 'Content-Length': '0'} 2024-04-12T03:07:07.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028e9c3cdf2c5b0cf-006618a55b-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:07:07.230 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.230 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.231 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.231 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.231 DEBUG:boto:Method: GET 2024-04-12T03:07:07.231 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.231 DEBUG:boto:Data: 2024-04-12T03:07:07.231 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.231 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.231 DEBUG:boto:Params: {} 2024-04-12T03:07:07.231 DEBUG:boto:Token: None 2024-04-12T03:07:07.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/?versions 2024-04-12T03:07:07.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc= 2024-04-12T03:07:07.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc=', 'Content-Length': '0'} 2024-04-12T03:07:07.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096b65fcc63bfbcbf-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.234 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.234 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:07.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:07:07.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:07:07.234 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:07:07.235 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.235 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.235 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.235 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.235 DEBUG:boto:Method: GET 2024-04-12T03:07:07.235 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.235 DEBUG:boto:Data: 2024-04-12T03:07:07.235 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.235 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.235 DEBUG:boto:Params: {} 2024-04-12T03:07:07.235 DEBUG:boto:Token: None 2024-04-12T03:07:07.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/?versions 2024-04-12T03:07:07.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc= 2024-04-12T03:07:07.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc=', 'Content-Length': '0'} 2024-04-12T03:07:07.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d866528969d3a6b4-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.238 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.238 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:07.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:07:07.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:07:07.239 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.239 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.239 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.239 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.239 DEBUG:boto:Method: GET 2024-04-12T03:07:07.239 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.239 DEBUG:boto:Data: 2024-04-12T03:07:07.239 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.239 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.239 DEBUG:boto:Params: {} 2024-04-12T03:07:07.240 DEBUG:boto:Token: None 2024-04-12T03:07:07.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/?versions 2024-04-12T03:07:07.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc= 2024-04-12T03:07:07.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc=', 'Content-Length': '0'} 2024-04-12T03:07:07.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11f14bada9b160c-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.242 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.243 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.243 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.243 DEBUG:boto:path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.243 DEBUG:boto:auth_path=/dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.243 DEBUG:boto:Method: GET 2024-04-12T03:07:07.243 DEBUG:boto:Path: /dgbpxg-14/?versions&max-keys=999 2024-04-12T03:07:07.243 DEBUG:boto:Data: 2024-04-12T03:07:07.243 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.244 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.244 DEBUG:boto:Params: {} 2024-04-12T03:07:07.244 DEBUG:boto:Token: None 2024-04-12T03:07:07.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/?versions 2024-04-12T03:07:07.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc= 2024-04-12T03:07:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ATRHk5+7kI8eEr1NkLAUXwxstc=', 'Content-Length': '0'} 2024-04-12T03:07:07.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c348c8a8b1bb9301-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.246 DEBUG:boto:b'dgbpxg-14999falsedummySo5uynX-HAY7KHWnNki39RXeEqs8CN7true2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1BA5eHZGRFuVxE2d2CpBOPmfBr0AWghUtrue2024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2fm9fz.Se6rpbuLcZyZup5nylTrecv0Ntrue2024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:07.247 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.247 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.247 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.247 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.247 DEBUG:boto:Method: GET 2024-04-12T03:07:07.247 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.247 DEBUG:boto:Data: 2024-04-12T03:07:07.247 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.248 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.248 DEBUG:boto:Params: {} 2024-04-12T03:07:07.248 DEBUG:boto:Token: None 2024-04-12T03:07:07.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004aeec262567deb6e-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.250 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.250 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.250 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.250 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.250 DEBUG:boto:Method: GET 2024-04-12T03:07:07.250 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.250 DEBUG:boto:Data: 2024-04-12T03:07:07.250 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.250 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.251 DEBUG:boto:Params: {} 2024-04-12T03:07:07.251 DEBUG:boto:Token: None 2024-04-12T03:07:07.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028a4baaa0b23a020-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.252 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.252 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.253 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.253 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.253 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.253 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.253 DEBUG:boto:Data: 2024-04-12T03:07:07.253 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.253 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.253 DEBUG:boto:Params: {} 2024-04-12T03:07:07.253 DEBUG:boto:Token: None 2024-04-12T03:07:07.253 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m+eCAUySgZcLjiDu0Hy2WhMEu0s= 2024-04-12T03:07:07.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m+eCAUySgZcLjiDu0Hy2WhMEu0s=', 'Content-Length': '0'} 2024-04-12T03:07:07.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d06e5137b37646d3-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.255 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.255 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.255 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.255 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.255 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.256 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.256 DEBUG:boto:Data: 2024-04-12T03:07:07.256 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.256 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.256 DEBUG:boto:Params: {} 2024-04-12T03:07:07.256 DEBUG:boto:Token: None 2024-04-12T03:07:07.256 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m+eCAUySgZcLjiDu0Hy2WhMEu0s= 2024-04-12T03:07:07.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m+eCAUySgZcLjiDu0Hy2WhMEu0s=', 'Content-Length': '0'} 2024-04-12T03:07:07.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d554b5248331755c-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:07.258 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.258 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.258 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.259 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.259 DEBUG:boto:Method: GET 2024-04-12T03:07:07.259 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.259 DEBUG:boto:Data: 2024-04-12T03:07:07.259 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.259 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.259 DEBUG:boto:Params: {} 2024-04-12T03:07:07.259 DEBUG:boto:Token: None 2024-04-12T03:07:07.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f1ed6472692a3de-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.262 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.262 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.262 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.262 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.262 DEBUG:boto:Method: GET 2024-04-12T03:07:07.262 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.262 DEBUG:boto:Data: 2024-04-12T03:07:07.262 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.262 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.262 DEBUG:boto:Params: {} 2024-04-12T03:07:07.263 DEBUG:boto:Token: None 2024-04-12T03:07:07.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea4bb2897641bab5-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.265 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.265 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.265 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.265 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.265 DEBUG:boto:Method: GET 2024-04-12T03:07:07.265 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.265 DEBUG:boto:Data: 2024-04-12T03:07:07.265 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.265 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.266 DEBUG:boto:Params: {} 2024-04-12T03:07:07.266 DEBUG:boto:Token: None 2024-04-12T03:07:07.266 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5d385bcec97bb0a-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.310 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.311 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.311 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.311 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.311 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.311 DEBUG:boto:Method: GET 2024-04-12T03:07:07.311 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.311 DEBUG:boto:Data: 2024-04-12T03:07:07.311 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.311 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.312 DEBUG:boto:Params: {} 2024-04-12T03:07:07.312 DEBUG:boto:Token: None 2024-04-12T03:07:07.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099793c53e6ecd53f-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.314 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.315 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.315 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.315 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.315 DEBUG:boto:Path: /dgbpxg-14/dummy 2024-04-12T03:07:07.315 DEBUG:boto:Data: 2024-04-12T03:07:07.315 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.315 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.315 DEBUG:boto:Params: {} 2024-04-12T03:07:07.315 DEBUG:boto:Token: None 2024-04-12T03:07:07.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy 2024-04-12T03:07:07.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i31jO9bJZ2uC07rWmMAE5MutIzQ= 2024-04-12T03:07:07.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i31jO9bJZ2uC07rWmMAE5MutIzQ=', 'Content-Length': '0'} 2024-04-12T03:07:07.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007bece2d00a8bc77b-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.318 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.318 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.318 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.318 DEBUG:boto:Path: /dgbpxg-14/dummy 2024-04-12T03:07:07.318 DEBUG:boto:Data: 2024-04-12T03:07:07.318 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.319 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.319 DEBUG:boto:Params: {} 2024-04-12T03:07:07.319 DEBUG:boto:Token: None 2024-04-12T03:07:07.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy 2024-04-12T03:07:07.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i31jO9bJZ2uC07rWmMAE5MutIzQ= 2024-04-12T03:07:07.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i31jO9bJZ2uC07rWmMAE5MutIzQ=', 'Content-Length': '0'} 2024-04-12T03:07:07.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033d9753c43da9178-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:07.321 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.321 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.322 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.322 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.322 DEBUG:boto:Method: GET 2024-04-12T03:07:07.322 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.322 DEBUG:boto:Data: 2024-04-12T03:07:07.322 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.322 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.322 DEBUG:boto:Params: {} 2024-04-12T03:07:07.322 DEBUG:boto:Token: None 2024-04-12T03:07:07.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a80d5f956bd45ed-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.324 DEBUG:boto:path=/dgbpxg-14/dummy 2024-04-12T03:07:07.325 DEBUG:boto:auth_path=/dgbpxg-14/dummy 2024-04-12T03:07:07.325 DEBUG:boto:path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.325 DEBUG:boto:auth_path=/dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.325 DEBUG:boto:Method: GET 2024-04-12T03:07:07.325 DEBUG:boto:Path: /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.325 DEBUG:boto:Data: 2024-04-12T03:07:07.325 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.325 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.325 DEBUG:boto:Params: {} 2024-04-12T03:07:07.325 DEBUG:boto:Token: None 2024-04-12T03:07:07.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/dummy?versionId=So5uynX-HAY7KHWnNki39RXeEqs8CN7 2024-04-12T03:07:07.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc= 2024-04-12T03:07:07.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8BXHMS62z8bg4l1jCrFstmRLhXc=', 'Content-Length': '0'} 2024-04-12T03:07:07.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:04 GMT'), ('x-amz-version-id', 'So5uynX-HAY7KHWnNki39RXeEqs8CN7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034e42384081f89b4-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.328 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.328 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.328 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.328 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.328 DEBUG:boto:Method: GET 2024-04-12T03:07:07.328 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.328 DEBUG:boto:Data: 2024-04-12T03:07:07.329 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.329 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.329 DEBUG:boto:Params: {} 2024-04-12T03:07:07.329 DEBUG:boto:Token: None 2024-04-12T03:07:07.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d40175fa07cef58-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.331 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.332 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.332 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.332 DEBUG:boto:path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.332 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.332 DEBUG:boto:Method: GET 2024-04-12T03:07:07.332 DEBUG:boto:Path: /dgbpxg-14/?prefix=dummy 2024-04-12T03:07:07.332 DEBUG:boto:Data: 2024-04-12T03:07:07.332 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.333 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.333 DEBUG:boto:Params: {} 2024-04-12T03:07:07.333 DEBUG:boto:Token: None 2024-04-12T03:07:07.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.333 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9144ce2e339b0f-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.335 DEBUG:boto:b'dgbpxg-14dummy1000falsedummy2024-04-12T03:06:04.288Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:07.336 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.336 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.336 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.336 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.336 DEBUG:boto:Method: GET 2024-04-12T03:07:07.336 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.336 DEBUG:boto:Data: 2024-04-12T03:07:07.337 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.337 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.337 DEBUG:boto:Params: {} 2024-04-12T03:07:07.337 DEBUG:boto:Token: None 2024-04-12T03:07:07.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082495e65b4858b05-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.339 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.339 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.339 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.340 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.340 DEBUG:boto:Method: GET 2024-04-12T03:07:07.340 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.340 DEBUG:boto:Data: 2024-04-12T03:07:07.340 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.340 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.340 DEBUG:boto:Params: {} 2024-04-12T03:07:07.340 DEBUG:boto:Token: None 2024-04-12T03:07:07.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073a5a1e1afb8ee8c-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.342 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.343 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.343 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.343 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.343 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.343 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.343 DEBUG:boto:Data: 2024-04-12T03:07:07.343 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.343 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.343 DEBUG:boto:Params: {} 2024-04-12T03:07:07.343 DEBUG:boto:Token: None 2024-04-12T03:07:07.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VKr4Hed20Knx8gEVGCXA2ZTI48c= 2024-04-12T03:07:07.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VKr4Hed20Knx8gEVGCXA2ZTI48c=', 'Content-Length': '0'} 2024-04-12T03:07:07.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5ec3a0677fce12c-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.345 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.345 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.345 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.346 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.346 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.346 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.346 DEBUG:boto:Data: 2024-04-12T03:07:07.346 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.346 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.346 DEBUG:boto:Params: {} 2024-04-12T03:07:07.346 DEBUG:boto:Token: None 2024-04-12T03:07:07.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VKr4Hed20Knx8gEVGCXA2ZTI48c= 2024-04-12T03:07:07.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VKr4Hed20Knx8gEVGCXA2ZTI48c=', 'Content-Length': '0'} 2024-04-12T03:07:07.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f62e954c7fe11c44-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:07.349 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.349 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.349 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.349 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.349 DEBUG:boto:Method: GET 2024-04-12T03:07:07.349 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.349 DEBUG:boto:Data: 2024-04-12T03:07:07.349 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.349 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.349 DEBUG:boto:Params: {} 2024-04-12T03:07:07.350 DEBUG:boto:Token: None 2024-04-12T03:07:07.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.350 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003abc0a80b3bbcbd5-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.394 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.394 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.395 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.395 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.395 DEBUG:boto:Method: GET 2024-04-12T03:07:07.395 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.395 DEBUG:boto:Data: 2024-04-12T03:07:07.395 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.395 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.395 DEBUG:boto:Params: {} 2024-04-12T03:07:07.395 DEBUG:boto:Token: None 2024-04-12T03:07:07.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c35f3f9b37180cb1-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.398 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.398 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.398 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.398 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.398 DEBUG:boto:Method: GET 2024-04-12T03:07:07.398 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.398 DEBUG:boto:Data: 2024-04-12T03:07:07.398 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.398 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.398 DEBUG:boto:Params: {} 2024-04-12T03:07:07.399 DEBUG:boto:Token: None 2024-04-12T03:07:07.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000526bb994f31cc28-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.401 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.401 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.401 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.401 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.402 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.402 DEBUG:boto:Method: GET 2024-04-12T03:07:07.402 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.402 DEBUG:boto:Data: 2024-04-12T03:07:07.402 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.402 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.402 DEBUG:boto:Params: {} 2024-04-12T03:07:07.402 DEBUG:boto:Token: None 2024-04-12T03:07:07.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe56c67becaaaadd-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.405 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.405 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.405 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.405 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.405 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:07:07.405 DEBUG:boto:Data: 2024-04-12T03:07:07.406 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.406 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.406 DEBUG:boto:Params: {} 2024-04-12T03:07:07.406 DEBUG:boto:Token: None 2024-04-12T03:07:07.406 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1 2024-04-12T03:07:07.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Iy3VTCwFJYcOCjVN779bubXw8Xw= 2024-04-12T03:07:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Iy3VTCwFJYcOCjVN779bubXw8Xw=', 'Content-Length': '0'} 2024-04-12T03:07:07.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4f0fd4e9f4c3192-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.408 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.408 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.408 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.409 DEBUG:boto:Path: /dgbpxg-14/obj-a1 2024-04-12T03:07:07.409 DEBUG:boto:Data: 2024-04-12T03:07:07.409 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.409 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.409 DEBUG:boto:Params: {} 2024-04-12T03:07:07.409 DEBUG:boto:Token: None 2024-04-12T03:07:07.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1 2024-04-12T03:07:07.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Iy3VTCwFJYcOCjVN779bubXw8Xw= 2024-04-12T03:07:07.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Iy3VTCwFJYcOCjVN779bubXw8Xw=', 'Content-Length': '0'} 2024-04-12T03:07:07.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6e14862652cc0df-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:07:07.411 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.411 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.412 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.412 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.412 DEBUG:boto:Method: GET 2024-04-12T03:07:07.412 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.412 DEBUG:boto:Data: 2024-04-12T03:07:07.412 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.412 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.412 DEBUG:boto:Params: {} 2024-04-12T03:07:07.412 DEBUG:boto:Token: None 2024-04-12T03:07:07.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043c2b33eb80217aa-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.458 DEBUG:boto:path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.458 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1 2024-04-12T03:07:07.459 DEBUG:boto:path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.459 DEBUG:boto:auth_path=/dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.459 DEBUG:boto:Method: GET 2024-04-12T03:07:07.459 DEBUG:boto:Path: /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.459 DEBUG:boto:Data: 2024-04-12T03:07:07.459 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.459 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.459 DEBUG:boto:Params: {} 2024-04-12T03:07:07.459 DEBUG:boto:Token: None 2024-04-12T03:07:07.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a1?versionId=BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU 2024-04-12T03:07:07.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU= 2024-04-12T03:07:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJlhH4QHa8XLIS3jSu9XUXksTRU=', 'Content-Length': '0'} 2024-04-12T03:07:07.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'BA5eHZGRFuVxE2d2CpBOPmfBr0AWghU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002905b86d4907c3e1-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.462 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.462 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.462 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.462 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.462 DEBUG:boto:Method: GET 2024-04-12T03:07:07.462 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.462 DEBUG:boto:Data: 2024-04-12T03:07:07.462 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.463 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.463 DEBUG:boto:Params: {} 2024-04-12T03:07:07.463 DEBUG:boto:Token: None 2024-04-12T03:07:07.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ea0820b9cf9197-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.465 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.465 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.466 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.466 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.466 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.466 DEBUG:boto:Method: GET 2024-04-12T03:07:07.466 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a1 2024-04-12T03:07:07.466 DEBUG:boto:Data: 2024-04-12T03:07:07.466 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.466 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.466 DEBUG:boto:Params: {} 2024-04-12T03:07:07.467 DEBUG:boto:Token: None 2024-04-12T03:07:07.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02ba193c04f3945-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.469 DEBUG:boto:b'dgbpxg-14obj-a11000falseobj-a12024-04-12T03:06:19.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:07.469 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.469 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.469 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.469 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.470 DEBUG:boto:Method: GET 2024-04-12T03:07:07.470 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.470 DEBUG:boto:Data: 2024-04-12T03:07:07.470 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.470 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.470 DEBUG:boto:Params: {} 2024-04-12T03:07:07.470 DEBUG:boto:Token: None 2024-04-12T03:07:07.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c3688cb8fc44d8f-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.472 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.472 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.473 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.473 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.473 DEBUG:boto:Method: GET 2024-04-12T03:07:07.473 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.473 DEBUG:boto:Data: 2024-04-12T03:07:07.473 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.473 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.473 DEBUG:boto:Params: {} 2024-04-12T03:07:07.473 DEBUG:boto:Token: None 2024-04-12T03:07:07.474 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.474 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d04933b52aa1f774-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.476 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.476 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.476 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.476 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.476 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.477 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.477 DEBUG:boto:Data: 2024-04-12T03:07:07.477 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.477 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.477 DEBUG:boto:Params: {} 2024-04-12T03:07:07.477 DEBUG:boto:Token: None 2024-04-12T03:07:07.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.478 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HXunmU+LlbyYyeMXd2bhqXdeF5I= 2024-04-12T03:07:07.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HXunmU+LlbyYyeMXd2bhqXdeF5I=', 'Content-Length': '0'} 2024-04-12T03:07:07.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a294e5d730b3e7b-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.479 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.479 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.480 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.480 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.480 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.480 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.480 DEBUG:boto:Data: 2024-04-12T03:07:07.480 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.480 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.480 DEBUG:boto:Params: {} 2024-04-12T03:07:07.480 DEBUG:boto:Token: None 2024-04-12T03:07:07.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HXunmU+LlbyYyeMXd2bhqXdeF5I= 2024-04-12T03:07:07.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HXunmU+LlbyYyeMXd2bhqXdeF5I=', 'Content-Length': '0'} 2024-04-12T03:07:07.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dffa2d4fc3870fd-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:07.483 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.483 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.483 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.483 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.483 DEBUG:boto:Method: GET 2024-04-12T03:07:07.483 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.484 DEBUG:boto:Data: 2024-04-12T03:07:07.484 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.484 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.484 DEBUG:boto:Params: {} 2024-04-12T03:07:07.484 DEBUG:boto:Token: None 2024-04-12T03:07:07.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015086751614ce948-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.486 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.486 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.487 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.487 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.487 DEBUG:boto:Method: GET 2024-04-12T03:07:07.487 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.487 DEBUG:boto:Data: 2024-04-12T03:07:07.487 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.487 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.487 DEBUG:boto:Params: {} 2024-04-12T03:07:07.487 DEBUG:boto:Token: None 2024-04-12T03:07:07.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bce85c5e8271ddc6-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.530 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.530 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.531 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.531 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.531 DEBUG:boto:Method: GET 2024-04-12T03:07:07.531 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.531 DEBUG:boto:Data: 2024-04-12T03:07:07.531 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.531 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.531 DEBUG:boto:Params: {} 2024-04-12T03:07:07.531 DEBUG:boto:Token: None 2024-04-12T03:07:07.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596d05ab236bc866-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.534 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.534 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.534 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.534 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.534 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.534 DEBUG:boto:Method: GET 2024-04-12T03:07:07.534 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.534 DEBUG:boto:Data: 2024-04-12T03:07:07.535 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.535 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.535 DEBUG:boto:Params: {} 2024-04-12T03:07:07.535 DEBUG:boto:Token: None 2024-04-12T03:07:07.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0de2cafb1196b6d-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.537 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.538 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.538 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.538 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.538 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:07:07.538 DEBUG:boto:Data: 2024-04-12T03:07:07.538 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.538 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.539 DEBUG:boto:Params: {} 2024-04-12T03:07:07.539 DEBUG:boto:Token: None 2024-04-12T03:07:07.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2 2024-04-12T03:07:07.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1FAyoG6DO1lvJq1SCaa/p6gEFaQ= 2024-04-12T03:07:07.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1FAyoG6DO1lvJq1SCaa/p6gEFaQ=', 'Content-Length': '0'} 2024-04-12T03:07:07.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3b1769300fce91f-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.541 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.541 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.541 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.541 DEBUG:boto:Path: /dgbpxg-14/obj-a2 2024-04-12T03:07:07.541 DEBUG:boto:Data: 2024-04-12T03:07:07.541 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.542 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.542 DEBUG:boto:Params: {} 2024-04-12T03:07:07.542 DEBUG:boto:Token: None 2024-04-12T03:07:07.542 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2 2024-04-12T03:07:07.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1FAyoG6DO1lvJq1SCaa/p6gEFaQ= 2024-04-12T03:07:07.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1FAyoG6DO1lvJq1SCaa/p6gEFaQ=', 'Content-Length': '0'} 2024-04-12T03:07:07.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000953f54d28cb9a79-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:07:07.544 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.544 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.544 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.544 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.544 DEBUG:boto:Method: GET 2024-04-12T03:07:07.545 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.545 DEBUG:boto:Data: 2024-04-12T03:07:07.545 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.545 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.545 DEBUG:boto:Params: {} 2024-04-12T03:07:07.545 DEBUG:boto:Token: None 2024-04-12T03:07:07.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075889710decb1eb7-006618a55b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.548 DEBUG:boto:path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.548 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2 2024-04-12T03:07:07.548 DEBUG:boto:path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.548 DEBUG:boto:auth_path=/dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.548 DEBUG:boto:Method: GET 2024-04-12T03:07:07.548 DEBUG:boto:Path: /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.548 DEBUG:boto:Data: 2024-04-12T03:07:07.548 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.548 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.548 DEBUG:boto:Params: {} 2024-04-12T03:07:07.549 DEBUG:boto:Token: None 2024-04-12T03:07:07.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/obj-a2?versionId=fm9fz.Se6rpbuLcZyZup5nylTrecv0N 2024-04-12T03:07:07.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w= 2024-04-12T03:07:07.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xT2ff5q5CqzP//dDFahcGkF5u9w=', 'Content-Length': '0'} 2024-04-12T03:07:07.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:06:19 GMT'), ('x-amz-version-id', 'fm9fz.Se6rpbuLcZyZup5nylTrecv0N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f331a3ffb3493397-006618a55b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.594 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.594 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.594 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.595 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.595 DEBUG:boto:Method: GET 2024-04-12T03:07:07.595 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.595 DEBUG:boto:Data: 2024-04-12T03:07:07.595 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:07.595 DEBUG:boto:Port: 8001 2024-04-12T03:07:07.595 DEBUG:boto:Params: {} 2024-04-12T03:07:07.595 DEBUG:boto:Token: None 2024-04-12T03:07:07.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5fc0ae6f4721338-006618a55b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.598 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.598 DEBUG:boto:path=/dgbpxg-14/ 2024-04-12T03:07:07.598 DEBUG:boto:auth_path=/dgbpxg-14/ 2024-04-12T03:07:07.599 DEBUG:boto:path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.599 DEBUG:boto:auth_path=/dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.599 DEBUG:boto:Method: GET 2024-04-12T03:07:07.599 DEBUG:boto:Path: /dgbpxg-14/?prefix=obj-a2 2024-04-12T03:07:07.599 DEBUG:boto:Data: 2024-04-12T03:07:07.599 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.599 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.599 DEBUG:boto:Params: {} 2024-04-12T03:07:07.599 DEBUG:boto:Token: None 2024-04-12T03:07:07.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-14/ 2024-04-12T03:07:07.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0= 2024-04-12T03:07:07.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:McuCDOYIFqMF+46O7zz5O3InFy0=', 'Content-Length': '0'} 2024-04-12T03:07:07.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f4581499b53dac-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.602 DEBUG:boto:b'dgbpxg-14obj-a21000falseobj-a22024-04-12T03:06:19.699Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:07.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-14 zones={a2, a1} 2024-04-12T03:07:07.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-12T03:07:07.604 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:07.604 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:07.604 DEBUG:boto:Method: PUT 2024-04-12T03:07:07.604 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:07.604 DEBUG:boto:Data: 2024-04-12T03:07:07.604 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.604 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.604 DEBUG:boto:Params: {} 2024-04-12T03:07:07.605 DEBUG:boto:Token: None 2024-04-12T03:07:07.605 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-15/ 2024-04-12T03:07:07.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zzqUFuiW9JraRjL0zGtRTuGh138= 2024-04-12T03:07:07.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zzqUFuiW9JraRjL0zGtRTuGh138=', 'Content-Length': '0'} 2024-04-12T03:07:07.617 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dfc8bc4b6864865b-006618a55b-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.618 DEBUG:rgw_multi.tests:created bucket=dgbpxg-15 2024-04-12T03:07:07.618 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:07.618 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:07.618 DEBUG:boto:path=/dgbpxg-15/?versioning 2024-04-12T03:07:07.618 DEBUG:boto:auth_path=/dgbpxg-15/?versioning 2024-04-12T03:07:07.618 DEBUG:boto:Method: PUT 2024-04-12T03:07:07.618 DEBUG:boto:Path: /dgbpxg-15/?versioning 2024-04-12T03:07:07.618 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:07:07.618 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.619 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.619 DEBUG:boto:Params: {} 2024-04-12T03:07:07.619 DEBUG:boto:Token: None 2024-04-12T03:07:07.619 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-15/?versioning 2024-04-12T03:07:07.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:064De9X+NYsJg7AxL3VGDkyHXN0= 2024-04-12T03:07:07.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:064De9X+NYsJg7AxL3VGDkyHXN0=', 'Content-Length': '228'} 2024-04-12T03:07:07.623 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078e495e9eacbd7e3-006618a55b-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:07:07.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:07:07.624 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:07:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:07.701 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:07:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891227.622790_266.1", 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:07:07.622790Z" 2024-04-12T03:07:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.715 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:07:07.716 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891227.622790_266.1'} 2024-04-12T03:07:07.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:07:07.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:07:07.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:07:07.716 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:07:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:07.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.792+0000 7f6abdf08780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:07:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.794+0000 7f6abdf08780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:07:07.807 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:07:07.807 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:07:07.807 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:07:07.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:07:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:07:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:07:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891227.622790_266.1", 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:07.622790Z", 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:07:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:07:07Z" 2024-04-12T03:07:07.814 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:07:07.815 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891227.622790_266.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:07.622790Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:07:07Z" } 2024-04-12T03:07:07.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891227.622790_266.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:07.622790Z', 'realm_epoch': 2}}] 2024-04-12T03:07:07.815 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891227.622790_266.1'} 2024-04-12T03:07:07.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891227.622790_266.1'} 2024-04-12T03:07:07.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:07:07.816 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:07.816 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:07.816 DEBUG:boto:Method: HEAD 2024-04-12T03:07:07.816 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:07.816 DEBUG:boto:Data: 2024-04-12T03:07:07.816 DEBUG:boto:Headers: {} 2024-04-12T03:07:07.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.817 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.817 DEBUG:boto:Params: {} 2024-04-12T03:07:07.817 DEBUG:boto:Token: None 2024-04-12T03:07:07.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-15/ 2024-04-12T03:07:07.817 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xNOD7haM1nwrKuGD0ZgbFpbLj4= 2024-04-12T03:07:07.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xNOD7haM1nwrKuGD0ZgbFpbLj4=', 'Content-Length': '0'} 2024-04-12T03:07:07.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035baa58d0aabbc2a-006618a55b-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.820 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:07.820 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:07.820 DEBUG:boto:Method: PUT 2024-04-12T03:07:07.820 DEBUG:boto:Path: /dgbpxg-15/dummy 2024-04-12T03:07:07.820 DEBUG:boto:Data: 2024-04-12T03:07:07.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:07:07.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:07.821 DEBUG:boto:Port: 8000 2024-04-12T03:07:07.821 DEBUG:boto:Params: {} 2024-04-12T03:07:07.821 DEBUG:boto:Token: None 2024-04-12T03:07:07.821 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:07:07 GMT /dgbpxg-15/dummy 2024-04-12T03:07:07.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H/Eem6orBCn9Vtmf9gfHPV/M6Zg= 2024-04-12T03:07:07.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H/Eem6orBCn9Vtmf9gfHPV/M6Zg='} 2024-04-12T03:07:07.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-amz-request-id', 'tx0000096f16f882ddf7d54-006618a55b-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:07.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-15 2024-04-12T03:07:07.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:07.832 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:07.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:07.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:07.917+0000 7f8240b04780 1 waiting to reach incremental sync.. 2024-04-12T03:07:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:12.918+0000 7f8240b04780 1 waiting to reach incremental sync.. 2024-04-12T03:07:17.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:17.919+0000 7f8240b04780 1 waiting to reach incremental sync.. 2024-04-12T03:07:22.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:22.919+0000 7f8240b04780 1 waiting to reach incremental sync.. 2024-04-12T03:07:24.797 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:07:24.796+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:07:27.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:27.921+0000 7f8240b04780 1 bucket sync caught up with source: 2024-04-12T03:07:27.923 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , , , ] 2024-04-12T03:07:27.923 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.24.11, , , , , ] 2024-04-12T03:07:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:27.921+0000 7f8240b04780 0 bucket checkpoint complete 2024-04-12T03:07:27.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-15 2024-04-12T03:07:27.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:27.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:27.939 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:28.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:28.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:28.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:28.036+0000 7fa45c99e780 1 waiting to reach incremental sync.. 2024-04-12T03:07:33.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:33.037+0000 7fa45c99e780 1 waiting to reach incremental sync.. 2024-04-12T03:07:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:38.037+0000 7fa45c99e780 1 waiting to reach incremental sync.. 2024-04-12T03:07:43.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:43.039+0000 7fa45c99e780 1 bucket sync caught up with source: 2024-04-12T03:07:43.041 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:07:43.041 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:07:43.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:43.039+0000 7fa45c99e780 0 bucket checkpoint complete 2024-04-12T03:07:43.056 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-15 zones={a2, a1} 2024-04-12T03:07:43.056 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.057 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.057 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.057 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.057 DEBUG:boto:Data: 2024-04-12T03:07:43.057 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.057 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.057 DEBUG:boto:Params: {} 2024-04-12T03:07:43.057 DEBUG:boto:Token: None 2024-04-12T03:07:43.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029eed153d426eaf1-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.061 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.061 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.061 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.061 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.061 DEBUG:boto:Data: 2024-04-12T03:07:43.061 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.061 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.061 DEBUG:boto:Params: {} 2024-04-12T03:07:43.062 DEBUG:boto:Token: None 2024-04-12T03:07:43.062 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bfcee03e4e9271a-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:07:43.065 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.065 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.065 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.065 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.065 DEBUG:boto:Method: GET 2024-04-12T03:07:43.065 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.065 DEBUG:boto:Data: 2024-04-12T03:07:43.065 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.065 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.065 DEBUG:boto:Params: {} 2024-04-12T03:07:43.066 DEBUG:boto:Token: None 2024-04-12T03:07:43.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/?versions 2024-04-12T03:07:43.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc= 2024-04-12T03:07:43.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc=', 'Content-Length': '0'} 2024-04-12T03:07:43.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acda26d218ef6d31-006618a57f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.068 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.069 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:43.069 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:07:43.069 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.069 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.069 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.069 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.069 DEBUG:boto:Method: GET 2024-04-12T03:07:43.069 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.070 DEBUG:boto:Data: 2024-04-12T03:07:43.070 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.070 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.070 DEBUG:boto:Params: {} 2024-04-12T03:07:43.070 DEBUG:boto:Token: None 2024-04-12T03:07:43.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/?versions 2024-04-12T03:07:43.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc= 2024-04-12T03:07:43.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc=', 'Content-Length': '0'} 2024-04-12T03:07:43.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5fdbaa2b2142c2-006618a57f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.073 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.073 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:07:43.073 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.073 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.073 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.073 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.074 DEBUG:boto:Method: GET 2024-04-12T03:07:43.074 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.074 DEBUG:boto:Data: 2024-04-12T03:07:43.074 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.074 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.074 DEBUG:boto:Params: {} 2024-04-12T03:07:43.074 DEBUG:boto:Token: None 2024-04-12T03:07:43.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/?versions 2024-04-12T03:07:43.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc= 2024-04-12T03:07:43.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc=', 'Content-Length': '0'} 2024-04-12T03:07:43.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2154274fb44113-006618a57f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.077 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.077 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.077 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.077 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.077 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.077 DEBUG:boto:Method: GET 2024-04-12T03:07:43.078 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:07:43.078 DEBUG:boto:Data: 2024-04-12T03:07:43.078 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.078 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.078 DEBUG:boto:Params: {} 2024-04-12T03:07:43.078 DEBUG:boto:Token: None 2024-04-12T03:07:43.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/?versions 2024-04-12T03:07:43.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc= 2024-04-12T03:07:43.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p2V49EK/0Otx+os5Sf+vyGlFnUc=', 'Content-Length': '0'} 2024-04-12T03:07:43.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001badd22af2cb1f78-006618a57f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.081 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:43.081 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.081 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.081 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.081 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.082 DEBUG:boto:Method: GET 2024-04-12T03:07:43.082 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.082 DEBUG:boto:Data: 2024-04-12T03:07:43.082 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.082 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.082 DEBUG:boto:Params: {} 2024-04-12T03:07:43.082 DEBUG:boto:Token: None 2024-04-12T03:07:43.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec63973dcb31f4e2-006618a57f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.085 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.085 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.085 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.085 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.085 DEBUG:boto:Method: GET 2024-04-12T03:07:43.085 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.085 DEBUG:boto:Data: 2024-04-12T03:07:43.085 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.086 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.086 DEBUG:boto:Params: {} 2024-04-12T03:07:43.086 DEBUG:boto:Token: None 2024-04-12T03:07:43.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7878bfa29d3137a-006618a57f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.088 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.088 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.088 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.088 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.088 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.088 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.088 DEBUG:boto:Data: 2024-04-12T03:07:43.088 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.089 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.089 DEBUG:boto:Params: {} 2024-04-12T03:07:43.089 DEBUG:boto:Token: None 2024-04-12T03:07:43.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3RGDY1ZB9G2/DHydzxl56Tz/+I= 2024-04-12T03:07:43.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3RGDY1ZB9G2/DHydzxl56Tz/+I=', 'Content-Length': '0'} 2024-04-12T03:07:43.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029256e450c3fdc72-006618a57f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.091 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.091 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.091 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.092 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.092 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.092 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.092 DEBUG:boto:Data: 2024-04-12T03:07:43.092 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.092 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.092 DEBUG:boto:Params: {} 2024-04-12T03:07:43.092 DEBUG:boto:Token: None 2024-04-12T03:07:43.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3RGDY1ZB9G2/DHydzxl56Tz/+I= 2024-04-12T03:07:43.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3RGDY1ZB9G2/DHydzxl56Tz/+I=', 'Content-Length': '0'} 2024-04-12T03:07:43.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000055aa4a6ee75546a-006618a57f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:43.095 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.095 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.096 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.096 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.096 DEBUG:boto:Method: GET 2024-04-12T03:07:43.096 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.096 DEBUG:boto:Data: 2024-04-12T03:07:43.096 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.096 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.096 DEBUG:boto:Params: {} 2024-04-12T03:07:43.096 DEBUG:boto:Token: None 2024-04-12T03:07:43.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015a139ff4148da31-006618a57f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.099 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.099 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.099 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.099 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.099 DEBUG:boto:Method: GET 2024-04-12T03:07:43.099 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.099 DEBUG:boto:Data: 2024-04-12T03:07:43.099 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.100 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.100 DEBUG:boto:Params: {} 2024-04-12T03:07:43.100 DEBUG:boto:Token: None 2024-04-12T03:07:43.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2286d62afacd07f-006618a57f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.102 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.102 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.102 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.102 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.102 DEBUG:boto:Method: GET 2024-04-12T03:07:43.102 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.103 DEBUG:boto:Data: 2024-04-12T03:07:43.103 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.103 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.103 DEBUG:boto:Params: {} 2024-04-12T03:07:43.103 DEBUG:boto:Token: None 2024-04-12T03:07:43.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8= 2024-04-12T03:07:43.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8=', 'Content-Length': '0'} 2024-04-12T03:07:43.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64eb08ad719a5e7-006618a57f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.146 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.147 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.147 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.147 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.147 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.147 DEBUG:boto:Method: GET 2024-04-12T03:07:43.147 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.147 DEBUG:boto:Data: 2024-04-12T03:07:43.147 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.147 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.147 DEBUG:boto:Params: {} 2024-04-12T03:07:43.148 DEBUG:boto:Token: None 2024-04-12T03:07:43.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8= 2024-04-12T03:07:43.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8=', 'Content-Length': '0'} 2024-04-12T03:07:43.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005377441d2855e397-006618a57f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.150 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.150 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.150 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.151 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.151 DEBUG:boto:Path: /dgbpxg-15/dummy 2024-04-12T03:07:43.151 DEBUG:boto:Data: 2024-04-12T03:07:43.151 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.151 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.151 DEBUG:boto:Params: {} 2024-04-12T03:07:43.151 DEBUG:boto:Token: None 2024-04-12T03:07:43.151 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy 2024-04-12T03:07:43.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2J6RhFe2J+aFNVjuVnT06Wsd6Ns= 2024-04-12T03:07:43.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2J6RhFe2J+aFNVjuVnT06Wsd6Ns=', 'Content-Length': '0'} 2024-04-12T03:07:43.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dc6fafbedf4e7a1-006618a57f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.154 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.154 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.154 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.154 DEBUG:boto:Path: /dgbpxg-15/dummy 2024-04-12T03:07:43.154 DEBUG:boto:Data: 2024-04-12T03:07:43.154 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.154 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.154 DEBUG:boto:Params: {} 2024-04-12T03:07:43.155 DEBUG:boto:Token: None 2024-04-12T03:07:43.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy 2024-04-12T03:07:43.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2J6RhFe2J+aFNVjuVnT06Wsd6Ns= 2024-04-12T03:07:43.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2J6RhFe2J+aFNVjuVnT06Wsd6Ns=', 'Content-Length': '0'} 2024-04-12T03:07:43.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbaa56f8e9c15ca9-006618a57f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:07:43.157 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.157 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.157 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.157 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.157 DEBUG:boto:Method: GET 2024-04-12T03:07:43.157 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.157 DEBUG:boto:Data: 2024-04-12T03:07:43.157 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.158 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.158 DEBUG:boto:Params: {} 2024-04-12T03:07:43.158 DEBUG:boto:Token: None 2024-04-12T03:07:43.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003cdf031037091601-006618a57f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.160 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:07:43.160 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:07:43.160 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.160 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.160 DEBUG:boto:Method: GET 2024-04-12T03:07:43.160 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.161 DEBUG:boto:Data: 2024-04-12T03:07:43.161 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.161 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.161 DEBUG:boto:Params: {} 2024-04-12T03:07:43.161 DEBUG:boto:Token: None 2024-04-12T03:07:43.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:07:43.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo= 2024-04-12T03:07:43.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpVVyfmFNVlvshgwo3SodG7ZbHo=', 'Content-Length': '0'} 2024-04-12T03:07:43.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076d3b4392e6d4740-006618a57f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.163 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.163 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.163 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.163 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.163 DEBUG:boto:Method: GET 2024-04-12T03:07:43.164 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.164 DEBUG:boto:Data: 2024-04-12T03:07:43.164 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.164 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.164 DEBUG:boto:Params: {} 2024-04-12T03:07:43.164 DEBUG:boto:Token: None 2024-04-12T03:07:43.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8= 2024-04-12T03:07:43.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8=', 'Content-Length': '0'} 2024-04-12T03:07:43.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034539cea69781441-006618a57f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.167 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.167 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.167 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.167 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.167 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.167 DEBUG:boto:Method: GET 2024-04-12T03:07:43.167 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:07:43.167 DEBUG:boto:Data: 2024-04-12T03:07:43.168 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.168 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.168 DEBUG:boto:Params: {} 2024-04-12T03:07:43.168 DEBUG:boto:Token: None 2024-04-12T03:07:43.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8= 2024-04-12T03:07:43.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWftZ5z6G0QEWW523c1dmaZDuw8=', 'Content-Length': '0'} 2024-04-12T03:07:43.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076406cc3a6493c8c-006618a57f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.214 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:07:43.215 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-15 zones={a2, a1} 2024-04-12T03:07:43.215 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.216 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.216 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.216 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.216 DEBUG:boto:Data: 2024-04-12T03:07:43.216 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.217 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.217 DEBUG:boto:Params: {} 2024-04-12T03:07:43.217 DEBUG:boto:Token: None 2024-04-12T03:07:43.218 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adb0e2f239462f4d-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.221 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.221 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.221 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.221 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.221 DEBUG:boto:Data: 2024-04-12T03:07:43.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.222 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.222 DEBUG:boto:Params: {} 2024-04-12T03:07:43.222 DEBUG:boto:Token: None 2024-04-12T03:07:43.223 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-amz-request-id', 'tx00000f1b2a47cc7d34d30-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.234 DEBUG:rgw_multi.tests:zone=a1 version=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:07:43.234 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.234 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.234 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.235 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.235 DEBUG:boto:Data: 2024-04-12T03:07:43.235 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.235 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.235 DEBUG:boto:Params: {} 2024-04-12T03:07:43.235 DEBUG:boto:Token: None 2024-04-12T03:07:43.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b73944af44b8c13-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.238 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.238 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.238 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.238 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.238 DEBUG:boto:Data: 2024-04-12T03:07:43.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.239 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.239 DEBUG:boto:Params: {} 2024-04-12T03:07:43.239 DEBUG:boto:Token: None 2024-04-12T03:07:43.239 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-amz-request-id', 'tx000003733c7c474320838-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.249 DEBUG:rgw_multi.tests:zone=a2 version=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:07:43.250 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.250 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.250 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.250 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.250 DEBUG:boto:Data: 2024-04-12T03:07:43.250 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.250 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.250 DEBUG:boto:Params: {} 2024-04-12T03:07:43.251 DEBUG:boto:Token: None 2024-04-12T03:07:43.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000216d939e3c372b41-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.253 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.253 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.253 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.254 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.254 DEBUG:boto:Data: 2024-04-12T03:07:43.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.254 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.254 DEBUG:boto:Params: {} 2024-04-12T03:07:43.254 DEBUG:boto:Token: None 2024-04-12T03:07:43.254 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.255 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-amz-request-id', 'tx00000752d824928d4c489-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.264 DEBUG:rgw_multi.tests:zone=a1 version=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:07:43.264 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.264 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.264 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.264 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.264 DEBUG:boto:Data: 2024-04-12T03:07:43.264 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.264 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.264 DEBUG:boto:Params: {} 2024-04-12T03:07:43.264 DEBUG:boto:Token: None 2024-04-12T03:07:43.265 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.265 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f29dc4bcdd4e32c-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.267 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.267 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.267 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.267 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.267 DEBUG:boto:Data: 2024-04-12T03:07:43.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.268 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.268 DEBUG:boto:Params: {} 2024-04-12T03:07:43.268 DEBUG:boto:Token: None 2024-04-12T03:07:43.268 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-amz-request-id', 'tx00000f8386d07201e2b6a-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.311 DEBUG:rgw_multi.tests:zone=a2 version=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:07:43.311 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.311 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.311 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.311 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.311 DEBUG:boto:Data: 2024-04-12T03:07:43.311 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.312 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.312 DEBUG:boto:Params: {} 2024-04-12T03:07:43.312 DEBUG:boto:Token: None 2024-04-12T03:07:43.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053740e0c02bd8faa-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.315 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.315 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.315 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.315 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.315 DEBUG:boto:Data: 2024-04-12T03:07:43.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.315 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.315 DEBUG:boto:Params: {} 2024-04-12T03:07:43.315 DEBUG:boto:Token: None 2024-04-12T03:07:43.316 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-amz-request-id', 'tx000009d6651b75b66f289-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.359 DEBUG:rgw_multi.tests:zone=a1 version=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:07:43.359 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.359 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.359 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.359 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.359 DEBUG:boto:Data: 2024-04-12T03:07:43.359 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.360 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.360 DEBUG:boto:Params: {} 2024-04-12T03:07:43.360 DEBUG:boto:Token: None 2024-04-12T03:07:43.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000980a4e1ab8f4cc4d-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.362 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.363 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.363 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.363 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.363 DEBUG:boto:Data: 2024-04-12T03:07:43.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.363 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.363 DEBUG:boto:Params: {} 2024-04-12T03:07:43.363 DEBUG:boto:Token: None 2024-04-12T03:07:43.363 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-amz-request-id', 'tx000006b562ef4a0f26684-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.372 DEBUG:rgw_multi.tests:zone=a2 version=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:07:43.372 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.372 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.372 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.372 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.372 DEBUG:boto:Data: 2024-04-12T03:07:43.372 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.373 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.373 DEBUG:boto:Params: {} 2024-04-12T03:07:43.373 DEBUG:boto:Token: None 2024-04-12T03:07:43.373 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079cc7d5c5944576a-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.375 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.376 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.376 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.376 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.376 DEBUG:boto:Data: 2024-04-12T03:07:43.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.376 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.376 DEBUG:boto:Params: {} 2024-04-12T03:07:43.376 DEBUG:boto:Token: None 2024-04-12T03:07:43.376 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.377 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-amz-request-id', 'tx000004e87efba42f8d0e9-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.385 DEBUG:rgw_multi.tests:zone=a1 version=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:07:43.385 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.385 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.385 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.385 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.385 DEBUG:boto:Data: 2024-04-12T03:07:43.385 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.385 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.385 DEBUG:boto:Params: {} 2024-04-12T03:07:43.386 DEBUG:boto:Token: None 2024-04-12T03:07:43.386 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000231f0664073dac3a-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.388 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.388 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.388 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.388 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.388 DEBUG:boto:Data: 2024-04-12T03:07:43.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.389 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.389 DEBUG:boto:Params: {} 2024-04-12T03:07:43.389 DEBUG:boto:Token: None 2024-04-12T03:07:43.389 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.389 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-amz-request-id', 'tx000001330ec4c150f874d-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.397 DEBUG:rgw_multi.tests:zone=a2 version=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:07:43.397 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.398 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.398 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.398 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.398 DEBUG:boto:Data: 2024-04-12T03:07:43.398 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.398 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.398 DEBUG:boto:Params: {} 2024-04-12T03:07:43.398 DEBUG:boto:Token: None 2024-04-12T03:07:43.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009480be70cead4388-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.401 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.401 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.401 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.401 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.401 DEBUG:boto:Data: 2024-04-12T03:07:43.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.402 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.402 DEBUG:boto:Params: {} 2024-04-12T03:07:43.402 DEBUG:boto:Token: None 2024-04-12T03:07:43.402 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-amz-request-id', 'tx00000a77ff63ae600170e-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.447 DEBUG:rgw_multi.tests:zone=a1 version=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:07:43.447 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.447 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.447 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.447 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.447 DEBUG:boto:Data: 2024-04-12T03:07:43.447 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.447 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.448 DEBUG:boto:Params: {} 2024-04-12T03:07:43.448 DEBUG:boto:Token: None 2024-04-12T03:07:43.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028064b50ca5ca31e-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.450 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.450 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.451 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.451 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.451 DEBUG:boto:Data: 2024-04-12T03:07:43.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.451 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.451 DEBUG:boto:Params: {} 2024-04-12T03:07:43.451 DEBUG:boto:Token: None 2024-04-12T03:07:43.451 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-amz-request-id', 'tx00000f0b7c8b120d65f36-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.462 DEBUG:rgw_multi.tests:zone=a2 version=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:07:43.462 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.462 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.462 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.462 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.463 DEBUG:boto:Data: 2024-04-12T03:07:43.463 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.463 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.463 DEBUG:boto:Params: {} 2024-04-12T03:07:43.463 DEBUG:boto:Token: None 2024-04-12T03:07:43.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b4b8497a8ae89ff-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.465 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.466 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.466 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.466 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.466 DEBUG:boto:Data: 2024-04-12T03:07:43.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.466 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.466 DEBUG:boto:Params: {} 2024-04-12T03:07:43.466 DEBUG:boto:Token: None 2024-04-12T03:07:43.466 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-amz-request-id', 'tx00000a3bcb24363c68581-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.474 DEBUG:rgw_multi.tests:zone=a1 version=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:07:43.474 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.475 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.475 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.475 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.475 DEBUG:boto:Data: 2024-04-12T03:07:43.475 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.475 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.475 DEBUG:boto:Params: {} 2024-04-12T03:07:43.475 DEBUG:boto:Token: None 2024-04-12T03:07:43.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b094ad731fd5d2e-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.478 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.478 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.478 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.478 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.478 DEBUG:boto:Data: 2024-04-12T03:07:43.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.479 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.479 DEBUG:boto:Params: {} 2024-04-12T03:07:43.479 DEBUG:boto:Token: None 2024-04-12T03:07:43.479 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-amz-request-id', 'tx000009f2e6dcac59d0c80-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.487 DEBUG:rgw_multi.tests:zone=a2 version=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:07:43.487 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.487 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.487 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.488 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.488 DEBUG:boto:Data: 2024-04-12T03:07:43.488 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.488 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.488 DEBUG:boto:Params: {} 2024-04-12T03:07:43.488 DEBUG:boto:Token: None 2024-04-12T03:07:43.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ce1e703928442d8-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.491 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.491 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.491 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.491 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.491 DEBUG:boto:Data: 2024-04-12T03:07:43.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.492 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.492 DEBUG:boto:Params: {} 2024-04-12T03:07:43.492 DEBUG:boto:Token: None 2024-04-12T03:07:43.492 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.492 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-amz-request-id', 'tx00000fe32e3c705488366-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.535 DEBUG:rgw_multi.tests:zone=a1 version=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:07:43.535 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.535 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.535 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.535 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.535 DEBUG:boto:Data: 2024-04-12T03:07:43.535 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.535 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.536 DEBUG:boto:Params: {} 2024-04-12T03:07:43.536 DEBUG:boto:Token: None 2024-04-12T03:07:43.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009b60df1f7c32990-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.538 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.538 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.539 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.539 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.539 DEBUG:boto:Data: 2024-04-12T03:07:43.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.539 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.539 DEBUG:boto:Params: {} 2024-04-12T03:07:43.539 DEBUG:boto:Token: None 2024-04-12T03:07:43.539 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-amz-request-id', 'tx00000f66107815ee73eca-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.551 DEBUG:rgw_multi.tests:zone=a2 version=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:07:43.551 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.551 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.551 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.551 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.551 DEBUG:boto:Data: 2024-04-12T03:07:43.552 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.552 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.552 DEBUG:boto:Params: {} 2024-04-12T03:07:43.552 DEBUG:boto:Token: None 2024-04-12T03:07:43.552 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a58427caf89318a5-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.555 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.555 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.555 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.555 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.555 DEBUG:boto:Data: 2024-04-12T03:07:43.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.556 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.556 DEBUG:boto:Params: {} 2024-04-12T03:07:43.556 DEBUG:boto:Token: None 2024-04-12T03:07:43.556 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-amz-request-id', 'tx0000096c3df7ccffdf752-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.564 DEBUG:rgw_multi.tests:zone=a1 version=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:07:43.564 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.564 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.565 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.565 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.565 DEBUG:boto:Data: 2024-04-12T03:07:43.565 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.565 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.565 DEBUG:boto:Params: {} 2024-04-12T03:07:43.565 DEBUG:boto:Token: None 2024-04-12T03:07:43.565 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.566 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd215db65e9cdd27-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.568 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.569 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.569 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.569 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.569 DEBUG:boto:Data: 2024-04-12T03:07:43.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.569 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.570 DEBUG:boto:Params: {} 2024-04-12T03:07:43.570 DEBUG:boto:Token: None 2024-04-12T03:07:43.570 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.570 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-amz-request-id', 'tx000009f83eff61ff0efeb-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.614 DEBUG:rgw_multi.tests:zone=a2 version=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:07:43.615 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.615 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.615 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.615 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.615 DEBUG:boto:Data: 2024-04-12T03:07:43.615 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.615 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.615 DEBUG:boto:Params: {} 2024-04-12T03:07:43.615 DEBUG:boto:Token: None 2024-04-12T03:07:43.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4cfc3b87539b9ab-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.618 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.618 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.619 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.619 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.619 DEBUG:boto:Data: 2024-04-12T03:07:43.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.619 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.619 DEBUG:boto:Params: {} 2024-04-12T03:07:43.619 DEBUG:boto:Token: None 2024-04-12T03:07:43.619 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-amz-request-id', 'tx000000623102cb769ba4e-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.629 DEBUG:rgw_multi.tests:zone=a1 version=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:07:43.629 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.629 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.629 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.630 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.630 DEBUG:boto:Data: 2024-04-12T03:07:43.630 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.630 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.630 DEBUG:boto:Params: {} 2024-04-12T03:07:43.631 DEBUG:boto:Token: None 2024-04-12T03:07:43.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.631 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004510c27f8a60c4cb-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.633 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.633 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.633 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.633 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.633 DEBUG:boto:Data: 2024-04-12T03:07:43.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.634 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.634 DEBUG:boto:Params: {} 2024-04-12T03:07:43.634 DEBUG:boto:Token: None 2024-04-12T03:07:43.634 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.634 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-amz-request-id', 'tx00000b7740e616aead60c-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.642 DEBUG:rgw_multi.tests:zone=a2 version=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:07:43.642 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.642 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.642 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.642 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.642 DEBUG:boto:Data: 2024-04-12T03:07:43.642 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.643 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.643 DEBUG:boto:Params: {} 2024-04-12T03:07:43.643 DEBUG:boto:Token: None 2024-04-12T03:07:43.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064a2624eaf4045a8-006618a57f-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.645 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.645 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.646 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.646 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.646 DEBUG:boto:Data: 2024-04-12T03:07:43.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:07:43.647 DEBUG:boto:Port: 8000 2024-04-12T03:07:43.647 DEBUG:boto:Params: {} 2024-04-12T03:07:43.647 DEBUG:boto:Token: None 2024-04-12T03:07:43.647 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-amz-request-id', 'tx000009a99ced2654dea78-006618a57f-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.691 DEBUG:rgw_multi.tests:zone=a1 version=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:07:43.691 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:07:43.691 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:07:43.691 DEBUG:boto:Method: HEAD 2024-04-12T03:07:43.692 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:07:43.692 DEBUG:boto:Data: 2024-04-12T03:07:43.692 DEBUG:boto:Headers: {} 2024-04-12T03:07:43.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.692 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.692 DEBUG:boto:Params: {} 2024-04-12T03:07:43.692 DEBUG:boto:Token: None 2024-04-12T03:07:43.693 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/ 2024-04-12T03:07:43.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno= 2024-04-12T03:07:43.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AGS7CLQ8rr/mQBK6oXbbXNcZQno=', 'Content-Length': '0'} 2024-04-12T03:07:43.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec839d85b7df716-006618a57f-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.696 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:07:43.696 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:07:43.696 DEBUG:boto:Method: PUT 2024-04-12T03:07:43.696 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:07:43.696 DEBUG:boto:Data: 2024-04-12T03:07:43.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:07:43.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:07:43.696 DEBUG:boto:Port: 8001 2024-04-12T03:07:43.697 DEBUG:boto:Params: {} 2024-04-12T03:07:43.697 DEBUG:boto:Token: None 2024-04-12T03:07:43.697 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:07:43 GMT /dgbpxg-15/obj 2024-04-12T03:07:43.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU= 2024-04-12T03:07:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:07:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3afTvl4FkQ+WQZw100u7sRVFcU='} 2024-04-12T03:07:43.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-amz-request-id', 'tx00000a958bda15cb33014-006618a57f-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:07:43.707 DEBUG:rgw_multi.tests:zone=a2 version=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:07:43.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-15 2024-04-12T03:07:43.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:43.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:07:43.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:07:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:07:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:07:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:43.796+0000 7f76d27fc780 1 waiting for incremental sync to catch up: 2024-04-12T03:07:43.797 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , , , ] 2024-04-12T03:07:43.797 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:07:48.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:48.796+0000 7f76d27fc780 1 waiting for incremental sync to catch up: 2024-04-12T03:07:48.798 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , 00000000020.61.12, , ] 2024-04-12T03:07:48.799 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:07:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:53.797+0000 7f76d27fc780 1 waiting for incremental sync to catch up: 2024-04-12T03:07:53.800 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , 00000000020.61.12, , ] 2024-04-12T03:07:53.800 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:07:58.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:07:58.799+0000 7f76d27fc780 1 waiting for incremental sync to catch up: 2024-04-12T03:07:58.801 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , 00000000020.61.12, , ] 2024-04-12T03:07:58.801 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:08:03.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:03.801+0000 7f76d27fc780 1 waiting for incremental sync to catch up: 2024-04-12T03:08:03.802 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.24.11, , , 00000000020.61.12, , ] 2024-04-12T03:08:03.802 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:08:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:08.802+0000 7f76d27fc780 1 bucket sync caught up with source: 2024-04-12T03:08:08.804 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.26.10, , , 00000000031.92.9, , ] 2024-04-12T03:08:08.804 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.26.10, , , 00000000020.61.12, , ] 2024-04-12T03:08:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:08.802+0000 7f76d27fc780 0 bucket checkpoint complete 2024-04-12T03:08:08.820 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-15 2024-04-12T03:08:08.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:08.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:08.821 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:08:08.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:08.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:08.917+0000 7fee1adbd780 1 bucket sync caught up with source: 2024-04-12T03:08:08.919 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.37.11, , , 00000000031.111.12, , ] 2024-04-12T03:08:08.919 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.37.11, , , 00000000031.111.12, , ] 2024-04-12T03:08:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:08.917+0000 7fee1adbd780 0 bucket checkpoint complete 2024-04-12T03:08:08.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-15 zones={a2, a1} 2024-04-12T03:08:08.934 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.934 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.934 DEBUG:boto:Method: HEAD 2024-04-12T03:08:08.934 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:08:08.935 DEBUG:boto:Data: 2024-04-12T03:08:08.935 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:08.935 DEBUG:boto:Port: 8001 2024-04-12T03:08:08.935 DEBUG:boto:Params: {} 2024-04-12T03:08:08.935 DEBUG:boto:Token: None 2024-04-12T03:08:08.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/ 2024-04-12T03:08:08.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:llpXsI+mX1aQRzyw+GGTjrZTAsg= 2024-04-12T03:08:08.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:llpXsI+mX1aQRzyw+GGTjrZTAsg=', 'Content-Length': '0'} 2024-04-12T03:08:08.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066e5dc3dde26d410-006618a598-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.939 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.939 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.939 DEBUG:boto:Method: HEAD 2024-04-12T03:08:08.939 DEBUG:boto:Path: /dgbpxg-15/ 2024-04-12T03:08:08.939 DEBUG:boto:Data: 2024-04-12T03:08:08.939 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:08.939 DEBUG:boto:Port: 8000 2024-04-12T03:08:08.940 DEBUG:boto:Params: {} 2024-04-12T03:08:08.940 DEBUG:boto:Token: None 2024-04-12T03:08:08.940 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/ 2024-04-12T03:08:08.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:llpXsI+mX1aQRzyw+GGTjrZTAsg= 2024-04-12T03:08:08.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:llpXsI+mX1aQRzyw+GGTjrZTAsg=', 'Content-Length': '0'} 2024-04-12T03:08:08.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be6716b7ca8306f-006618a598-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:08:08.943 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.943 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.944 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.944 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.944 DEBUG:boto:Method: GET 2024-04-12T03:08:08.944 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.944 DEBUG:boto:Data: 2024-04-12T03:08:08.944 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:08.945 DEBUG:boto:Port: 8001 2024-04-12T03:08:08.945 DEBUG:boto:Params: {} 2024-04-12T03:08:08.945 DEBUG:boto:Token: None 2024-04-12T03:08:08.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/?versions 2024-04-12T03:08:08.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8= 2024-04-12T03:08:08.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8=', 'Content-Length': '0'} 2024-04-12T03:08:08.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c608a30f29f32af4-006618a598-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.948 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGQK1DlZF21SeIkechhmXlaU1KROzKxztrue2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeIjMcOCeqSDUqA-0n2CvttoJOI6GNxffalse2024-04-12T03:07:43.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjQLilKfoKgGsBNPhQ.r1u4mN4A760aWfalse2024-04-12T03:07:43.635Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyApN1qfcLlzZUxxFPmrPO6qfFmhYsR6false2024-04-12T03:07:43.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-cAPUVSS2ci.HqftZCMgVG2X9O4.zkCfalse2024-04-12T03:07:43.571Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY4UsZptQCnyCfND-3uZAipSEiHu4LFDfalse2024-04-12T03:07:43.557Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJfalse2024-04-12T03:07:43.540Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjffvLWMVrdZDe13V1v1UbT78s2d98pF-false2024-04-12T03:07:43.493Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47QlbJPF6KNDjYM8z8QahgoYTX4c1WXfalse2024-04-12T03:07:43.480Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjV1pRzYXnGlieidUQDQ-AjtYhKK-BGGfalse2024-04-12T03:07:43.467Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoaMQ.eux27q2OxMdGhmDSwI.QvJdy-Ffalse2024-04-12T03:07:43.403Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwrHehFqeBQkfs1.ar4QEDn6bOCWN2mCfalse2024-04-12T03:07:43.452Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlkvI3dmhci4zQk8IEwXAj9LrvAIJCuLfalse2024-04-12T03:07:43.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpak4GQgdOu63EMuX67LGYKjVi9OFJirfalse2024-04-12T03:07:43.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9IV19jO8jWopyR.dsl0X2CCjrKwR8xWfalse2024-04-12T03:07:43.365Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhiI6JTYES9BqLUDKwnK0RcVlWcWJWKLfalse2024-04-12T03:07:43.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-kcLOF-uBDPIeu9dWArWlO6KUUrsGSgfalse2024-04-12T03:07:43.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcaJEOLhbPMl5nr7xTZPU-.Uh0k1woZPfalse2024-04-12T03:07:43.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPQyVfrdwJE7QIbPm0v64BtD0mpy58hrfalse2024-04-12T03:07:43.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwlQG40RppM7AX0WKjPtx4kQYRknCEJGfalse2024-04-12T03:07:43.240Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:08.950 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:08:08.952 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.952 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.953 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.953 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.953 DEBUG:boto:Method: GET 2024-04-12T03:08:08.953 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.953 DEBUG:boto:Data: 2024-04-12T03:08:08.953 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:08.953 DEBUG:boto:Port: 8000 2024-04-12T03:08:08.953 DEBUG:boto:Params: {} 2024-04-12T03:08:08.953 DEBUG:boto:Token: None 2024-04-12T03:08:08.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/?versions 2024-04-12T03:08:08.954 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8= 2024-04-12T03:08:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8=', 'Content-Length': '0'} 2024-04-12T03:08:08.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005126a8b63d838217-006618a598-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.963 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGQK1DlZF21SeIkechhmXlaU1KROzKxztrue2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeIjMcOCeqSDUqA-0n2CvttoJOI6GNxffalse2024-04-12T03:07:43.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjQLilKfoKgGsBNPhQ.r1u4mN4A760aWfalse2024-04-12T03:07:43.635Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyApN1qfcLlzZUxxFPmrPO6qfFmhYsR6false2024-04-12T03:07:43.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-cAPUVSS2ci.HqftZCMgVG2X9O4.zkCfalse2024-04-12T03:07:43.571Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY4UsZptQCnyCfND-3uZAipSEiHu4LFDfalse2024-04-12T03:07:43.557Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJfalse2024-04-12T03:07:43.540Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjffvLWMVrdZDe13V1v1UbT78s2d98pF-false2024-04-12T03:07:43.493Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47QlbJPF6KNDjYM8z8QahgoYTX4c1WXfalse2024-04-12T03:07:43.480Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjV1pRzYXnGlieidUQDQ-AjtYhKK-BGGfalse2024-04-12T03:07:43.467Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoaMQ.eux27q2OxMdGhmDSwI.QvJdy-Ffalse2024-04-12T03:07:43.403Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwrHehFqeBQkfs1.ar4QEDn6bOCWN2mCfalse2024-04-12T03:07:43.452Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlkvI3dmhci4zQk8IEwXAj9LrvAIJCuLfalse2024-04-12T03:07:43.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpak4GQgdOu63EMuX67LGYKjVi9OFJirfalse2024-04-12T03:07:43.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9IV19jO8jWopyR.dsl0X2CCjrKwR8xWfalse2024-04-12T03:07:43.365Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhiI6JTYES9BqLUDKwnK0RcVlWcWJWKLfalse2024-04-12T03:07:43.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-kcLOF-uBDPIeu9dWArWlO6KUUrsGSgfalse2024-04-12T03:07:43.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcaJEOLhbPMl5nr7xTZPU-.Uh0k1woZPfalse2024-04-12T03:07:43.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPQyVfrdwJE7QIbPm0v64BtD0mpy58hrfalse2024-04-12T03:07:43.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwlQG40RppM7AX0WKjPtx4kQYRknCEJGfalse2024-04-12T03:07:43.240Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:08.965 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:08.969 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.969 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.969 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.970 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.970 DEBUG:boto:Method: GET 2024-04-12T03:08:08.970 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.970 DEBUG:boto:Data: 2024-04-12T03:08:08.970 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:08.971 DEBUG:boto:Port: 8001 2024-04-12T03:08:08.971 DEBUG:boto:Params: {} 2024-04-12T03:08:08.971 DEBUG:boto:Token: None 2024-04-12T03:08:08.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/?versions 2024-04-12T03:08:08.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8= 2024-04-12T03:08:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8=', 'Content-Length': '0'} 2024-04-12T03:08:08.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad87878e83b17f0-006618a598-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.976 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGQK1DlZF21SeIkechhmXlaU1KROzKxztrue2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeIjMcOCeqSDUqA-0n2CvttoJOI6GNxffalse2024-04-12T03:07:43.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjQLilKfoKgGsBNPhQ.r1u4mN4A760aWfalse2024-04-12T03:07:43.635Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyApN1qfcLlzZUxxFPmrPO6qfFmhYsR6false2024-04-12T03:07:43.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-cAPUVSS2ci.HqftZCMgVG2X9O4.zkCfalse2024-04-12T03:07:43.571Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY4UsZptQCnyCfND-3uZAipSEiHu4LFDfalse2024-04-12T03:07:43.557Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJfalse2024-04-12T03:07:43.540Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjffvLWMVrdZDe13V1v1UbT78s2d98pF-false2024-04-12T03:07:43.493Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47QlbJPF6KNDjYM8z8QahgoYTX4c1WXfalse2024-04-12T03:07:43.480Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjV1pRzYXnGlieidUQDQ-AjtYhKK-BGGfalse2024-04-12T03:07:43.467Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoaMQ.eux27q2OxMdGhmDSwI.QvJdy-Ffalse2024-04-12T03:07:43.403Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwrHehFqeBQkfs1.ar4QEDn6bOCWN2mCfalse2024-04-12T03:07:43.452Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlkvI3dmhci4zQk8IEwXAj9LrvAIJCuLfalse2024-04-12T03:07:43.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpak4GQgdOu63EMuX67LGYKjVi9OFJirfalse2024-04-12T03:07:43.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9IV19jO8jWopyR.dsl0X2CCjrKwR8xWfalse2024-04-12T03:07:43.365Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhiI6JTYES9BqLUDKwnK0RcVlWcWJWKLfalse2024-04-12T03:07:43.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-kcLOF-uBDPIeu9dWArWlO6KUUrsGSgfalse2024-04-12T03:07:43.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcaJEOLhbPMl5nr7xTZPU-.Uh0k1woZPfalse2024-04-12T03:07:43.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPQyVfrdwJE7QIbPm0v64BtD0mpy58hrfalse2024-04-12T03:07:43.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwlQG40RppM7AX0WKjPtx4kQYRknCEJGfalse2024-04-12T03:07:43.240Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:08.978 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:08.979 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:08.979 DEBUG:boto:path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.979 DEBUG:boto:auth_path=/dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.980 DEBUG:boto:Method: GET 2024-04-12T03:08:08.980 DEBUG:boto:Path: /dgbpxg-15/?versions&max-keys=999 2024-04-12T03:08:08.980 DEBUG:boto:Data: 2024-04-12T03:08:08.980 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:08.981 DEBUG:boto:Port: 8000 2024-04-12T03:08:08.981 DEBUG:boto:Params: {} 2024-04-12T03:08:08.981 DEBUG:boto:Token: None 2024-04-12T03:08:08.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/?versions 2024-04-12T03:08:08.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8= 2024-04-12T03:08:08.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893gl00h4pDSKuVgIrG322dRWK8=', 'Content-Length': '0'} 2024-04-12T03:08:08.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b203f8297d324d0-006618a598-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:08.992 DEBUG:boto:b'dgbpxg-15999falsedummyULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.true2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGQK1DlZF21SeIkechhmXlaU1KROzKxztrue2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeIjMcOCeqSDUqA-0n2CvttoJOI6GNxffalse2024-04-12T03:07:43.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjQLilKfoKgGsBNPhQ.r1u4mN4A760aWfalse2024-04-12T03:07:43.635Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyApN1qfcLlzZUxxFPmrPO6qfFmhYsR6false2024-04-12T03:07:43.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-cAPUVSS2ci.HqftZCMgVG2X9O4.zkCfalse2024-04-12T03:07:43.571Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY4UsZptQCnyCfND-3uZAipSEiHu4LFDfalse2024-04-12T03:07:43.557Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJfalse2024-04-12T03:07:43.540Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjffvLWMVrdZDe13V1v1UbT78s2d98pF-false2024-04-12T03:07:43.493Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47QlbJPF6KNDjYM8z8QahgoYTX4c1WXfalse2024-04-12T03:07:43.480Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjV1pRzYXnGlieidUQDQ-AjtYhKK-BGGfalse2024-04-12T03:07:43.467Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoaMQ.eux27q2OxMdGhmDSwI.QvJdy-Ffalse2024-04-12T03:07:43.403Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwrHehFqeBQkfs1.ar4QEDn6bOCWN2mCfalse2024-04-12T03:07:43.452Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlkvI3dmhci4zQk8IEwXAj9LrvAIJCuLfalse2024-04-12T03:07:43.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpak4GQgdOu63EMuX67LGYKjVi9OFJirfalse2024-04-12T03:07:43.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9IV19jO8jWopyR.dsl0X2CCjrKwR8xWfalse2024-04-12T03:07:43.365Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhiI6JTYES9BqLUDKwnK0RcVlWcWJWKLfalse2024-04-12T03:07:43.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-kcLOF-uBDPIeu9dWArWlO6KUUrsGSgfalse2024-04-12T03:07:43.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcaJEOLhbPMl5nr7xTZPU-.Uh0k1woZPfalse2024-04-12T03:07:43.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPQyVfrdwJE7QIbPm0v64BtD0mpy58hrfalse2024-04-12T03:07:43.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwlQG40RppM7AX0WKjPtx4kQYRknCEJGfalse2024-04-12T03:07:43.240Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:08.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:08:08.994 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:08.994 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:08.994 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:08.994 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:08.994 DEBUG:boto:Method: GET 2024-04-12T03:08:08.995 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:08.995 DEBUG:boto:Data: 2024-04-12T03:08:08.995 DEBUG:boto:Headers: {} 2024-04-12T03:08:08.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:08.995 DEBUG:boto:Port: 8001 2024-04-12T03:08:08.995 DEBUG:boto:Params: {} 2024-04-12T03:08:08.998 DEBUG:boto:Token: None 2024-04-12T03:08:08.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:08 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:08.998 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j4ktmfqMCjVWKOvXevrImO2jYzw= 2024-04-12T03:08:08.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:08 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j4ktmfqMCjVWKOvXevrImO2jYzw=', 'Content-Length': '0'} 2024-04-12T03:08:09.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb9bfca46d62f8dc-006618a598-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.002 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.002 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.002 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.003 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.003 DEBUG:boto:Method: GET 2024-04-12T03:08:09.003 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.003 DEBUG:boto:Data: 2024-04-12T03:08:09.003 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.003 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.003 DEBUG:boto:Params: {} 2024-04-12T03:08:09.004 DEBUG:boto:Token: None 2024-04-12T03:08:09.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo= 2024-04-12T03:08:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo=', 'Content-Length': '0'} 2024-04-12T03:08:09.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c9f12278c45c483-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.012 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.012 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.013 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.013 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.013 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.013 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.013 DEBUG:boto:Data: 2024-04-12T03:08:09.013 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.013 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.013 DEBUG:boto:Params: {} 2024-04-12T03:08:09.013 DEBUG:boto:Token: None 2024-04-12T03:08:09.014 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wS39/rqR85bH0HiEj7966KMPV0Y= 2024-04-12T03:08:09.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wS39/rqR85bH0HiEj7966KMPV0Y=', 'Content-Length': '0'} 2024-04-12T03:08:09.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b9b48fb713c292a-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.021 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.021 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.021 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.021 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.021 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.021 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.021 DEBUG:boto:Data: 2024-04-12T03:08:09.021 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.022 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.022 DEBUG:boto:Params: {} 2024-04-12T03:08:09.022 DEBUG:boto:Token: None 2024-04-12T03:08:09.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wS39/rqR85bH0HiEj7966KMPV0Y= 2024-04-12T03:08:09.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wS39/rqR85bH0HiEj7966KMPV0Y=', 'Content-Length': '0'} 2024-04-12T03:08:09.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000aefe0596924f33d-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:08:09.028 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.028 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.028 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.028 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.028 DEBUG:boto:Method: GET 2024-04-12T03:08:09.028 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.028 DEBUG:boto:Data: 2024-04-12T03:08:09.028 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.028 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.029 DEBUG:boto:Params: {} 2024-04-12T03:08:09.029 DEBUG:boto:Token: None 2024-04-12T03:08:09.029 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo= 2024-04-12T03:08:09.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo=', 'Content-Length': '0'} 2024-04-12T03:08:09.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008dc9c6d2bd138d53-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.034 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.034 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.034 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.034 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.034 DEBUG:boto:Method: GET 2024-04-12T03:08:09.034 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.034 DEBUG:boto:Data: 2024-04-12T03:08:09.034 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.035 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.035 DEBUG:boto:Params: {} 2024-04-12T03:08:09.035 DEBUG:boto:Token: None 2024-04-12T03:08:09.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo= 2024-04-12T03:08:09.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo=', 'Content-Length': '0'} 2024-04-12T03:08:09.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d748a83b691a8103-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.043 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.044 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.044 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.044 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.044 DEBUG:boto:Method: GET 2024-04-12T03:08:09.044 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.044 DEBUG:boto:Data: 2024-04-12T03:08:09.044 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.044 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.044 DEBUG:boto:Params: {} 2024-04-12T03:08:09.045 DEBUG:boto:Token: None 2024-04-12T03:08:09.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57278b986e0996d-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.091 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.091 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.091 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.091 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.091 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.091 DEBUG:boto:Method: GET 2024-04-12T03:08:09.092 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.092 DEBUG:boto:Data: 2024-04-12T03:08:09.092 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.092 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.092 DEBUG:boto:Params: {} 2024-04-12T03:08:09.092 DEBUG:boto:Token: None 2024-04-12T03:08:09.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1a816639e4b0e2-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.095 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.096 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.096 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.096 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.097 DEBUG:boto:Path: /dgbpxg-15/dummy 2024-04-12T03:08:09.097 DEBUG:boto:Data: 2024-04-12T03:08:09.097 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.097 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.097 DEBUG:boto:Params: {} 2024-04-12T03:08:09.098 DEBUG:boto:Token: None 2024-04-12T03:08:09.098 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy 2024-04-12T03:08:09.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hM7P/jIBOtgmYbz8Fcypfgl/MwA= 2024-04-12T03:08:09.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hM7P/jIBOtgmYbz8Fcypfgl/MwA=', 'Content-Length': '0'} 2024-04-12T03:08:09.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f8999058715b654-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.102 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.102 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.103 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.103 DEBUG:boto:Path: /dgbpxg-15/dummy 2024-04-12T03:08:09.103 DEBUG:boto:Data: 2024-04-12T03:08:09.103 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.104 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.104 DEBUG:boto:Params: {} 2024-04-12T03:08:09.104 DEBUG:boto:Token: None 2024-04-12T03:08:09.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy 2024-04-12T03:08:09.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hM7P/jIBOtgmYbz8Fcypfgl/MwA= 2024-04-12T03:08:09.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hM7P/jIBOtgmYbz8Fcypfgl/MwA=', 'Content-Length': '0'} 2024-04-12T03:08:09.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a7fbd4e66ce3dcc-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:08:09.113 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.113 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.113 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.113 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.114 DEBUG:boto:Method: GET 2024-04-12T03:08:09.114 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.114 DEBUG:boto:Data: 2024-04-12T03:08:09.114 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.114 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.115 DEBUG:boto:Params: {} 2024-04-12T03:08:09.115 DEBUG:boto:Token: None 2024-04-12T03:08:09.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo= 2024-04-12T03:08:09.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo=', 'Content-Length': '0'} 2024-04-12T03:08:09.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000125cac9e3b1f55cd-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.120 DEBUG:boto:path=/dgbpxg-15/dummy 2024-04-12T03:08:09.120 DEBUG:boto:auth_path=/dgbpxg-15/dummy 2024-04-12T03:08:09.121 DEBUG:boto:path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.121 DEBUG:boto:auth_path=/dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.121 DEBUG:boto:Method: GET 2024-04-12T03:08:09.121 DEBUG:boto:Path: /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.121 DEBUG:boto:Data: 2024-04-12T03:08:09.122 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.122 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.122 DEBUG:boto:Params: {} 2024-04-12T03:08:09.123 DEBUG:boto:Token: None 2024-04-12T03:08:09.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/dummy?versionId=ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT. 2024-04-12T03:08:09.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo= 2024-04-12T03:08:09.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMFd4cwTEL7f7VhZnJ1uh/5m3mo=', 'Content-Length': '0'} 2024-04-12T03:08:09.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:07 GMT'), ('x-amz-version-id', 'ULhZ29LD2VhWtl5ikzcuZPa1QH5SjT.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e66832716bf267c8-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.129 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.130 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.130 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.130 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.130 DEBUG:boto:Method: GET 2024-04-12T03:08:09.130 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.131 DEBUG:boto:Data: 2024-04-12T03:08:09.131 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.131 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.131 DEBUG:boto:Params: {} 2024-04-12T03:08:09.131 DEBUG:boto:Token: None 2024-04-12T03:08:09.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca349850dd28da50-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.139 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.140 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.140 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.141 DEBUG:boto:path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.141 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.141 DEBUG:boto:Method: GET 2024-04-12T03:08:09.141 DEBUG:boto:Path: /dgbpxg-15/?prefix=dummy 2024-04-12T03:08:09.142 DEBUG:boto:Data: 2024-04-12T03:08:09.142 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.142 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.142 DEBUG:boto:Params: {} 2024-04-12T03:08:09.143 DEBUG:boto:Token: None 2024-04-12T03:08:09.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e78f296131a3872-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.187 DEBUG:boto:b'dgbpxg-15dummy1000falsedummy2024-04-12T03:07:07.823Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.187 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.188 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.188 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.188 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.188 DEBUG:boto:Method: GET 2024-04-12T03:08:09.188 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.188 DEBUG:boto:Data: 2024-04-12T03:08:09.188 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.188 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.188 DEBUG:boto:Params: {} 2024-04-12T03:08:09.189 DEBUG:boto:Token: None 2024-04-12T03:08:09.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000988cfb203d44a0ea-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.235 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.236 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.236 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.236 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.236 DEBUG:boto:Method: GET 2024-04-12T03:08:09.236 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.236 DEBUG:boto:Data: 2024-04-12T03:08:09.236 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.237 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.237 DEBUG:boto:Params: {} 2024-04-12T03:08:09.237 DEBUG:boto:Token: None 2024-04-12T03:08:09.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.237 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076f095bffe2e4bbe-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.242 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.242 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.242 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.243 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.243 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.243 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.243 DEBUG:boto:Data: 2024-04-12T03:08:09.244 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.244 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.244 DEBUG:boto:Params: {} 2024-04-12T03:08:09.245 DEBUG:boto:Token: None 2024-04-12T03:08:09.245 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EtGr7SL3sP3Uiafdk15YfhRRn6Y= 2024-04-12T03:08:09.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EtGr7SL3sP3Uiafdk15YfhRRn6Y=', 'Content-Length': '0'} 2024-04-12T03:08:09.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000355eca4aeeeb94b8-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.249 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.250 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.250 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.250 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.250 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.251 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.251 DEBUG:boto:Data: 2024-04-12T03:08:09.251 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.251 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.251 DEBUG:boto:Params: {} 2024-04-12T03:08:09.252 DEBUG:boto:Token: None 2024-04-12T03:08:09.252 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EtGr7SL3sP3Uiafdk15YfhRRn6Y= 2024-04-12T03:08:09.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EtGr7SL3sP3Uiafdk15YfhRRn6Y=', 'Content-Length': '0'} 2024-04-12T03:08:09.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006f86f0e24a48cd8-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.255 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.255 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.256 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.256 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.256 DEBUG:boto:Method: GET 2024-04-12T03:08:09.256 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.256 DEBUG:boto:Data: 2024-04-12T03:08:09.256 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.257 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.257 DEBUG:boto:Params: {} 2024-04-12T03:08:09.257 DEBUG:boto:Token: None 2024-04-12T03:08:09.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.258 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa9132e91b08ae7f-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.313 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.313 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.313 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.314 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.314 DEBUG:boto:Method: GET 2024-04-12T03:08:09.314 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.314 DEBUG:boto:Data: 2024-04-12T03:08:09.314 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.314 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.314 DEBUG:boto:Params: {} 2024-04-12T03:08:09.314 DEBUG:boto:Token: None 2024-04-12T03:08:09.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009116b9eb11b75f4b-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.319 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.320 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.320 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.320 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.320 DEBUG:boto:Method: GET 2024-04-12T03:08:09.320 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.320 DEBUG:boto:Data: 2024-04-12T03:08:09.320 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.320 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.320 DEBUG:boto:Params: {} 2024-04-12T03:08:09.320 DEBUG:boto:Token: None 2024-04-12T03:08:09.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.321 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553749faecd8318f-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.324 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.324 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.324 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.324 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.324 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.325 DEBUG:boto:Method: GET 2024-04-12T03:08:09.325 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.325 DEBUG:boto:Data: 2024-04-12T03:08:09.325 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.325 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.325 DEBUG:boto:Params: {} 2024-04-12T03:08:09.325 DEBUG:boto:Token: None 2024-04-12T03:08:09.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806723db652bbd8d-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.329 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.330 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.330 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.330 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.330 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.330 DEBUG:boto:Data: 2024-04-12T03:08:09.330 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.330 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.331 DEBUG:boto:Params: {} 2024-04-12T03:08:09.331 DEBUG:boto:Token: None 2024-04-12T03:08:09.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a283ef107d8d753-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.336 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.336 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.337 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.337 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.337 DEBUG:boto:Data: 2024-04-12T03:08:09.337 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.337 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.337 DEBUG:boto:Params: {} 2024-04-12T03:08:09.338 DEBUG:boto:Token: None 2024-04-12T03:08:09.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e985076e13453176-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.345 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.345 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.345 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.345 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.346 DEBUG:boto:Method: GET 2024-04-12T03:08:09.346 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.346 DEBUG:boto:Data: 2024-04-12T03:08:09.346 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.347 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.347 DEBUG:boto:Params: {} 2024-04-12T03:08:09.348 DEBUG:boto:Token: None 2024-04-12T03:08:09.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fee8519390ea2863-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.398 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.399 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.399 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.399 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.399 DEBUG:boto:Method: GET 2024-04-12T03:08:09.399 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.399 DEBUG:boto:Data: 2024-04-12T03:08:09.399 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.399 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.399 DEBUG:boto:Params: {} 2024-04-12T03:08:09.400 DEBUG:boto:Token: None 2024-04-12T03:08:09.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e05920387566d24-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.407 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.407 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.407 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.407 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.408 DEBUG:boto:Method: GET 2024-04-12T03:08:09.408 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.408 DEBUG:boto:Data: 2024-04-12T03:08:09.408 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.408 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.408 DEBUG:boto:Params: {} 2024-04-12T03:08:09.408 DEBUG:boto:Token: None 2024-04-12T03:08:09.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8faa692bbb42a6c-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.414 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.414 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.414 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.414 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.414 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.415 DEBUG:boto:Method: GET 2024-04-12T03:08:09.415 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.415 DEBUG:boto:Data: 2024-04-12T03:08:09.415 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.415 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.415 DEBUG:boto:Params: {} 2024-04-12T03:08:09.415 DEBUG:boto:Token: None 2024-04-12T03:08:09.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.415 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e69f52643155a8a-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.464 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.464 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.464 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.465 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.465 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.465 DEBUG:boto:Method: GET 2024-04-12T03:08:09.465 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.465 DEBUG:boto:Data: 2024-04-12T03:08:09.465 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.465 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.465 DEBUG:boto:Params: {} 2024-04-12T03:08:09.465 DEBUG:boto:Token: None 2024-04-12T03:08:09.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY= 2024-04-12T03:08:09.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY=', 'Content-Length': '0'} 2024-04-12T03:08:09.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a0ff9f9b5898ca6-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.470 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.470 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.470 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.470 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.470 DEBUG:boto:Method: GET 2024-04-12T03:08:09.470 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.470 DEBUG:boto:Data: 2024-04-12T03:08:09.470 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.471 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.471 DEBUG:boto:Params: {} 2024-04-12T03:08:09.471 DEBUG:boto:Token: None 2024-04-12T03:08:09.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY= 2024-04-12T03:08:09.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY=', 'Content-Length': '0'} 2024-04-12T03:08:09.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000669aba2555fc6f2e-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.475 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.475 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.475 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.475 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.475 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.475 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.475 DEBUG:boto:Data: 2024-04-12T03:08:09.475 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.476 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.476 DEBUG:boto:Params: {} 2024-04-12T03:08:09.476 DEBUG:boto:Token: None 2024-04-12T03:08:09.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FSNI5/z4FLOlIctoajaG6PdKn0A= 2024-04-12T03:08:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FSNI5/z4FLOlIctoajaG6PdKn0A=', 'Content-Length': '0'} 2024-04-12T03:08:09.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a89834be1eff330b-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.478 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.479 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.479 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.479 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.479 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.479 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.479 DEBUG:boto:Data: 2024-04-12T03:08:09.479 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.479 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.479 DEBUG:boto:Params: {} 2024-04-12T03:08:09.479 DEBUG:boto:Token: None 2024-04-12T03:08:09.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FSNI5/z4FLOlIctoajaG6PdKn0A= 2024-04-12T03:08:09.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FSNI5/z4FLOlIctoajaG6PdKn0A=', 'Content-Length': '0'} 2024-04-12T03:08:09.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080e141d005ccc5dd-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.482 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.482 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.482 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.482 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.482 DEBUG:boto:Method: GET 2024-04-12T03:08:09.482 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.482 DEBUG:boto:Data: 2024-04-12T03:08:09.482 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.483 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.483 DEBUG:boto:Params: {} 2024-04-12T03:08:09.483 DEBUG:boto:Token: None 2024-04-12T03:08:09.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY= 2024-04-12T03:08:09.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY=', 'Content-Length': '0'} 2024-04-12T03:08:09.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a6b4775489487af-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.528 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.528 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.528 DEBUG:boto:path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.528 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.528 DEBUG:boto:Method: GET 2024-04-12T03:08:09.528 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.528 DEBUG:boto:Data: 2024-04-12T03:08:09.528 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.529 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.529 DEBUG:boto:Params: {} 2024-04-12T03:08:09.529 DEBUG:boto:Token: None 2024-04-12T03:08:09.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf 2024-04-12T03:08:09.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY= 2024-04-12T03:08:09.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X3G4rYNKsg0s4Gdatwz9zK9eCKY=', 'Content-Length': '0'} 2024-04-12T03:08:09.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'eIjMcOCeqSDUqA-0n2CvttoJOI6GNxf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002547f8d69b249cb-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.580 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.580 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.580 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.580 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.580 DEBUG:boto:Method: GET 2024-04-12T03:08:09.580 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.580 DEBUG:boto:Data: 2024-04-12T03:08:09.580 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.581 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.581 DEBUG:boto:Params: {} 2024-04-12T03:08:09.581 DEBUG:boto:Token: None 2024-04-12T03:08:09.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.581 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef093068cc77669-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.588 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.589 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.589 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.589 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.589 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.589 DEBUG:boto:Method: GET 2024-04-12T03:08:09.589 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.589 DEBUG:boto:Data: 2024-04-12T03:08:09.589 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.590 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.590 DEBUG:boto:Params: {} 2024-04-12T03:08:09.590 DEBUG:boto:Token: None 2024-04-12T03:08:09.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.590 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5db33b5dcd62e66-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.636 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.636 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.636 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.637 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.637 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.637 DEBUG:boto:Data: 2024-04-12T03:08:09.637 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.637 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.637 DEBUG:boto:Params: {} 2024-04-12T03:08:09.637 DEBUG:boto:Token: None 2024-04-12T03:08:09.637 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1ff4eab991143d7-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.641 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.641 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.641 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.641 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.641 DEBUG:boto:Data: 2024-04-12T03:08:09.641 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.642 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.642 DEBUG:boto:Params: {} 2024-04-12T03:08:09.642 DEBUG:boto:Token: None 2024-04-12T03:08:09.642 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003feadcf9a8f89320-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.645 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.645 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.645 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.645 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.645 DEBUG:boto:Method: GET 2024-04-12T03:08:09.646 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.646 DEBUG:boto:Data: 2024-04-12T03:08:09.646 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.646 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.646 DEBUG:boto:Params: {} 2024-04-12T03:08:09.646 DEBUG:boto:Token: None 2024-04-12T03:08:09.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.646 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a64601bb6321e8cb-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.691 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.691 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.691 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.691 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.691 DEBUG:boto:Method: GET 2024-04-12T03:08:09.691 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.691 DEBUG:boto:Data: 2024-04-12T03:08:09.691 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.692 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.692 DEBUG:boto:Params: {} 2024-04-12T03:08:09.692 DEBUG:boto:Token: None 2024-04-12T03:08:09.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc5e50b37692a0ea-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.695 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.695 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.695 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.695 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.695 DEBUG:boto:Method: GET 2024-04-12T03:08:09.695 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.696 DEBUG:boto:Data: 2024-04-12T03:08:09.696 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.696 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.696 DEBUG:boto:Params: {} 2024-04-12T03:08:09.696 DEBUG:boto:Token: None 2024-04-12T03:08:09.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002662d28caa4f6cc8-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.742 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.743 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.743 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.743 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.743 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.743 DEBUG:boto:Method: GET 2024-04-12T03:08:09.743 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.743 DEBUG:boto:Data: 2024-04-12T03:08:09.743 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.744 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.744 DEBUG:boto:Params: {} 2024-04-12T03:08:09.744 DEBUG:boto:Token: None 2024-04-12T03:08:09.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029e0ebbf33a79076-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.749 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.750 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.750 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.750 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.750 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.750 DEBUG:boto:Method: GET 2024-04-12T03:08:09.750 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.750 DEBUG:boto:Data: 2024-04-12T03:08:09.750 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.751 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.751 DEBUG:boto:Params: {} 2024-04-12T03:08:09.751 DEBUG:boto:Token: None 2024-04-12T03:08:09.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk= 2024-04-12T03:08:09.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk=', 'Content-Length': '0'} 2024-04-12T03:08:09.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbf165c0fadc8e78-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.754 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.754 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.754 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.754 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.755 DEBUG:boto:Method: GET 2024-04-12T03:08:09.755 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.755 DEBUG:boto:Data: 2024-04-12T03:08:09.755 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.755 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.755 DEBUG:boto:Params: {} 2024-04-12T03:08:09.755 DEBUG:boto:Token: None 2024-04-12T03:08:09.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk= 2024-04-12T03:08:09.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk=', 'Content-Length': '0'} 2024-04-12T03:08:09.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b381dbbb56fd3c04-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.758 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.758 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.758 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.758 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.759 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.759 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.759 DEBUG:boto:Data: 2024-04-12T03:08:09.759 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.759 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.759 DEBUG:boto:Params: {} 2024-04-12T03:08:09.759 DEBUG:boto:Token: None 2024-04-12T03:08:09.759 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NHvIrY0cySLw5czHiMCTTB/oZpU= 2024-04-12T03:08:09.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NHvIrY0cySLw5czHiMCTTB/oZpU=', 'Content-Length': '0'} 2024-04-12T03:08:09.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8a4bc052bd8fb45-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.762 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.762 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.762 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.762 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.763 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.763 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.763 DEBUG:boto:Data: 2024-04-12T03:08:09.763 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.763 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.763 DEBUG:boto:Params: {} 2024-04-12T03:08:09.763 DEBUG:boto:Token: None 2024-04-12T03:08:09.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NHvIrY0cySLw5czHiMCTTB/oZpU= 2024-04-12T03:08:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NHvIrY0cySLw5czHiMCTTB/oZpU=', 'Content-Length': '0'} 2024-04-12T03:08:09.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7a16cc18f532d91-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.767 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.768 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.768 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.768 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.768 DEBUG:boto:Method: GET 2024-04-12T03:08:09.768 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.768 DEBUG:boto:Data: 2024-04-12T03:08:09.768 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.768 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.768 DEBUG:boto:Params: {} 2024-04-12T03:08:09.768 DEBUG:boto:Token: None 2024-04-12T03:08:09.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk= 2024-04-12T03:08:09.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk=', 'Content-Length': '0'} 2024-04-12T03:08:09.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046c809f7d2edd73d-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.771 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.771 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.771 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.772 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.772 DEBUG:boto:Method: GET 2024-04-12T03:08:09.772 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.772 DEBUG:boto:Data: 2024-04-12T03:08:09.772 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.772 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.772 DEBUG:boto:Params: {} 2024-04-12T03:08:09.772 DEBUG:boto:Token: None 2024-04-12T03:08:09.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=jQLilKfoKgGsBNPhQ.r1u4mN4A760aW 2024-04-12T03:08:09.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk= 2024-04-12T03:08:09.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2obo/9jCIvDManSjGIu80pk4fMk=', 'Content-Length': '0'} 2024-04-12T03:08:09.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jQLilKfoKgGsBNPhQ.r1u4mN4A760aW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000930036a01a060c93-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.820 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.820 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.820 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.820 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.820 DEBUG:boto:Method: GET 2024-04-12T03:08:09.820 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.820 DEBUG:boto:Data: 2024-04-12T03:08:09.821 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.821 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.821 DEBUG:boto:Params: {} 2024-04-12T03:08:09.821 DEBUG:boto:Token: None 2024-04-12T03:08:09.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a796da66a3604e38-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.827 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.827 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.827 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.828 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.828 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.828 DEBUG:boto:Method: GET 2024-04-12T03:08:09.828 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.828 DEBUG:boto:Data: 2024-04-12T03:08:09.828 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.828 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.828 DEBUG:boto:Params: {} 2024-04-12T03:08:09.828 DEBUG:boto:Token: None 2024-04-12T03:08:09.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d76ee27983f51116-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.834 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.834 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.834 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.834 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.834 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.834 DEBUG:boto:Data: 2024-04-12T03:08:09.835 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.835 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.835 DEBUG:boto:Params: {} 2024-04-12T03:08:09.835 DEBUG:boto:Token: None 2024-04-12T03:08:09.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ae1e9365fa4aa7a-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.841 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.841 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.842 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.842 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:09.842 DEBUG:boto:Data: 2024-04-12T03:08:09.842 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.842 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.842 DEBUG:boto:Params: {} 2024-04-12T03:08:09.843 DEBUG:boto:Token: None 2024-04-12T03:08:09.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj 2024-04-12T03:08:09.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U= 2024-04-12T03:08:09.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W/SLTH2U+fLqG+NH2u66bVXYg/U=', 'Content-Length': '0'} 2024-04-12T03:08:09.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e47d2729973b4b00-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.847 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.847 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.847 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.847 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.847 DEBUG:boto:Method: GET 2024-04-12T03:08:09.847 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.847 DEBUG:boto:Data: 2024-04-12T03:08:09.848 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.848 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.848 DEBUG:boto:Params: {} 2024-04-12T03:08:09.848 DEBUG:boto:Token: None 2024-04-12T03:08:09.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041e029a7c01e0b15-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.895 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.895 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.895 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.895 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.895 DEBUG:boto:Method: GET 2024-04-12T03:08:09.896 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.896 DEBUG:boto:Data: 2024-04-12T03:08:09.896 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.896 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.896 DEBUG:boto:Params: {} 2024-04-12T03:08:09.897 DEBUG:boto:Token: None 2024-04-12T03:08:09.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:09.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE= 2024-04-12T03:08:09.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MRMky1xDhKAV5oH88c66EMpXEE=', 'Content-Length': '0'} 2024-04-12T03:08:09.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f05a1b1d98984e-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.900 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.900 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.900 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.900 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.901 DEBUG:boto:Method: GET 2024-04-12T03:08:09.901 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.901 DEBUG:boto:Data: 2024-04-12T03:08:09.901 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.901 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.901 DEBUG:boto:Params: {} 2024-04-12T03:08:09.901 DEBUG:boto:Token: None 2024-04-12T03:08:09.901 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdae33ff7934913-006618a599-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.905 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.905 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:09.906 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:09.906 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.906 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.906 DEBUG:boto:Method: GET 2024-04-12T03:08:09.906 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:09.906 DEBUG:boto:Data: 2024-04-12T03:08:09.906 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.907 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.907 DEBUG:boto:Params: {} 2024-04-12T03:08:09.907 DEBUG:boto:Token: None 2024-04-12T03:08:09.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/ 2024-04-12T03:08:09.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg= 2024-04-12T03:08:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGe0YkHGsiVd5WlEw7U3Z28ZQFg=', 'Content-Length': '0'} 2024-04-12T03:08:09.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c4bbc6d4f57794-006618a599-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.959 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:09.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.960 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.960 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.960 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.960 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.960 DEBUG:boto:Method: GET 2024-04-12T03:08:09.961 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.961 DEBUG:boto:Data: 2024-04-12T03:08:09.961 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.961 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.961 DEBUG:boto:Params: {} 2024-04-12T03:08:09.961 DEBUG:boto:Token: None 2024-04-12T03:08:09.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.962 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY= 2024-04-12T03:08:09.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY=', 'Content-Length': '0'} 2024-04-12T03:08:09.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a951b695c213580-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.966 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.966 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.966 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.966 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.966 DEBUG:boto:Method: GET 2024-04-12T03:08:09.967 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.967 DEBUG:boto:Data: 2024-04-12T03:08:09.967 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.967 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.967 DEBUG:boto:Params: {} 2024-04-12T03:08:09.967 DEBUG:boto:Token: None 2024-04-12T03:08:09.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY= 2024-04-12T03:08:09.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY=', 'Content-Length': '0'} 2024-04-12T03:08:09.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5eef4f33e6ddea2-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.971 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.971 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.972 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.972 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.972 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.972 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.972 DEBUG:boto:Data: 2024-04-12T03:08:09.972 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.972 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.972 DEBUG:boto:Params: {} 2024-04-12T03:08:09.972 DEBUG:boto:Token: None 2024-04-12T03:08:09.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.973 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WkVu6A1ZD5fq2TMn5w12DaoWbj4= 2024-04-12T03:08:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WkVu6A1ZD5fq2TMn5w12DaoWbj4=', 'Content-Length': '0'} 2024-04-12T03:08:09.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c61ceaf5109aa585-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.976 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.977 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.977 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.977 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.977 DEBUG:boto:Method: HEAD 2024-04-12T03:08:09.977 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.977 DEBUG:boto:Data: 2024-04-12T03:08:09.977 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:09.977 DEBUG:boto:Port: 8000 2024-04-12T03:08:09.977 DEBUG:boto:Params: {} 2024-04-12T03:08:09.978 DEBUG:boto:Token: None 2024-04-12T03:08:09.978 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.978 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WkVu6A1ZD5fq2TMn5w12DaoWbj4= 2024-04-12T03:08:09.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WkVu6A1ZD5fq2TMn5w12DaoWbj4=', 'Content-Length': '0'} 2024-04-12T03:08:09.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e50e2ee3781bb559-006618a599-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:09.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:09.983 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:09.983 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:09.983 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.983 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.983 DEBUG:boto:Method: GET 2024-04-12T03:08:09.983 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.983 DEBUG:boto:Data: 2024-04-12T03:08:09.983 DEBUG:boto:Headers: {} 2024-04-12T03:08:09.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:09.984 DEBUG:boto:Port: 8001 2024-04-12T03:08:09.984 DEBUG:boto:Params: {} 2024-04-12T03:08:09.984 DEBUG:boto:Token: None 2024-04-12T03:08:09.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:09 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:09.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY= 2024-04-12T03:08:09.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X6Li4d6elOxQ0M/IEMScJbn62iY=', 'Content-Length': '0'} 2024-04-12T03:08:09.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bdd6e53bd76ec74-006618a599-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.031 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.031 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.031 DEBUG:boto:path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:10.031 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:10.031 DEBUG:boto:Method: GET 2024-04-12T03:08:10.031 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:10.032 DEBUG:boto:Data: 2024-04-12T03:08:10.032 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.032 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.032 DEBUG:boto:Params: {} 2024-04-12T03:08:10.032 DEBUG:boto:Token: None 2024-04-12T03:08:10.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6 2024-04-12T03:08:10.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:glTKRceluwI675NEEjrRydxOJs0= 2024-04-12T03:08:10.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:glTKRceluwI675NEEjrRydxOJs0=', 'Content-Length': '0'} 2024-04-12T03:08:10.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'yApN1qfcLlzZUxxFPmrPO6qfFmhYsR6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e66ad0af43c5faa-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.034 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.035 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.035 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.035 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.035 DEBUG:boto:Method: GET 2024-04-12T03:08:10.035 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.035 DEBUG:boto:Data: 2024-04-12T03:08:10.035 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.035 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.035 DEBUG:boto:Params: {} 2024-04-12T03:08:10.036 DEBUG:boto:Token: None 2024-04-12T03:08:10.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d2bbc77104b785-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.087 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.088 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.088 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.088 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.088 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.088 DEBUG:boto:Method: GET 2024-04-12T03:08:10.088 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.088 DEBUG:boto:Data: 2024-04-12T03:08:10.088 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.089 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.089 DEBUG:boto:Params: {} 2024-04-12T03:08:10.089 DEBUG:boto:Token: None 2024-04-12T03:08:10.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de533c8cdc0f1c4b-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.092 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.092 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.093 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.093 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.093 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.093 DEBUG:boto:Data: 2024-04-12T03:08:10.093 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.093 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.093 DEBUG:boto:Params: {} 2024-04-12T03:08:10.093 DEBUG:boto:Token: None 2024-04-12T03:08:10.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a48628db35b29640-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.097 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.097 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.097 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.097 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.097 DEBUG:boto:Data: 2024-04-12T03:08:10.097 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.098 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.098 DEBUG:boto:Params: {} 2024-04-12T03:08:10.098 DEBUG:boto:Token: None 2024-04-12T03:08:10.098 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c70577d753eb05c7-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.101 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.101 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.101 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.101 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.101 DEBUG:boto:Method: GET 2024-04-12T03:08:10.102 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.102 DEBUG:boto:Data: 2024-04-12T03:08:10.102 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.102 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.102 DEBUG:boto:Params: {} 2024-04-12T03:08:10.102 DEBUG:boto:Token: None 2024-04-12T03:08:10.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3f84e71b5ee1acf-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.147 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.147 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.148 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.148 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.148 DEBUG:boto:Method: GET 2024-04-12T03:08:10.148 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.148 DEBUG:boto:Data: 2024-04-12T03:08:10.148 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.148 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.148 DEBUG:boto:Params: {} 2024-04-12T03:08:10.148 DEBUG:boto:Token: None 2024-04-12T03:08:10.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df4f24e10cc3833b-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.152 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.152 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.152 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.152 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.152 DEBUG:boto:Method: GET 2024-04-12T03:08:10.152 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.152 DEBUG:boto:Data: 2024-04-12T03:08:10.152 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.153 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.153 DEBUG:boto:Params: {} 2024-04-12T03:08:10.153 DEBUG:boto:Token: None 2024-04-12T03:08:10.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6be79468b218ea3-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.157 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.157 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.158 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.158 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.158 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.158 DEBUG:boto:Method: GET 2024-04-12T03:08:10.158 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.158 DEBUG:boto:Data: 2024-04-12T03:08:10.158 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.158 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.158 DEBUG:boto:Params: {} 2024-04-12T03:08:10.159 DEBUG:boto:Token: None 2024-04-12T03:08:10.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8739213752c051e-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.162 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.162 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.162 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.162 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.162 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.163 DEBUG:boto:Method: GET 2024-04-12T03:08:10.163 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.163 DEBUG:boto:Data: 2024-04-12T03:08:10.163 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.163 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.163 DEBUG:boto:Params: {} 2024-04-12T03:08:10.163 DEBUG:boto:Token: None 2024-04-12T03:08:10.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4= 2024-04-12T03:08:10.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4=', 'Content-Length': '0'} 2024-04-12T03:08:10.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4d0635b47180624-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.207 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.207 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.207 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.207 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.207 DEBUG:boto:Method: GET 2024-04-12T03:08:10.207 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.208 DEBUG:boto:Data: 2024-04-12T03:08:10.208 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.208 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.208 DEBUG:boto:Params: {} 2024-04-12T03:08:10.208 DEBUG:boto:Token: None 2024-04-12T03:08:10.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4= 2024-04-12T03:08:10.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4=', 'Content-Length': '0'} 2024-04-12T03:08:10.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee8ffe8524e16848-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.210 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.210 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.211 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.211 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.211 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.211 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.211 DEBUG:boto:Data: 2024-04-12T03:08:10.211 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.211 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.211 DEBUG:boto:Params: {} 2024-04-12T03:08:10.211 DEBUG:boto:Token: None 2024-04-12T03:08:10.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PVCYjqIY0RKyPNWL6Qrde3EfId8= 2024-04-12T03:08:10.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PVCYjqIY0RKyPNWL6Qrde3EfId8=', 'Content-Length': '0'} 2024-04-12T03:08:10.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5df5498fadc1aae-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.214 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.215 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.215 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.215 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.215 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.215 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.215 DEBUG:boto:Data: 2024-04-12T03:08:10.215 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.215 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.215 DEBUG:boto:Params: {} 2024-04-12T03:08:10.215 DEBUG:boto:Token: None 2024-04-12T03:08:10.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PVCYjqIY0RKyPNWL6Qrde3EfId8= 2024-04-12T03:08:10.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PVCYjqIY0RKyPNWL6Qrde3EfId8=', 'Content-Length': '0'} 2024-04-12T03:08:10.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ca527d6ab20961f-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.218 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.218 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.218 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.218 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.218 DEBUG:boto:Method: GET 2024-04-12T03:08:10.218 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.218 DEBUG:boto:Data: 2024-04-12T03:08:10.218 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.219 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.219 DEBUG:boto:Params: {} 2024-04-12T03:08:10.222 DEBUG:boto:Token: None 2024-04-12T03:08:10.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4= 2024-04-12T03:08:10.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4=', 'Content-Length': '0'} 2024-04-12T03:08:10.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000904e7546dcb39b16-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.224 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.224 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.224 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.225 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.225 DEBUG:boto:Method: GET 2024-04-12T03:08:10.225 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.225 DEBUG:boto:Data: 2024-04-12T03:08:10.225 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.225 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.225 DEBUG:boto:Params: {} 2024-04-12T03:08:10.225 DEBUG:boto:Token: None 2024-04-12T03:08:10.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC 2024-04-12T03:08:10.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4= 2024-04-12T03:08:10.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SMu37RcatlxAhTA5EErQYtrKU+4=', 'Content-Length': '0'} 2024-04-12T03:08:10.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-cAPUVSS2ci.HqftZCMgVG2X9O4.zkC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4e7022c00e6f20f-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.270 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.270 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.271 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.271 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.271 DEBUG:boto:Method: GET 2024-04-12T03:08:10.271 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.271 DEBUG:boto:Data: 2024-04-12T03:08:10.271 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.271 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.271 DEBUG:boto:Params: {} 2024-04-12T03:08:10.271 DEBUG:boto:Token: None 2024-04-12T03:08:10.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e643ac6730ce14a-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.274 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.275 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.275 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.275 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.275 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.275 DEBUG:boto:Method: GET 2024-04-12T03:08:10.275 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.275 DEBUG:boto:Data: 2024-04-12T03:08:10.275 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.275 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.275 DEBUG:boto:Params: {} 2024-04-12T03:08:10.275 DEBUG:boto:Token: None 2024-04-12T03:08:10.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9f8764e6689387-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.318 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.318 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.319 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.319 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.319 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.319 DEBUG:boto:Data: 2024-04-12T03:08:10.319 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.319 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.319 DEBUG:boto:Params: {} 2024-04-12T03:08:10.319 DEBUG:boto:Token: None 2024-04-12T03:08:10.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000028295a68c53b9da-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.323 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.323 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.323 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.323 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.323 DEBUG:boto:Data: 2024-04-12T03:08:10.323 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.323 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.323 DEBUG:boto:Params: {} 2024-04-12T03:08:10.323 DEBUG:boto:Token: None 2024-04-12T03:08:10.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0f7b8560af2a148-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.328 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.328 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.328 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.328 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.328 DEBUG:boto:Method: GET 2024-04-12T03:08:10.328 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.328 DEBUG:boto:Data: 2024-04-12T03:08:10.328 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.328 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.328 DEBUG:boto:Params: {} 2024-04-12T03:08:10.329 DEBUG:boto:Token: None 2024-04-12T03:08:10.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b690b764d3b61ee-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.374 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.374 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.374 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.374 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.375 DEBUG:boto:Method: GET 2024-04-12T03:08:10.375 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.375 DEBUG:boto:Data: 2024-04-12T03:08:10.375 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.375 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.375 DEBUG:boto:Params: {} 2024-04-12T03:08:10.375 DEBUG:boto:Token: None 2024-04-12T03:08:10.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093ca1db0a5a049a7-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.377 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.378 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.378 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.378 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.378 DEBUG:boto:Method: GET 2024-04-12T03:08:10.378 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.378 DEBUG:boto:Data: 2024-04-12T03:08:10.378 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.378 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.378 DEBUG:boto:Params: {} 2024-04-12T03:08:10.378 DEBUG:boto:Token: None 2024-04-12T03:08:10.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79f667725858e95-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.381 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.381 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.382 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.382 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.382 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.382 DEBUG:boto:Method: GET 2024-04-12T03:08:10.382 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.382 DEBUG:boto:Data: 2024-04-12T03:08:10.382 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.382 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.382 DEBUG:boto:Params: {} 2024-04-12T03:08:10.382 DEBUG:boto:Token: None 2024-04-12T03:08:10.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8b3dbe1ad85a419-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.385 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.386 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.386 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.386 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.386 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.386 DEBUG:boto:Method: GET 2024-04-12T03:08:10.386 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.386 DEBUG:boto:Data: 2024-04-12T03:08:10.386 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.386 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.386 DEBUG:boto:Params: {} 2024-04-12T03:08:10.387 DEBUG:boto:Token: None 2024-04-12T03:08:10.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA= 2024-04-12T03:08:10.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA=', 'Content-Length': '0'} 2024-04-12T03:08:10.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080831b88bcff1e51-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.389 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.389 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.389 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.389 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.389 DEBUG:boto:Method: GET 2024-04-12T03:08:10.389 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.390 DEBUG:boto:Data: 2024-04-12T03:08:10.390 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.390 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.390 DEBUG:boto:Params: {} 2024-04-12T03:08:10.390 DEBUG:boto:Token: None 2024-04-12T03:08:10.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA= 2024-04-12T03:08:10.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA=', 'Content-Length': '0'} 2024-04-12T03:08:10.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000443fddcee8f815ca-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.392 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.392 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.392 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.392 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.392 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.393 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.393 DEBUG:boto:Data: 2024-04-12T03:08:10.393 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.393 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.393 DEBUG:boto:Params: {} 2024-04-12T03:08:10.393 DEBUG:boto:Token: None 2024-04-12T03:08:10.393 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+h/t2C8c0JGGbpptQs/yCp4W9Q= 2024-04-12T03:08:10.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+h/t2C8c0JGGbpptQs/yCp4W9Q=', 'Content-Length': '0'} 2024-04-12T03:08:10.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000667f4c7944efb75a-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.395 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.395 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.395 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.395 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.396 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.396 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.396 DEBUG:boto:Data: 2024-04-12T03:08:10.396 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.396 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.396 DEBUG:boto:Params: {} 2024-04-12T03:08:10.396 DEBUG:boto:Token: None 2024-04-12T03:08:10.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+h/t2C8c0JGGbpptQs/yCp4W9Q= 2024-04-12T03:08:10.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+h/t2C8c0JGGbpptQs/yCp4W9Q=', 'Content-Length': '0'} 2024-04-12T03:08:10.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c42f427464adeb40-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.398 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.398 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.398 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.398 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.399 DEBUG:boto:Method: GET 2024-04-12T03:08:10.399 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.399 DEBUG:boto:Data: 2024-04-12T03:08:10.399 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.399 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.399 DEBUG:boto:Params: {} 2024-04-12T03:08:10.399 DEBUG:boto:Token: None 2024-04-12T03:08:10.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA= 2024-04-12T03:08:10.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA=', 'Content-Length': '0'} 2024-04-12T03:08:10.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000268bac621d96a529-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.442 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.442 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.443 DEBUG:boto:path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.443 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.443 DEBUG:boto:Method: GET 2024-04-12T03:08:10.443 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.443 DEBUG:boto:Data: 2024-04-12T03:08:10.443 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.443 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.443 DEBUG:boto:Params: {} 2024-04-12T03:08:10.443 DEBUG:boto:Token: None 2024-04-12T03:08:10.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=Y4UsZptQCnyCfND-3uZAipSEiHu4LFD 2024-04-12T03:08:10.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA= 2024-04-12T03:08:10.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMvp/t7GxLzEQ83H1mtP1zQPuKA=', 'Content-Length': '0'} 2024-04-12T03:08:10.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'Y4UsZptQCnyCfND-3uZAipSEiHu4LFD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e79fe2d1c54d081-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.446 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.446 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.446 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.446 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.446 DEBUG:boto:Method: GET 2024-04-12T03:08:10.446 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.447 DEBUG:boto:Data: 2024-04-12T03:08:10.447 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.447 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.447 DEBUG:boto:Params: {} 2024-04-12T03:08:10.447 DEBUG:boto:Token: None 2024-04-12T03:08:10.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9b6ea2d68b8590c-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.490 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.490 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.491 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.491 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.491 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.491 DEBUG:boto:Method: GET 2024-04-12T03:08:10.491 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.491 DEBUG:boto:Data: 2024-04-12T03:08:10.491 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.491 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.491 DEBUG:boto:Params: {} 2024-04-12T03:08:10.491 DEBUG:boto:Token: None 2024-04-12T03:08:10.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.492 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047857f2afcab1363-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.494 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.494 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.495 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.495 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.495 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.495 DEBUG:boto:Data: 2024-04-12T03:08:10.495 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.495 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.495 DEBUG:boto:Params: {} 2024-04-12T03:08:10.495 DEBUG:boto:Token: None 2024-04-12T03:08:10.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbead86cfea52473-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.499 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.499 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.499 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.499 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.499 DEBUG:boto:Data: 2024-04-12T03:08:10.499 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.499 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.500 DEBUG:boto:Params: {} 2024-04-12T03:08:10.500 DEBUG:boto:Token: None 2024-04-12T03:08:10.500 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001de04aef692e9806-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.503 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.503 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.503 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.503 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.503 DEBUG:boto:Method: GET 2024-04-12T03:08:10.503 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.503 DEBUG:boto:Data: 2024-04-12T03:08:10.503 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.504 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.504 DEBUG:boto:Params: {} 2024-04-12T03:08:10.504 DEBUG:boto:Token: None 2024-04-12T03:08:10.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000051642465c594119-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.546 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.546 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.546 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.547 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.547 DEBUG:boto:Method: GET 2024-04-12T03:08:10.547 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.547 DEBUG:boto:Data: 2024-04-12T03:08:10.547 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.547 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.547 DEBUG:boto:Params: {} 2024-04-12T03:08:10.547 DEBUG:boto:Token: None 2024-04-12T03:08:10.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a239d1967b12afc-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.550 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.550 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.550 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.550 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.550 DEBUG:boto:Method: GET 2024-04-12T03:08:10.550 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.550 DEBUG:boto:Data: 2024-04-12T03:08:10.550 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.550 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.550 DEBUG:boto:Params: {} 2024-04-12T03:08:10.551 DEBUG:boto:Token: None 2024-04-12T03:08:10.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbbd7a12ba40d6d9-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.594 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.595 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.595 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.595 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.595 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.595 DEBUG:boto:Method: GET 2024-04-12T03:08:10.595 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.595 DEBUG:boto:Data: 2024-04-12T03:08:10.595 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.595 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.595 DEBUG:boto:Params: {} 2024-04-12T03:08:10.595 DEBUG:boto:Token: None 2024-04-12T03:08:10.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2781061671912e0-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.598 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.599 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.599 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.599 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.599 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.599 DEBUG:boto:Method: GET 2024-04-12T03:08:10.599 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.599 DEBUG:boto:Data: 2024-04-12T03:08:10.599 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.599 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.599 DEBUG:boto:Params: {} 2024-04-12T03:08:10.600 DEBUG:boto:Token: None 2024-04-12T03:08:10.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4= 2024-04-12T03:08:10.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4=', 'Content-Length': '0'} 2024-04-12T03:08:10.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a92eae78cac8d30e-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.602 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.602 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.602 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.602 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.602 DEBUG:boto:Method: GET 2024-04-12T03:08:10.602 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.602 DEBUG:boto:Data: 2024-04-12T03:08:10.603 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.603 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.603 DEBUG:boto:Params: {} 2024-04-12T03:08:10.603 DEBUG:boto:Token: None 2024-04-12T03:08:10.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4= 2024-04-12T03:08:10.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4=', 'Content-Length': '0'} 2024-04-12T03:08:10.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e763cbfaf45eb86-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.646 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.646 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.646 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.646 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.647 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.647 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.647 DEBUG:boto:Data: 2024-04-12T03:08:10.647 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.647 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.647 DEBUG:boto:Params: {} 2024-04-12T03:08:10.647 DEBUG:boto:Token: None 2024-04-12T03:08:10.647 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Db6uvYZoc1jfaLipQtFoZ1S0GTo= 2024-04-12T03:08:10.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Db6uvYZoc1jfaLipQtFoZ1S0GTo=', 'Content-Length': '0'} 2024-04-12T03:08:10.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b32836375d7abdf-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.649 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.650 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.650 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.650 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.650 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.650 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.650 DEBUG:boto:Data: 2024-04-12T03:08:10.650 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.650 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.650 DEBUG:boto:Params: {} 2024-04-12T03:08:10.650 DEBUG:boto:Token: None 2024-04-12T03:08:10.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Db6uvYZoc1jfaLipQtFoZ1S0GTo= 2024-04-12T03:08:10.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Db6uvYZoc1jfaLipQtFoZ1S0GTo=', 'Content-Length': '0'} 2024-04-12T03:08:10.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001da71eb997086fd5-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.653 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.653 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.653 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.653 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.653 DEBUG:boto:Method: GET 2024-04-12T03:08:10.653 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.653 DEBUG:boto:Data: 2024-04-12T03:08:10.653 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.653 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.653 DEBUG:boto:Params: {} 2024-04-12T03:08:10.654 DEBUG:boto:Token: None 2024-04-12T03:08:10.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.654 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4= 2024-04-12T03:08:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4=', 'Content-Length': '0'} 2024-04-12T03:08:10.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000423db56cb9ae4167-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.656 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.656 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.656 DEBUG:boto:path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.656 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.656 DEBUG:boto:Method: GET 2024-04-12T03:08:10.657 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.657 DEBUG:boto:Data: 2024-04-12T03:08:10.657 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.657 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.657 DEBUG:boto:Params: {} 2024-04-12T03:08:10.657 DEBUG:boto:Token: None 2024-04-12T03:08:10.657 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ 2024-04-12T03:08:10.657 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4= 2024-04-12T03:08:10.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iOUjfkdtegTd6rl5bztq2TPLjq4=', 'Content-Length': '0'} 2024-04-12T03:08:10.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '1NzaxpEYA.-7CjOQu2yMRfoEbrpF2RJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072537a3dc3b0cd6f-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.702 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.702 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.702 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.703 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.703 DEBUG:boto:Method: GET 2024-04-12T03:08:10.703 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.703 DEBUG:boto:Data: 2024-04-12T03:08:10.703 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.703 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.703 DEBUG:boto:Params: {} 2024-04-12T03:08:10.703 DEBUG:boto:Token: None 2024-04-12T03:08:10.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001525e4f2811c9cf0-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.706 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.707 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.707 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.707 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.707 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.707 DEBUG:boto:Method: GET 2024-04-12T03:08:10.707 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.707 DEBUG:boto:Data: 2024-04-12T03:08:10.707 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.707 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.707 DEBUG:boto:Params: {} 2024-04-12T03:08:10.707 DEBUG:boto:Token: None 2024-04-12T03:08:10.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003342caffff3065e0-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.710 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.711 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.711 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.711 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.711 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.711 DEBUG:boto:Data: 2024-04-12T03:08:10.711 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.711 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.711 DEBUG:boto:Params: {} 2024-04-12T03:08:10.711 DEBUG:boto:Token: None 2024-04-12T03:08:10.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c9859d23a0213ec-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.715 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.715 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.715 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.715 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.715 DEBUG:boto:Data: 2024-04-12T03:08:10.715 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.715 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.715 DEBUG:boto:Params: {} 2024-04-12T03:08:10.716 DEBUG:boto:Token: None 2024-04-12T03:08:10.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000748f6d80167fe6cd-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.719 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.719 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.719 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.720 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.720 DEBUG:boto:Method: GET 2024-04-12T03:08:10.720 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.720 DEBUG:boto:Data: 2024-04-12T03:08:10.720 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.720 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.720 DEBUG:boto:Params: {} 2024-04-12T03:08:10.720 DEBUG:boto:Token: None 2024-04-12T03:08:10.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071afd04cd46b5a1d-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.762 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.762 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.762 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.763 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.763 DEBUG:boto:Method: GET 2024-04-12T03:08:10.763 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.763 DEBUG:boto:Data: 2024-04-12T03:08:10.763 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.763 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.763 DEBUG:boto:Params: {} 2024-04-12T03:08:10.763 DEBUG:boto:Token: None 2024-04-12T03:08:10.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f84a87fde938d29-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.766 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.766 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.766 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.766 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.766 DEBUG:boto:Method: GET 2024-04-12T03:08:10.766 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.766 DEBUG:boto:Data: 2024-04-12T03:08:10.766 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.766 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.766 DEBUG:boto:Params: {} 2024-04-12T03:08:10.767 DEBUG:boto:Token: None 2024-04-12T03:08:10.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756affcb31d4d402-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.769 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.770 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.770 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.770 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.770 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.770 DEBUG:boto:Method: GET 2024-04-12T03:08:10.770 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.770 DEBUG:boto:Data: 2024-04-12T03:08:10.770 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.771 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.771 DEBUG:boto:Params: {} 2024-04-12T03:08:10.771 DEBUG:boto:Token: None 2024-04-12T03:08:10.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008680278f754c337c-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.774 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.774 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.775 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.775 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.775 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.775 DEBUG:boto:Method: GET 2024-04-12T03:08:10.775 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.775 DEBUG:boto:Data: 2024-04-12T03:08:10.775 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.775 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.775 DEBUG:boto:Params: {} 2024-04-12T03:08:10.775 DEBUG:boto:Token: None 2024-04-12T03:08:10.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU= 2024-04-12T03:08:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU=', 'Content-Length': '0'} 2024-04-12T03:08:10.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000283ccdaabed4ae3-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.778 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.778 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.778 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.778 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.778 DEBUG:boto:Method: GET 2024-04-12T03:08:10.778 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.779 DEBUG:boto:Data: 2024-04-12T03:08:10.779 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.779 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.779 DEBUG:boto:Params: {} 2024-04-12T03:08:10.779 DEBUG:boto:Token: None 2024-04-12T03:08:10.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU= 2024-04-12T03:08:10.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU=', 'Content-Length': '0'} 2024-04-12T03:08:10.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dfe1cdcb439cf4c-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.781 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.781 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.782 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.782 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.782 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.782 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.782 DEBUG:boto:Data: 2024-04-12T03:08:10.782 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.782 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.782 DEBUG:boto:Params: {} 2024-04-12T03:08:10.782 DEBUG:boto:Token: None 2024-04-12T03:08:10.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vohRs7k3vLceJlMynwUv7cOKpg= 2024-04-12T03:08:10.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vohRs7k3vLceJlMynwUv7cOKpg=', 'Content-Length': '0'} 2024-04-12T03:08:10.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f16d2ae8a688311-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.785 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.785 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.785 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.785 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.785 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.785 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.785 DEBUG:boto:Data: 2024-04-12T03:08:10.785 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.785 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.786 DEBUG:boto:Params: {} 2024-04-12T03:08:10.786 DEBUG:boto:Token: None 2024-04-12T03:08:10.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vohRs7k3vLceJlMynwUv7cOKpg= 2024-04-12T03:08:10.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vohRs7k3vLceJlMynwUv7cOKpg=', 'Content-Length': '0'} 2024-04-12T03:08:10.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020a9adecc50cf0a9-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.791 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.791 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.791 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.791 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.791 DEBUG:boto:Method: GET 2024-04-12T03:08:10.791 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.791 DEBUG:boto:Data: 2024-04-12T03:08:10.792 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.792 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.792 DEBUG:boto:Params: {} 2024-04-12T03:08:10.792 DEBUG:boto:Token: None 2024-04-12T03:08:10.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU= 2024-04-12T03:08:10.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU=', 'Content-Length': '0'} 2024-04-12T03:08:10.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000021df3497026f1eb-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.838 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.839 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.839 DEBUG:boto:path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.839 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.839 DEBUG:boto:Method: GET 2024-04-12T03:08:10.839 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.839 DEBUG:boto:Data: 2024-04-12T03:08:10.839 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.839 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.839 DEBUG:boto:Params: {} 2024-04-12T03:08:10.840 DEBUG:boto:Token: None 2024-04-12T03:08:10.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=ffvLWMVrdZDe13V1v1UbT78s2d98pF- 2024-04-12T03:08:10.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU= 2024-04-12T03:08:10.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzlmV6e+RIiQUlYPeE6wC/uruAU=', 'Content-Length': '0'} 2024-04-12T03:08:10.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'ffvLWMVrdZDe13V1v1UbT78s2d98pF-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000651c7e0acebf086-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.842 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.842 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.843 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.843 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.843 DEBUG:boto:Method: GET 2024-04-12T03:08:10.843 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.843 DEBUG:boto:Data: 2024-04-12T03:08:10.843 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.843 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.843 DEBUG:boto:Params: {} 2024-04-12T03:08:10.843 DEBUG:boto:Token: None 2024-04-12T03:08:10.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094214dafdc66373-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.846 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.847 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.847 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.847 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.847 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.847 DEBUG:boto:Method: GET 2024-04-12T03:08:10.847 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.847 DEBUG:boto:Data: 2024-04-12T03:08:10.847 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.847 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.848 DEBUG:boto:Params: {} 2024-04-12T03:08:10.848 DEBUG:boto:Token: None 2024-04-12T03:08:10.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0790d3594c387e1-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.851 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.851 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.851 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.851 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.852 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.852 DEBUG:boto:Data: 2024-04-12T03:08:10.852 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.852 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.852 DEBUG:boto:Params: {} 2024-04-12T03:08:10.852 DEBUG:boto:Token: None 2024-04-12T03:08:10.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002334f00e44253574-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.856 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.856 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.856 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.856 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:10.856 DEBUG:boto:Data: 2024-04-12T03:08:10.856 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.857 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.857 DEBUG:boto:Params: {} 2024-04-12T03:08:10.857 DEBUG:boto:Token: None 2024-04-12T03:08:10.857 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj 2024-04-12T03:08:10.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs= 2024-04-12T03:08:10.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:smpndZOyyBcT8PSb3peQBn7Mgjs=', 'Content-Length': '0'} 2024-04-12T03:08:10.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4f45e923b0ac297-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.861 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.861 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.861 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.861 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.861 DEBUG:boto:Method: GET 2024-04-12T03:08:10.861 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.861 DEBUG:boto:Data: 2024-04-12T03:08:10.861 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.862 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.862 DEBUG:boto:Params: {} 2024-04-12T03:08:10.862 DEBUG:boto:Token: None 2024-04-12T03:08:10.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b151ae538dd3e1b5-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.906 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.907 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.907 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.907 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.907 DEBUG:boto:Method: GET 2024-04-12T03:08:10.907 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.907 DEBUG:boto:Data: 2024-04-12T03:08:10.907 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.907 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.907 DEBUG:boto:Params: {} 2024-04-12T03:08:10.908 DEBUG:boto:Token: None 2024-04-12T03:08:10.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:10.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ= 2024-04-12T03:08:10.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cCMilehaEG1ofxmeRuJEVi4ZiYQ=', 'Content-Length': '0'} 2024-04-12T03:08:10.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f61d178b0778bdc3-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.910 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.910 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.910 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.910 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.910 DEBUG:boto:Method: GET 2024-04-12T03:08:10.911 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.911 DEBUG:boto:Data: 2024-04-12T03:08:10.911 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.911 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.911 DEBUG:boto:Params: {} 2024-04-12T03:08:10.911 DEBUG:boto:Token: None 2024-04-12T03:08:10.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000796a198563bc75df-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.914 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.914 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.915 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.915 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.915 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.915 DEBUG:boto:Method: GET 2024-04-12T03:08:10.915 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.915 DEBUG:boto:Data: 2024-04-12T03:08:10.915 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.915 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.915 DEBUG:boto:Params: {} 2024-04-12T03:08:10.916 DEBUG:boto:Token: None 2024-04-12T03:08:10.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f494d7545d3a48f0-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.918 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.919 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.919 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.919 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.919 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.919 DEBUG:boto:Method: GET 2024-04-12T03:08:10.919 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.919 DEBUG:boto:Data: 2024-04-12T03:08:10.919 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.920 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.920 DEBUG:boto:Params: {} 2024-04-12T03:08:10.920 DEBUG:boto:Token: None 2024-04-12T03:08:10.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8= 2024-04-12T03:08:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8=', 'Content-Length': '0'} 2024-04-12T03:08:10.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bb7eeb0824f9e30-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.922 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.922 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.922 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.922 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.922 DEBUG:boto:Method: GET 2024-04-12T03:08:10.923 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.923 DEBUG:boto:Data: 2024-04-12T03:08:10.923 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.923 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.923 DEBUG:boto:Params: {} 2024-04-12T03:08:10.923 DEBUG:boto:Token: None 2024-04-12T03:08:10.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8= 2024-04-12T03:08:10.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8=', 'Content-Length': '0'} 2024-04-12T03:08:10.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f05970293e5fa67-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.926 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.926 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.926 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.926 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.926 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.926 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.926 DEBUG:boto:Data: 2024-04-12T03:08:10.926 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.926 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.926 DEBUG:boto:Params: {} 2024-04-12T03:08:10.927 DEBUG:boto:Token: None 2024-04-12T03:08:10.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VX3bt2gybx2RLKe+LSUwUJ+yZk0= 2024-04-12T03:08:10.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VX3bt2gybx2RLKe+LSUwUJ+yZk0=', 'Content-Length': '0'} 2024-04-12T03:08:10.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f09d26f4f8606ac3-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.929 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.929 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.929 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.929 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.929 DEBUG:boto:Method: HEAD 2024-04-12T03:08:10.929 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.929 DEBUG:boto:Data: 2024-04-12T03:08:10.930 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.930 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.930 DEBUG:boto:Params: {} 2024-04-12T03:08:10.930 DEBUG:boto:Token: None 2024-04-12T03:08:10.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VX3bt2gybx2RLKe+LSUwUJ+yZk0= 2024-04-12T03:08:10.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VX3bt2gybx2RLKe+LSUwUJ+yZk0=', 'Content-Length': '0'} 2024-04-12T03:08:10.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a30b0adc47c26c05-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:10.932 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.932 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.932 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.932 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.932 DEBUG:boto:Method: GET 2024-04-12T03:08:10.933 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.933 DEBUG:boto:Data: 2024-04-12T03:08:10.933 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.933 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.933 DEBUG:boto:Params: {} 2024-04-12T03:08:10.933 DEBUG:boto:Token: None 2024-04-12T03:08:10.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8= 2024-04-12T03:08:10.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8=', 'Content-Length': '0'} 2024-04-12T03:08:10.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008188d7d17800a5d2-006618a59a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.935 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:10.935 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:10.936 DEBUG:boto:path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.936 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.936 DEBUG:boto:Method: GET 2024-04-12T03:08:10.936 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.936 DEBUG:boto:Data: 2024-04-12T03:08:10.936 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.936 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.936 DEBUG:boto:Params: {} 2024-04-12T03:08:10.936 DEBUG:boto:Token: None 2024-04-12T03:08:10.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/obj?versionId=47QlbJPF6KNDjYM8z8QahgoYTX4c1WX 2024-04-12T03:08:10.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8= 2024-04-12T03:08:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xAKDm4y5g3YTFHO3gDWxy0ccXW8=', 'Content-Length': '0'} 2024-04-12T03:08:10.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '47QlbJPF6KNDjYM8z8QahgoYTX4c1WX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000445caeb130305ee1-006618a59a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.978 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.979 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.979 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.979 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.979 DEBUG:boto:Method: GET 2024-04-12T03:08:10.979 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.979 DEBUG:boto:Data: 2024-04-12T03:08:10.979 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:10.979 DEBUG:boto:Port: 8001 2024-04-12T03:08:10.979 DEBUG:boto:Params: {} 2024-04-12T03:08:10.980 DEBUG:boto:Token: None 2024-04-12T03:08:10.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064752f2200ffb51d-006618a59a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:10.983 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:10.983 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:10.983 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:10.983 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.983 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.984 DEBUG:boto:Method: GET 2024-04-12T03:08:10.984 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:10.984 DEBUG:boto:Data: 2024-04-12T03:08:10.984 DEBUG:boto:Headers: {} 2024-04-12T03:08:10.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:10.984 DEBUG:boto:Port: 8000 2024-04-12T03:08:10.984 DEBUG:boto:Params: {} 2024-04-12T03:08:10.984 DEBUG:boto:Token: None 2024-04-12T03:08:10.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:10 GMT /dgbpxg-15/ 2024-04-12T03:08:10.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs= 2024-04-12T03:08:10.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:10 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cXkpKlTtPq1oAERd5d7ffe7l3Zs=', 'Content-Length': '0'} 2024-04-12T03:08:10.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f8641ee708dd514-006618a59a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.030 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.031 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.031 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.031 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.031 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.031 DEBUG:boto:Data: 2024-04-12T03:08:11.031 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.031 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.031 DEBUG:boto:Params: {} 2024-04-12T03:08:11.032 DEBUG:boto:Token: None 2024-04-12T03:08:11.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd1a2f85afe66ff5-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.036 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.036 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.036 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.036 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.036 DEBUG:boto:Data: 2024-04-12T03:08:11.036 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.036 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.036 DEBUG:boto:Params: {} 2024-04-12T03:08:11.036 DEBUG:boto:Token: None 2024-04-12T03:08:11.037 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d7ae4b268b75e0-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.040 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.041 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.041 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.041 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.041 DEBUG:boto:Method: GET 2024-04-12T03:08:11.041 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.041 DEBUG:boto:Data: 2024-04-12T03:08:11.041 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.041 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.041 DEBUG:boto:Params: {} 2024-04-12T03:08:11.041 DEBUG:boto:Token: None 2024-04-12T03:08:11.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000418c7870543d6a34-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.086 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.087 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.087 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.087 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.087 DEBUG:boto:Method: GET 2024-04-12T03:08:11.087 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.087 DEBUG:boto:Data: 2024-04-12T03:08:11.088 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.088 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.088 DEBUG:boto:Params: {} 2024-04-12T03:08:11.088 DEBUG:boto:Token: None 2024-04-12T03:08:11.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a2f7ff10033fd3b-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.091 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.091 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.091 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.091 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.091 DEBUG:boto:Method: GET 2024-04-12T03:08:11.091 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.092 DEBUG:boto:Data: 2024-04-12T03:08:11.092 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.092 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.092 DEBUG:boto:Params: {} 2024-04-12T03:08:11.092 DEBUG:boto:Token: None 2024-04-12T03:08:11.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd0a5cb5929de30-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.095 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.095 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.095 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.095 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.095 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.095 DEBUG:boto:Method: GET 2024-04-12T03:08:11.095 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.096 DEBUG:boto:Data: 2024-04-12T03:08:11.096 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.096 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.096 DEBUG:boto:Params: {} 2024-04-12T03:08:11.096 DEBUG:boto:Token: None 2024-04-12T03:08:11.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bcb85440e07a5ff-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.099 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.099 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.100 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.100 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.100 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.100 DEBUG:boto:Method: GET 2024-04-12T03:08:11.100 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.100 DEBUG:boto:Data: 2024-04-12T03:08:11.100 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.100 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.100 DEBUG:boto:Params: {} 2024-04-12T03:08:11.100 DEBUG:boto:Token: None 2024-04-12T03:08:11.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y= 2024-04-12T03:08:11.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d7cf324768b5c8a-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.103 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.103 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.103 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.103 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.104 DEBUG:boto:Method: GET 2024-04-12T03:08:11.104 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.104 DEBUG:boto:Data: 2024-04-12T03:08:11.104 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.104 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.104 DEBUG:boto:Params: {} 2024-04-12T03:08:11.104 DEBUG:boto:Token: None 2024-04-12T03:08:11.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y= 2024-04-12T03:08:11.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad165db67a6e38fc-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.150 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.151 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.151 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.151 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.151 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.151 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.151 DEBUG:boto:Data: 2024-04-12T03:08:11.151 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.151 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.151 DEBUG:boto:Params: {} 2024-04-12T03:08:11.152 DEBUG:boto:Token: None 2024-04-12T03:08:11.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lkFhgNvWVpk1JrCP2zCf5DmAWPI= 2024-04-12T03:08:11.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lkFhgNvWVpk1JrCP2zCf5DmAWPI=', 'Content-Length': '0'} 2024-04-12T03:08:11.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019bb56524d538764-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.154 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.154 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.154 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.154 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.154 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.155 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.155 DEBUG:boto:Data: 2024-04-12T03:08:11.155 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.155 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.155 DEBUG:boto:Params: {} 2024-04-12T03:08:11.155 DEBUG:boto:Token: None 2024-04-12T03:08:11.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lkFhgNvWVpk1JrCP2zCf5DmAWPI= 2024-04-12T03:08:11.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lkFhgNvWVpk1JrCP2zCf5DmAWPI=', 'Content-Length': '0'} 2024-04-12T03:08:11.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae15ecc88a83cb36-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.158 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.158 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.158 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.158 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.158 DEBUG:boto:Method: GET 2024-04-12T03:08:11.158 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.158 DEBUG:boto:Data: 2024-04-12T03:08:11.158 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.158 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.159 DEBUG:boto:Params: {} 2024-04-12T03:08:11.159 DEBUG:boto:Token: None 2024-04-12T03:08:11.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y= 2024-04-12T03:08:11.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6537fc2ee87862b-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.202 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.203 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.203 DEBUG:boto:path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.203 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.203 DEBUG:boto:Method: GET 2024-04-12T03:08:11.203 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.203 DEBUG:boto:Data: 2024-04-12T03:08:11.203 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.203 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.204 DEBUG:boto:Params: {} 2024-04-12T03:08:11.204 DEBUG:boto:Token: None 2024-04-12T03:08:11.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG 2024-04-12T03:08:11.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y= 2024-04-12T03:08:11.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nK1wJ2LDHgkMxyVfR3J45Osey7Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'jV1pRzYXnGlieidUQDQ-AjtYhKK-BGG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056ed15d9f2b83ec6-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.206 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.206 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.207 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.207 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.207 DEBUG:boto:Method: GET 2024-04-12T03:08:11.207 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.207 DEBUG:boto:Data: 2024-04-12T03:08:11.207 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.207 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.207 DEBUG:boto:Params: {} 2024-04-12T03:08:11.207 DEBUG:boto:Token: None 2024-04-12T03:08:11.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7036ed608b8445c-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.254 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.255 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.255 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.255 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.255 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.255 DEBUG:boto:Method: GET 2024-04-12T03:08:11.255 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.256 DEBUG:boto:Data: 2024-04-12T03:08:11.256 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.256 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.256 DEBUG:boto:Params: {} 2024-04-12T03:08:11.256 DEBUG:boto:Token: None 2024-04-12T03:08:11.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a45530711a2e54-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.259 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.259 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.259 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.260 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.260 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.260 DEBUG:boto:Data: 2024-04-12T03:08:11.260 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.260 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.260 DEBUG:boto:Params: {} 2024-04-12T03:08:11.260 DEBUG:boto:Token: None 2024-04-12T03:08:11.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000949a3695de8842c-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.264 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.264 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.264 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.264 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.264 DEBUG:boto:Data: 2024-04-12T03:08:11.264 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.264 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.264 DEBUG:boto:Params: {} 2024-04-12T03:08:11.264 DEBUG:boto:Token: None 2024-04-12T03:08:11.265 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.265 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bee0190dd9a0536-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.268 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.268 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.268 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.268 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.268 DEBUG:boto:Method: GET 2024-04-12T03:08:11.268 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.268 DEBUG:boto:Data: 2024-04-12T03:08:11.268 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.269 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.269 DEBUG:boto:Params: {} 2024-04-12T03:08:11.269 DEBUG:boto:Token: None 2024-04-12T03:08:11.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000730284368863d977-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.314 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.314 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.315 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.315 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.315 DEBUG:boto:Method: GET 2024-04-12T03:08:11.315 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.315 DEBUG:boto:Data: 2024-04-12T03:08:11.315 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.315 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.315 DEBUG:boto:Params: {} 2024-04-12T03:08:11.316 DEBUG:boto:Token: None 2024-04-12T03:08:11.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055d0d158297e2bb4-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.318 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.318 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.318 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.318 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.318 DEBUG:boto:Method: GET 2024-04-12T03:08:11.318 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.318 DEBUG:boto:Data: 2024-04-12T03:08:11.319 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.319 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.319 DEBUG:boto:Params: {} 2024-04-12T03:08:11.319 DEBUG:boto:Token: None 2024-04-12T03:08:11.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e2998bca141006-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.362 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.363 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.363 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.363 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.363 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.363 DEBUG:boto:Method: GET 2024-04-12T03:08:11.363 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.363 DEBUG:boto:Data: 2024-04-12T03:08:11.363 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.364 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.364 DEBUG:boto:Params: {} 2024-04-12T03:08:11.364 DEBUG:boto:Token: None 2024-04-12T03:08:11.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d83ab4548b62da4-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.367 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.367 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.367 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.368 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.368 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.368 DEBUG:boto:Method: GET 2024-04-12T03:08:11.368 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.368 DEBUG:boto:Data: 2024-04-12T03:08:11.368 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.368 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.368 DEBUG:boto:Params: {} 2024-04-12T03:08:11.368 DEBUG:boto:Token: None 2024-04-12T03:08:11.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k= 2024-04-12T03:08:11.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k=', 'Content-Length': '0'} 2024-04-12T03:08:11.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e18de1e04d9ca43e-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.371 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.371 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.371 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.371 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.372 DEBUG:boto:Method: GET 2024-04-12T03:08:11.372 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.372 DEBUG:boto:Data: 2024-04-12T03:08:11.372 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.372 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.372 DEBUG:boto:Params: {} 2024-04-12T03:08:11.372 DEBUG:boto:Token: None 2024-04-12T03:08:11.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k= 2024-04-12T03:08:11.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k=', 'Content-Length': '0'} 2024-04-12T03:08:11.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042ccea25ab0e2a74-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.414 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.415 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.415 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.415 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.415 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.415 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.415 DEBUG:boto:Data: 2024-04-12T03:08:11.415 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.415 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.415 DEBUG:boto:Params: {} 2024-04-12T03:08:11.415 DEBUG:boto:Token: None 2024-04-12T03:08:11.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Y2E7EFtfrobCCic6ExF2aTyBgo= 2024-04-12T03:08:11.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Y2E7EFtfrobCCic6ExF2aTyBgo=', 'Content-Length': '0'} 2024-04-12T03:08:11.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000519b5bf896fa19a2-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.418 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.418 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.418 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.418 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.418 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.418 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.418 DEBUG:boto:Data: 2024-04-12T03:08:11.419 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.419 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.419 DEBUG:boto:Params: {} 2024-04-12T03:08:11.419 DEBUG:boto:Token: None 2024-04-12T03:08:11.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Y2E7EFtfrobCCic6ExF2aTyBgo= 2024-04-12T03:08:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Y2E7EFtfrobCCic6ExF2aTyBgo=', 'Content-Length': '0'} 2024-04-12T03:08:11.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2167b847bb84c41-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.421 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.421 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.421 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.421 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.422 DEBUG:boto:Method: GET 2024-04-12T03:08:11.422 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.422 DEBUG:boto:Data: 2024-04-12T03:08:11.422 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.422 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.422 DEBUG:boto:Params: {} 2024-04-12T03:08:11.422 DEBUG:boto:Token: None 2024-04-12T03:08:11.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k= 2024-04-12T03:08:11.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k=', 'Content-Length': '0'} 2024-04-12T03:08:11.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c33f7625e9ea134-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.470 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.471 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.471 DEBUG:boto:path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.471 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.471 DEBUG:boto:Method: GET 2024-04-12T03:08:11.471 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.471 DEBUG:boto:Data: 2024-04-12T03:08:11.471 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.471 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.471 DEBUG:boto:Params: {} 2024-04-12T03:08:11.471 DEBUG:boto:Token: None 2024-04-12T03:08:11.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F 2024-04-12T03:08:11.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k= 2024-04-12T03:08:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aXAw8GyQwkoAu1w/u+mgrCMsl4k=', 'Content-Length': '0'} 2024-04-12T03:08:11.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'oaMQ.eux27q2OxMdGhmDSwI.QvJdy-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0c9bb38b521c08a-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.474 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.474 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.474 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.474 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.474 DEBUG:boto:Method: GET 2024-04-12T03:08:11.475 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.475 DEBUG:boto:Data: 2024-04-12T03:08:11.475 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.475 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.475 DEBUG:boto:Params: {} 2024-04-12T03:08:11.475 DEBUG:boto:Token: None 2024-04-12T03:08:11.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e54b4aa1cefd98c-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.478 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.478 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.478 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.478 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.479 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.479 DEBUG:boto:Method: GET 2024-04-12T03:08:11.479 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.479 DEBUG:boto:Data: 2024-04-12T03:08:11.479 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.479 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.479 DEBUG:boto:Params: {} 2024-04-12T03:08:11.479 DEBUG:boto:Token: None 2024-04-12T03:08:11.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d772365db8d93ea7-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.482 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.483 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.483 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.483 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.483 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.483 DEBUG:boto:Data: 2024-04-12T03:08:11.483 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.483 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.483 DEBUG:boto:Params: {} 2024-04-12T03:08:11.484 DEBUG:boto:Token: None 2024-04-12T03:08:11.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055904a6691c9537d-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.487 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.487 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.488 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.488 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.488 DEBUG:boto:Data: 2024-04-12T03:08:11.488 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.488 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.488 DEBUG:boto:Params: {} 2024-04-12T03:08:11.488 DEBUG:boto:Token: None 2024-04-12T03:08:11.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000776a19d1cfe3af6f-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.492 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.492 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.492 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.492 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.492 DEBUG:boto:Method: GET 2024-04-12T03:08:11.492 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.492 DEBUG:boto:Data: 2024-04-12T03:08:11.492 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.493 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.493 DEBUG:boto:Params: {} 2024-04-12T03:08:11.493 DEBUG:boto:Token: None 2024-04-12T03:08:11.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3af7d0a9db9399f-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.538 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.539 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.539 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.539 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.539 DEBUG:boto:Method: GET 2024-04-12T03:08:11.539 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.539 DEBUG:boto:Data: 2024-04-12T03:08:11.539 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.539 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.540 DEBUG:boto:Params: {} 2024-04-12T03:08:11.540 DEBUG:boto:Token: None 2024-04-12T03:08:11.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd6785ebb9785048-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.542 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.543 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.543 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.543 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.543 DEBUG:boto:Method: GET 2024-04-12T03:08:11.543 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.543 DEBUG:boto:Data: 2024-04-12T03:08:11.543 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.543 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.544 DEBUG:boto:Params: {} 2024-04-12T03:08:11.544 DEBUG:boto:Token: None 2024-04-12T03:08:11.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e197b8de5978c3b4-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.547 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.547 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.547 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.547 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.547 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.548 DEBUG:boto:Method: GET 2024-04-12T03:08:11.548 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.548 DEBUG:boto:Data: 2024-04-12T03:08:11.548 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.548 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.548 DEBUG:boto:Params: {} 2024-04-12T03:08:11.548 DEBUG:boto:Token: None 2024-04-12T03:08:11.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81297fe192aa99a-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.594 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.595 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.595 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.595 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.595 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.595 DEBUG:boto:Method: GET 2024-04-12T03:08:11.595 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.595 DEBUG:boto:Data: 2024-04-12T03:08:11.596 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.596 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.596 DEBUG:boto:Params: {} 2024-04-12T03:08:11.596 DEBUG:boto:Token: None 2024-04-12T03:08:11.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8= 2024-04-12T03:08:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8=', 'Content-Length': '0'} 2024-04-12T03:08:11.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a27fb1f950fe90d8-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.599 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.599 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.599 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.599 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.599 DEBUG:boto:Method: GET 2024-04-12T03:08:11.599 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.599 DEBUG:boto:Data: 2024-04-12T03:08:11.599 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.600 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.600 DEBUG:boto:Params: {} 2024-04-12T03:08:11.600 DEBUG:boto:Token: None 2024-04-12T03:08:11.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8= 2024-04-12T03:08:11.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8=', 'Content-Length': '0'} 2024-04-12T03:08:11.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b04492628abd79f7-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.603 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.603 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.603 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.603 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.603 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.603 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.603 DEBUG:boto:Data: 2024-04-12T03:08:11.603 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.603 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.603 DEBUG:boto:Params: {} 2024-04-12T03:08:11.604 DEBUG:boto:Token: None 2024-04-12T03:08:11.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uNtuZtsuLZY0N0bn1YNqLnGSAaE= 2024-04-12T03:08:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uNtuZtsuLZY0N0bn1YNqLnGSAaE=', 'Content-Length': '0'} 2024-04-12T03:08:11.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053d25205dac6244f-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.606 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.606 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.606 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.606 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.606 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.606 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.606 DEBUG:boto:Data: 2024-04-12T03:08:11.606 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.607 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.607 DEBUG:boto:Params: {} 2024-04-12T03:08:11.607 DEBUG:boto:Token: None 2024-04-12T03:08:11.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uNtuZtsuLZY0N0bn1YNqLnGSAaE= 2024-04-12T03:08:11.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uNtuZtsuLZY0N0bn1YNqLnGSAaE=', 'Content-Length': '0'} 2024-04-12T03:08:11.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008955e3a326528e87-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.609 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.610 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.610 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.610 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.610 DEBUG:boto:Method: GET 2024-04-12T03:08:11.610 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.610 DEBUG:boto:Data: 2024-04-12T03:08:11.610 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.610 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.610 DEBUG:boto:Params: {} 2024-04-12T03:08:11.610 DEBUG:boto:Token: None 2024-04-12T03:08:11.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8= 2024-04-12T03:08:11.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8=', 'Content-Length': '0'} 2024-04-12T03:08:11.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c748985bc0f639d9-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.654 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.654 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.655 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.655 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.655 DEBUG:boto:Method: GET 2024-04-12T03:08:11.655 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.655 DEBUG:boto:Data: 2024-04-12T03:08:11.655 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.655 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.655 DEBUG:boto:Params: {} 2024-04-12T03:08:11.655 DEBUG:boto:Token: None 2024-04-12T03:08:11.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC 2024-04-12T03:08:11.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8= 2024-04-12T03:08:11.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0IFp6Lw3C8DsRd/HeBNh2fykBZ8=', 'Content-Length': '0'} 2024-04-12T03:08:11.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wrHehFqeBQkfs1.ar4QEDn6bOCWN2mC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc2c8f47b26fc3e-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.698 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.698 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.699 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.699 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.699 DEBUG:boto:Method: GET 2024-04-12T03:08:11.699 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.699 DEBUG:boto:Data: 2024-04-12T03:08:11.699 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.699 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.699 DEBUG:boto:Params: {} 2024-04-12T03:08:11.699 DEBUG:boto:Token: None 2024-04-12T03:08:11.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003305dc733f818b85-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.702 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.703 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.703 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.703 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.703 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.703 DEBUG:boto:Method: GET 2024-04-12T03:08:11.703 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.703 DEBUG:boto:Data: 2024-04-12T03:08:11.703 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.704 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.704 DEBUG:boto:Params: {} 2024-04-12T03:08:11.704 DEBUG:boto:Token: None 2024-04-12T03:08:11.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269556f492aaeb0c-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.750 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.751 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.751 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.751 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.751 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.751 DEBUG:boto:Data: 2024-04-12T03:08:11.751 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.751 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.751 DEBUG:boto:Params: {} 2024-04-12T03:08:11.751 DEBUG:boto:Token: None 2024-04-12T03:08:11.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ad1a82109a5e6e2-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.755 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.755 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.755 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.755 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.755 DEBUG:boto:Data: 2024-04-12T03:08:11.755 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.756 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.756 DEBUG:boto:Params: {} 2024-04-12T03:08:11.756 DEBUG:boto:Token: None 2024-04-12T03:08:11.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068bb47141b837b73-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.759 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.760 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.760 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.760 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.760 DEBUG:boto:Method: GET 2024-04-12T03:08:11.760 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.760 DEBUG:boto:Data: 2024-04-12T03:08:11.760 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.760 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.761 DEBUG:boto:Params: {} 2024-04-12T03:08:11.761 DEBUG:boto:Token: None 2024-04-12T03:08:11.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb155da8a31dad01-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.806 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.806 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.807 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.807 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.807 DEBUG:boto:Method: GET 2024-04-12T03:08:11.807 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.807 DEBUG:boto:Data: 2024-04-12T03:08:11.807 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.807 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.807 DEBUG:boto:Params: {} 2024-04-12T03:08:11.807 DEBUG:boto:Token: None 2024-04-12T03:08:11.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ddf57496c3b3ea1-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.810 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.810 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.810 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.810 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.811 DEBUG:boto:Method: GET 2024-04-12T03:08:11.811 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.811 DEBUG:boto:Data: 2024-04-12T03:08:11.811 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.811 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.811 DEBUG:boto:Params: {} 2024-04-12T03:08:11.811 DEBUG:boto:Token: None 2024-04-12T03:08:11.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b970ab3843c0f6-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.858 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.859 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.859 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.859 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.859 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.859 DEBUG:boto:Method: GET 2024-04-12T03:08:11.859 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.859 DEBUG:boto:Data: 2024-04-12T03:08:11.859 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.859 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.860 DEBUG:boto:Params: {} 2024-04-12T03:08:11.860 DEBUG:boto:Token: None 2024-04-12T03:08:11.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002635ba1c89c50c3a-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.863 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.863 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.863 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.863 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.863 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.864 DEBUG:boto:Method: GET 2024-04-12T03:08:11.864 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.864 DEBUG:boto:Data: 2024-04-12T03:08:11.864 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.864 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.864 DEBUG:boto:Params: {} 2024-04-12T03:08:11.864 DEBUG:boto:Token: None 2024-04-12T03:08:11.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4= 2024-04-12T03:08:11.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4=', 'Content-Length': '0'} 2024-04-12T03:08:11.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063fad10e3f7c46d7-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.867 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.867 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.867 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.867 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.867 DEBUG:boto:Method: GET 2024-04-12T03:08:11.867 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.867 DEBUG:boto:Data: 2024-04-12T03:08:11.867 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.867 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.867 DEBUG:boto:Params: {} 2024-04-12T03:08:11.868 DEBUG:boto:Token: None 2024-04-12T03:08:11.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4= 2024-04-12T03:08:11.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4=', 'Content-Length': '0'} 2024-04-12T03:08:11.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d525c4bbf09a7d77-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.870 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.870 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.870 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.870 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.870 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.870 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.870 DEBUG:boto:Data: 2024-04-12T03:08:11.871 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.871 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.871 DEBUG:boto:Params: {} 2024-04-12T03:08:11.871 DEBUG:boto:Token: None 2024-04-12T03:08:11.871 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tFsmYYX8ZL0+55pbjZBTz52HC3Y= 2024-04-12T03:08:11.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tFsmYYX8ZL0+55pbjZBTz52HC3Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007659d7a72f54ce8f-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.873 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.874 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.874 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.874 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.874 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.874 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.874 DEBUG:boto:Data: 2024-04-12T03:08:11.874 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.874 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.874 DEBUG:boto:Params: {} 2024-04-12T03:08:11.874 DEBUG:boto:Token: None 2024-04-12T03:08:11.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tFsmYYX8ZL0+55pbjZBTz52HC3Y= 2024-04-12T03:08:11.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tFsmYYX8ZL0+55pbjZBTz52HC3Y=', 'Content-Length': '0'} 2024-04-12T03:08:11.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66f843f0cebe3a7-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.877 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.877 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.877 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.877 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.877 DEBUG:boto:Method: GET 2024-04-12T03:08:11.877 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.877 DEBUG:boto:Data: 2024-04-12T03:08:11.877 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.878 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.878 DEBUG:boto:Params: {} 2024-04-12T03:08:11.878 DEBUG:boto:Token: None 2024-04-12T03:08:11.878 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.878 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4= 2024-04-12T03:08:11.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4=', 'Content-Length': '0'} 2024-04-12T03:08:11.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005407aeabb89bde82-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.922 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.922 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.923 DEBUG:boto:path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.923 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.923 DEBUG:boto:Method: GET 2024-04-12T03:08:11.923 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.923 DEBUG:boto:Data: 2024-04-12T03:08:11.923 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.923 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.923 DEBUG:boto:Params: {} 2024-04-12T03:08:11.923 DEBUG:boto:Token: None 2024-04-12T03:08:11.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL 2024-04-12T03:08:11.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4= 2024-04-12T03:08:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5937Zsyd5zJo9eGXHVO4FI4THT4=', 'Content-Length': '0'} 2024-04-12T03:08:11.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'lkvI3dmhci4zQk8IEwXAj9LrvAIJCuL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000318d56df264fef95-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.926 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.926 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.926 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.926 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.927 DEBUG:boto:Method: GET 2024-04-12T03:08:11.927 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.927 DEBUG:boto:Data: 2024-04-12T03:08:11.927 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.927 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.927 DEBUG:boto:Params: {} 2024-04-12T03:08:11.927 DEBUG:boto:Token: None 2024-04-12T03:08:11.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056638c1fe97c4075-006618a59b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.970 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.971 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:11.971 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:11.971 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.971 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.971 DEBUG:boto:Method: GET 2024-04-12T03:08:11.971 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:11.971 DEBUG:boto:Data: 2024-04-12T03:08:11.972 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.972 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.972 DEBUG:boto:Params: {} 2024-04-12T03:08:11.972 DEBUG:boto:Token: None 2024-04-12T03:08:11.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/ 2024-04-12T03:08:11.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw= 2024-04-12T03:08:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xv+XDc8YLKJmjgvbjv0MeGZk8Uw=', 'Content-Length': '0'} 2024-04-12T03:08:11.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f206a6795d3623c-006618a59b-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.975 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:11.975 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.975 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.976 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.976 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.976 DEBUG:boto:Data: 2024-04-12T03:08:11.976 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.976 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.976 DEBUG:boto:Params: {} 2024-04-12T03:08:11.976 DEBUG:boto:Token: None 2024-04-12T03:08:11.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba4509fd2c146808-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.980 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.980 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.980 DEBUG:boto:Method: HEAD 2024-04-12T03:08:11.980 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:11.980 DEBUG:boto:Data: 2024-04-12T03:08:11.980 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:11.981 DEBUG:boto:Port: 8000 2024-04-12T03:08:11.981 DEBUG:boto:Params: {} 2024-04-12T03:08:11.981 DEBUG:boto:Token: None 2024-04-12T03:08:11.981 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj 2024-04-12T03:08:11.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU= 2024-04-12T03:08:11.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rzDRJr3yTkRPZon81rYrfo1tkzU=', 'Content-Length': '0'} 2024-04-12T03:08:11.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000899f12f92eb33fae-006618a59b-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:11.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:11.984 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:11.984 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:11.984 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.985 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.985 DEBUG:boto:Method: GET 2024-04-12T03:08:11.985 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.985 DEBUG:boto:Data: 2024-04-12T03:08:11.985 DEBUG:boto:Headers: {} 2024-04-12T03:08:11.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:11.985 DEBUG:boto:Port: 8001 2024-04-12T03:08:11.985 DEBUG:boto:Params: {} 2024-04-12T03:08:11.985 DEBUG:boto:Token: None 2024-04-12T03:08:11.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:11 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:11.986 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg= 2024-04-12T03:08:11.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2lRS5XoX7unvNFXJ05wjEaVdmg=', 'Content-Length': '0'} 2024-04-12T03:08:11.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e7389becce3e547-006618a59b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.034 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.034 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.034 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.035 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.035 DEBUG:boto:Method: GET 2024-04-12T03:08:12.035 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.035 DEBUG:boto:Data: 2024-04-12T03:08:12.035 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.035 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.035 DEBUG:boto:Params: {} 2024-04-12T03:08:12.035 DEBUG:boto:Token: None 2024-04-12T03:08:12.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069642015335db96a-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.038 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.038 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.038 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.038 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.038 DEBUG:boto:Method: GET 2024-04-12T03:08:12.038 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.038 DEBUG:boto:Data: 2024-04-12T03:08:12.038 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.039 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.039 DEBUG:boto:Params: {} 2024-04-12T03:08:12.039 DEBUG:boto:Token: None 2024-04-12T03:08:12.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca21a69fa61a8b9-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.082 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.083 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.083 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.083 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.083 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.083 DEBUG:boto:Method: GET 2024-04-12T03:08:12.083 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.083 DEBUG:boto:Data: 2024-04-12T03:08:12.083 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.084 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.084 DEBUG:boto:Params: {} 2024-04-12T03:08:12.084 DEBUG:boto:Token: None 2024-04-12T03:08:12.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bb7a6ee3dbfd3a-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.087 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.087 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.087 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.087 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.088 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.088 DEBUG:boto:Method: GET 2024-04-12T03:08:12.088 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.088 DEBUG:boto:Data: 2024-04-12T03:08:12.088 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.088 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.088 DEBUG:boto:Params: {} 2024-04-12T03:08:12.088 DEBUG:boto:Token: None 2024-04-12T03:08:12.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0= 2024-04-12T03:08:12.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0=', 'Content-Length': '0'} 2024-04-12T03:08:12.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1ea5056b4da2fc6-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.091 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.091 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.091 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.091 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.091 DEBUG:boto:Method: GET 2024-04-12T03:08:12.091 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.091 DEBUG:boto:Data: 2024-04-12T03:08:12.091 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.092 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.092 DEBUG:boto:Params: {} 2024-04-12T03:08:12.092 DEBUG:boto:Token: None 2024-04-12T03:08:12.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0= 2024-04-12T03:08:12.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0=', 'Content-Length': '0'} 2024-04-12T03:08:12.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7807676e7c48086-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.095 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.095 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.095 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.095 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.095 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.095 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.095 DEBUG:boto:Data: 2024-04-12T03:08:12.095 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.096 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.096 DEBUG:boto:Params: {} 2024-04-12T03:08:12.096 DEBUG:boto:Token: None 2024-04-12T03:08:12.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G8Lc5gPd3XcLgoFAUG7EPvGNW4Y= 2024-04-12T03:08:12.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G8Lc5gPd3XcLgoFAUG7EPvGNW4Y=', 'Content-Length': '0'} 2024-04-12T03:08:12.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000162996fb3048af95-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.098 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.098 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.099 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.099 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.099 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.099 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.099 DEBUG:boto:Data: 2024-04-12T03:08:12.099 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.099 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.099 DEBUG:boto:Params: {} 2024-04-12T03:08:12.099 DEBUG:boto:Token: None 2024-04-12T03:08:12.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G8Lc5gPd3XcLgoFAUG7EPvGNW4Y= 2024-04-12T03:08:12.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G8Lc5gPd3XcLgoFAUG7EPvGNW4Y=', 'Content-Length': '0'} 2024-04-12T03:08:12.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9e6582e1bb91b69-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.102 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.102 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.102 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.102 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.102 DEBUG:boto:Method: GET 2024-04-12T03:08:12.102 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.102 DEBUG:boto:Data: 2024-04-12T03:08:12.103 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.103 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.103 DEBUG:boto:Params: {} 2024-04-12T03:08:12.103 DEBUG:boto:Token: None 2024-04-12T03:08:12.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0= 2024-04-12T03:08:12.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0=', 'Content-Length': '0'} 2024-04-12T03:08:12.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d6f3f1e0f76512a-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.105 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.105 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.105 DEBUG:boto:path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.105 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.105 DEBUG:boto:Method: GET 2024-04-12T03:08:12.105 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.106 DEBUG:boto:Data: 2024-04-12T03:08:12.106 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.106 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.106 DEBUG:boto:Params: {} 2024-04-12T03:08:12.106 DEBUG:boto:Token: None 2024-04-12T03:08:12.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=pak4GQgdOu63EMuX67LGYKjVi9OFJir 2024-04-12T03:08:12.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0= 2024-04-12T03:08:12.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5rhYJKZQ+lFK6qs+gvSLmvinX0=', 'Content-Length': '0'} 2024-04-12T03:08:12.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'pak4GQgdOu63EMuX67LGYKjVi9OFJir'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb4ff9dab207f3d-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.150 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.150 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.150 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.151 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.151 DEBUG:boto:Method: GET 2024-04-12T03:08:12.151 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.151 DEBUG:boto:Data: 2024-04-12T03:08:12.151 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.151 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.151 DEBUG:boto:Params: {} 2024-04-12T03:08:12.151 DEBUG:boto:Token: None 2024-04-12T03:08:12.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1f4b277881b2ef-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.154 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.155 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.155 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.155 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.155 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.155 DEBUG:boto:Method: GET 2024-04-12T03:08:12.155 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.155 DEBUG:boto:Data: 2024-04-12T03:08:12.155 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.156 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.156 DEBUG:boto:Params: {} 2024-04-12T03:08:12.156 DEBUG:boto:Token: None 2024-04-12T03:08:12.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009faafc9a26b2837b-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.202 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.203 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.203 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.203 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.203 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.203 DEBUG:boto:Data: 2024-04-12T03:08:12.203 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.203 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.203 DEBUG:boto:Params: {} 2024-04-12T03:08:12.203 DEBUG:boto:Token: None 2024-04-12T03:08:12.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068edb3abf7a8a797-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.207 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.207 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.207 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.207 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.208 DEBUG:boto:Data: 2024-04-12T03:08:12.208 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.208 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.208 DEBUG:boto:Params: {} 2024-04-12T03:08:12.208 DEBUG:boto:Token: None 2024-04-12T03:08:12.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005bda1bcb2947d86-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.212 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.212 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.212 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.212 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.212 DEBUG:boto:Method: GET 2024-04-12T03:08:12.212 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.212 DEBUG:boto:Data: 2024-04-12T03:08:12.212 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.212 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.212 DEBUG:boto:Params: {} 2024-04-12T03:08:12.213 DEBUG:boto:Token: None 2024-04-12T03:08:12.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b32affa9cb44aa1c-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.258 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.258 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.259 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.259 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.259 DEBUG:boto:Method: GET 2024-04-12T03:08:12.259 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.259 DEBUG:boto:Data: 2024-04-12T03:08:12.259 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.259 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.259 DEBUG:boto:Params: {} 2024-04-12T03:08:12.259 DEBUG:boto:Token: None 2024-04-12T03:08:12.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd61f86de075efd6-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.262 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.262 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.262 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.262 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.262 DEBUG:boto:Method: GET 2024-04-12T03:08:12.262 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.262 DEBUG:boto:Data: 2024-04-12T03:08:12.262 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.263 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.263 DEBUG:boto:Params: {} 2024-04-12T03:08:12.263 DEBUG:boto:Token: None 2024-04-12T03:08:12.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e73941db235f49f-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.266 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.266 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.266 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.266 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.266 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.267 DEBUG:boto:Method: GET 2024-04-12T03:08:12.267 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.267 DEBUG:boto:Data: 2024-04-12T03:08:12.267 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.267 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.267 DEBUG:boto:Params: {} 2024-04-12T03:08:12.267 DEBUG:boto:Token: None 2024-04-12T03:08:12.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ce72c33a68aa23-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.270 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.270 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.270 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.271 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.271 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.271 DEBUG:boto:Method: GET 2024-04-12T03:08:12.271 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.271 DEBUG:boto:Data: 2024-04-12T03:08:12.271 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.271 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.271 DEBUG:boto:Params: {} 2024-04-12T03:08:12.271 DEBUG:boto:Token: None 2024-04-12T03:08:12.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4= 2024-04-12T03:08:12.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4=', 'Content-Length': '0'} 2024-04-12T03:08:12.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070f4436a251c76be-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.274 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.274 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.274 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.274 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.274 DEBUG:boto:Method: GET 2024-04-12T03:08:12.274 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.274 DEBUG:boto:Data: 2024-04-12T03:08:12.274 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.275 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.275 DEBUG:boto:Params: {} 2024-04-12T03:08:12.275 DEBUG:boto:Token: None 2024-04-12T03:08:12.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4= 2024-04-12T03:08:12.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4=', 'Content-Length': '0'} 2024-04-12T03:08:12.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2ececee15bd9f3a-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.277 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.278 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.278 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.278 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.278 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.278 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.278 DEBUG:boto:Data: 2024-04-12T03:08:12.278 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.278 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.278 DEBUG:boto:Params: {} 2024-04-12T03:08:12.279 DEBUG:boto:Token: None 2024-04-12T03:08:12.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ghFBcDOkIo7Fbr7KYFFXbS9HAiY= 2024-04-12T03:08:12.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ghFBcDOkIo7Fbr7KYFFXbS9HAiY=', 'Content-Length': '0'} 2024-04-12T03:08:12.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004caa03d5a91cb172-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.281 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.281 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.281 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.281 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.281 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.281 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.281 DEBUG:boto:Data: 2024-04-12T03:08:12.282 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.282 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.282 DEBUG:boto:Params: {} 2024-04-12T03:08:12.282 DEBUG:boto:Token: None 2024-04-12T03:08:12.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.282 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ghFBcDOkIo7Fbr7KYFFXbS9HAiY= 2024-04-12T03:08:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ghFBcDOkIo7Fbr7KYFFXbS9HAiY=', 'Content-Length': '0'} 2024-04-12T03:08:12.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa5fbb45dc84c730-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.284 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.284 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.285 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.285 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.285 DEBUG:boto:Method: GET 2024-04-12T03:08:12.285 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.285 DEBUG:boto:Data: 2024-04-12T03:08:12.285 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.285 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.285 DEBUG:boto:Params: {} 2024-04-12T03:08:12.285 DEBUG:boto:Token: None 2024-04-12T03:08:12.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4= 2024-04-12T03:08:12.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4=', 'Content-Length': '0'} 2024-04-12T03:08:12.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000189a3e100a587b82-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.288 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.288 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.288 DEBUG:boto:path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.288 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.288 DEBUG:boto:Method: GET 2024-04-12T03:08:12.288 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.288 DEBUG:boto:Data: 2024-04-12T03:08:12.288 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.288 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.289 DEBUG:boto:Params: {} 2024-04-12T03:08:12.289 DEBUG:boto:Token: None 2024-04-12T03:08:12.289 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=9IV19jO8jWopyR.dsl0X2CCjrKwR8xW 2024-04-12T03:08:12.289 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4= 2024-04-12T03:08:12.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4NoUnsyz9vI+dhZRUcQwbXtKbK4=', 'Content-Length': '0'} 2024-04-12T03:08:12.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '9IV19jO8jWopyR.dsl0X2CCjrKwR8xW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fab71edbadbd52e6-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.334 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.334 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.334 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.335 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.335 DEBUG:boto:Method: GET 2024-04-12T03:08:12.335 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.335 DEBUG:boto:Data: 2024-04-12T03:08:12.335 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.335 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.335 DEBUG:boto:Params: {} 2024-04-12T03:08:12.335 DEBUG:boto:Token: None 2024-04-12T03:08:12.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000222bd0b6cc785e99-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.338 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.338 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.339 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.339 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.339 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.339 DEBUG:boto:Method: GET 2024-04-12T03:08:12.339 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.339 DEBUG:boto:Data: 2024-04-12T03:08:12.339 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.339 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.339 DEBUG:boto:Params: {} 2024-04-12T03:08:12.339 DEBUG:boto:Token: None 2024-04-12T03:08:12.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5716cdf7d83a500-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.342 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.342 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.343 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.343 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.343 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.343 DEBUG:boto:Data: 2024-04-12T03:08:12.343 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.343 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.343 DEBUG:boto:Params: {} 2024-04-12T03:08:12.343 DEBUG:boto:Token: None 2024-04-12T03:08:12.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3945604b0e7dfc7-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.347 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.347 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.347 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.347 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.347 DEBUG:boto:Data: 2024-04-12T03:08:12.347 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.347 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.347 DEBUG:boto:Params: {} 2024-04-12T03:08:12.348 DEBUG:boto:Token: None 2024-04-12T03:08:12.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000917312cfede88fb-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.351 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.351 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.351 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.351 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.351 DEBUG:boto:Method: GET 2024-04-12T03:08:12.352 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.352 DEBUG:boto:Data: 2024-04-12T03:08:12.352 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.352 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.352 DEBUG:boto:Params: {} 2024-04-12T03:08:12.352 DEBUG:boto:Token: None 2024-04-12T03:08:12.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4c654e7fecaadcd-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.394 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.394 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.394 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.395 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.395 DEBUG:boto:Method: GET 2024-04-12T03:08:12.395 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.395 DEBUG:boto:Data: 2024-04-12T03:08:12.395 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.395 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.395 DEBUG:boto:Params: {} 2024-04-12T03:08:12.395 DEBUG:boto:Token: None 2024-04-12T03:08:12.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098b81b8ddb54093d-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.398 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.398 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.398 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.398 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.398 DEBUG:boto:Method: GET 2024-04-12T03:08:12.398 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.398 DEBUG:boto:Data: 2024-04-12T03:08:12.398 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.398 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.399 DEBUG:boto:Params: {} 2024-04-12T03:08:12.399 DEBUG:boto:Token: None 2024-04-12T03:08:12.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a3a2af7b8e88ce-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.442 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.443 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.443 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.443 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.443 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.443 DEBUG:boto:Method: GET 2024-04-12T03:08:12.443 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.443 DEBUG:boto:Data: 2024-04-12T03:08:12.443 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.444 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.444 DEBUG:boto:Params: {} 2024-04-12T03:08:12.444 DEBUG:boto:Token: None 2024-04-12T03:08:12.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3111060de85d9e-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.447 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.447 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.447 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.447 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.448 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.448 DEBUG:boto:Method: GET 2024-04-12T03:08:12.448 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.448 DEBUG:boto:Data: 2024-04-12T03:08:12.448 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.448 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.448 DEBUG:boto:Params: {} 2024-04-12T03:08:12.448 DEBUG:boto:Token: None 2024-04-12T03:08:12.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.449 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ= 2024-04-12T03:08:12.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ=', 'Content-Length': '0'} 2024-04-12T03:08:12.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dddae96326b8d551-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.451 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.451 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.451 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.451 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.451 DEBUG:boto:Method: GET 2024-04-12T03:08:12.451 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.451 DEBUG:boto:Data: 2024-04-12T03:08:12.451 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.452 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.452 DEBUG:boto:Params: {} 2024-04-12T03:08:12.452 DEBUG:boto:Token: None 2024-04-12T03:08:12.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ= 2024-04-12T03:08:12.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ=', 'Content-Length': '0'} 2024-04-12T03:08:12.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fabfb97717579662-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.454 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.455 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.455 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.455 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.455 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.455 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.455 DEBUG:boto:Data: 2024-04-12T03:08:12.455 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.455 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.455 DEBUG:boto:Params: {} 2024-04-12T03:08:12.456 DEBUG:boto:Token: None 2024-04-12T03:08:12.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfVBvpHnwEwRbcr8teEK9DqmN7g= 2024-04-12T03:08:12.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfVBvpHnwEwRbcr8teEK9DqmN7g=', 'Content-Length': '0'} 2024-04-12T03:08:12.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f557c824da863c-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.458 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.458 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.458 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.458 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.458 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.458 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.458 DEBUG:boto:Data: 2024-04-12T03:08:12.459 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.459 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.459 DEBUG:boto:Params: {} 2024-04-12T03:08:12.459 DEBUG:boto:Token: None 2024-04-12T03:08:12.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfVBvpHnwEwRbcr8teEK9DqmN7g= 2024-04-12T03:08:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfVBvpHnwEwRbcr8teEK9DqmN7g=', 'Content-Length': '0'} 2024-04-12T03:08:12.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2096713467989f2-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.461 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.461 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.461 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.461 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.462 DEBUG:boto:Method: GET 2024-04-12T03:08:12.462 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.462 DEBUG:boto:Data: 2024-04-12T03:08:12.462 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.462 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.462 DEBUG:boto:Params: {} 2024-04-12T03:08:12.462 DEBUG:boto:Token: None 2024-04-12T03:08:12.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ= 2024-04-12T03:08:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ=', 'Content-Length': '0'} 2024-04-12T03:08:12.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000397156e09a38bc9c-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.506 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.506 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.506 DEBUG:boto:path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.507 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.507 DEBUG:boto:Method: GET 2024-04-12T03:08:12.507 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.507 DEBUG:boto:Data: 2024-04-12T03:08:12.507 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.507 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.507 DEBUG:boto:Params: {} 2024-04-12T03:08:12.507 DEBUG:boto:Token: None 2024-04-12T03:08:12.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL 2024-04-12T03:08:12.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ= 2024-04-12T03:08:12.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HmAEUtoO+r26lwiHU8EQVKXWoqQ=', 'Content-Length': '0'} 2024-04-12T03:08:12.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'hiI6JTYES9BqLUDKwnK0RcVlWcWJWKL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c568df59ba9eb759-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.510 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.510 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.510 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.510 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.510 DEBUG:boto:Method: GET 2024-04-12T03:08:12.510 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.510 DEBUG:boto:Data: 2024-04-12T03:08:12.510 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.510 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.511 DEBUG:boto:Params: {} 2024-04-12T03:08:12.511 DEBUG:boto:Token: None 2024-04-12T03:08:12.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd2f8f2c8610398-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.554 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.555 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.555 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.555 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.555 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.555 DEBUG:boto:Method: GET 2024-04-12T03:08:12.555 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.555 DEBUG:boto:Data: 2024-04-12T03:08:12.555 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.556 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.556 DEBUG:boto:Params: {} 2024-04-12T03:08:12.556 DEBUG:boto:Token: None 2024-04-12T03:08:12.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b5f7720238e1c90-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.559 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.559 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.559 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.559 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.559 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.559 DEBUG:boto:Data: 2024-04-12T03:08:12.559 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.560 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.560 DEBUG:boto:Params: {} 2024-04-12T03:08:12.560 DEBUG:boto:Token: None 2024-04-12T03:08:12.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e33b9db7b0f438c2-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.563 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.563 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.563 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.563 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.564 DEBUG:boto:Data: 2024-04-12T03:08:12.564 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.564 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.564 DEBUG:boto:Params: {} 2024-04-12T03:08:12.564 DEBUG:boto:Token: None 2024-04-12T03:08:12.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099ea2b7b51f32d9e-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.568 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.568 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.568 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.568 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.568 DEBUG:boto:Method: GET 2024-04-12T03:08:12.568 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.568 DEBUG:boto:Data: 2024-04-12T03:08:12.568 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.568 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.569 DEBUG:boto:Params: {} 2024-04-12T03:08:12.569 DEBUG:boto:Token: None 2024-04-12T03:08:12.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.569 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de381cacf2e8456f-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.614 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.614 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.614 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.615 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.615 DEBUG:boto:Method: GET 2024-04-12T03:08:12.615 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.615 DEBUG:boto:Data: 2024-04-12T03:08:12.615 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.615 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.615 DEBUG:boto:Params: {} 2024-04-12T03:08:12.615 DEBUG:boto:Token: None 2024-04-12T03:08:12.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a86845e40a064bd-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.618 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.618 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.618 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.618 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.618 DEBUG:boto:Method: GET 2024-04-12T03:08:12.618 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.618 DEBUG:boto:Data: 2024-04-12T03:08:12.619 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.619 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.619 DEBUG:boto:Params: {} 2024-04-12T03:08:12.619 DEBUG:boto:Token: None 2024-04-12T03:08:12.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094539fc4aaf01ffc-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.662 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.663 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.663 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.663 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.663 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.663 DEBUG:boto:Method: GET 2024-04-12T03:08:12.663 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.663 DEBUG:boto:Data: 2024-04-12T03:08:12.663 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.663 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.663 DEBUG:boto:Params: {} 2024-04-12T03:08:12.664 DEBUG:boto:Token: None 2024-04-12T03:08:12.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d019ef216f8254-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.666 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.667 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.667 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.667 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.667 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.667 DEBUG:boto:Method: GET 2024-04-12T03:08:12.667 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.667 DEBUG:boto:Data: 2024-04-12T03:08:12.667 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.668 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.668 DEBUG:boto:Params: {} 2024-04-12T03:08:12.668 DEBUG:boto:Token: None 2024-04-12T03:08:12.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4= 2024-04-12T03:08:12.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4=', 'Content-Length': '0'} 2024-04-12T03:08:12.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009f69cea13956582-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.670 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.671 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.671 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.671 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.671 DEBUG:boto:Method: GET 2024-04-12T03:08:12.671 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.671 DEBUG:boto:Data: 2024-04-12T03:08:12.671 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.671 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.671 DEBUG:boto:Params: {} 2024-04-12T03:08:12.672 DEBUG:boto:Token: None 2024-04-12T03:08:12.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4= 2024-04-12T03:08:12.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4=', 'Content-Length': '0'} 2024-04-12T03:08:12.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a90bcc39e70f4af-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.674 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.674 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.674 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.674 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.675 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.675 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.675 DEBUG:boto:Data: 2024-04-12T03:08:12.675 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.675 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.675 DEBUG:boto:Params: {} 2024-04-12T03:08:12.675 DEBUG:boto:Token: None 2024-04-12T03:08:12.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XcYybHhPxNHDVZrcPyJWKuSN1ck= 2024-04-12T03:08:12.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XcYybHhPxNHDVZrcPyJWKuSN1ck=', 'Content-Length': '0'} 2024-04-12T03:08:12.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa6eff2cb0ad9d06-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.678 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.678 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.678 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.678 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.678 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.678 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.678 DEBUG:boto:Data: 2024-04-12T03:08:12.678 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.679 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.679 DEBUG:boto:Params: {} 2024-04-12T03:08:12.679 DEBUG:boto:Token: None 2024-04-12T03:08:12.679 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.679 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XcYybHhPxNHDVZrcPyJWKuSN1ck= 2024-04-12T03:08:12.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XcYybHhPxNHDVZrcPyJWKuSN1ck=', 'Content-Length': '0'} 2024-04-12T03:08:12.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a1363d575b1789c-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.681 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.681 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.681 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.681 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.682 DEBUG:boto:Method: GET 2024-04-12T03:08:12.682 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.682 DEBUG:boto:Data: 2024-04-12T03:08:12.682 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.682 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.682 DEBUG:boto:Params: {} 2024-04-12T03:08:12.682 DEBUG:boto:Token: None 2024-04-12T03:08:12.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.683 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4= 2024-04-12T03:08:12.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4=', 'Content-Length': '0'} 2024-04-12T03:08:12.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000241b46aae50c551d-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.726 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.726 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.726 DEBUG:boto:path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.727 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.727 DEBUG:boto:Method: GET 2024-04-12T03:08:12.727 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.727 DEBUG:boto:Data: 2024-04-12T03:08:12.727 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.727 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.727 DEBUG:boto:Params: {} 2024-04-12T03:08:12.727 DEBUG:boto:Token: None 2024-04-12T03:08:12.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg 2024-04-12T03:08:12.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4= 2024-04-12T03:08:12.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPTUPN6vd0T9SGnRGNuU2W883n4=', 'Content-Length': '0'} 2024-04-12T03:08:12.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', '-kcLOF-uBDPIeu9dWArWlO6KUUrsGSg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6b4b1b0072bd998-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.730 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.730 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.730 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.730 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.730 DEBUG:boto:Method: GET 2024-04-12T03:08:12.730 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.730 DEBUG:boto:Data: 2024-04-12T03:08:12.730 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.731 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.731 DEBUG:boto:Params: {} 2024-04-12T03:08:12.731 DEBUG:boto:Token: None 2024-04-12T03:08:12.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af354a57ea754658-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.774 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.775 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.775 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.775 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.775 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.775 DEBUG:boto:Method: GET 2024-04-12T03:08:12.775 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.775 DEBUG:boto:Data: 2024-04-12T03:08:12.775 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.776 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.776 DEBUG:boto:Params: {} 2024-04-12T03:08:12.776 DEBUG:boto:Token: None 2024-04-12T03:08:12.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9a82f449af55fe-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.779 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.779 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.779 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.779 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.779 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.779 DEBUG:boto:Data: 2024-04-12T03:08:12.779 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.780 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.780 DEBUG:boto:Params: {} 2024-04-12T03:08:12.780 DEBUG:boto:Token: None 2024-04-12T03:08:12.780 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.780 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c0c3b45ddde52a7-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.783 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.783 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.783 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.783 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:12.783 DEBUG:boto:Data: 2024-04-12T03:08:12.784 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.784 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.784 DEBUG:boto:Params: {} 2024-04-12T03:08:12.784 DEBUG:boto:Token: None 2024-04-12T03:08:12.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj 2024-04-12T03:08:12.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw= 2024-04-12T03:08:12.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gfqILWXH30+wx5nKslkkRSjgSEw=', 'Content-Length': '0'} 2024-04-12T03:08:12.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000836d196e518ef625-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.788 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.788 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.788 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.788 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.788 DEBUG:boto:Method: GET 2024-04-12T03:08:12.788 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.788 DEBUG:boto:Data: 2024-04-12T03:08:12.788 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.788 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.788 DEBUG:boto:Params: {} 2024-04-12T03:08:12.788 DEBUG:boto:Token: None 2024-04-12T03:08:12.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071eda92e0c472e66-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.834 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.834 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.835 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.835 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.835 DEBUG:boto:Method: GET 2024-04-12T03:08:12.835 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.835 DEBUG:boto:Data: 2024-04-12T03:08:12.835 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.835 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.835 DEBUG:boto:Params: {} 2024-04-12T03:08:12.835 DEBUG:boto:Token: None 2024-04-12T03:08:12.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:12.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8= 2024-04-12T03:08:12.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWUG5BtKn3fIulSSVsP+X36d/H8=', 'Content-Length': '0'} 2024-04-12T03:08:12.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068d45bb0784f864a-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.838 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.838 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.838 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.838 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.838 DEBUG:boto:Method: GET 2024-04-12T03:08:12.838 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.838 DEBUG:boto:Data: 2024-04-12T03:08:12.838 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.838 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.839 DEBUG:boto:Params: {} 2024-04-12T03:08:12.839 DEBUG:boto:Token: None 2024-04-12T03:08:12.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037db1fb7c6930fc7-006618a59c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.886 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.887 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:12.887 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:12.887 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.887 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.887 DEBUG:boto:Method: GET 2024-04-12T03:08:12.887 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:12.887 DEBUG:boto:Data: 2024-04-12T03:08:12.887 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.888 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.888 DEBUG:boto:Params: {} 2024-04-12T03:08:12.888 DEBUG:boto:Token: None 2024-04-12T03:08:12.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/ 2024-04-12T03:08:12.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug= 2024-04-12T03:08:12.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NRKisoaA5noZYrjTeNVeMmP7lug=', 'Content-Length': '0'} 2024-04-12T03:08:12.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009272dd909c975f07-006618a59c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.891 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:12.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.892 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.892 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.892 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.892 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.892 DEBUG:boto:Method: GET 2024-04-12T03:08:12.892 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.892 DEBUG:boto:Data: 2024-04-12T03:08:12.892 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.893 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.893 DEBUG:boto:Params: {} 2024-04-12T03:08:12.893 DEBUG:boto:Token: None 2024-04-12T03:08:12.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4= 2024-04-12T03:08:12.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4=', 'Content-Length': '0'} 2024-04-12T03:08:12.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000042235e4b377c770-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.895 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.895 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.895 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.895 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.896 DEBUG:boto:Method: GET 2024-04-12T03:08:12.896 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.896 DEBUG:boto:Data: 2024-04-12T03:08:12.896 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.896 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.896 DEBUG:boto:Params: {} 2024-04-12T03:08:12.896 DEBUG:boto:Token: None 2024-04-12T03:08:12.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4= 2024-04-12T03:08:12.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4=', 'Content-Length': '0'} 2024-04-12T03:08:12.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f7631b4939a5f3e-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.946 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.946 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.947 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.947 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.947 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.947 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.947 DEBUG:boto:Data: 2024-04-12T03:08:12.947 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.947 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.947 DEBUG:boto:Params: {} 2024-04-12T03:08:12.947 DEBUG:boto:Token: None 2024-04-12T03:08:12.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fl949K0Kxc+0ITbqiuhMbOWnxNU= 2024-04-12T03:08:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fl949K0Kxc+0ITbqiuhMbOWnxNU=', 'Content-Length': '0'} 2024-04-12T03:08:12.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017d327507027e8db-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.950 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.950 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.950 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.950 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.950 DEBUG:boto:Method: HEAD 2024-04-12T03:08:12.950 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.951 DEBUG:boto:Data: 2024-04-12T03:08:12.951 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.951 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.951 DEBUG:boto:Params: {} 2024-04-12T03:08:12.951 DEBUG:boto:Token: None 2024-04-12T03:08:12.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fl949K0Kxc+0ITbqiuhMbOWnxNU= 2024-04-12T03:08:12.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fl949K0Kxc+0ITbqiuhMbOWnxNU=', 'Content-Length': '0'} 2024-04-12T03:08:12.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000c21d340d0e4f75-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:12.954 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.954 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.954 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.954 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.954 DEBUG:boto:Method: GET 2024-04-12T03:08:12.954 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.955 DEBUG:boto:Data: 2024-04-12T03:08:12.955 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:12.955 DEBUG:boto:Port: 8001 2024-04-12T03:08:12.955 DEBUG:boto:Params: {} 2024-04-12T03:08:12.955 DEBUG:boto:Token: None 2024-04-12T03:08:12.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4= 2024-04-12T03:08:12.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4=', 'Content-Length': '0'} 2024-04-12T03:08:12.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000150d0c75432611b5-006618a59c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:12.958 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:12.958 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:12.958 DEBUG:boto:path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.958 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.958 DEBUG:boto:Method: GET 2024-04-12T03:08:12.958 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.959 DEBUG:boto:Data: 2024-04-12T03:08:12.959 DEBUG:boto:Headers: {} 2024-04-12T03:08:12.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:12.959 DEBUG:boto:Port: 8000 2024-04-12T03:08:12.959 DEBUG:boto:Params: {} 2024-04-12T03:08:12.959 DEBUG:boto:Token: None 2024-04-12T03:08:12.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:12 GMT /dgbpxg-15/obj?versionId=caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP 2024-04-12T03:08:12.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4= 2024-04-12T03:08:12.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaTAtKvCD6wrIqbjhL3SHKMJIh4=', 'Content-Length': '0'} 2024-04-12T03:08:12.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'caJEOLhbPMl5nr7xTZPU-.Uh0k1woZP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efbb907047251a0f-006618a59c-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.002 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.002 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.003 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.003 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.003 DEBUG:boto:Method: GET 2024-04-12T03:08:13.003 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.003 DEBUG:boto:Data: 2024-04-12T03:08:13.003 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.003 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.003 DEBUG:boto:Params: {} 2024-04-12T03:08:13.003 DEBUG:boto:Token: None 2024-04-12T03:08:13.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad34bc500dc725b-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.006 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.007 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.007 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.007 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.007 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.007 DEBUG:boto:Method: GET 2024-04-12T03:08:13.007 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.008 DEBUG:boto:Data: 2024-04-12T03:08:13.008 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.008 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.008 DEBUG:boto:Params: {} 2024-04-12T03:08:13.008 DEBUG:boto:Token: None 2024-04-12T03:08:13.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aec26c6201d3355-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.011 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.012 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.012 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.012 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.012 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.012 DEBUG:boto:Data: 2024-04-12T03:08:13.012 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.012 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.012 DEBUG:boto:Params: {} 2024-04-12T03:08:13.012 DEBUG:boto:Token: None 2024-04-12T03:08:13.013 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000193e770a42fbb49-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.016 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.016 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.016 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.016 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.016 DEBUG:boto:Data: 2024-04-12T03:08:13.016 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.017 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.017 DEBUG:boto:Params: {} 2024-04-12T03:08:13.017 DEBUG:boto:Token: None 2024-04-12T03:08:13.017 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6bbab523aed33ea-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.021 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.021 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.021 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.021 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.021 DEBUG:boto:Method: GET 2024-04-12T03:08:13.022 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.022 DEBUG:boto:Data: 2024-04-12T03:08:13.022 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.022 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.022 DEBUG:boto:Params: {} 2024-04-12T03:08:13.022 DEBUG:boto:Token: None 2024-04-12T03:08:13.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5e08eb59bf37c6c-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.066 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.067 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.067 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.067 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.067 DEBUG:boto:Method: GET 2024-04-12T03:08:13.067 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.068 DEBUG:boto:Data: 2024-04-12T03:08:13.068 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.068 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.068 DEBUG:boto:Params: {} 2024-04-12T03:08:13.068 DEBUG:boto:Token: None 2024-04-12T03:08:13.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084f600cbd186b13f-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.071 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.072 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.072 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.072 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.072 DEBUG:boto:Method: GET 2024-04-12T03:08:13.072 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.072 DEBUG:boto:Data: 2024-04-12T03:08:13.072 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.073 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.073 DEBUG:boto:Params: {} 2024-04-12T03:08:13.073 DEBUG:boto:Token: None 2024-04-12T03:08:13.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629cfed1d7c88a73-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.076 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.077 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.077 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.077 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.077 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.077 DEBUG:boto:Method: GET 2024-04-12T03:08:13.077 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.077 DEBUG:boto:Data: 2024-04-12T03:08:13.077 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.077 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.077 DEBUG:boto:Params: {} 2024-04-12T03:08:13.078 DEBUG:boto:Token: None 2024-04-12T03:08:13.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a964cb0f0b0e1a47-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.081 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.081 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.081 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.081 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.081 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.082 DEBUG:boto:Method: GET 2024-04-12T03:08:13.082 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.082 DEBUG:boto:Data: 2024-04-12T03:08:13.082 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.082 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.082 DEBUG:boto:Params: {} 2024-04-12T03:08:13.082 DEBUG:boto:Token: None 2024-04-12T03:08:13.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0= 2024-04-12T03:08:13.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0=', 'Content-Length': '0'} 2024-04-12T03:08:13.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cda1e48748e46e3-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.085 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.085 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.085 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.085 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.085 DEBUG:boto:Method: GET 2024-04-12T03:08:13.085 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.085 DEBUG:boto:Data: 2024-04-12T03:08:13.085 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.086 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.086 DEBUG:boto:Params: {} 2024-04-12T03:08:13.086 DEBUG:boto:Token: None 2024-04-12T03:08:13.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0= 2024-04-12T03:08:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0=', 'Content-Length': '0'} 2024-04-12T03:08:13.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076be4000adc0c555-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.088 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.088 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.088 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.089 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.089 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.089 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.089 DEBUG:boto:Data: 2024-04-12T03:08:13.089 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.089 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.089 DEBUG:boto:Params: {} 2024-04-12T03:08:13.089 DEBUG:boto:Token: None 2024-04-12T03:08:13.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9H1OEhuGCQ7NZ3No0Z4Z0Ng/2Gw= 2024-04-12T03:08:13.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9H1OEhuGCQ7NZ3No0Z4Z0Ng/2Gw=', 'Content-Length': '0'} 2024-04-12T03:08:13.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000158ae2d44c3a309-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.092 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.092 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.092 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.092 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.092 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.093 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.093 DEBUG:boto:Data: 2024-04-12T03:08:13.093 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.093 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.093 DEBUG:boto:Params: {} 2024-04-12T03:08:13.093 DEBUG:boto:Token: None 2024-04-12T03:08:13.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9H1OEhuGCQ7NZ3No0Z4Z0Ng/2Gw= 2024-04-12T03:08:13.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9H1OEhuGCQ7NZ3No0Z4Z0Ng/2Gw=', 'Content-Length': '0'} 2024-04-12T03:08:13.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016e68f861a1767df-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.095 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.096 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.096 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.096 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.096 DEBUG:boto:Method: GET 2024-04-12T03:08:13.096 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.096 DEBUG:boto:Data: 2024-04-12T03:08:13.096 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.096 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.096 DEBUG:boto:Params: {} 2024-04-12T03:08:13.096 DEBUG:boto:Token: None 2024-04-12T03:08:13.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0= 2024-04-12T03:08:13.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0=', 'Content-Length': '0'} 2024-04-12T03:08:13.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d69ae7574d2dddf-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.142 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.142 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.142 DEBUG:boto:path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.143 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.143 DEBUG:boto:Method: GET 2024-04-12T03:08:13.143 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.143 DEBUG:boto:Data: 2024-04-12T03:08:13.143 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.143 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.143 DEBUG:boto:Params: {} 2024-04-12T03:08:13.143 DEBUG:boto:Token: None 2024-04-12T03:08:13.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=PQyVfrdwJE7QIbPm0v64BtD0mpy58hr 2024-04-12T03:08:13.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0= 2024-04-12T03:08:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DglTqHK4tK/1ln1mHzx7A89QYP0=', 'Content-Length': '0'} 2024-04-12T03:08:13.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'PQyVfrdwJE7QIbPm0v64BtD0mpy58hr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003a03c768b42631f-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.146 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.146 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.146 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.146 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.146 DEBUG:boto:Method: GET 2024-04-12T03:08:13.146 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.146 DEBUG:boto:Data: 2024-04-12T03:08:13.146 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.147 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.147 DEBUG:boto:Params: {} 2024-04-12T03:08:13.147 DEBUG:boto:Token: None 2024-04-12T03:08:13.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028120187cbfd7410-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.150 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.150 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.151 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.151 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.151 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.151 DEBUG:boto:Method: GET 2024-04-12T03:08:13.151 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.151 DEBUG:boto:Data: 2024-04-12T03:08:13.151 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.151 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.151 DEBUG:boto:Params: {} 2024-04-12T03:08:13.151 DEBUG:boto:Token: None 2024-04-12T03:08:13.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b4e5ffe3fc8c1ca-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.154 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.155 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.155 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.155 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.155 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.155 DEBUG:boto:Data: 2024-04-12T03:08:13.155 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.155 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.155 DEBUG:boto:Params: {} 2024-04-12T03:08:13.156 DEBUG:boto:Token: None 2024-04-12T03:08:13.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000766634c886922da-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.159 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.159 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.159 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.159 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.159 DEBUG:boto:Data: 2024-04-12T03:08:13.160 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.160 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.160 DEBUG:boto:Params: {} 2024-04-12T03:08:13.160 DEBUG:boto:Token: None 2024-04-12T03:08:13.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3bcb1d5838f745c-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.163 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.164 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.164 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.164 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.164 DEBUG:boto:Method: GET 2024-04-12T03:08:13.164 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.164 DEBUG:boto:Data: 2024-04-12T03:08:13.164 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.164 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.164 DEBUG:boto:Params: {} 2024-04-12T03:08:13.164 DEBUG:boto:Token: None 2024-04-12T03:08:13.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042b328aa759a4bcc-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.210 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.210 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.211 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.211 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.211 DEBUG:boto:Method: GET 2024-04-12T03:08:13.211 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.211 DEBUG:boto:Data: 2024-04-12T03:08:13.211 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.211 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.211 DEBUG:boto:Params: {} 2024-04-12T03:08:13.211 DEBUG:boto:Token: None 2024-04-12T03:08:13.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ee7774b40f53435-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.214 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.214 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.214 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.214 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.214 DEBUG:boto:Method: GET 2024-04-12T03:08:13.214 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.215 DEBUG:boto:Data: 2024-04-12T03:08:13.215 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.215 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.215 DEBUG:boto:Params: {} 2024-04-12T03:08:13.215 DEBUG:boto:Token: None 2024-04-12T03:08:13.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f234f2bf1c60d9-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.218 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.218 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.218 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.218 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.219 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.219 DEBUG:boto:Method: GET 2024-04-12T03:08:13.219 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.219 DEBUG:boto:Data: 2024-04-12T03:08:13.219 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.219 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.219 DEBUG:boto:Params: {} 2024-04-12T03:08:13.219 DEBUG:boto:Token: None 2024-04-12T03:08:13.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c872d2fb332acba6-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.222 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.222 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.223 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.223 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.223 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.223 DEBUG:boto:Method: GET 2024-04-12T03:08:13.223 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.223 DEBUG:boto:Data: 2024-04-12T03:08:13.223 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.223 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.223 DEBUG:boto:Params: {} 2024-04-12T03:08:13.223 DEBUG:boto:Token: None 2024-04-12T03:08:13.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8= 2024-04-12T03:08:13.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8=', 'Content-Length': '0'} 2024-04-12T03:08:13.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034bbab07c6c541b3-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.226 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.226 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.226 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.226 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.226 DEBUG:boto:Method: GET 2024-04-12T03:08:13.226 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.226 DEBUG:boto:Data: 2024-04-12T03:08:13.227 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.227 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.227 DEBUG:boto:Params: {} 2024-04-12T03:08:13.227 DEBUG:boto:Token: None 2024-04-12T03:08:13.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8= 2024-04-12T03:08:13.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8=', 'Content-Length': '0'} 2024-04-12T03:08:13.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009716c5cfd3d104a5-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.229 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.229 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.230 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.230 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.230 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.230 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.230 DEBUG:boto:Data: 2024-04-12T03:08:13.230 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.230 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.230 DEBUG:boto:Params: {} 2024-04-12T03:08:13.230 DEBUG:boto:Token: None 2024-04-12T03:08:13.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsMcCPzgGZvk279w8fOHxzLujUM= 2024-04-12T03:08:13.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsMcCPzgGZvk279w8fOHxzLujUM=', 'Content-Length': '0'} 2024-04-12T03:08:13.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000264b13b6fccb0279-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.233 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.233 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.233 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.233 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.233 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.233 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.233 DEBUG:boto:Data: 2024-04-12T03:08:13.233 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.234 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.234 DEBUG:boto:Params: {} 2024-04-12T03:08:13.234 DEBUG:boto:Token: None 2024-04-12T03:08:13.234 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.234 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsMcCPzgGZvk279w8fOHxzLujUM= 2024-04-12T03:08:13.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsMcCPzgGZvk279w8fOHxzLujUM=', 'Content-Length': '0'} 2024-04-12T03:08:13.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053f1431439123e20-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.236 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.236 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.237 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.237 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.237 DEBUG:boto:Method: GET 2024-04-12T03:08:13.237 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.237 DEBUG:boto:Data: 2024-04-12T03:08:13.237 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.237 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.237 DEBUG:boto:Params: {} 2024-04-12T03:08:13.237 DEBUG:boto:Token: None 2024-04-12T03:08:13.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8= 2024-04-12T03:08:13.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8=', 'Content-Length': '0'} 2024-04-12T03:08:13.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3913973b863bce9-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.240 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.240 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.240 DEBUG:boto:path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.240 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.240 DEBUG:boto:Method: GET 2024-04-12T03:08:13.240 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.240 DEBUG:boto:Data: 2024-04-12T03:08:13.240 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.241 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.241 DEBUG:boto:Params: {} 2024-04-12T03:08:13.241 DEBUG:boto:Token: None 2024-04-12T03:08:13.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=wlQG40RppM7AX0WKjPtx4kQYRknCEJG 2024-04-12T03:08:13.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8= 2024-04-12T03:08:13.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5irjOOHClRHyoCa0ic1awi4FtR8=', 'Content-Length': '0'} 2024-04-12T03:08:13.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'wlQG40RppM7AX0WKjPtx4kQYRknCEJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011f30f9a113ea493-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.286 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.286 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.287 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.287 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.287 DEBUG:boto:Method: GET 2024-04-12T03:08:13.287 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.287 DEBUG:boto:Data: 2024-04-12T03:08:13.287 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.287 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.287 DEBUG:boto:Params: {} 2024-04-12T03:08:13.287 DEBUG:boto:Token: None 2024-04-12T03:08:13.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089e53fe02bf8be9-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.290 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.291 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.291 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.291 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.291 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.291 DEBUG:boto:Method: GET 2024-04-12T03:08:13.291 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.291 DEBUG:boto:Data: 2024-04-12T03:08:13.291 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.292 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.292 DEBUG:boto:Params: {} 2024-04-12T03:08:13.292 DEBUG:boto:Token: None 2024-04-12T03:08:13.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019cf35447e27cf6c-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.338 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.339 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.339 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.339 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.339 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.339 DEBUG:boto:Data: 2024-04-12T03:08:13.339 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.339 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.340 DEBUG:boto:Params: {} 2024-04-12T03:08:13.340 DEBUG:boto:Token: None 2024-04-12T03:08:13.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e9a772b63a2161d-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.343 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.343 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.343 DEBUG:boto:Method: HEAD 2024-04-12T03:08:13.344 DEBUG:boto:Path: /dgbpxg-15/obj 2024-04-12T03:08:13.344 DEBUG:boto:Data: 2024-04-12T03:08:13.344 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.344 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.344 DEBUG:boto:Params: {} 2024-04-12T03:08:13.344 DEBUG:boto:Token: None 2024-04-12T03:08:13.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj 2024-04-12T03:08:13.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw= 2024-04-12T03:08:13.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:334y8okXvxM3owTxDjevtJI5whw=', 'Content-Length': '0'} 2024-04-12T03:08:13.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc8ef02fe63d7099-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:13.348 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.348 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.348 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.348 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.348 DEBUG:boto:Method: GET 2024-04-12T03:08:13.348 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.348 DEBUG:boto:Data: 2024-04-12T03:08:13.348 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.348 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.348 DEBUG:boto:Params: {} 2024-04-12T03:08:13.349 DEBUG:boto:Token: None 2024-04-12T03:08:13.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029eb1fad6f2d9ef2-006618a59d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.394 DEBUG:boto:path=/dgbpxg-15/obj 2024-04-12T03:08:13.394 DEBUG:boto:auth_path=/dgbpxg-15/obj 2024-04-12T03:08:13.395 DEBUG:boto:path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.395 DEBUG:boto:auth_path=/dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.395 DEBUG:boto:Method: GET 2024-04-12T03:08:13.395 DEBUG:boto:Path: /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.395 DEBUG:boto:Data: 2024-04-12T03:08:13.395 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.395 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.395 DEBUG:boto:Params: {} 2024-04-12T03:08:13.395 DEBUG:boto:Token: None 2024-04-12T03:08:13.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/obj?versionId=GQK1DlZF21SeIkechhmXlaU1KROzKxz 2024-04-12T03:08:13.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg= 2024-04-12T03:08:13.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0hCwHkmOqtPX1YJquq4xCLcnAg=', 'Content-Length': '0'} 2024-04-12T03:08:13.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:07:43 GMT'), ('x-amz-version-id', 'GQK1DlZF21SeIkechhmXlaU1KROzKxz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db5f8b4f4deb64fd-006618a59d-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.398 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.398 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.398 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.398 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.398 DEBUG:boto:Method: GET 2024-04-12T03:08:13.399 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.399 DEBUG:boto:Data: 2024-04-12T03:08:13.399 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:13.399 DEBUG:boto:Port: 8001 2024-04-12T03:08:13.399 DEBUG:boto:Params: {} 2024-04-12T03:08:13.399 DEBUG:boto:Token: None 2024-04-12T03:08:13.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f3958ff5f356b1-006618a59d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.402 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.402 DEBUG:boto:path=/dgbpxg-15/ 2024-04-12T03:08:13.402 DEBUG:boto:auth_path=/dgbpxg-15/ 2024-04-12T03:08:13.402 DEBUG:boto:path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.403 DEBUG:boto:auth_path=/dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.403 DEBUG:boto:Method: GET 2024-04-12T03:08:13.403 DEBUG:boto:Path: /dgbpxg-15/?prefix=obj 2024-04-12T03:08:13.403 DEBUG:boto:Data: 2024-04-12T03:08:13.403 DEBUG:boto:Headers: {} 2024-04-12T03:08:13.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:13.403 DEBUG:boto:Port: 8000 2024-04-12T03:08:13.403 DEBUG:boto:Params: {} 2024-04-12T03:08:13.403 DEBUG:boto:Token: None 2024-04-12T03:08:13.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:13 GMT /dgbpxg-15/ 2024-04-12T03:08:13.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw= 2024-04-12T03:08:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Riwq7KBHU0O6lxMb1GoV0GrEDRw=', 'Content-Length': '0'} 2024-04-12T03:08:13.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27fe70aec36e2ff-006618a59d-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:13.406 DEBUG:boto:b'dgbpxg-15obj1000falseobj2024-04-12T03:07:43.648Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:13.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-15 zones={a2, a1} 2024-04-12T03:08:13.407 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:08:13.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:13.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:13.407 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:13.461 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:13.461 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:13.509 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:08:09.327152Z" 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:07:47.325574Z" 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.511 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:08:13.512 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:13.512 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:08:13.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:13.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:13.513 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:13.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:13.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:13.606 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:13.607 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:13.608 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:13.609 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.324062Z" 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:13.610 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.325574Z" 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.611 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:13.612 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:08:13.612 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:08:13.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:13.612 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:13Z" 2024-04-12T03:08:13.612 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:13.613 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.324062Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.325574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:08:13Z" } 2024-04-12T03:08:13.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.324062Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.325574Z'}}] 2024-04-12T03:08:13.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:13.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003606', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:13.614 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-12T03:08:18.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:18.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:18.615 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:08:18.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:08:18.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:18.718 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:18.719 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.324062Z" 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:18.720 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.325574Z" 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:18Z" 2024-04-12T03:08:18.721 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:18.722 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.324062Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.325574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:08:18Z" } 2024-04-12T03:08:18.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.324062Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.325574Z'}}] 2024-04-12T03:08:18.723 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:18.723 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003606', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:18.723 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-12T03:08:23.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:23.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:23.725 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:23.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:23.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:23.827 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:08:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.324062Z" 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.325574Z" 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:23Z" 2024-04-12T03:08:23.833 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:23.834 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.324062Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.325574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:08:23Z" } 2024-04-12T03:08:23.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.324062Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.325574Z'}}] 2024-04-12T03:08:23.834 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:23.834 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003606', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:23.834 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-12T03:08:28.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:28.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:28.836 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:28.893 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:28.894 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.939 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:08:09.327152Z" 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.325574Z" 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:28Z" 2024-04-12T03:08:28.944 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:28.945 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:08:09.327152Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.325574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:08:28Z" } 2024-04-12T03:08:28.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:08:09.327152Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.325574Z'}}] 2024-04-12T03:08:28.946 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:28.946 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:08:28.946 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:08:28.946 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:08:28.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:28.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:28.946 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:28.994 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:28.994 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:01:11.506001Z" 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:01:11.507550Z" 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:07:47.524248Z" 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:08:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:07:25.521966Z" 2024-04-12T03:08:29.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.042 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:08:29.042 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:08:29.043 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:08:29.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.043 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:29.093 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:29.093 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:29.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.506001Z" 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.135 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.507550Z" 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:47.524248Z" 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:07:25.521966Z" 2024-04-12T03:08:29.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:29Z" 2024-04-12T03:08:29.139 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:29.140 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.506001Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.507550Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:47.524248Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:07:25.521966Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:08:29Z" } 2024-04-12T03:08:29.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.506001Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.507550Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:47.524248Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:07:25.521966Z'}}] 2024-04-12T03:08:29.140 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:08:29.141 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000003747', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:08:29.141 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:08:29.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-12T03:08:29.143 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:29.143 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:29.143 DEBUG:boto:Method: PUT 2024-04-12T03:08:29.143 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:08:29.143 DEBUG:boto:Data: 2024-04-12T03:08:29.143 DEBUG:boto:Headers: {} 2024-04-12T03:08:29.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:29.143 DEBUG:boto:Port: 8000 2024-04-12T03:08:29.143 DEBUG:boto:Params: {} 2024-04-12T03:08:29.144 DEBUG:boto:Token: None 2024-04-12T03:08:29.144 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:08:29 GMT /dgbpxg-16/ 2024-04-12T03:08:29.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KaCu0Muz73WcZmw7pZveZRgd2Zc= 2024-04-12T03:08:29.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KaCu0Muz73WcZmw7pZveZRgd2Zc=', 'Content-Length': '0'} 2024-04-12T03:08:29.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e3fc1007e32c4f9-006618a5ad-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:29.160 DEBUG:rgw_multi.tests:created bucket=dgbpxg-16 2024-04-12T03:08:29.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.161 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:29.237 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:08:29.248 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:08:29.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.248 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:08:29.248 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891309.158391_279.1", 2024-04-12T03:08:29.250 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:08:29.158391Z" 2024-04-12T03:08:29.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.250 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:08:29.251 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891309.158391_279.1'} 2024-04-12T03:08:29.251 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:08:29.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:29.251 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:29.299 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:29.299 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:29.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:29.324+0000 7f9e4a098780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:08:29.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:29.325+0000 7f9e4a098780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:29.339 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:08:29.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:08:29.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891309.151876_276.1", 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:29.344 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:08:29.151876Z", 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:08:29.345 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:08:29.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:29.346 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:29Z" 2024-04-12T03:08:29.346 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:29.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891309.151876_276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:08:29.151876Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:08:29Z" } 2024-04-12T03:08:29.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891309.151876_276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:08:29.151876Z', 'realm_epoch': 2}}] 2024-04-12T03:08:29.347 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891309.158391_279.1'} 2024-04-12T03:08:29.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891309.151876_276.1'} 2024-04-12T03:08:29.347 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891309.158391_279.1 target=1_1712891309.151876_276.1 2024-04-12T03:08:34.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:34.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:34.348 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:34.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:34.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:34.432+0000 7ff02a8fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:08:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:34.433+0000 7ff02a8fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:08:34.447 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:08:34.447 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:08:34.447 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:08:34.447 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:08:34.447 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:08:34.448 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:08:34.449 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:34.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:34.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891309.158391_279.1", 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:08:29.158391Z", 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:08:34.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:34Z" 2024-04-12T03:08:34.453 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:08:34.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891309.158391_279.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:08:29.158391Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:08:34Z" } 2024-04-12T03:08:34.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891309.158391_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:08:29.158391Z', 'realm_epoch': 2}}] 2024-04-12T03:08:34.454 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891309.158391_279.1'} 2024-04-12T03:08:34.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891309.158391_279.1'} 2024-04-12T03:08:34.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:08:34.455 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:34.455 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:34.455 DEBUG:boto:Method: HEAD 2024-04-12T03:08:34.455 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:08:34.455 DEBUG:boto:Data: 2024-04-12T03:08:34.455 DEBUG:boto:Headers: {} 2024-04-12T03:08:34.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:34.455 DEBUG:boto:Port: 8000 2024-04-12T03:08:34.455 DEBUG:boto:Params: {} 2024-04-12T03:08:34.456 DEBUG:boto:Token: None 2024-04-12T03:08:34.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:34 GMT /dgbpxg-16/ 2024-04-12T03:08:34.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jnkHAc6GtF0dozigCx484clT1/c= 2024-04-12T03:08:34.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:34 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jnkHAc6GtF0dozigCx484clT1/c=', 'Content-Length': '0'} 2024-04-12T03:08:34.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3b18e982d4ebba0-006618a5b2-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:34.458 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:34.459 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:34.459 DEBUG:boto:Method: PUT 2024-04-12T03:08:34.459 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:08:34.459 DEBUG:boto:Data: 2024-04-12T03:08:34.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:08:34.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:34.459 DEBUG:boto:Port: 8000 2024-04-12T03:08:34.459 DEBUG:boto:Params: {} 2024-04-12T03:08:34.459 DEBUG:boto:Token: None 2024-04-12T03:08:34.460 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:08:34 GMT /dgbpxg-16/obj 2024-04-12T03:08:34.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XKHJDr4AnQRzciILrUFXeZcKJtY= 2024-04-12T03:08:34.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:08:34 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XKHJDr4AnQRzciILrUFXeZcKJtY='} 2024-04-12T03:08:34.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085749ae5befa8b45-006618a5b2-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:34.465 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-12T03:08:34.466 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-16 2024-04-12T03:08:34.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:34.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:34.466 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:08:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:34.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:34.553+0000 7fb7ccb20780 1 waiting to reach incremental sync.. 2024-04-12T03:08:39.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:39.554+0000 7fb7ccb20780 1 waiting to reach incremental sync.. 2024-04-12T03:08:44.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:44.555+0000 7fb7ccb20780 1 waiting to reach incremental sync.. 2024-04-12T03:08:49.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:49.557+0000 7fb7ccb20780 1 bucket sync caught up with source: 2024-04-12T03:08:49.558 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:08:49.558 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:08:49.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:49.557+0000 7fb7ccb20780 0 bucket checkpoint complete 2024-04-12T03:08:49.575 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-16 2024-04-12T03:08:49.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:49.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:08:49.576 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:08:49.637 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:49.638 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:49.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:49.673+0000 7f55c5bc3780 1 waiting to reach incremental sync.. 2024-04-12T03:08:54.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:54.673+0000 7f55c5bc3780 1 waiting to reach incremental sync.. 2024-04-12T03:08:59.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:59.675+0000 7f55c5bc3780 1 bucket sync caught up with source: 2024-04-12T03:08:59.677 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.42.6, , ] 2024-04-12T03:08:59.677 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.42.6, , ] 2024-04-12T03:08:59.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:59.675+0000 7f55c5bc3780 0 bucket checkpoint complete 2024-04-12T03:08:59.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:08:59.692 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.692 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.692 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.693 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:08:59.693 DEBUG:boto:Data: 2024-04-12T03:08:59.693 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.693 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.693 DEBUG:boto:Params: {} 2024-04-12T03:08:59.693 DEBUG:boto:Token: None 2024-04-12T03:08:59.694 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.694 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ma+0a9CK1shIRksTmHJs5PgHKAM= 2024-04-12T03:08:59.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma+0a9CK1shIRksTmHJs5PgHKAM=', 'Content-Length': '0'} 2024-04-12T03:08:59.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b100f01ea53b02b-006618a5cb-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.697 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.697 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.697 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.697 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:08:59.697 DEBUG:boto:Data: 2024-04-12T03:08:59.697 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.697 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.697 DEBUG:boto:Params: {} 2024-04-12T03:08:59.698 DEBUG:boto:Token: None 2024-04-12T03:08:59.698 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ma+0a9CK1shIRksTmHJs5PgHKAM= 2024-04-12T03:08:59.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma+0a9CK1shIRksTmHJs5PgHKAM=', 'Content-Length': '0'} 2024-04-12T03:08:59.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1f2786e03b9dd83-006618a5cb-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.701 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:08:59.701 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.701 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.701 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.701 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.701 DEBUG:boto:Method: GET 2024-04-12T03:08:59.701 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.702 DEBUG:boto:Data: 2024-04-12T03:08:59.702 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.702 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.702 DEBUG:boto:Params: {} 2024-04-12T03:08:59.702 DEBUG:boto:Token: None 2024-04-12T03:08:59.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/?versions 2024-04-12T03:08:59.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho= 2024-04-12T03:08:59.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho=', 'Content-Length': '0'} 2024-04-12T03:08:59.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d6ca09457943152-006618a5cb-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.705 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:59.706 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:08:59.706 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.706 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.706 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.706 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.706 DEBUG:boto:Method: GET 2024-04-12T03:08:59.707 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.707 DEBUG:boto:Data: 2024-04-12T03:08:59.707 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.707 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.707 DEBUG:boto:Params: {} 2024-04-12T03:08:59.707 DEBUG:boto:Token: None 2024-04-12T03:08:59.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/?versions 2024-04-12T03:08:59.707 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho= 2024-04-12T03:08:59.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho=', 'Content-Length': '0'} 2024-04-12T03:08:59.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0eeb8b48db47f0-006618a5cb-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.710 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:08:59.710 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.710 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.710 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.711 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.711 DEBUG:boto:Method: GET 2024-04-12T03:08:59.711 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.711 DEBUG:boto:Data: 2024-04-12T03:08:59.711 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.711 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.711 DEBUG:boto:Params: {} 2024-04-12T03:08:59.711 DEBUG:boto:Token: None 2024-04-12T03:08:59.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/?versions 2024-04-12T03:08:59.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho= 2024-04-12T03:08:59.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho=', 'Content-Length': '0'} 2024-04-12T03:08:59.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003253390f2241e240-006618a5cb-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.714 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.714 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.714 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.714 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.715 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.715 DEBUG:boto:Method: GET 2024-04-12T03:08:59.715 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:08:59.715 DEBUG:boto:Data: 2024-04-12T03:08:59.715 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.715 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.715 DEBUG:boto:Params: {} 2024-04-12T03:08:59.715 DEBUG:boto:Token: None 2024-04-12T03:08:59.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/?versions 2024-04-12T03:08:59.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho= 2024-04-12T03:08:59.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z5hFZ0B6WGVObnX+NFfoqWjY9ho=', 'Content-Length': '0'} 2024-04-12T03:08:59.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0685a40e4692ff-006618a5cb-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.718 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:59.718 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.718 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.719 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.719 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.719 DEBUG:boto:Method: GET 2024-04-12T03:08:59.719 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.719 DEBUG:boto:Data: 2024-04-12T03:08:59.719 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.719 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.719 DEBUG:boto:Params: {} 2024-04-12T03:08:59.719 DEBUG:boto:Token: None 2024-04-12T03:08:59.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ= 2024-04-12T03:08:59.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ=', 'Content-Length': '0'} 2024-04-12T03:08:59.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a47629bc018c21c-006618a5cb-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.722 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.722 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.722 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.722 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.722 DEBUG:boto:Method: GET 2024-04-12T03:08:59.722 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.723 DEBUG:boto:Data: 2024-04-12T03:08:59.723 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.723 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.723 DEBUG:boto:Params: {} 2024-04-12T03:08:59.723 DEBUG:boto:Token: None 2024-04-12T03:08:59.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.723 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ= 2024-04-12T03:08:59.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ=', 'Content-Length': '0'} 2024-04-12T03:08:59.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cee47457b595cc7f-006618a5cb-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.725 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.726 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.726 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.726 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.726 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.726 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.726 DEBUG:boto:Data: 2024-04-12T03:08:59.726 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.726 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.727 DEBUG:boto:Params: {} 2024-04-12T03:08:59.727 DEBUG:boto:Token: None 2024-04-12T03:08:59.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.727 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gf5XUV/wqjcmLhAGMzYVz9jyoz0= 2024-04-12T03:08:59.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gf5XUV/wqjcmLhAGMzYVz9jyoz0=', 'Content-Length': '0'} 2024-04-12T03:08:59.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024d542e146df211d-006618a5cb-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.729 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.729 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.729 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.729 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.729 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.730 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.730 DEBUG:boto:Data: 2024-04-12T03:08:59.730 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.730 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.730 DEBUG:boto:Params: {} 2024-04-12T03:08:59.730 DEBUG:boto:Token: None 2024-04-12T03:08:59.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gf5XUV/wqjcmLhAGMzYVz9jyoz0= 2024-04-12T03:08:59.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gf5XUV/wqjcmLhAGMzYVz9jyoz0=', 'Content-Length': '0'} 2024-04-12T03:08:59.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ca5011ab74381d6-006618a5cb-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:59.732 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.733 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.733 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.733 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.733 DEBUG:boto:Method: GET 2024-04-12T03:08:59.733 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.733 DEBUG:boto:Data: 2024-04-12T03:08:59.733 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.733 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.733 DEBUG:boto:Params: {} 2024-04-12T03:08:59.734 DEBUG:boto:Token: None 2024-04-12T03:08:59.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ= 2024-04-12T03:08:59.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ=', 'Content-Length': '0'} 2024-04-12T03:08:59.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068cb5ab5da34ccb2-006618a5cb-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.736 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.736 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.736 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.736 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.736 DEBUG:boto:Method: GET 2024-04-12T03:08:59.736 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.736 DEBUG:boto:Data: 2024-04-12T03:08:59.736 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.736 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.736 DEBUG:boto:Params: {} 2024-04-12T03:08:59.737 DEBUG:boto:Token: None 2024-04-12T03:08:59.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:08:59.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ= 2024-04-12T03:08:59.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rysnwEHY/wxe3kCXOFlKb+lhaUQ=', 'Content-Length': '0'} 2024-04-12T03:08:59.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003995ab68bbfb05b7-006618a5cb-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.739 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.739 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.739 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.739 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.739 DEBUG:boto:Method: GET 2024-04-12T03:08:59.739 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.739 DEBUG:boto:Data: 2024-04-12T03:08:59.739 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.740 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.740 DEBUG:boto:Params: {} 2024-04-12T03:08:59.740 DEBUG:boto:Token: None 2024-04-12T03:08:59.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU= 2024-04-12T03:08:59.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU=', 'Content-Length': '0'} 2024-04-12T03:08:59.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d068a2334942a3a6-006618a5cb-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.742 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.743 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.743 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.743 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.743 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.743 DEBUG:boto:Method: GET 2024-04-12T03:08:59.743 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.743 DEBUG:boto:Data: 2024-04-12T03:08:59.743 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.743 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.743 DEBUG:boto:Params: {} 2024-04-12T03:08:59.744 DEBUG:boto:Token: None 2024-04-12T03:08:59.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU= 2024-04-12T03:08:59.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU=', 'Content-Length': '0'} 2024-04-12T03:08:59.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081831e238855b47-006618a5cb-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.746 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.746 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.746 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.747 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.747 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:08:59.747 DEBUG:boto:Data: 2024-04-12T03:08:59.747 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.747 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.747 DEBUG:boto:Params: {} 2024-04-12T03:08:59.748 DEBUG:boto:Token: None 2024-04-12T03:08:59.748 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj 2024-04-12T03:08:59.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Xfk9rhRquTggb5YfGwouLjNYwE= 2024-04-12T03:08:59.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Xfk9rhRquTggb5YfGwouLjNYwE=', 'Content-Length': '0'} 2024-04-12T03:08:59.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6e71b8901e7c8fd-006618a5cb-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.750 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.750 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.750 DEBUG:boto:Method: HEAD 2024-04-12T03:08:59.750 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:08:59.750 DEBUG:boto:Data: 2024-04-12T03:08:59.750 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.750 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.750 DEBUG:boto:Params: {} 2024-04-12T03:08:59.751 DEBUG:boto:Token: None 2024-04-12T03:08:59.751 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj 2024-04-12T03:08:59.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Xfk9rhRquTggb5YfGwouLjNYwE= 2024-04-12T03:08:59.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Xfk9rhRquTggb5YfGwouLjNYwE=', 'Content-Length': '0'} 2024-04-12T03:08:59.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058635815e2cc199e-006618a5cb-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:08:59.753 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.753 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.753 DEBUG:boto:Method: GET 2024-04-12T03:08:59.753 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:08:59.753 DEBUG:boto:Data: 2024-04-12T03:08:59.753 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.753 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.753 DEBUG:boto:Params: {} 2024-04-12T03:08:59.753 DEBUG:boto:Token: None 2024-04-12T03:08:59.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj 2024-04-12T03:08:59.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQKM89uZgPlkQMEKTaWUfmRT6ME= 2024-04-12T03:08:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQKM89uZgPlkQMEKTaWUfmRT6ME=', 'Content-Length': '0'} 2024-04-12T03:08:59.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c0dcb9c0b2dfd1b-006618a5cb-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.756 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:08:59.756 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:08:59.756 DEBUG:boto:Method: GET 2024-04-12T03:08:59.756 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:08:59.756 DEBUG:boto:Data: 2024-04-12T03:08:59.756 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.756 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.756 DEBUG:boto:Params: {} 2024-04-12T03:08:59.757 DEBUG:boto:Token: None 2024-04-12T03:08:59.757 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/obj 2024-04-12T03:08:59.757 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQKM89uZgPlkQMEKTaWUfmRT6ME= 2024-04-12T03:08:59.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQKM89uZgPlkQMEKTaWUfmRT6ME=', 'Content-Length': '0'} 2024-04-12T03:08:59.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3589df8307656d6-006618a5cb-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.759 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.759 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.759 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.759 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.759 DEBUG:boto:Method: GET 2024-04-12T03:08:59.759 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.759 DEBUG:boto:Data: 2024-04-12T03:08:59.759 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:08:59.760 DEBUG:boto:Port: 8001 2024-04-12T03:08:59.760 DEBUG:boto:Params: {} 2024-04-12T03:08:59.760 DEBUG:boto:Token: None 2024-04-12T03:08:59.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU= 2024-04-12T03:08:59.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU=', 'Content-Length': '0'} 2024-04-12T03:08:59.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4511957df6cc5b5-006618a5cb-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.763 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.763 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.763 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.764 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.764 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.764 DEBUG:boto:Method: GET 2024-04-12T03:08:59.764 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:08:59.764 DEBUG:boto:Data: 2024-04-12T03:08:59.764 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.764 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.764 DEBUG:boto:Params: {} 2024-04-12T03:08:59.764 DEBUG:boto:Token: None 2024-04-12T03:08:59.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/ 2024-04-12T03:08:59.765 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU= 2024-04-12T03:08:59.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DuADgKG+IbnBByH3udJM12VROmU=', 'Content-Length': '0'} 2024-04-12T03:08:59.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b567b8f603a4205f-006618a5cb-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.810 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:08:59.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:08:59.811 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:08:59.811 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:08:59.811 DEBUG:boto:path=/dgbpxg-16/?versioning 2024-04-12T03:08:59.811 DEBUG:boto:auth_path=/dgbpxg-16/?versioning 2024-04-12T03:08:59.811 DEBUG:boto:Method: PUT 2024-04-12T03:08:59.811 DEBUG:boto:Path: /dgbpxg-16/?versioning 2024-04-12T03:08:59.811 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:08:59.811 DEBUG:boto:Headers: {} 2024-04-12T03:08:59.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:08:59.811 DEBUG:boto:Port: 8000 2024-04-12T03:08:59.812 DEBUG:boto:Params: {} 2024-04-12T03:08:59.812 DEBUG:boto:Token: None 2024-04-12T03:08:59.812 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:08:59 GMT /dgbpxg-16/?versioning 2024-04-12T03:08:59.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Hv9u1C6kMT44bgC1G6h15wZSSnk= 2024-04-12T03:08:59.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:08:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hv9u1C6kMT44bgC1G6h15wZSSnk=', 'Content-Length': '228'} 2024-04-12T03:08:59.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cb07f7b8c4146593-006618a5cb-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:08:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:08:59.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:59.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:59.818 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:59.900 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:08:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:08:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:08:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891339.816831_287.1", 2024-04-12T03:08:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:08:59.816831Z" 2024-04-12T03:08:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:08:59.914 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:08:59.915 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891339.816831_287.1'} 2024-04-12T03:08:59.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:08:59.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:59.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:08:59.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:08:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:08:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:08:59.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:59.993+0000 7f90ca920780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:08:59.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:08:59.994+0000 7f90ca920780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:00.006 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:09:00.007 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:09:00.007 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:09:00.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:09:00.007 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:09:00.007 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:09:00.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:09:00.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:00.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:00.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:00.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:00.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:00.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:00.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:09:00.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:09:00.013 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:00.014 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891339.813533_286.1", 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:08:59.813533Z", 2024-04-12T03:09:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:09:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:09:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:08:59Z" 2024-04-12T03:09:00.017 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:09:00.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891339.813533_286.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:08:59.813533Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:08:59Z" } 2024-04-12T03:09:00.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891339.813533_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:08:59.813533Z', 'realm_epoch': 2}}] 2024-04-12T03:09:00.019 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891339.816831_287.1'} 2024-04-12T03:09:00.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891339.813533_286.1'} 2024-04-12T03:09:00.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891339.816831_287.1 target=1_1712891339.813533_286.1 2024-04-12T03:09:05.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:05.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:05.021 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:05.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:05.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:05.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:05.110+0000 7f1a438d5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:05.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:05.111+0000 7f1a438d5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:05.124 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:09:05.124 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:09:05.124 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:09:05.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:09:05.124 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:05.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:05.126 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:05.127 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:05.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891339.816831_287.1", 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:08:59.816831Z", 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:09:05.129 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:09:05.130 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:09:05.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:05.130 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:09:05Z" 2024-04-12T03:09:05.130 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:09:05.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891339.816831_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:08:59.816831Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:09:05Z" } 2024-04-12T03:09:05.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891339.816831_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:08:59.816831Z', 'realm_epoch': 2}}] 2024-04-12T03:09:05.132 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891339.816831_287.1'} 2024-04-12T03:09:05.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891339.816831_287.1'} 2024-04-12T03:09:05.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:09:05.132 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:05.132 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:05.132 DEBUG:boto:Method: HEAD 2024-04-12T03:09:05.132 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:05.132 DEBUG:boto:Data: 2024-04-12T03:09:05.132 DEBUG:boto:Headers: {} 2024-04-12T03:09:05.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:05.133 DEBUG:boto:Port: 8000 2024-04-12T03:09:05.133 DEBUG:boto:Params: {} 2024-04-12T03:09:05.133 DEBUG:boto:Token: None 2024-04-12T03:09:05.133 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:05 GMT /dgbpxg-16/ 2024-04-12T03:09:05.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2JZNL/p78lKT+9EZSpLHnU8ye7w= 2024-04-12T03:09:05.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2JZNL/p78lKT+9EZSpLHnU8ye7w=', 'Content-Length': '0'} 2024-04-12T03:09:05.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5e8b3f2095b9152-006618a5d1-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:05.136 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:05.136 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:05.136 DEBUG:boto:Method: PUT 2024-04-12T03:09:05.136 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:05.136 DEBUG:boto:Data: 2024-04-12T03:09:05.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:09:05.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:05.136 DEBUG:boto:Port: 8000 2024-04-12T03:09:05.136 DEBUG:boto:Params: {} 2024-04-12T03:09:05.136 DEBUG:boto:Token: None 2024-04-12T03:09:05.137 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:09:05 GMT /dgbpxg-16/obj 2024-04-12T03:09:05.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1J1xnrWo4CBIHwYjAQE3h8CtvsU= 2024-04-12T03:09:05.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:09:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1J1xnrWo4CBIHwYjAQE3h8CtvsU='} 2024-04-12T03:09:05.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-amz-request-id', 'tx0000061830f00c09143e8-006618a5d1-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:05.146 DEBUG:rgw_multi.tests:created new version id=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:05.147 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-16 2024-04-12T03:09:05.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:05.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:05.147 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:05.203 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:05.203 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:05.235+0000 7f2d8db80780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:05.237 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:09:05.237 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:09:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:10.236+0000 7f2d8db80780 1 bucket sync caught up with source: 2024-04-12T03:09:10.238 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:09:10.239 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:09:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:10.236+0000 7f2d8db80780 0 bucket checkpoint complete 2024-04-12T03:09:10.254 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-16 2024-04-12T03:09:10.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:10.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:10.255 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:10.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:10.350+0000 7fbe57dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:10.352 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.42.6, , ] 2024-04-12T03:09:10.352 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:15.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:15.351+0000 7fbe57dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:15.354 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.42.6, , ] 2024-04-12T03:09:15.354 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:20.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:20.352+0000 7fbe57dc2780 1 bucket sync caught up with source: 2024-04-12T03:09:20.355 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:20.355 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:20.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:20.352+0000 7fbe57dc2780 0 bucket checkpoint complete 2024-04-12T03:09:20.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:09:20.370 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.370 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.370 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.371 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:20.371 DEBUG:boto:Data: 2024-04-12T03:09:20.371 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.371 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.371 DEBUG:boto:Params: {} 2024-04-12T03:09:20.371 DEBUG:boto:Token: None 2024-04-12T03:09:20.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y= 2024-04-12T03:09:20.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y=', 'Content-Length': '0'} 2024-04-12T03:09:20.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e528f7cd34a94573-006618a5e0-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.374 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.375 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.375 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.375 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:20.375 DEBUG:boto:Data: 2024-04-12T03:09:20.375 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.375 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.375 DEBUG:boto:Params: {} 2024-04-12T03:09:20.375 DEBUG:boto:Token: None 2024-04-12T03:09:20.375 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y= 2024-04-12T03:09:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y=', 'Content-Length': '0'} 2024-04-12T03:09:20.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000483addd6808a3430-006618a5e0-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.378 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:09:20.378 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.378 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.378 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.379 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.379 DEBUG:boto:Method: GET 2024-04-12T03:09:20.379 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.379 DEBUG:boto:Data: 2024-04-12T03:09:20.379 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.379 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.379 DEBUG:boto:Params: {} 2024-04-12T03:09:20.379 DEBUG:boto:Token: None 2024-04-12T03:09:20.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/?versions 2024-04-12T03:09:20.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs= 2024-04-12T03:09:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs=', 'Content-Length': '0'} 2024-04-12T03:09:20.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e03b75b1be6d91f2-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.426 DEBUG:boto:b'dgbpxg-16999falseobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegtrue2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:20.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:20.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:09:20.428 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.428 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.428 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.428 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.428 DEBUG:boto:Method: GET 2024-04-12T03:09:20.428 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.428 DEBUG:boto:Data: 2024-04-12T03:09:20.428 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.429 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.429 DEBUG:boto:Params: {} 2024-04-12T03:09:20.429 DEBUG:boto:Token: None 2024-04-12T03:09:20.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/?versions 2024-04-12T03:09:20.429 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs= 2024-04-12T03:09:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs=', 'Content-Length': '0'} 2024-04-12T03:09:20.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001159899cd54d71d4-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.432 DEBUG:boto:b'dgbpxg-16999falseobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegtrue2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:20.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:20.433 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.433 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.433 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.433 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.433 DEBUG:boto:Method: GET 2024-04-12T03:09:20.433 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.434 DEBUG:boto:Data: 2024-04-12T03:09:20.434 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.434 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.434 DEBUG:boto:Params: {} 2024-04-12T03:09:20.434 DEBUG:boto:Token: None 2024-04-12T03:09:20.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/?versions 2024-04-12T03:09:20.434 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs= 2024-04-12T03:09:20.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs=', 'Content-Length': '0'} 2024-04-12T03:09:20.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d765c9dcd9bbde50-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.437 DEBUG:boto:b'dgbpxg-16999falseobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegtrue2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.437 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.437 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.437 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.438 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.438 DEBUG:boto:Method: GET 2024-04-12T03:09:20.438 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:20.438 DEBUG:boto:Data: 2024-04-12T03:09:20.438 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.438 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.438 DEBUG:boto:Params: {} 2024-04-12T03:09:20.439 DEBUG:boto:Token: None 2024-04-12T03:09:20.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/?versions 2024-04-12T03:09:20.439 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs= 2024-04-12T03:09:20.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vI0kBvmGCaWJMRIN1CZLdtxTcSs=', 'Content-Length': '0'} 2024-04-12T03:09:20.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647182d01ec1ef65-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.486 DEBUG:boto:b'dgbpxg-16999falseobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegtrue2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:08:34.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.487 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.487 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.487 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.487 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.488 DEBUG:boto:Method: GET 2024-04-12T03:09:20.488 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.488 DEBUG:boto:Data: 2024-04-12T03:09:20.488 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.488 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.488 DEBUG:boto:Params: {} 2024-04-12T03:09:20.488 DEBUG:boto:Token: None 2024-04-12T03:09:20.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.489 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008334624d3b742361-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.491 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.491 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.491 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.491 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.492 DEBUG:boto:Method: GET 2024-04-12T03:09:20.492 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.492 DEBUG:boto:Data: 2024-04-12T03:09:20.492 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.492 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.492 DEBUG:boto:Params: {} 2024-04-12T03:09:20.492 DEBUG:boto:Token: None 2024-04-12T03:09:20.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.492 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065dbb28ddde1b61f-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.494 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.494 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.494 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.495 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.495 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.495 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.495 DEBUG:boto:Data: 2024-04-12T03:09:20.495 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.495 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.495 DEBUG:boto:Params: {} 2024-04-12T03:09:20.495 DEBUG:boto:Token: None 2024-04-12T03:09:20.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DghpxPbn8e/TzgE7QBAMuZaEIa8= 2024-04-12T03:09:20.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DghpxPbn8e/TzgE7QBAMuZaEIa8=', 'Content-Length': '0'} 2024-04-12T03:09:20.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000375b1f81a1771c02-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.498 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.498 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.498 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.498 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.498 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.498 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.498 DEBUG:boto:Data: 2024-04-12T03:09:20.498 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.498 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.499 DEBUG:boto:Params: {} 2024-04-12T03:09:20.499 DEBUG:boto:Token: None 2024-04-12T03:09:20.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DghpxPbn8e/TzgE7QBAMuZaEIa8= 2024-04-12T03:09:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DghpxPbn8e/TzgE7QBAMuZaEIa8=', 'Content-Length': '0'} 2024-04-12T03:09:20.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a709966cb0c1b963-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.501 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.501 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.501 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.501 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.501 DEBUG:boto:Method: GET 2024-04-12T03:09:20.501 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.501 DEBUG:boto:Data: 2024-04-12T03:09:20.501 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.501 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.502 DEBUG:boto:Params: {} 2024-04-12T03:09:20.502 DEBUG:boto:Token: None 2024-04-12T03:09:20.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.502 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0dd1a5b04699608-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.504 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.504 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.504 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.504 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.504 DEBUG:boto:Method: GET 2024-04-12T03:09:20.504 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.505 DEBUG:boto:Data: 2024-04-12T03:09:20.505 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.505 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.505 DEBUG:boto:Params: {} 2024-04-12T03:09:20.505 DEBUG:boto:Token: None 2024-04-12T03:09:20.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.505 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f79feca16a54db51-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.507 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.507 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.507 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.508 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.508 DEBUG:boto:Method: GET 2024-04-12T03:09:20.508 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.508 DEBUG:boto:Data: 2024-04-12T03:09:20.508 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.508 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.508 DEBUG:boto:Params: {} 2024-04-12T03:09:20.508 DEBUG:boto:Token: None 2024-04-12T03:09:20.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000838cb4c707333b2-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.516 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.516 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.516 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.516 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.516 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.516 DEBUG:boto:Method: GET 2024-04-12T03:09:20.517 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.517 DEBUG:boto:Data: 2024-04-12T03:09:20.517 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.517 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.517 DEBUG:boto:Params: {} 2024-04-12T03:09:20.517 DEBUG:boto:Token: None 2024-04-12T03:09:20.517 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.517 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d55e51eff134eab9-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.520 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.520 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.520 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.520 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.521 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:20.521 DEBUG:boto:Data: 2024-04-12T03:09:20.521 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.521 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.521 DEBUG:boto:Params: {} 2024-04-12T03:09:20.521 DEBUG:boto:Token: None 2024-04-12T03:09:20.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj 2024-04-12T03:09:20.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU= 2024-04-12T03:09:20.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU=', 'Content-Length': '0'} 2024-04-12T03:09:20.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3c160ae6a630a1b-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.524 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.524 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.524 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.524 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:20.524 DEBUG:boto:Data: 2024-04-12T03:09:20.524 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.524 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.524 DEBUG:boto:Params: {} 2024-04-12T03:09:20.524 DEBUG:boto:Token: None 2024-04-12T03:09:20.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj 2024-04-12T03:09:20.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU= 2024-04-12T03:09:20.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU=', 'Content-Length': '0'} 2024-04-12T03:09:20.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000136e0f1f6f8dd32f-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.527 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.527 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.527 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.527 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.527 DEBUG:boto:Method: GET 2024-04-12T03:09:20.528 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.528 DEBUG:boto:Data: 2024-04-12T03:09:20.528 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.528 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.528 DEBUG:boto:Params: {} 2024-04-12T03:09:20.528 DEBUG:boto:Token: None 2024-04-12T03:09:20.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000cb87c801509cbb2-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.530 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.530 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.531 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.531 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.531 DEBUG:boto:Method: GET 2024-04-12T03:09:20.531 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.531 DEBUG:boto:Data: 2024-04-12T03:09:20.531 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.531 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.531 DEBUG:boto:Params: {} 2024-04-12T03:09:20.531 DEBUG:boto:Token: None 2024-04-12T03:09:20.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b59916c254e8e32f-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.534 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.534 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.534 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.534 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.534 DEBUG:boto:Method: GET 2024-04-12T03:09:20.534 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.534 DEBUG:boto:Data: 2024-04-12T03:09:20.534 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.534 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.535 DEBUG:boto:Params: {} 2024-04-12T03:09:20.535 DEBUG:boto:Token: None 2024-04-12T03:09:20.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d984cbd5d62499-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.578 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.579 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.579 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.579 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.579 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.579 DEBUG:boto:Method: GET 2024-04-12T03:09:20.579 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.579 DEBUG:boto:Data: 2024-04-12T03:09:20.579 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.580 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.580 DEBUG:boto:Params: {} 2024-04-12T03:09:20.580 DEBUG:boto:Token: None 2024-04-12T03:09:20.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb3f3773f9dfcfe-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.583 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.583 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.583 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.583 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.584 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.584 DEBUG:boto:Method: GET 2024-04-12T03:09:20.584 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.584 DEBUG:boto:Data: 2024-04-12T03:09:20.584 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.584 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.584 DEBUG:boto:Params: {} 2024-04-12T03:09:20.584 DEBUG:boto:Token: None 2024-04-12T03:09:20.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8= 2024-04-12T03:09:20.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8=', 'Content-Length': '0'} 2024-04-12T03:09:20.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b98c80908869e6c8-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.586 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.587 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.587 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.587 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.587 DEBUG:boto:Method: GET 2024-04-12T03:09:20.587 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.587 DEBUG:boto:Data: 2024-04-12T03:09:20.587 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.587 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.587 DEBUG:boto:Params: {} 2024-04-12T03:09:20.587 DEBUG:boto:Token: None 2024-04-12T03:09:20.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8= 2024-04-12T03:09:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8=', 'Content-Length': '0'} 2024-04-12T03:09:20.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc5f2457c31ca9e9-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.590 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.590 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.590 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.590 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.590 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.590 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.590 DEBUG:boto:Data: 2024-04-12T03:09:20.590 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.590 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.591 DEBUG:boto:Params: {} 2024-04-12T03:09:20.591 DEBUG:boto:Token: None 2024-04-12T03:09:20.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eG99sC3NDOVA1arQ3tAFlKo5jn0= 2024-04-12T03:09:20.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eG99sC3NDOVA1arQ3tAFlKo5jn0=', 'Content-Length': '0'} 2024-04-12T03:09:20.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a4608a0ceaaf944-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.593 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.593 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.593 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.593 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.593 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.593 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.593 DEBUG:boto:Data: 2024-04-12T03:09:20.593 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.594 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.594 DEBUG:boto:Params: {} 2024-04-12T03:09:20.594 DEBUG:boto:Token: None 2024-04-12T03:09:20.594 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eG99sC3NDOVA1arQ3tAFlKo5jn0= 2024-04-12T03:09:20.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eG99sC3NDOVA1arQ3tAFlKo5jn0=', 'Content-Length': '0'} 2024-04-12T03:09:20.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041e912eed210cea4-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.596 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.596 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.596 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.596 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.597 DEBUG:boto:Method: GET 2024-04-12T03:09:20.597 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.597 DEBUG:boto:Data: 2024-04-12T03:09:20.597 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.597 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.597 DEBUG:boto:Params: {} 2024-04-12T03:09:20.597 DEBUG:boto:Token: None 2024-04-12T03:09:20.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.597 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8= 2024-04-12T03:09:20.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8=', 'Content-Length': '0'} 2024-04-12T03:09:20.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000010d3995924a6871-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.599 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.599 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.600 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.600 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.600 DEBUG:boto:Method: GET 2024-04-12T03:09:20.600 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.600 DEBUG:boto:Data: 2024-04-12T03:09:20.600 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.600 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.600 DEBUG:boto:Params: {} 2024-04-12T03:09:20.600 DEBUG:boto:Token: None 2024-04-12T03:09:20.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:20.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8= 2024-04-12T03:09:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh3C+EusHM/T1fmhcIpa8+68Nw8=', 'Content-Length': '0'} 2024-04-12T03:09:20.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002504066bc27386d6-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.602 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.603 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.603 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.603 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.603 DEBUG:boto:Method: GET 2024-04-12T03:09:20.603 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.603 DEBUG:boto:Data: 2024-04-12T03:09:20.603 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.603 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.603 DEBUG:boto:Params: {} 2024-04-12T03:09:20.603 DEBUG:boto:Token: None 2024-04-12T03:09:20.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f3b470a5c7e50d0-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.606 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.607 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.607 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.607 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.607 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.607 DEBUG:boto:Method: GET 2024-04-12T03:09:20.607 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.607 DEBUG:boto:Data: 2024-04-12T03:09:20.607 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.607 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.608 DEBUG:boto:Params: {} 2024-04-12T03:09:20.608 DEBUG:boto:Token: None 2024-04-12T03:09:20.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a752933d9f39c118-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.610 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.610 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.611 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.611 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.611 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:20.611 DEBUG:boto:Data: 2024-04-12T03:09:20.611 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.611 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.611 DEBUG:boto:Params: {} 2024-04-12T03:09:20.611 DEBUG:boto:Token: None 2024-04-12T03:09:20.611 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj 2024-04-12T03:09:20.612 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU= 2024-04-12T03:09:20.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU=', 'Content-Length': '0'} 2024-04-12T03:09:20.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092010678ba9e2b48-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.614 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.614 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.614 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.614 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:20.614 DEBUG:boto:Data: 2024-04-12T03:09:20.614 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.614 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.615 DEBUG:boto:Params: {} 2024-04-12T03:09:20.615 DEBUG:boto:Token: None 2024-04-12T03:09:20.615 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj 2024-04-12T03:09:20.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU= 2024-04-12T03:09:20.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c04d9HOUhlyZu0tBAPJKb8BRctU=', 'Content-Length': '0'} 2024-04-12T03:09:20.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007772383ff9cec543-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:20.617 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.617 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.617 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.617 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.617 DEBUG:boto:Method: GET 2024-04-12T03:09:20.618 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.618 DEBUG:boto:Data: 2024-04-12T03:09:20.618 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.618 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.618 DEBUG:boto:Params: {} 2024-04-12T03:09:20.618 DEBUG:boto:Token: None 2024-04-12T03:09:20.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a54f230347d4097-006618a5e0-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.620 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.620 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.620 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.620 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.621 DEBUG:boto:Method: GET 2024-04-12T03:09:20.621 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.621 DEBUG:boto:Data: 2024-04-12T03:09:20.621 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.621 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.621 DEBUG:boto:Params: {} 2024-04-12T03:09:20.621 DEBUG:boto:Token: None 2024-04-12T03:09:20.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:20.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg= 2024-04-12T03:09:20.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T5LWJI37c4cDCoXnmrptgqzEoCg=', 'Content-Length': '0'} 2024-04-12T03:09:20.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ecb5173b174ab3d-006618a5e0-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.623 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.624 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.624 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.624 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.624 DEBUG:boto:Method: GET 2024-04-12T03:09:20.624 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.624 DEBUG:boto:Data: 2024-04-12T03:09:20.624 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:20.624 DEBUG:boto:Port: 8001 2024-04-12T03:09:20.624 DEBUG:boto:Params: {} 2024-04-12T03:09:20.625 DEBUG:boto:Token: None 2024-04-12T03:09:20.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.625 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000381508ab798500e5-006618a5e0-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.674 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.675 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.675 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.675 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.675 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.675 DEBUG:boto:Method: GET 2024-04-12T03:09:20.675 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:20.676 DEBUG:boto:Data: 2024-04-12T03:09:20.676 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.676 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.676 DEBUG:boto:Params: {} 2024-04-12T03:09:20.676 DEBUG:boto:Token: None 2024-04-12T03:09:20.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw= 2024-04-12T03:09:20.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fJ5A99V3WZZXOj6APrTw/jb2Xaw=', 'Content-Length': '0'} 2024-04-12T03:09:20.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed3ca1ca9a75fc8-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.679 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:20.679 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:09:20.679 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.680 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.680 DEBUG:boto:path=/dgbpxg-16/?versioning 2024-04-12T03:09:20.680 DEBUG:boto:auth_path=/dgbpxg-16/?versioning 2024-04-12T03:09:20.680 DEBUG:boto:Method: PUT 2024-04-12T03:09:20.680 DEBUG:boto:Path: /dgbpxg-16/?versioning 2024-04-12T03:09:20.680 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:09:20.680 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.680 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.680 DEBUG:boto:Params: {} 2024-04-12T03:09:20.680 DEBUG:boto:Token: None 2024-04-12T03:09:20.681 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/?versioning 2024-04-12T03:09:20.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1MWnOED4gK0N+Du/86wSgxX9rSc= 2024-04-12T03:09:20.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1MWnOED4gK0N+Du/86wSgxX9rSc=', 'Content-Length': '230'} 2024-04-12T03:09:20.686 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000015792258b197d067-006618a5e0-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:20.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:20.687 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:09:20.783 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:09:20.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.783 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:20.783 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:09:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891360.685438_294.1", 2024-04-12T03:09:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:09:20.685438Z" 2024-04-12T03:09:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.785 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:09:20.786 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891360.685438_294.1'} 2024-04-12T03:09:20.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:09:20.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:20.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:20.787 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:20.865+0000 7f3b86aff780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:20.866+0000 7f3b86aff780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:09:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891360.685438_294.1", 2024-04-12T03:09:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:09:20.685438Z", 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:09:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:09:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:09:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:09:20Z" 2024-04-12T03:09:20.886 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:09:20.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891360.685438_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:09:20.685438Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:09:20Z" } 2024-04-12T03:09:20.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891360.685438_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:09:20.685438Z', 'realm_epoch': 2}}] 2024-04-12T03:09:20.887 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891360.685438_294.1'} 2024-04-12T03:09:20.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891360.685438_294.1'} 2024-04-12T03:09:20.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:09:20.888 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:20.888 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:20.888 DEBUG:boto:Method: HEAD 2024-04-12T03:09:20.888 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:20.888 DEBUG:boto:Data: 2024-04-12T03:09:20.888 DEBUG:boto:Headers: {} 2024-04-12T03:09:20.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.888 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.889 DEBUG:boto:Params: {} 2024-04-12T03:09:20.889 DEBUG:boto:Token: None 2024-04-12T03:09:20.889 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/ 2024-04-12T03:09:20.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y= 2024-04-12T03:09:20.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aqXSomNmbhj0GJQivSWX81iFo7Y=', 'Content-Length': '0'} 2024-04-12T03:09:20.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004acb84fa43e1b194-006618a5e0-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.892 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:20.892 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:20.892 DEBUG:boto:Method: PUT 2024-04-12T03:09:20.892 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:20.892 DEBUG:boto:Data: 2024-04-12T03:09:20.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:09:20.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:20.892 DEBUG:boto:Port: 8000 2024-04-12T03:09:20.892 DEBUG:boto:Params: {} 2024-04-12T03:09:20.892 DEBUG:boto:Token: None 2024-04-12T03:09:20.893 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:09:20 GMT /dgbpxg-16/obj 2024-04-12T03:09:20.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wIIM7Yrr8zU0Gb6Hogv7kI0bDmU= 2024-04-12T03:09:20.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:09:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wIIM7Yrr8zU0Gb6Hogv7kI0bDmU='} 2024-04-12T03:09:20.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1acb31f1b985d94-006618a5e0-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:20.902 DEBUG:rgw_multi.tests:created null version id=None 2024-04-12T03:09:20.902 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-16 2024-04-12T03:09:20.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:20.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:20.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:20.990+0000 7fda6e83a780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:20.991 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:09:20.991 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:09:25.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:25.991+0000 7fda6e83a780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:25.993 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:09:25.993 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:09:30.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:30.992+0000 7fda6e83a780 1 bucket sync caught up with source: 2024-04-12T03:09:30.994 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:09:30.994 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:09:30.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:30.992+0000 7fda6e83a780 0 bucket checkpoint complete 2024-04-12T03:09:31.007 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-16 2024-04-12T03:09:31.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:31.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:31.008 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:31.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:31.097+0000 7f92050a3780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:31.098 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:31.098 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.51.18, , ] 2024-04-12T03:09:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:36.098+0000 7f92050a3780 1 waiting for incremental sync to catch up: 2024-04-12T03:09:36.100 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.46.13, , ] 2024-04-12T03:09:36.100 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.51.18, , ] 2024-04-12T03:09:41.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:41.099+0000 7f92050a3780 1 bucket sync caught up with source: 2024-04-12T03:09:41.101 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.51.18, , ] 2024-04-12T03:09:41.101 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.51.18, , ] 2024-04-12T03:09:41.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:41.100+0000 7f92050a3780 0 bucket checkpoint complete 2024-04-12T03:09:41.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:09:41.117 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.117 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.117 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.117 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:41.117 DEBUG:boto:Data: 2024-04-12T03:09:41.117 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.117 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.117 DEBUG:boto:Params: {} 2024-04-12T03:09:41.118 DEBUG:boto:Token: None 2024-04-12T03:09:41.118 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.118 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mFxab0AoesWPxYZEeYwJNT3nGsM= 2024-04-12T03:09:41.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mFxab0AoesWPxYZEeYwJNT3nGsM=', 'Content-Length': '0'} 2024-04-12T03:09:41.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c3794a4d92227c2-006618a5f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.121 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.121 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.121 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.121 DEBUG:boto:Path: /dgbpxg-16/ 2024-04-12T03:09:41.121 DEBUG:boto:Data: 2024-04-12T03:09:41.121 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.121 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.121 DEBUG:boto:Params: {} 2024-04-12T03:09:41.122 DEBUG:boto:Token: None 2024-04-12T03:09:41.122 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mFxab0AoesWPxYZEeYwJNT3nGsM= 2024-04-12T03:09:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mFxab0AoesWPxYZEeYwJNT3nGsM=', 'Content-Length': '0'} 2024-04-12T03:09:41.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000218996a699dd6553-006618a5f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.124 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:09:41.124 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.124 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.125 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.125 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.125 DEBUG:boto:Method: GET 2024-04-12T03:09:41.125 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.125 DEBUG:boto:Data: 2024-04-12T03:09:41.125 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.126 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.126 DEBUG:boto:Params: {} 2024-04-12T03:09:41.126 DEBUG:boto:Token: None 2024-04-12T03:09:41.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/?versions 2024-04-12T03:09:41.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk= 2024-04-12T03:09:41.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk=', 'Content-Length': '0'} 2024-04-12T03:09:41.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f4e26986a7d76e-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.128 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegfalse2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:41.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:41.129 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:09:41.129 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.130 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.130 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.130 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.130 DEBUG:boto:Method: GET 2024-04-12T03:09:41.130 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.130 DEBUG:boto:Data: 2024-04-12T03:09:41.130 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.130 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.130 DEBUG:boto:Params: {} 2024-04-12T03:09:41.131 DEBUG:boto:Token: None 2024-04-12T03:09:41.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/?versions 2024-04-12T03:09:41.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk= 2024-04-12T03:09:41.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk=', 'Content-Length': '0'} 2024-04-12T03:09:41.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4cd937cd6089f1-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.133 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegfalse2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:41.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:09:41.134 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.134 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.134 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.134 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.134 DEBUG:boto:Method: GET 2024-04-12T03:09:41.134 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.134 DEBUG:boto:Data: 2024-04-12T03:09:41.134 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.135 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.135 DEBUG:boto:Params: {} 2024-04-12T03:09:41.135 DEBUG:boto:Token: None 2024-04-12T03:09:41.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/?versions 2024-04-12T03:09:41.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk= 2024-04-12T03:09:41.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk=', 'Content-Length': '0'} 2024-04-12T03:09:41.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057bd08f6183bd9b3-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.137 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegfalse2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.138 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.138 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.138 DEBUG:boto:path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.138 DEBUG:boto:auth_path=/dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.138 DEBUG:boto:Method: GET 2024-04-12T03:09:41.138 DEBUG:boto:Path: /dgbpxg-16/?versions&max-keys=999 2024-04-12T03:09:41.138 DEBUG:boto:Data: 2024-04-12T03:09:41.138 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.138 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.139 DEBUG:boto:Params: {} 2024-04-12T03:09:41.139 DEBUG:boto:Token: None 2024-04-12T03:09:41.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/?versions 2024-04-12T03:09:41.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk= 2024-04-12T03:09:41.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X/oG7Qr9PDgT8jfqITTgn55BLxk=', 'Content-Length': '0'} 2024-04-12T03:09:41.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008833349212b63f73-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.144 DEBUG:boto:b'dgbpxg-16999falseobjnulltrue2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4wlAa8cpjXzid6WdRKF5zyjcQsc2qegfalse2024-04-12T03:09:05.138Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.145 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.145 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.145 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.145 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.145 DEBUG:boto:Method: GET 2024-04-12T03:09:41.145 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.145 DEBUG:boto:Data: 2024-04-12T03:09:41.145 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.146 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.146 DEBUG:boto:Params: {} 2024-04-12T03:09:41.146 DEBUG:boto:Token: None 2024-04-12T03:09:41.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08= 2024-04-12T03:09:41.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08=', 'Content-Length': '0'} 2024-04-12T03:09:41.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002297359ee0264217-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.148 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.148 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.148 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.149 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.149 DEBUG:boto:Method: GET 2024-04-12T03:09:41.149 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.149 DEBUG:boto:Data: 2024-04-12T03:09:41.149 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.149 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.149 DEBUG:boto:Params: {} 2024-04-12T03:09:41.149 DEBUG:boto:Token: None 2024-04-12T03:09:41.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08= 2024-04-12T03:09:41.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08=', 'Content-Length': '0'} 2024-04-12T03:09:41.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ac68f3ff8b4de44-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.151 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.152 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.152 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.152 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.152 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.152 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.152 DEBUG:boto:Data: 2024-04-12T03:09:41.152 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.152 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.152 DEBUG:boto:Params: {} 2024-04-12T03:09:41.152 DEBUG:boto:Token: None 2024-04-12T03:09:41.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o65Ql8xMqtDDVYQzN5IdsBA42fw= 2024-04-12T03:09:41.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o65Ql8xMqtDDVYQzN5IdsBA42fw=', 'Content-Length': '0'} 2024-04-12T03:09:41.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b37326e7cc36c98-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.154 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.154 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.155 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.155 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.155 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.155 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.155 DEBUG:boto:Data: 2024-04-12T03:09:41.155 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.155 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.155 DEBUG:boto:Params: {} 2024-04-12T03:09:41.155 DEBUG:boto:Token: None 2024-04-12T03:09:41.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o65Ql8xMqtDDVYQzN5IdsBA42fw= 2024-04-12T03:09:41.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o65Ql8xMqtDDVYQzN5IdsBA42fw=', 'Content-Length': '0'} 2024-04-12T03:09:41.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000222fda265e772758-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.157 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.158 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.158 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.158 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.158 DEBUG:boto:Method: GET 2024-04-12T03:09:41.158 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.158 DEBUG:boto:Data: 2024-04-12T03:09:41.158 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.158 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.158 DEBUG:boto:Params: {} 2024-04-12T03:09:41.158 DEBUG:boto:Token: None 2024-04-12T03:09:41.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08= 2024-04-12T03:09:41.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08=', 'Content-Length': '0'} 2024-04-12T03:09:41.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007bd60e88d55215de-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.161 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.161 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.161 DEBUG:boto:path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.161 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.161 DEBUG:boto:Method: GET 2024-04-12T03:09:41.161 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.161 DEBUG:boto:Data: 2024-04-12T03:09:41.161 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.162 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.162 DEBUG:boto:Params: {} 2024-04-12T03:09:41.162 DEBUG:boto:Token: None 2024-04-12T03:09:41.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=null 2024-04-12T03:09:41.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08= 2024-04-12T03:09:41.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XVkDq9lZNmopJQ2/JNpnSW4hT08=', 'Content-Length': '0'} 2024-04-12T03:09:41.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e088b449dbc97231-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.164 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.164 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.164 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.164 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.164 DEBUG:boto:Method: GET 2024-04-12T03:09:41.164 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.165 DEBUG:boto:Data: 2024-04-12T03:09:41.165 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.165 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.165 DEBUG:boto:Params: {} 2024-04-12T03:09:41.165 DEBUG:boto:Token: None 2024-04-12T03:09:41.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0386e53b0c856c3-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.167 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.168 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.168 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.168 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.168 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.168 DEBUG:boto:Method: GET 2024-04-12T03:09:41.168 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.168 DEBUG:boto:Data: 2024-04-12T03:09:41.168 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.169 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.169 DEBUG:boto:Params: {} 2024-04-12T03:09:41.169 DEBUG:boto:Token: None 2024-04-12T03:09:41.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e668df6e44a99e45-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.171 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.172 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.172 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.172 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.172 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.172 DEBUG:boto:Data: 2024-04-12T03:09:41.172 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.172 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.172 DEBUG:boto:Params: {} 2024-04-12T03:09:41.172 DEBUG:boto:Token: None 2024-04-12T03:09:41.173 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.173 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8= 2024-04-12T03:09:41.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8=', 'Content-Length': '0'} 2024-04-12T03:09:41.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8da8363e069856b-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.174 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.175 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.175 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.175 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.175 DEBUG:boto:Data: 2024-04-12T03:09:41.175 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.175 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.175 DEBUG:boto:Params: {} 2024-04-12T03:09:41.175 DEBUG:boto:Token: None 2024-04-12T03:09:41.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8= 2024-04-12T03:09:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8=', 'Content-Length': '0'} 2024-04-12T03:09:41.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a27fdb2a663623a-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.177 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.177 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.178 DEBUG:boto:Method: GET 2024-04-12T03:09:41.178 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.178 DEBUG:boto:Data: 2024-04-12T03:09:41.178 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.178 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.178 DEBUG:boto:Params: {} 2024-04-12T03:09:41.178 DEBUG:boto:Token: None 2024-04-12T03:09:41.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk= 2024-04-12T03:09:41.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk=', 'Content-Length': '0'} 2024-04-12T03:09:41.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009dab6ef44d579d72-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.181 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.181 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.181 DEBUG:boto:Method: GET 2024-04-12T03:09:41.181 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.181 DEBUG:boto:Data: 2024-04-12T03:09:41.181 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.181 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.181 DEBUG:boto:Params: {} 2024-04-12T03:09:41.181 DEBUG:boto:Token: None 2024-04-12T03:09:41.182 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk= 2024-04-12T03:09:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk=', 'Content-Length': '0'} 2024-04-12T03:09:41.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be2649840eb2de09-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.184 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.184 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.184 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.184 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.184 DEBUG:boto:Method: GET 2024-04-12T03:09:41.184 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.184 DEBUG:boto:Data: 2024-04-12T03:09:41.184 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.184 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.185 DEBUG:boto:Params: {} 2024-04-12T03:09:41.185 DEBUG:boto:Token: None 2024-04-12T03:09:41.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.185 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd322e247e216704-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.187 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.188 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.188 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.188 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.188 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.188 DEBUG:boto:Method: GET 2024-04-12T03:09:41.188 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.188 DEBUG:boto:Data: 2024-04-12T03:09:41.188 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.189 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.189 DEBUG:boto:Params: {} 2024-04-12T03:09:41.189 DEBUG:boto:Token: None 2024-04-12T03:09:41.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412eaa88b8c26e69-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.234 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.235 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.235 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.235 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.235 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.235 DEBUG:boto:Method: GET 2024-04-12T03:09:41.235 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.236 DEBUG:boto:Data: 2024-04-12T03:09:41.236 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.236 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.236 DEBUG:boto:Params: {} 2024-04-12T03:09:41.236 DEBUG:boto:Token: None 2024-04-12T03:09:41.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo= 2024-04-12T03:09:41.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo=', 'Content-Length': '0'} 2024-04-12T03:09:41.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001da8e5b3805da075-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.239 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.239 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.239 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.239 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.239 DEBUG:boto:Method: GET 2024-04-12T03:09:41.239 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.239 DEBUG:boto:Data: 2024-04-12T03:09:41.239 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.239 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.239 DEBUG:boto:Params: {} 2024-04-12T03:09:41.240 DEBUG:boto:Token: None 2024-04-12T03:09:41.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo= 2024-04-12T03:09:41.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo=', 'Content-Length': '0'} 2024-04-12T03:09:41.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9901fdec3fbd386-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.242 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.242 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.242 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.242 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.242 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.242 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.242 DEBUG:boto:Data: 2024-04-12T03:09:41.242 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.243 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.243 DEBUG:boto:Params: {} 2024-04-12T03:09:41.243 DEBUG:boto:Token: None 2024-04-12T03:09:41.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zdxxEiB3lE1s2PVwdy9A0b+SIq8= 2024-04-12T03:09:41.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zdxxEiB3lE1s2PVwdy9A0b+SIq8=', 'Content-Length': '0'} 2024-04-12T03:09:41.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c50b5d33c608acd-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.245 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.245 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.245 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.245 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.245 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.245 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.246 DEBUG:boto:Data: 2024-04-12T03:09:41.246 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.246 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.246 DEBUG:boto:Params: {} 2024-04-12T03:09:41.246 DEBUG:boto:Token: None 2024-04-12T03:09:41.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zdxxEiB3lE1s2PVwdy9A0b+SIq8= 2024-04-12T03:09:41.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zdxxEiB3lE1s2PVwdy9A0b+SIq8=', 'Content-Length': '0'} 2024-04-12T03:09:41.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000066bcf7b065881a-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.249 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.249 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.249 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.249 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.249 DEBUG:boto:Method: GET 2024-04-12T03:09:41.249 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.249 DEBUG:boto:Data: 2024-04-12T03:09:41.249 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.249 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.249 DEBUG:boto:Params: {} 2024-04-12T03:09:41.250 DEBUG:boto:Token: None 2024-04-12T03:09:41.250 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.250 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo= 2024-04-12T03:09:41.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo=', 'Content-Length': '0'} 2024-04-12T03:09:41.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006077cef031884c91-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.252 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.252 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.252 DEBUG:boto:path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.252 DEBUG:boto:auth_path=/dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.252 DEBUG:boto:Method: GET 2024-04-12T03:09:41.252 DEBUG:boto:Path: /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.252 DEBUG:boto:Data: 2024-04-12T03:09:41.252 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.253 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.253 DEBUG:boto:Params: {} 2024-04-12T03:09:41.253 DEBUG:boto:Token: None 2024-04-12T03:09:41.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj?versionId=4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg 2024-04-12T03:09:41.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo= 2024-04-12T03:09:41.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tDh0zrlfdeou8M8dX9+os5RmZeo=', 'Content-Length': '0'} 2024-04-12T03:09:41.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:05 GMT'), ('x-amz-version-id', '4wlAa8cpjXzid6WdRKF5zyjcQsc2qeg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dde7bbea3a28c564-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.255 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.256 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.256 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.256 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.256 DEBUG:boto:Method: GET 2024-04-12T03:09:41.256 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.256 DEBUG:boto:Data: 2024-04-12T03:09:41.256 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.256 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.256 DEBUG:boto:Params: {} 2024-04-12T03:09:41.257 DEBUG:boto:Token: None 2024-04-12T03:09:41.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d54cab21e5697c-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.306 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.307 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.307 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.307 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.307 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.307 DEBUG:boto:Method: GET 2024-04-12T03:09:41.307 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.307 DEBUG:boto:Data: 2024-04-12T03:09:41.307 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.308 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.308 DEBUG:boto:Params: {} 2024-04-12T03:09:41.308 DEBUG:boto:Token: None 2024-04-12T03:09:41.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000884e97041e010a83-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.311 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.311 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.311 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.311 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.311 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.311 DEBUG:boto:Data: 2024-04-12T03:09:41.312 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.312 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.312 DEBUG:boto:Params: {} 2024-04-12T03:09:41.312 DEBUG:boto:Token: None 2024-04-12T03:09:41.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8= 2024-04-12T03:09:41.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8=', 'Content-Length': '0'} 2024-04-12T03:09:41.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d70879437e9a547-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.314 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.314 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.315 DEBUG:boto:Method: HEAD 2024-04-12T03:09:41.315 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.315 DEBUG:boto:Data: 2024-04-12T03:09:41.315 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.315 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.315 DEBUG:boto:Params: {} 2024-04-12T03:09:41.315 DEBUG:boto:Token: None 2024-04-12T03:09:41.315 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8= 2024-04-12T03:09:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/XaGQa52K+T03saMHrV1IyYkZA8=', 'Content-Length': '0'} 2024-04-12T03:09:41.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000166d8a71898271f6-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:09:41.317 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.317 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.318 DEBUG:boto:Method: GET 2024-04-12T03:09:41.318 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.318 DEBUG:boto:Data: 2024-04-12T03:09:41.318 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.318 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.318 DEBUG:boto:Params: {} 2024-04-12T03:09:41.318 DEBUG:boto:Token: None 2024-04-12T03:09:41.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk= 2024-04-12T03:09:41.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk=', 'Content-Length': '0'} 2024-04-12T03:09:41.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086e601f4b17f0416-006618a5f5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.320 DEBUG:boto:path=/dgbpxg-16/obj 2024-04-12T03:09:41.321 DEBUG:boto:auth_path=/dgbpxg-16/obj 2024-04-12T03:09:41.321 DEBUG:boto:Method: GET 2024-04-12T03:09:41.321 DEBUG:boto:Path: /dgbpxg-16/obj 2024-04-12T03:09:41.321 DEBUG:boto:Data: 2024-04-12T03:09:41.321 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.321 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.321 DEBUG:boto:Params: {} 2024-04-12T03:09:41.321 DEBUG:boto:Token: None 2024-04-12T03:09:41.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/obj 2024-04-12T03:09:41.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk= 2024-04-12T03:09:41.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSbEWaLj6bQHC5yMB4syTZ0bnvk=', 'Content-Length': '0'} 2024-04-12T03:09:41.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb036f1630da218e-006618a5f5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.324 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.324 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.324 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.324 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.324 DEBUG:boto:Method: GET 2024-04-12T03:09:41.324 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.324 DEBUG:boto:Data: 2024-04-12T03:09:41.325 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.325 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.325 DEBUG:boto:Params: {} 2024-04-12T03:09:41.325 DEBUG:boto:Token: None 2024-04-12T03:09:41.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dea1896b0412374-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.328 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.328 DEBUG:boto:path=/dgbpxg-16/ 2024-04-12T03:09:41.328 DEBUG:boto:auth_path=/dgbpxg-16/ 2024-04-12T03:09:41.328 DEBUG:boto:path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.328 DEBUG:boto:auth_path=/dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.328 DEBUG:boto:Method: GET 2024-04-12T03:09:41.328 DEBUG:boto:Path: /dgbpxg-16/?prefix=obj 2024-04-12T03:09:41.329 DEBUG:boto:Data: 2024-04-12T03:09:41.329 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.329 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.329 DEBUG:boto:Params: {} 2024-04-12T03:09:41.329 DEBUG:boto:Token: None 2024-04-12T03:09:41.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-16/ 2024-04-12T03:09:41.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY= 2024-04-12T03:09:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbsXrfondBBuIuCBZ49Ryz/sZrY=', 'Content-Length': '0'} 2024-04-12T03:09:41.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0552a046d5260f-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.374 DEBUG:boto:b'dgbpxg-16obj1000falseobj2024-04-12T03:09:20.894Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:09:41.375 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-16 zones={a2, a1} 2024-04-12T03:09:41.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-12T03:09:41.377 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-17 2024-04-12T03:09:41.377 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:09:41.377 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:09:41.377 DEBUG:boto:Method: PUT 2024-04-12T03:09:41.377 DEBUG:boto:Path: /dgbpxg-17/ 2024-04-12T03:09:41.377 DEBUG:boto:Data: 2024-04-12T03:09:41.377 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.377 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.377 DEBUG:boto:Params: {} 2024-04-12T03:09:41.378 DEBUG:boto:Token: None 2024-04-12T03:09:41.378 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-17/ 2024-04-12T03:09:41.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xhKcdNN50gXBpJ01FwiTseR8AsY= 2024-04-12T03:09:41.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xhKcdNN50gXBpJ01FwiTseR8AsY=', 'Content-Length': '0'} 2024-04-12T03:09:41.393 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000efc5f9601dd9fdd4-006618a5f5-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.393 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-18 2024-04-12T03:09:41.394 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:09:41.394 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:09:41.394 DEBUG:boto:Method: PUT 2024-04-12T03:09:41.394 DEBUG:boto:Path: /dgbpxg-18/ 2024-04-12T03:09:41.394 DEBUG:boto:Data: 2024-04-12T03:09:41.394 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.394 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.394 DEBUG:boto:Params: {} 2024-04-12T03:09:41.395 DEBUG:boto:Token: None 2024-04-12T03:09:41.395 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-18/ 2024-04-12T03:09:41.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X9xGi1qz7+OBaVfMaA2QMqFkELA= 2024-04-12T03:09:41.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X9xGi1qz7+OBaVfMaA2QMqFkELA=', 'Content-Length': '0'} 2024-04-12T03:09:41.418 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ff94d95d0c59866-006618a5f5-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.418 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:09:41.418 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:09:41.418 DEBUG:boto:path=/dgbpxg-17/?versioning 2024-04-12T03:09:41.418 DEBUG:boto:auth_path=/dgbpxg-17/?versioning 2024-04-12T03:09:41.418 DEBUG:boto:Method: PUT 2024-04-12T03:09:41.418 DEBUG:boto:Path: /dgbpxg-17/?versioning 2024-04-12T03:09:41.419 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:09:41.419 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:41.419 DEBUG:boto:Port: 8000 2024-04-12T03:09:41.419 DEBUG:boto:Params: {} 2024-04-12T03:09:41.419 DEBUG:boto:Token: None 2024-04-12T03:09:41.419 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-17/?versioning 2024-04-12T03:09:41.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8ErIapU1PHB7QU0cr4X1/eqNQwg= 2024-04-12T03:09:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8ErIapU1PHB7QU0cr4X1/eqNQwg=', 'Content-Length': '228'} 2024-04-12T03:09:41.425 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dac63989acc1b0c9-006618a5f5-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.425 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:09:41.425 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:09:41.425 DEBUG:boto:path=/dgbpxg-18/?versioning 2024-04-12T03:09:41.425 DEBUG:boto:auth_path=/dgbpxg-18/?versioning 2024-04-12T03:09:41.425 DEBUG:boto:Method: PUT 2024-04-12T03:09:41.425 DEBUG:boto:Path: /dgbpxg-18/?versioning 2024-04-12T03:09:41.425 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:09:41.426 DEBUG:boto:Headers: {} 2024-04-12T03:09:41.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:41.426 DEBUG:boto:Port: 8001 2024-04-12T03:09:41.426 DEBUG:boto:Params: {} 2024-04-12T03:09:41.426 DEBUG:boto:Token: None 2024-04-12T03:09:41.426 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:09:41 GMT /dgbpxg-18/?versioning 2024-04-12T03:09:41.426 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yNOwRO3XSc/c4WDlaHpHJa5QC0A= 2024-04-12T03:09:41.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yNOwRO3XSc/c4WDlaHpHJa5QC0A=', 'Content-Length': '228'} 2024-04-12T03:09:41.434 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ea8c99a7bf519ba-006618a5f5-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:41.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:41.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:41.434 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891381.431052_310.1", 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:09:41.431052Z" 2024-04-12T03:09:41.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.527 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:09:41.527 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891381.431052_310.1'} 2024-04-12T03:09:41.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:09:41.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:41.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:41.528 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:41.611+0000 7fbed7085780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:41.612+0000 7fbed7085780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:09:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:09:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891381.383627_299.1", 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:09:41.383627Z", 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:09:41Z" 2024-04-12T03:09:41.631 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:09:41.631 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891381.383627_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:09:41.383627Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:09:41Z" } 2024-04-12T03:09:41.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891381.383627_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:09:41.383627Z', 'realm_epoch': 2}}] 2024-04-12T03:09:41.632 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891381.431052_310.1'} 2024-04-12T03:09:41.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891381.383627_299.1'} 2024-04-12T03:09:41.632 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891381.431052_310.1 target=1_1712891381.383627_299.1 2024-04-12T03:09:46.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:46.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:09:46.634 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:09:46.689 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:46.689 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:46.718+0000 7f61c0cbc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:46.720+0000 7f61c0cbc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:09:46.733 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:09:46.733 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:09:46.733 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:46.735 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:09:46.736 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:09:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891381.431052_310.1", 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:09:41.431052Z", 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:09:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:09:46Z" 2024-04-12T03:09:46.739 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:09:46.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891381.431052_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:09:41.431052Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:09:46Z" } 2024-04-12T03:09:46.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891381.431052_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:09:41.431052Z', 'realm_epoch': 2}}] 2024-04-12T03:09:46.740 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891381.431052_310.1'} 2024-04-12T03:09:46.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891381.431052_310.1'} 2024-04-12T03:09:46.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:09:46.741 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:09:46.741 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:09:46.741 DEBUG:boto:Method: HEAD 2024-04-12T03:09:46.741 DEBUG:boto:Path: /dgbpxg-17/ 2024-04-12T03:09:46.741 DEBUG:boto:Data: 2024-04-12T03:09:46.741 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:46.741 DEBUG:boto:Port: 8000 2024-04-12T03:09:46.741 DEBUG:boto:Params: {} 2024-04-12T03:09:46.742 DEBUG:boto:Token: None 2024-04-12T03:09:46.742 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-17/ 2024-04-12T03:09:46.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lQmrKZlBXqd3RnN+w7sIr3CVOn0= 2024-04-12T03:09:46.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lQmrKZlBXqd3RnN+w7sIr3CVOn0=', 'Content-Length': '0'} 2024-04-12T03:09:46.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002eb6ead51b0c9a6-006618a5fa-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.745 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:09:46.745 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:09:46.746 DEBUG:boto:Method: PUT 2024-04-12T03:09:46.746 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:09:46.746 DEBUG:boto:Data: 2024-04-12T03:09:46.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:09:46.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:46.746 DEBUG:boto:Port: 8000 2024-04-12T03:09:46.746 DEBUG:boto:Params: {} 2024-04-12T03:09:46.746 DEBUG:boto:Token: None 2024-04-12T03:09:46.746 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-17/obj 2024-04-12T03:09:46.746 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LqeLweHnWlZybSSMbbjuTDDXoqY= 2024-04-12T03:09:46.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LqeLweHnWlZybSSMbbjuTDDXoqY='} 2024-04-12T03:09:46.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-amz-request-id', 'tx00000af8fdb40dcca650f-006618a5fa-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.757 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:09:46.757 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:09:46.757 DEBUG:boto:Method: HEAD 2024-04-12T03:09:46.758 DEBUG:boto:Path: /dgbpxg-17/ 2024-04-12T03:09:46.758 DEBUG:boto:Data: 2024-04-12T03:09:46.758 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:46.758 DEBUG:boto:Port: 8000 2024-04-12T03:09:46.758 DEBUG:boto:Params: {} 2024-04-12T03:09:46.758 DEBUG:boto:Token: None 2024-04-12T03:09:46.758 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-17/ 2024-04-12T03:09:46.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lQmrKZlBXqd3RnN+w7sIr3CVOn0= 2024-04-12T03:09:46.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lQmrKZlBXqd3RnN+w7sIr3CVOn0=', 'Content-Length': '0'} 2024-04-12T03:09:46.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc6a133afe8432a1-006618a5fa-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.760 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:09:46.761 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:09:46.761 DEBUG:boto:Method: DELETE 2024-04-12T03:09:46.761 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:09:46.761 DEBUG:boto:Data: 2024-04-12T03:09:46.761 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:09:46.761 DEBUG:boto:Port: 8000 2024-04-12T03:09:46.761 DEBUG:boto:Params: {} 2024-04-12T03:09:46.761 DEBUG:boto:Token: None 2024-04-12T03:09:46.761 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-17/obj 2024-04-12T03:09:46.762 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6e7MK4tKjxSJEHgWI6vmfmZkX+c= 2024-04-12T03:09:46.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6e7MK4tKjxSJEHgWI6vmfmZkX+c=', 'Content-Length': '0'} 2024-04-12T03:09:46.767 DEBUG:boto:Response headers: [('x-amz-version-id', 'p9snhHbRnV4x5dME8jLiBSwxhoZ16mU'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000dc6f731dc0231345-006618a5fa-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.768 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:09:46.768 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:09:46.768 DEBUG:boto:Method: HEAD 2024-04-12T03:09:46.768 DEBUG:boto:Path: /dgbpxg-18/ 2024-04-12T03:09:46.768 DEBUG:boto:Data: 2024-04-12T03:09:46.768 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:46.768 DEBUG:boto:Port: 8001 2024-04-12T03:09:46.768 DEBUG:boto:Params: {} 2024-04-12T03:09:46.768 DEBUG:boto:Token: None 2024-04-12T03:09:46.769 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-18/ 2024-04-12T03:09:46.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I1cdzZlYpBnhGJI4rMTV4CwzPKw= 2024-04-12T03:09:46.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I1cdzZlYpBnhGJI4rMTV4CwzPKw=', 'Content-Length': '0'} 2024-04-12T03:09:46.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514376398300f312-006618a5fa-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.771 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:09:46.771 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:09:46.771 DEBUG:boto:Method: PUT 2024-04-12T03:09:46.771 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:09:46.771 DEBUG:boto:Data: 2024-04-12T03:09:46.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:09:46.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:46.772 DEBUG:boto:Port: 8001 2024-04-12T03:09:46.772 DEBUG:boto:Params: {} 2024-04-12T03:09:46.772 DEBUG:boto:Token: None 2024-04-12T03:09:46.772 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-18/obj 2024-04-12T03:09:46.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nNcP9lVp2eosYHjq1w2hIcY5meY= 2024-04-12T03:09:46.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nNcP9lVp2eosYHjq1w2hIcY5meY='} 2024-04-12T03:09:46.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-amz-request-id', 'tx00000de55abd21bb3d6e5-006618a5fa-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.782 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:09:46.782 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:09:46.782 DEBUG:boto:Method: HEAD 2024-04-12T03:09:46.782 DEBUG:boto:Path: /dgbpxg-18/ 2024-04-12T03:09:46.782 DEBUG:boto:Data: 2024-04-12T03:09:46.783 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:46.783 DEBUG:boto:Port: 8001 2024-04-12T03:09:46.783 DEBUG:boto:Params: {} 2024-04-12T03:09:46.783 DEBUG:boto:Token: None 2024-04-12T03:09:46.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-18/ 2024-04-12T03:09:46.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I1cdzZlYpBnhGJI4rMTV4CwzPKw= 2024-04-12T03:09:46.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I1cdzZlYpBnhGJI4rMTV4CwzPKw=', 'Content-Length': '0'} 2024-04-12T03:09:46.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb8408ee24c02282-006618a5fa-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.785 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:09:46.785 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:09:46.785 DEBUG:boto:Method: DELETE 2024-04-12T03:09:46.786 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:09:46.786 DEBUG:boto:Data: 2024-04-12T03:09:46.786 DEBUG:boto:Headers: {} 2024-04-12T03:09:46.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:09:46.786 DEBUG:boto:Port: 8001 2024-04-12T03:09:46.786 DEBUG:boto:Params: {} 2024-04-12T03:09:46.786 DEBUG:boto:Token: None 2024-04-12T03:09:46.786 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:09:46 GMT /dgbpxg-18/obj 2024-04-12T03:09:46.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w6G0dxis/zVue3kf+ozsfm3MQo8= 2024-04-12T03:09:46.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:09:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w6G0dxis/zVue3kf+ozsfm3MQo8=', 'Content-Length': '0'} 2024-04-12T03:09:46.792 DEBUG:boto:Response headers: [('x-amz-version-id', 'Smp4360u7BMuCukzL6xcnUVFxo.9GmP'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000002b68f1cd40127c14-006618a5fa-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:09:46.792 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-17 2024-04-12T03:09:46.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:46.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:46.793 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:46.848 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:46.848 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:46.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:46.879+0000 7f0014800780 1 waiting to reach incremental sync.. 2024-04-12T03:09:51.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:51.879+0000 7f0014800780 1 waiting to reach incremental sync.. 2024-04-12T03:09:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:56.880+0000 7f0014800780 1 bucket sync caught up with source: 2024-04-12T03:09:56.883 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.23.12, , ] 2024-04-12T03:09:56.883 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.23.12, , ] 2024-04-12T03:09:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:56.880+0000 7f0014800780 0 bucket checkpoint complete 2024-04-12T03:09:56.899 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-17 2024-04-12T03:09:56.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:56.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:09:56.899 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:09:56.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:09:56.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:09:56.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:09:56.991+0000 7fe7c4e15780 1 waiting to reach incremental sync.. 2024-04-12T03:10:01.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:01.992+0000 7fe7c4e15780 1 bucket sync caught up with source: 2024-04-12T03:10:01.995 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.53.6, , ] 2024-04-12T03:10:01.995 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.53.6, , ] 2024-04-12T03:10:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:01.992+0000 7fe7c4e15780 0 bucket checkpoint complete 2024-04-12T03:10:02.011 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-17 zones={a2, a1} 2024-04-12T03:10:02.011 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.011 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.011 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.011 DEBUG:boto:Path: /dgbpxg-17/ 2024-04-12T03:10:02.012 DEBUG:boto:Data: 2024-04-12T03:10:02.012 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.012 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.012 DEBUG:boto:Params: {} 2024-04-12T03:10:02.012 DEBUG:boto:Token: None 2024-04-12T03:10:02.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/ 2024-04-12T03:10:02.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj2Rm0Z6Sq/LLkCoj7sidJ3QPqU= 2024-04-12T03:10:02.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj2Rm0Z6Sq/LLkCoj7sidJ3QPqU=', 'Content-Length': '0'} 2024-04-12T03:10:02.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074fc49feeacca0f3-006618a60a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.017 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.017 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.017 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.017 DEBUG:boto:Path: /dgbpxg-17/ 2024-04-12T03:10:02.017 DEBUG:boto:Data: 2024-04-12T03:10:02.017 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.018 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.018 DEBUG:boto:Params: {} 2024-04-12T03:10:02.018 DEBUG:boto:Token: None 2024-04-12T03:10:02.018 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/ 2024-04-12T03:10:02.018 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj2Rm0Z6Sq/LLkCoj7sidJ3QPqU= 2024-04-12T03:10:02.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj2Rm0Z6Sq/LLkCoj7sidJ3QPqU=', 'Content-Length': '0'} 2024-04-12T03:10:02.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049394b277c06419f-006618a60a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:10:02.021 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.021 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.021 DEBUG:boto:path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.021 DEBUG:boto:auth_path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.022 DEBUG:boto:Method: GET 2024-04-12T03:10:02.022 DEBUG:boto:Path: /dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.022 DEBUG:boto:Data: 2024-04-12T03:10:02.022 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.022 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.022 DEBUG:boto:Params: {} 2024-04-12T03:10:02.022 DEBUG:boto:Token: None 2024-04-12T03:10:02.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/?versions 2024-04-12T03:10:02.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks= 2024-04-12T03:10:02.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks=', 'Content-Length': '0'} 2024-04-12T03:10:02.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e27dee2deb87b8-006618a60a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.066 DEBUG:boto:b'dgbpxg-17999falseobjp9snhHbRnV4x5dME8jLiBSwxhoZ16mUtrue2024-04-12T03:09:46.763Zrgw-multisite-test-userMultisite Test UserNormalobj9cm2wv7un9L9uMHXneaI6JBcB9ZBPqpfalse2024-04-12T03:09:46.748Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:02.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:02.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:02.068 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:10:02.068 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.068 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.068 DEBUG:boto:path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.068 DEBUG:boto:auth_path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.068 DEBUG:boto:Method: GET 2024-04-12T03:10:02.068 DEBUG:boto:Path: /dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.068 DEBUG:boto:Data: 2024-04-12T03:10:02.068 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.068 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.069 DEBUG:boto:Params: {} 2024-04-12T03:10:02.069 DEBUG:boto:Token: None 2024-04-12T03:10:02.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/?versions 2024-04-12T03:10:02.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks= 2024-04-12T03:10:02.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks=', 'Content-Length': '0'} 2024-04-12T03:10:02.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3991d954468d320-006618a60a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.072 DEBUG:boto:b'dgbpxg-17999falseobjp9snhHbRnV4x5dME8jLiBSwxhoZ16mUtrue2024-04-12T03:09:46.763Zrgw-multisite-test-userMultisite Test UserNormalobj9cm2wv7un9L9uMHXneaI6JBcB9ZBPqpfalse2024-04-12T03:09:46.748Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:02.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:02.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:02.073 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.073 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.073 DEBUG:boto:path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.073 DEBUG:boto:auth_path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.073 DEBUG:boto:Method: GET 2024-04-12T03:10:02.073 DEBUG:boto:Path: /dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.073 DEBUG:boto:Data: 2024-04-12T03:10:02.073 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.073 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.073 DEBUG:boto:Params: {} 2024-04-12T03:10:02.074 DEBUG:boto:Token: None 2024-04-12T03:10:02.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/?versions 2024-04-12T03:10:02.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks= 2024-04-12T03:10:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks=', 'Content-Length': '0'} 2024-04-12T03:10:02.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf6a8d8b7b75d47-006618a60a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.076 DEBUG:boto:b'dgbpxg-17999falseobjp9snhHbRnV4x5dME8jLiBSwxhoZ16mUtrue2024-04-12T03:09:46.763Zrgw-multisite-test-userMultisite Test UserNormalobj9cm2wv7un9L9uMHXneaI6JBcB9ZBPqpfalse2024-04-12T03:09:46.748Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:02.077 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.077 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.077 DEBUG:boto:path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.077 DEBUG:boto:auth_path=/dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.077 DEBUG:boto:Method: GET 2024-04-12T03:10:02.077 DEBUG:boto:Path: /dgbpxg-17/?versions&max-keys=999 2024-04-12T03:10:02.077 DEBUG:boto:Data: 2024-04-12T03:10:02.077 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.078 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.078 DEBUG:boto:Params: {} 2024-04-12T03:10:02.078 DEBUG:boto:Token: None 2024-04-12T03:10:02.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/?versions 2024-04-12T03:10:02.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks= 2024-04-12T03:10:02.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7dz63ZAbPrpq4o4DGAFQulWOks=', 'Content-Length': '0'} 2024-04-12T03:10:02.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007560e9f205bc3698-006618a60a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.081 DEBUG:boto:b'dgbpxg-17999falseobjp9snhHbRnV4x5dME8jLiBSwxhoZ16mUtrue2024-04-12T03:09:46.763Zrgw-multisite-test-userMultisite Test UserNormalobj9cm2wv7un9L9uMHXneaI6JBcB9ZBPqpfalse2024-04-12T03:09:46.748Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:02.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:02.082 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.082 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.082 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.082 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:10:02.082 DEBUG:boto:Data: 2024-04-12T03:10:02.083 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.083 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.083 DEBUG:boto:Params: {} 2024-04-12T03:10:02.083 DEBUG:boto:Token: None 2024-04-12T03:10:02.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj 2024-04-12T03:10:02.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk= 2024-04-12T03:10:02.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk=', 'Content-Length': '0'} 2024-04-12T03:10:02.086 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000094d7e5bb80f9beea-006618a60a-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.086 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.086 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.086 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.086 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:10:02.086 DEBUG:boto:Data: 2024-04-12T03:10:02.087 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.087 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.087 DEBUG:boto:Params: {} 2024-04-12T03:10:02.087 DEBUG:boto:Token: None 2024-04-12T03:10:02.087 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj 2024-04-12T03:10:02.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk= 2024-04-12T03:10:02.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk=', 'Content-Length': '0'} 2024-04-12T03:10:02.089 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005c02a51e7343bb4a-006618a60a-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:02.090 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.090 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.090 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.090 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.090 DEBUG:boto:Method: GET 2024-04-12T03:10:02.090 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.090 DEBUG:boto:Data: 2024-04-12T03:10:02.090 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.091 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.091 DEBUG:boto:Params: {} 2024-04-12T03:10:02.091 DEBUG:boto:Token: None 2024-04-12T03:10:02.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.091 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U= 2024-04-12T03:10:02.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U=', 'Content-Length': '0'} 2024-04-12T03:10:02.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5b1fb61d9193c03-006618a60a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.094 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.094 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.094 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.094 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.094 DEBUG:boto:Method: GET 2024-04-12T03:10:02.095 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.095 DEBUG:boto:Data: 2024-04-12T03:10:02.095 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.095 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.095 DEBUG:boto:Params: {} 2024-04-12T03:10:02.096 DEBUG:boto:Token: None 2024-04-12T03:10:02.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U= 2024-04-12T03:10:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U=', 'Content-Length': '0'} 2024-04-12T03:10:02.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bb0decd0323ac4f-006618a60a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.098 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.098 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.098 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.099 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.099 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.099 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.099 DEBUG:boto:Data: 2024-04-12T03:10:02.099 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.099 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.099 DEBUG:boto:Params: {} 2024-04-12T03:10:02.099 DEBUG:boto:Token: None 2024-04-12T03:10:02.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vHnbB7AUrKTIsEVGxImAnIi8fc= 2024-04-12T03:10:02.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vHnbB7AUrKTIsEVGxImAnIi8fc=', 'Content-Length': '0'} 2024-04-12T03:10:02.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fec268df9fe0c834-006618a60a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.102 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.102 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.102 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.102 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.102 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.102 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.102 DEBUG:boto:Data: 2024-04-12T03:10:02.102 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.102 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.103 DEBUG:boto:Params: {} 2024-04-12T03:10:02.103 DEBUG:boto:Token: None 2024-04-12T03:10:02.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vHnbB7AUrKTIsEVGxImAnIi8fc= 2024-04-12T03:10:02.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vHnbB7AUrKTIsEVGxImAnIi8fc=', 'Content-Length': '0'} 2024-04-12T03:10:02.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017840af45dc6a1f2-006618a60a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:02.106 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.107 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.107 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.107 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.107 DEBUG:boto:Method: GET 2024-04-12T03:10:02.107 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.107 DEBUG:boto:Data: 2024-04-12T03:10:02.107 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.107 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.108 DEBUG:boto:Params: {} 2024-04-12T03:10:02.108 DEBUG:boto:Token: None 2024-04-12T03:10:02.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U= 2024-04-12T03:10:02.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U=', 'Content-Length': '0'} 2024-04-12T03:10:02.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d915e620f5326f5-006618a60a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.110 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.110 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.110 DEBUG:boto:path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.110 DEBUG:boto:auth_path=/dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.110 DEBUG:boto:Method: GET 2024-04-12T03:10:02.111 DEBUG:boto:Path: /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.111 DEBUG:boto:Data: 2024-04-12T03:10:02.111 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.111 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.111 DEBUG:boto:Params: {} 2024-04-12T03:10:02.111 DEBUG:boto:Token: None 2024-04-12T03:10:02.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj?versionId=9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp 2024-04-12T03:10:02.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U= 2024-04-12T03:10:02.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W0pjakR4FYgZ54qbwR36kAA03U=', 'Content-Length': '0'} 2024-04-12T03:10:02.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '9cm2wv7un9L9uMHXneaI6JBcB9ZBPqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acd6648e2b707985-006618a60a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.114 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.114 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.114 DEBUG:boto:path=/dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.114 DEBUG:boto:auth_path=/dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.114 DEBUG:boto:Method: GET 2024-04-12T03:10:02.114 DEBUG:boto:Path: /dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.114 DEBUG:boto:Data: 2024-04-12T03:10:02.115 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.115 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.115 DEBUG:boto:Params: {} 2024-04-12T03:10:02.115 DEBUG:boto:Token: None 2024-04-12T03:10:02.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/ 2024-04-12T03:10:02.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEHo5f6RQUEvvmQT6i27JbdkDTo= 2024-04-12T03:10:02.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEHo5f6RQUEvvmQT6i27JbdkDTo=', 'Content-Length': '0'} 2024-04-12T03:10:02.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971eb3c5e4e0afc9-006618a60a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.118 DEBUG:boto:b'dgbpxg-17obj1000false' 2024-04-12T03:10:02.118 DEBUG:boto:path=/dgbpxg-17/ 2024-04-12T03:10:02.119 DEBUG:boto:auth_path=/dgbpxg-17/ 2024-04-12T03:10:02.119 DEBUG:boto:path=/dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.119 DEBUG:boto:auth_path=/dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.119 DEBUG:boto:Method: GET 2024-04-12T03:10:02.119 DEBUG:boto:Path: /dgbpxg-17/?prefix=obj 2024-04-12T03:10:02.119 DEBUG:boto:Data: 2024-04-12T03:10:02.119 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.119 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.119 DEBUG:boto:Params: {} 2024-04-12T03:10:02.120 DEBUG:boto:Token: None 2024-04-12T03:10:02.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/ 2024-04-12T03:10:02.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEHo5f6RQUEvvmQT6i27JbdkDTo= 2024-04-12T03:10:02.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEHo5f6RQUEvvmQT6i27JbdkDTo=', 'Content-Length': '0'} 2024-04-12T03:10:02.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453a5f15e188ca51-006618a60a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.122 DEBUG:boto:b'dgbpxg-17obj1000false' 2024-04-12T03:10:02.123 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.123 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.123 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.123 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:10:02.123 DEBUG:boto:Data: 2024-04-12T03:10:02.123 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:02.124 DEBUG:boto:Port: 8001 2024-04-12T03:10:02.124 DEBUG:boto:Params: {} 2024-04-12T03:10:02.124 DEBUG:boto:Token: None 2024-04-12T03:10:02.124 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj 2024-04-12T03:10:02.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk= 2024-04-12T03:10:02.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk=', 'Content-Length': '0'} 2024-04-12T03:10:02.126 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000185c1272a80e88e1-006618a60a-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.126 DEBUG:boto:path=/dgbpxg-17/obj 2024-04-12T03:10:02.126 DEBUG:boto:auth_path=/dgbpxg-17/obj 2024-04-12T03:10:02.127 DEBUG:boto:Method: HEAD 2024-04-12T03:10:02.127 DEBUG:boto:Path: /dgbpxg-17/obj 2024-04-12T03:10:02.127 DEBUG:boto:Data: 2024-04-12T03:10:02.127 DEBUG:boto:Headers: {} 2024-04-12T03:10:02.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:02.127 DEBUG:boto:Port: 8000 2024-04-12T03:10:02.127 DEBUG:boto:Params: {} 2024-04-12T03:10:02.127 DEBUG:boto:Token: None 2024-04-12T03:10:02.127 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:02 GMT /dgbpxg-17/obj 2024-04-12T03:10:02.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk= 2024-04-12T03:10:02.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4bgmJOBOhW7a4Vtspc1Edz2bmsk=', 'Content-Length': '0'} 2024-04-12T03:10:02.130 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000aad6e54b5976a6c0-006618a60a-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:02.130 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-17 zones={a2, a1} 2024-04-12T03:10:02.130 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-18 2024-04-12T03:10:02.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:02.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:02.131 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:02.190 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:02.191 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:02.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:02.222+0000 7f394062c780 1 waiting to reach incremental sync.. 2024-04-12T03:10:07.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:07.223+0000 7f394062c780 1 waiting to reach incremental sync.. 2024-04-12T03:10:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.225+0000 7f394062c780 1 bucket sync caught up with source: 2024-04-12T03:10:12.227 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.34.6, , ] 2024-04-12T03:10:12.227 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.34.6, , ] 2024-04-12T03:10:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.225+0000 7f394062c780 0 bucket checkpoint complete 2024-04-12T03:10:12.243 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-18 2024-04-12T03:10:12.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:12.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:12.243 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:12.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:12.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:12.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.339+0000 7f9f62312780 1 bucket sync caught up with source: 2024-04-12T03:10:12.341 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.76.12, , ] 2024-04-12T03:10:12.341 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.76.12, , ] 2024-04-12T03:10:12.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.339+0000 7f9f62312780 0 bucket checkpoint complete 2024-04-12T03:10:12.354 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-18 zones={a2, a1} 2024-04-12T03:10:12.354 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.354 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.354 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.354 DEBUG:boto:Path: /dgbpxg-18/ 2024-04-12T03:10:12.354 DEBUG:boto:Data: 2024-04-12T03:10:12.354 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.355 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.355 DEBUG:boto:Params: {} 2024-04-12T03:10:12.355 DEBUG:boto:Token: None 2024-04-12T03:10:12.355 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/ 2024-04-12T03:10:12.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:htdAF5t/9Of6V4C0GS8zt+QSTBQ= 2024-04-12T03:10:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:htdAF5t/9Of6V4C0GS8zt+QSTBQ=', 'Content-Length': '0'} 2024-04-12T03:10:12.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031ec05a9351fd45a-006618a614-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.358 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.358 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.358 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.358 DEBUG:boto:Path: /dgbpxg-18/ 2024-04-12T03:10:12.358 DEBUG:boto:Data: 2024-04-12T03:10:12.358 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.359 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.359 DEBUG:boto:Params: {} 2024-04-12T03:10:12.359 DEBUG:boto:Token: None 2024-04-12T03:10:12.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/ 2024-04-12T03:10:12.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:htdAF5t/9Of6V4C0GS8zt+QSTBQ= 2024-04-12T03:10:12.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:htdAF5t/9Of6V4C0GS8zt+QSTBQ=', 'Content-Length': '0'} 2024-04-12T03:10:12.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f9c7c2136c26eda-006618a614-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:10:12.362 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.362 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.362 DEBUG:boto:path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.362 DEBUG:boto:auth_path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.362 DEBUG:boto:Method: GET 2024-04-12T03:10:12.363 DEBUG:boto:Path: /dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.363 DEBUG:boto:Data: 2024-04-12T03:10:12.363 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.363 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.363 DEBUG:boto:Params: {} 2024-04-12T03:10:12.363 DEBUG:boto:Token: None 2024-04-12T03:10:12.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/?versions 2024-04-12T03:10:12.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM= 2024-04-12T03:10:12.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM=', 'Content-Length': '0'} 2024-04-12T03:10:12.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c3310aa087e9ed-006618a614-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.366 DEBUG:boto:b'dgbpxg-18999falseobjSmp4360u7BMuCukzL6xcnUVFxo.9GmPtrue2024-04-12T03:09:46.788Zrgw-multisite-test-userMultisite Test UserNormalobj1KxI2DJZCb19MEof19gmxCmYVwHF.3Xfalse2024-04-12T03:09:46.773Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:12.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:12.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:12.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:10:12.366 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.367 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.367 DEBUG:boto:path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.367 DEBUG:boto:auth_path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.367 DEBUG:boto:Method: GET 2024-04-12T03:10:12.367 DEBUG:boto:Path: /dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.367 DEBUG:boto:Data: 2024-04-12T03:10:12.367 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.367 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.367 DEBUG:boto:Params: {} 2024-04-12T03:10:12.367 DEBUG:boto:Token: None 2024-04-12T03:10:12.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/?versions 2024-04-12T03:10:12.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM= 2024-04-12T03:10:12.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM=', 'Content-Length': '0'} 2024-04-12T03:10:12.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3124c9bc0811007-006618a614-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.370 DEBUG:boto:b'dgbpxg-18999falseobjSmp4360u7BMuCukzL6xcnUVFxo.9GmPtrue2024-04-12T03:09:46.788Zrgw-multisite-test-userMultisite Test UserNormalobj1KxI2DJZCb19MEof19gmxCmYVwHF.3Xfalse2024-04-12T03:09:46.773Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:12.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:12.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:12.371 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.371 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.371 DEBUG:boto:path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.371 DEBUG:boto:auth_path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.371 DEBUG:boto:Method: GET 2024-04-12T03:10:12.371 DEBUG:boto:Path: /dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.371 DEBUG:boto:Data: 2024-04-12T03:10:12.371 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.371 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.372 DEBUG:boto:Params: {} 2024-04-12T03:10:12.372 DEBUG:boto:Token: None 2024-04-12T03:10:12.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/?versions 2024-04-12T03:10:12.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM= 2024-04-12T03:10:12.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM=', 'Content-Length': '0'} 2024-04-12T03:10:12.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a61d210eec62b18-006618a614-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.374 DEBUG:boto:b'dgbpxg-18999falseobjSmp4360u7BMuCukzL6xcnUVFxo.9GmPtrue2024-04-12T03:09:46.788Zrgw-multisite-test-userMultisite Test UserNormalobj1KxI2DJZCb19MEof19gmxCmYVwHF.3Xfalse2024-04-12T03:09:46.773Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:12.374 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.375 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.375 DEBUG:boto:path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.375 DEBUG:boto:auth_path=/dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.375 DEBUG:boto:Method: GET 2024-04-12T03:10:12.375 DEBUG:boto:Path: /dgbpxg-18/?versions&max-keys=999 2024-04-12T03:10:12.375 DEBUG:boto:Data: 2024-04-12T03:10:12.375 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.375 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.375 DEBUG:boto:Params: {} 2024-04-12T03:10:12.376 DEBUG:boto:Token: None 2024-04-12T03:10:12.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/?versions 2024-04-12T03:10:12.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM= 2024-04-12T03:10:12.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/Rik1CsF9mCoQW8wEaN1EWJ2jM=', 'Content-Length': '0'} 2024-04-12T03:10:12.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f4fd34a066cdb6-006618a614-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.378 DEBUG:boto:b'dgbpxg-18999falseobjSmp4360u7BMuCukzL6xcnUVFxo.9GmPtrue2024-04-12T03:09:46.788Zrgw-multisite-test-userMultisite Test UserNormalobj1KxI2DJZCb19MEof19gmxCmYVwHF.3Xfalse2024-04-12T03:09:46.773Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:12.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:12.383 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.383 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.383 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.383 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:10:12.383 DEBUG:boto:Data: 2024-04-12T03:10:12.383 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.383 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.384 DEBUG:boto:Params: {} 2024-04-12T03:10:12.384 DEBUG:boto:Token: None 2024-04-12T03:10:12.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj 2024-04-12T03:10:12.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k= 2024-04-12T03:10:12.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k=', 'Content-Length': '0'} 2024-04-12T03:10:12.386 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006d35e12323c9b561-006618a614-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.386 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.386 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.386 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.386 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:10:12.386 DEBUG:boto:Data: 2024-04-12T03:10:12.386 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.386 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.387 DEBUG:boto:Params: {} 2024-04-12T03:10:12.387 DEBUG:boto:Token: None 2024-04-12T03:10:12.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj 2024-04-12T03:10:12.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k= 2024-04-12T03:10:12.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k=', 'Content-Length': '0'} 2024-04-12T03:10:12.389 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ffd19ac534cda621-006618a614-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:12.389 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.389 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.389 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.389 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.389 DEBUG:boto:Method: GET 2024-04-12T03:10:12.389 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.389 DEBUG:boto:Data: 2024-04-12T03:10:12.390 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.390 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.390 DEBUG:boto:Params: {} 2024-04-12T03:10:12.390 DEBUG:boto:Token: None 2024-04-12T03:10:12.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8= 2024-04-12T03:10:12.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8=', 'Content-Length': '0'} 2024-04-12T03:10:12.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005060f6fb5a6b703e-006618a614-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.392 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.392 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.392 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.392 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.392 DEBUG:boto:Method: GET 2024-04-12T03:10:12.392 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.393 DEBUG:boto:Data: 2024-04-12T03:10:12.393 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.393 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.393 DEBUG:boto:Params: {} 2024-04-12T03:10:12.393 DEBUG:boto:Token: None 2024-04-12T03:10:12.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8= 2024-04-12T03:10:12.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8=', 'Content-Length': '0'} 2024-04-12T03:10:12.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000706dba87a62cfd3b-006618a614-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.395 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.395 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.395 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.395 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.395 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.396 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.396 DEBUG:boto:Data: 2024-04-12T03:10:12.396 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.396 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.396 DEBUG:boto:Params: {} 2024-04-12T03:10:12.396 DEBUG:boto:Token: None 2024-04-12T03:10:12.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l87HCAgofyXcLH7DspklHrQ919A= 2024-04-12T03:10:12.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l87HCAgofyXcLH7DspklHrQ919A=', 'Content-Length': '0'} 2024-04-12T03:10:12.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071ea012b2c3f1484-006618a614-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.398 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.398 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.399 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.399 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.399 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.399 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.399 DEBUG:boto:Data: 2024-04-12T03:10:12.399 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.399 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.399 DEBUG:boto:Params: {} 2024-04-12T03:10:12.399 DEBUG:boto:Token: None 2024-04-12T03:10:12.400 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l87HCAgofyXcLH7DspklHrQ919A= 2024-04-12T03:10:12.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l87HCAgofyXcLH7DspklHrQ919A=', 'Content-Length': '0'} 2024-04-12T03:10:12.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c56a4d74767b493-006618a614-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:12.401 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.401 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.401 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.402 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.402 DEBUG:boto:Method: GET 2024-04-12T03:10:12.402 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.402 DEBUG:boto:Data: 2024-04-12T03:10:12.402 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.402 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.402 DEBUG:boto:Params: {} 2024-04-12T03:10:12.402 DEBUG:boto:Token: None 2024-04-12T03:10:12.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8= 2024-04-12T03:10:12.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8=', 'Content-Length': '0'} 2024-04-12T03:10:12.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e41f656701fac0e7-006618a614-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.405 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.405 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.405 DEBUG:boto:path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.405 DEBUG:boto:auth_path=/dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.405 DEBUG:boto:Method: GET 2024-04-12T03:10:12.405 DEBUG:boto:Path: /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.405 DEBUG:boto:Data: 2024-04-12T03:10:12.405 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.405 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.405 DEBUG:boto:Params: {} 2024-04-12T03:10:12.406 DEBUG:boto:Token: None 2024-04-12T03:10:12.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj?versionId=1KxI2DJZCb19MEof19gmxCmYVwHF.3X 2024-04-12T03:10:12.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8= 2024-04-12T03:10:12.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nrpPCJ4u3iPXQv9xy90UO3VbWh8=', 'Content-Length': '0'} 2024-04-12T03:10:12.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:09:46 GMT'), ('x-amz-version-id', '1KxI2DJZCb19MEof19gmxCmYVwHF.3X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063c1730753c98396-006618a614-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.407 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.408 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.408 DEBUG:boto:path=/dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.408 DEBUG:boto:auth_path=/dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.408 DEBUG:boto:Method: GET 2024-04-12T03:10:12.408 DEBUG:boto:Path: /dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.408 DEBUG:boto:Data: 2024-04-12T03:10:12.408 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.408 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.408 DEBUG:boto:Params: {} 2024-04-12T03:10:12.409 DEBUG:boto:Token: None 2024-04-12T03:10:12.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/ 2024-04-12T03:10:12.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OvrLroSKunwy1oRipEwSKv3QGMM= 2024-04-12T03:10:12.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OvrLroSKunwy1oRipEwSKv3QGMM=', 'Content-Length': '0'} 2024-04-12T03:10:12.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003acf8d5335515442-006618a614-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.411 DEBUG:boto:b'dgbpxg-18obj1000false' 2024-04-12T03:10:12.411 DEBUG:boto:path=/dgbpxg-18/ 2024-04-12T03:10:12.411 DEBUG:boto:auth_path=/dgbpxg-18/ 2024-04-12T03:10:12.411 DEBUG:boto:path=/dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.412 DEBUG:boto:auth_path=/dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.412 DEBUG:boto:Method: GET 2024-04-12T03:10:12.412 DEBUG:boto:Path: /dgbpxg-18/?prefix=obj 2024-04-12T03:10:12.412 DEBUG:boto:Data: 2024-04-12T03:10:12.412 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.412 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.412 DEBUG:boto:Params: {} 2024-04-12T03:10:12.412 DEBUG:boto:Token: None 2024-04-12T03:10:12.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/ 2024-04-12T03:10:12.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OvrLroSKunwy1oRipEwSKv3QGMM= 2024-04-12T03:10:12.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OvrLroSKunwy1oRipEwSKv3QGMM=', 'Content-Length': '0'} 2024-04-12T03:10:12.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd19f9c0345a963-006618a614-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.459 DEBUG:boto:b'dgbpxg-18obj1000false' 2024-04-12T03:10:12.459 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.459 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.459 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.460 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:10:12.460 DEBUG:boto:Data: 2024-04-12T03:10:12.460 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.460 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.460 DEBUG:boto:Params: {} 2024-04-12T03:10:12.460 DEBUG:boto:Token: None 2024-04-12T03:10:12.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj 2024-04-12T03:10:12.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k= 2024-04-12T03:10:12.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k=', 'Content-Length': '0'} 2024-04-12T03:10:12.462 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b7aeec3f9b779274-006618a614-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.463 DEBUG:boto:path=/dgbpxg-18/obj 2024-04-12T03:10:12.463 DEBUG:boto:auth_path=/dgbpxg-18/obj 2024-04-12T03:10:12.463 DEBUG:boto:Method: HEAD 2024-04-12T03:10:12.463 DEBUG:boto:Path: /dgbpxg-18/obj 2024-04-12T03:10:12.463 DEBUG:boto:Data: 2024-04-12T03:10:12.463 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.463 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.463 DEBUG:boto:Params: {} 2024-04-12T03:10:12.463 DEBUG:boto:Token: None 2024-04-12T03:10:12.464 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-18/obj 2024-04-12T03:10:12.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k= 2024-04-12T03:10:12.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BzAL8FtVsyhEA804koD3EpIA09k=', 'Content-Length': '0'} 2024-04-12T03:10:12.465 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008d040eadd6b1a47c-006618a614-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-18 zones={a2, a1} 2024-04-12T03:10:12.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-12T03:10:12.467 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-19 2024-04-12T03:10:12.467 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:12.467 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:12.467 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.467 DEBUG:boto:Path: /dgbpxg-19/ 2024-04-12T03:10:12.467 DEBUG:boto:Data: 2024-04-12T03:10:12.467 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.468 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.468 DEBUG:boto:Params: {} 2024-04-12T03:10:12.468 DEBUG:boto:Token: None 2024-04-12T03:10:12.468 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-19/ 2024-04-12T03:10:12.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dC94+MD2v9UV3JRfZK3fLLj0D88= 2024-04-12T03:10:12.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dC94+MD2v9UV3JRfZK3fLLj0D88=', 'Content-Length': '0'} 2024-04-12T03:10:12.480 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000804c934fc586929f-006618a614-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.480 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-20 2024-04-12T03:10:12.480 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:12.480 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:12.480 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.481 DEBUG:boto:Path: /dgbpxg-20/ 2024-04-12T03:10:12.481 DEBUG:boto:Data: 2024-04-12T03:10:12.481 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.481 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.481 DEBUG:boto:Params: {} 2024-04-12T03:10:12.481 DEBUG:boto:Token: None 2024-04-12T03:10:12.481 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-20/ 2024-04-12T03:10:12.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wx4vvS6LDtKv6Cs44iLu0wPECeM= 2024-04-12T03:10:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wx4vvS6LDtKv6Cs44iLu0wPECeM=', 'Content-Length': '0'} 2024-04-12T03:10:12.502 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c343527be4be8ad-006618a614-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.502 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:12.503 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:12.503 DEBUG:boto:path=/dgbpxg-19/?versioning 2024-04-12T03:10:12.503 DEBUG:boto:auth_path=/dgbpxg-19/?versioning 2024-04-12T03:10:12.503 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.503 DEBUG:boto:Path: /dgbpxg-19/?versioning 2024-04-12T03:10:12.503 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:12.503 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.503 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.503 DEBUG:boto:Params: {} 2024-04-12T03:10:12.503 DEBUG:boto:Token: None 2024-04-12T03:10:12.504 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-19/?versioning 2024-04-12T03:10:12.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kqLr0skosxqdCRI4oqZZgfgLUyk= 2024-04-12T03:10:12.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kqLr0skosxqdCRI4oqZZgfgLUyk=', 'Content-Length': '228'} 2024-04-12T03:10:12.508 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7a007e3d84f71fb-006618a614-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.509 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:12.509 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:12.509 DEBUG:boto:path=/dgbpxg-19/?versioning 2024-04-12T03:10:12.509 DEBUG:boto:auth_path=/dgbpxg-19/?versioning 2024-04-12T03:10:12.509 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.509 DEBUG:boto:Path: /dgbpxg-19/?versioning 2024-04-12T03:10:12.509 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:10:12.509 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:12.510 DEBUG:boto:Port: 8000 2024-04-12T03:10:12.510 DEBUG:boto:Params: {} 2024-04-12T03:10:12.510 DEBUG:boto:Token: None 2024-04-12T03:10:12.510 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-19/?versioning 2024-04-12T03:10:12.510 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kqLr0skosxqdCRI4oqZZgfgLUyk= 2024-04-12T03:10:12.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kqLr0skosxqdCRI4oqZZgfgLUyk=', 'Content-Length': '230'} 2024-04-12T03:10:12.514 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d397a087c0fd68d7-006618a614-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.514 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:12.514 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:12.514 DEBUG:boto:path=/dgbpxg-20/?versioning 2024-04-12T03:10:12.515 DEBUG:boto:auth_path=/dgbpxg-20/?versioning 2024-04-12T03:10:12.515 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.515 DEBUG:boto:Path: /dgbpxg-20/?versioning 2024-04-12T03:10:12.515 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:12.515 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.515 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.515 DEBUG:boto:Params: {} 2024-04-12T03:10:12.515 DEBUG:boto:Token: None 2024-04-12T03:10:12.515 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-20/?versioning 2024-04-12T03:10:12.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nRYVcgisN6N0y1WNjcDOReiMWuQ= 2024-04-12T03:10:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nRYVcgisN6N0y1WNjcDOReiMWuQ=', 'Content-Length': '228'} 2024-04-12T03:10:12.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8605cb3fb4ea886-006618a614-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.522 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:12.522 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:12.522 DEBUG:boto:path=/dgbpxg-20/?versioning 2024-04-12T03:10:12.522 DEBUG:boto:auth_path=/dgbpxg-20/?versioning 2024-04-12T03:10:12.522 DEBUG:boto:Method: PUT 2024-04-12T03:10:12.522 DEBUG:boto:Path: /dgbpxg-20/?versioning 2024-04-12T03:10:12.523 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:10:12.523 DEBUG:boto:Headers: {} 2024-04-12T03:10:12.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:12.523 DEBUG:boto:Port: 8001 2024-04-12T03:10:12.523 DEBUG:boto:Params: {} 2024-04-12T03:10:12.523 DEBUG:boto:Token: None 2024-04-12T03:10:12.523 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:12 GMT /dgbpxg-20/?versioning 2024-04-12T03:10:12.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nRYVcgisN6N0y1WNjcDOReiMWuQ= 2024-04-12T03:10:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nRYVcgisN6N0y1WNjcDOReiMWuQ=', 'Content-Length': '230'} 2024-04-12T03:10:12.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a4b1b5508b2c230-006618a614-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:12.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:12.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:12.530 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891412.527516_334.1", 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:10:12.527516Z" 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.627 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:10:12.628 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891412.527516_334.1'} 2024-04-12T03:10:12.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:10:12.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:12.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:12.629 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:12.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.708+0000 7fe42dd5f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:12.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:12.709+0000 7fe42dd5f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:12.722 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:10:12.722 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:10:12.722 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:10:12.722 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:12.723 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:10:12.724 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891412.505217_327.1", 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:12.505217Z", 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:12.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:10:12Z" 2024-04-12T03:10:12.728 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:10:12.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891412.505217_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:12.505217Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:10:12Z" } 2024-04-12T03:10:12.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891412.505217_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:12.505217Z', 'realm_epoch': 2}}] 2024-04-12T03:10:12.730 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891412.527516_334.1'} 2024-04-12T03:10:12.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891412.505217_327.1'} 2024-04-12T03:10:12.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891412.527516_334.1 target=1_1712891412.505217_327.1 2024-04-12T03:10:17.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:17.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:17.732 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:17.787 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:17.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:17.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:17.817+0000 7f88631f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:17.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:17.818+0000 7f88631f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:17.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:17.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891412.527516_334.1", 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:17.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:12.527516Z", 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:17.837 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:10:17Z" 2024-04-12T03:10:17.838 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:10:17.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891412.527516_334.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:12.527516Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:10:17Z" } 2024-04-12T03:10:17.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891412.527516_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:12.527516Z', 'realm_epoch': 2}}] 2024-04-12T03:10:17.839 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891412.527516_334.1'} 2024-04-12T03:10:17.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891412.527516_334.1'} 2024-04-12T03:10:17.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:10:17.839 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:17.839 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:17.839 DEBUG:boto:Method: HEAD 2024-04-12T03:10:17.840 DEBUG:boto:Path: /dgbpxg-19/ 2024-04-12T03:10:17.840 DEBUG:boto:Data: 2024-04-12T03:10:17.840 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:17.840 DEBUG:boto:Port: 8000 2024-04-12T03:10:17.840 DEBUG:boto:Params: {} 2024-04-12T03:10:17.840 DEBUG:boto:Token: None 2024-04-12T03:10:17.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-19/ 2024-04-12T03:10:17.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gsEIH3OT/1b7CBq131FxD1AeoQc= 2024-04-12T03:10:17.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gsEIH3OT/1b7CBq131FxD1AeoQc=', 'Content-Length': '0'} 2024-04-12T03:10:17.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee24a16e9535fbd9-006618a619-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.843 DEBUG:boto:path=/dgbpxg-19/obj 2024-04-12T03:10:17.843 DEBUG:boto:auth_path=/dgbpxg-19/obj 2024-04-12T03:10:17.843 DEBUG:boto:Method: PUT 2024-04-12T03:10:17.844 DEBUG:boto:Path: /dgbpxg-19/obj 2024-04-12T03:10:17.844 DEBUG:boto:Data: 2024-04-12T03:10:17.844 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:10:17.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:17.844 DEBUG:boto:Port: 8000 2024-04-12T03:10:17.844 DEBUG:boto:Params: {} 2024-04-12T03:10:17.844 DEBUG:boto:Token: None 2024-04-12T03:10:17.845 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-19/obj 2024-04-12T03:10:17.845 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAJayWluHqmMRMnVBg0GnTeH4Js= 2024-04-12T03:10:17.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAJayWluHqmMRMnVBg0GnTeH4Js='} 2024-04-12T03:10:17.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000920f2352f776105a-006618a619-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.851 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:17.851 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:17.851 DEBUG:boto:Method: HEAD 2024-04-12T03:10:17.851 DEBUG:boto:Path: /dgbpxg-19/ 2024-04-12T03:10:17.851 DEBUG:boto:Data: 2024-04-12T03:10:17.851 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:17.852 DEBUG:boto:Port: 8000 2024-04-12T03:10:17.852 DEBUG:boto:Params: {} 2024-04-12T03:10:17.852 DEBUG:boto:Token: None 2024-04-12T03:10:17.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-19/ 2024-04-12T03:10:17.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gsEIH3OT/1b7CBq131FxD1AeoQc= 2024-04-12T03:10:17.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gsEIH3OT/1b7CBq131FxD1AeoQc=', 'Content-Length': '0'} 2024-04-12T03:10:17.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d72e1e494cd7caf4-006618a619-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.854 DEBUG:boto:path=/dgbpxg-19/obj 2024-04-12T03:10:17.855 DEBUG:boto:auth_path=/dgbpxg-19/obj 2024-04-12T03:10:17.855 DEBUG:boto:Method: DELETE 2024-04-12T03:10:17.855 DEBUG:boto:Path: /dgbpxg-19/obj 2024-04-12T03:10:17.855 DEBUG:boto:Data: 2024-04-12T03:10:17.855 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:17.855 DEBUG:boto:Port: 8000 2024-04-12T03:10:17.856 DEBUG:boto:Params: {} 2024-04-12T03:10:17.856 DEBUG:boto:Token: None 2024-04-12T03:10:17.856 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-19/obj 2024-04-12T03:10:17.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUHi7c7khA15Q1ZjxOTgjFCazdo= 2024-04-12T03:10:17.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUHi7c7khA15Q1ZjxOTgjFCazdo=', 'Content-Length': '0'} 2024-04-12T03:10:17.864 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000039c948bb3467b745-006618a619-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.865 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:17.865 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:17.865 DEBUG:boto:Method: HEAD 2024-04-12T03:10:17.865 DEBUG:boto:Path: /dgbpxg-20/ 2024-04-12T03:10:17.865 DEBUG:boto:Data: 2024-04-12T03:10:17.865 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:17.865 DEBUG:boto:Port: 8001 2024-04-12T03:10:17.865 DEBUG:boto:Params: {} 2024-04-12T03:10:17.866 DEBUG:boto:Token: None 2024-04-12T03:10:17.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-20/ 2024-04-12T03:10:17.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P5/a4Yyfz0fqs8FN7tFAtbiWjsA= 2024-04-12T03:10:17.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P5/a4Yyfz0fqs8FN7tFAtbiWjsA=', 'Content-Length': '0'} 2024-04-12T03:10:17.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f5ef5cdc93d3207-006618a619-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.868 DEBUG:boto:path=/dgbpxg-20/obj 2024-04-12T03:10:17.868 DEBUG:boto:auth_path=/dgbpxg-20/obj 2024-04-12T03:10:17.868 DEBUG:boto:Method: PUT 2024-04-12T03:10:17.869 DEBUG:boto:Path: /dgbpxg-20/obj 2024-04-12T03:10:17.869 DEBUG:boto:Data: 2024-04-12T03:10:17.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:10:17.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:17.869 DEBUG:boto:Port: 8001 2024-04-12T03:10:17.869 DEBUG:boto:Params: {} 2024-04-12T03:10:17.869 DEBUG:boto:Token: None 2024-04-12T03:10:17.869 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-20/obj 2024-04-12T03:10:17.869 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UQVFMPy52FHZuFG6duQgPM7dOR8= 2024-04-12T03:10:17.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UQVFMPy52FHZuFG6duQgPM7dOR8='} 2024-04-12T03:10:17.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036d3a5faa71e2717-006618a619-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.875 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:17.875 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:17.875 DEBUG:boto:Method: HEAD 2024-04-12T03:10:17.875 DEBUG:boto:Path: /dgbpxg-20/ 2024-04-12T03:10:17.875 DEBUG:boto:Data: 2024-04-12T03:10:17.875 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:17.875 DEBUG:boto:Port: 8001 2024-04-12T03:10:17.876 DEBUG:boto:Params: {} 2024-04-12T03:10:17.876 DEBUG:boto:Token: None 2024-04-12T03:10:17.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-20/ 2024-04-12T03:10:17.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P5/a4Yyfz0fqs8FN7tFAtbiWjsA= 2024-04-12T03:10:17.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P5/a4Yyfz0fqs8FN7tFAtbiWjsA=', 'Content-Length': '0'} 2024-04-12T03:10:17.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087bf180c068d6e40-006618a619-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.878 DEBUG:boto:path=/dgbpxg-20/obj 2024-04-12T03:10:17.878 DEBUG:boto:auth_path=/dgbpxg-20/obj 2024-04-12T03:10:17.879 DEBUG:boto:Method: DELETE 2024-04-12T03:10:17.879 DEBUG:boto:Path: /dgbpxg-20/obj 2024-04-12T03:10:17.879 DEBUG:boto:Data: 2024-04-12T03:10:17.879 DEBUG:boto:Headers: {} 2024-04-12T03:10:17.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:17.879 DEBUG:boto:Port: 8001 2024-04-12T03:10:17.879 DEBUG:boto:Params: {} 2024-04-12T03:10:17.879 DEBUG:boto:Token: None 2024-04-12T03:10:17.880 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:10:17 GMT /dgbpxg-20/obj 2024-04-12T03:10:17.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xnKh04CUkf3Ng9/6OJTPN4pCdTE= 2024-04-12T03:10:17.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xnKh04CUkf3Ng9/6OJTPN4pCdTE=', 'Content-Length': '0'} 2024-04-12T03:10:17.887 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000000dc5980897ea9bbb-006618a619-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:10:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:17.888 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-19 2024-04-12T03:10:17.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:17.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:17.888 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-19 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:17.944 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:17.944 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:17.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:17.975+0000 7fc707727780 1 waiting to reach incremental sync.. 2024-04-12T03:10:22.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:22.976+0000 7fc707727780 1 waiting to reach incremental sync.. 2024-04-12T03:10:27.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:27.976+0000 7fc707727780 1 waiting to reach incremental sync.. 2024-04-12T03:10:32.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:32.977+0000 7fc707727780 1 waiting to reach incremental sync.. 2024-04-12T03:10:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:37.978+0000 7fc707727780 1 bucket sync caught up with source: 2024-04-12T03:10:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.18.7, , ] 2024-04-12T03:10:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.18.7, , ] 2024-04-12T03:10:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:37.978+0000 7fc707727780 0 bucket checkpoint complete 2024-04-12T03:10:37.996 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-19 2024-04-12T03:10:37.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:37.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:37.997 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-19 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:38.090+0000 7fe5f5ba8780 1 waiting to reach incremental sync.. 2024-04-12T03:10:43.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.092+0000 7fe5f5ba8780 1 bucket sync caught up with source: 2024-04-12T03:10:43.094 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.19.14, , ] 2024-04-12T03:10:43.094 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.19.14, , ] 2024-04-12T03:10:43.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.092+0000 7fe5f5ba8780 0 bucket checkpoint complete 2024-04-12T03:10:43.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-19 zones={a2, a1} 2024-04-12T03:10:43.111 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.111 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.111 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.111 DEBUG:boto:Path: /dgbpxg-19/ 2024-04-12T03:10:43.111 DEBUG:boto:Data: 2024-04-12T03:10:43.111 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.112 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.112 DEBUG:boto:Params: {} 2024-04-12T03:10:43.112 DEBUG:boto:Token: None 2024-04-12T03:10:43.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/ 2024-04-12T03:10:43.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hTAu+TBAzJziTSQZ2OZOmoKcvGI= 2024-04-12T03:10:43.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hTAu+TBAzJziTSQZ2OZOmoKcvGI=', 'Content-Length': '0'} 2024-04-12T03:10:43.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000491be3fe67bc74a7-006618a633-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.115 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.115 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.115 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.116 DEBUG:boto:Path: /dgbpxg-19/ 2024-04-12T03:10:43.116 DEBUG:boto:Data: 2024-04-12T03:10:43.116 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.116 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.116 DEBUG:boto:Params: {} 2024-04-12T03:10:43.116 DEBUG:boto:Token: None 2024-04-12T03:10:43.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/ 2024-04-12T03:10:43.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hTAu+TBAzJziTSQZ2OZOmoKcvGI= 2024-04-12T03:10:43.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hTAu+TBAzJziTSQZ2OZOmoKcvGI=', 'Content-Length': '0'} 2024-04-12T03:10:43.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a76e0714b370425a-006618a633-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.119 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:10:43.119 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.119 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.119 DEBUG:boto:path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.119 DEBUG:boto:auth_path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.119 DEBUG:boto:Method: GET 2024-04-12T03:10:43.120 DEBUG:boto:Path: /dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.120 DEBUG:boto:Data: 2024-04-12T03:10:43.120 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.120 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.120 DEBUG:boto:Params: {} 2024-04-12T03:10:43.120 DEBUG:boto:Token: None 2024-04-12T03:10:43.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/?versions 2024-04-12T03:10:43.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0= 2024-04-12T03:10:43.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0=', 'Content-Length': '0'} 2024-04-12T03:10:43.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525af56fd9e49d72-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.123 DEBUG:boto:b'dgbpxg-19999falseobjnulltrue2024-04-12T03:10:17.857Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:43.123 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:10:43.124 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.124 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.124 DEBUG:boto:path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.124 DEBUG:boto:auth_path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.124 DEBUG:boto:Method: GET 2024-04-12T03:10:43.124 DEBUG:boto:Path: /dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.124 DEBUG:boto:Data: 2024-04-12T03:10:43.124 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.124 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.124 DEBUG:boto:Params: {} 2024-04-12T03:10:43.125 DEBUG:boto:Token: None 2024-04-12T03:10:43.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/?versions 2024-04-12T03:10:43.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0= 2024-04-12T03:10:43.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0=', 'Content-Length': '0'} 2024-04-12T03:10:43.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9730c40da8a8f62-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.127 DEBUG:boto:b'dgbpxg-19999falseobjnulltrue2024-04-12T03:10:17.857Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:43.128 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.128 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.128 DEBUG:boto:path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.128 DEBUG:boto:auth_path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.128 DEBUG:boto:Method: GET 2024-04-12T03:10:43.128 DEBUG:boto:Path: /dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.128 DEBUG:boto:Data: 2024-04-12T03:10:43.128 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.128 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.128 DEBUG:boto:Params: {} 2024-04-12T03:10:43.129 DEBUG:boto:Token: None 2024-04-12T03:10:43.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/?versions 2024-04-12T03:10:43.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0= 2024-04-12T03:10:43.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0=', 'Content-Length': '0'} 2024-04-12T03:10:43.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd09859b045f08d-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.174 DEBUG:boto:b'dgbpxg-19999falseobjnulltrue2024-04-12T03:10:17.857Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.175 DEBUG:boto:path=/dgbpxg-19/ 2024-04-12T03:10:43.175 DEBUG:boto:auth_path=/dgbpxg-19/ 2024-04-12T03:10:43.175 DEBUG:boto:path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.175 DEBUG:boto:auth_path=/dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.175 DEBUG:boto:Method: GET 2024-04-12T03:10:43.175 DEBUG:boto:Path: /dgbpxg-19/?versions&max-keys=999 2024-04-12T03:10:43.175 DEBUG:boto:Data: 2024-04-12T03:10:43.175 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.176 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.176 DEBUG:boto:Params: {} 2024-04-12T03:10:43.176 DEBUG:boto:Token: None 2024-04-12T03:10:43.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/?versions 2024-04-12T03:10:43.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0= 2024-04-12T03:10:43.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pUqxLJ1vhYHUonEZ8ix2LHqYDG0=', 'Content-Length': '0'} 2024-04-12T03:10:43.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a711184f0a642c64-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.179 DEBUG:boto:b'dgbpxg-19999falseobjnulltrue2024-04-12T03:10:17.857Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:43.179 DEBUG:boto:path=/dgbpxg-19/obj 2024-04-12T03:10:43.179 DEBUG:boto:auth_path=/dgbpxg-19/obj 2024-04-12T03:10:43.180 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.180 DEBUG:boto:Path: /dgbpxg-19/obj 2024-04-12T03:10:43.180 DEBUG:boto:Data: 2024-04-12T03:10:43.180 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.180 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.180 DEBUG:boto:Params: {} 2024-04-12T03:10:43.180 DEBUG:boto:Token: None 2024-04-12T03:10:43.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/obj 2024-04-12T03:10:43.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+FyC+MEhLVzzzzQ6E/qrt3oxr4= 2024-04-12T03:10:43.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+FyC+MEhLVzzzzQ6E/qrt3oxr4=', 'Content-Length': '0'} 2024-04-12T03:10:43.182 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000096a30531f6ca7c82-006618a633-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.182 DEBUG:boto:path=/dgbpxg-19/obj 2024-04-12T03:10:43.183 DEBUG:boto:auth_path=/dgbpxg-19/obj 2024-04-12T03:10:43.183 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.183 DEBUG:boto:Path: /dgbpxg-19/obj 2024-04-12T03:10:43.183 DEBUG:boto:Data: 2024-04-12T03:10:43.183 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.183 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.183 DEBUG:boto:Params: {} 2024-04-12T03:10:43.183 DEBUG:boto:Token: None 2024-04-12T03:10:43.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-19/obj 2024-04-12T03:10:43.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+FyC+MEhLVzzzzQ6E/qrt3oxr4= 2024-04-12T03:10:43.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+FyC+MEhLVzzzzQ6E/qrt3oxr4=', 'Content-Length': '0'} 2024-04-12T03:10:43.185 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004d0e1496fa9c8e48-006618a633-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-19 zones={a2, a1} 2024-04-12T03:10:43.186 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-20 2024-04-12T03:10:43.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:43.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:43.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-20 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:43.249 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:43.249 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.283+0000 7fe13faad780 1 bucket sync caught up with source: 2024-04-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.4.14, , ] 2024-04-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.4.14, , ] 2024-04-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.283+0000 7fe13faad780 0 bucket checkpoint complete 2024-04-12T03:10:43.299 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-20 2024-04-12T03:10:43.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:43.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:10:43.299 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-20 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:10:43.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:43.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:43.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.386+0000 7f33c7a25780 1 bucket sync caught up with source: 2024-04-12T03:10:43.389 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.15.7, , ] 2024-04-12T03:10:43.389 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.7, , ] 2024-04-12T03:10:43.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:43.386+0000 7f33c7a25780 0 bucket checkpoint complete 2024-04-12T03:10:43.401 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-20 zones={a2, a1} 2024-04-12T03:10:43.401 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.401 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.401 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.401 DEBUG:boto:Path: /dgbpxg-20/ 2024-04-12T03:10:43.401 DEBUG:boto:Data: 2024-04-12T03:10:43.401 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.401 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.402 DEBUG:boto:Params: {} 2024-04-12T03:10:43.402 DEBUG:boto:Token: None 2024-04-12T03:10:43.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/ 2024-04-12T03:10:43.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/NwqPNjFR3JpgPods0WMQfQF348= 2024-04-12T03:10:43.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/NwqPNjFR3JpgPods0WMQfQF348=', 'Content-Length': '0'} 2024-04-12T03:10:43.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fce0587de62f60e-006618a633-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.404 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.404 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.404 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.404 DEBUG:boto:Path: /dgbpxg-20/ 2024-04-12T03:10:43.405 DEBUG:boto:Data: 2024-04-12T03:10:43.405 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.405 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.405 DEBUG:boto:Params: {} 2024-04-12T03:10:43.405 DEBUG:boto:Token: None 2024-04-12T03:10:43.405 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/ 2024-04-12T03:10:43.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/NwqPNjFR3JpgPods0WMQfQF348= 2024-04-12T03:10:43.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/NwqPNjFR3JpgPods0WMQfQF348=', 'Content-Length': '0'} 2024-04-12T03:10:43.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aed50c300ed4d665-006618a633-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:10:43.407 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.407 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.408 DEBUG:boto:path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.408 DEBUG:boto:auth_path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.409 DEBUG:boto:Method: GET 2024-04-12T03:10:43.409 DEBUG:boto:Path: /dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.409 DEBUG:boto:Data: 2024-04-12T03:10:43.409 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.409 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.410 DEBUG:boto:Params: {} 2024-04-12T03:10:43.410 DEBUG:boto:Token: None 2024-04-12T03:10:43.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/?versions 2024-04-12T03:10:43.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4= 2024-04-12T03:10:43.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4=', 'Content-Length': '0'} 2024-04-12T03:10:43.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061dcfb19de604e7e-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.412 DEBUG:boto:b'dgbpxg-20999falseobjnulltrue2024-04-12T03:10:17.881Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:43.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:10:43.413 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.413 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.413 DEBUG:boto:path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.413 DEBUG:boto:auth_path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.413 DEBUG:boto:Method: GET 2024-04-12T03:10:43.413 DEBUG:boto:Path: /dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.413 DEBUG:boto:Data: 2024-04-12T03:10:43.413 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.414 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.414 DEBUG:boto:Params: {} 2024-04-12T03:10:43.414 DEBUG:boto:Token: None 2024-04-12T03:10:43.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/?versions 2024-04-12T03:10:43.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4= 2024-04-12T03:10:43.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4=', 'Content-Length': '0'} 2024-04-12T03:10:43.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b73fa8d68f4e95-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.458 DEBUG:boto:b'dgbpxg-20999falseobjnulltrue2024-04-12T03:10:17.881Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:10:43.459 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.459 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.459 DEBUG:boto:path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.459 DEBUG:boto:auth_path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.459 DEBUG:boto:Method: GET 2024-04-12T03:10:43.459 DEBUG:boto:Path: /dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.460 DEBUG:boto:Data: 2024-04-12T03:10:43.460 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.460 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.460 DEBUG:boto:Params: {} 2024-04-12T03:10:43.460 DEBUG:boto:Token: None 2024-04-12T03:10:43.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/?versions 2024-04-12T03:10:43.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4= 2024-04-12T03:10:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4=', 'Content-Length': '0'} 2024-04-12T03:10:43.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a7e4ed4cb13135-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.463 DEBUG:boto:b'dgbpxg-20999falseobjnulltrue2024-04-12T03:10:17.881Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.463 DEBUG:boto:path=/dgbpxg-20/ 2024-04-12T03:10:43.463 DEBUG:boto:auth_path=/dgbpxg-20/ 2024-04-12T03:10:43.463 DEBUG:boto:path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.463 DEBUG:boto:auth_path=/dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.464 DEBUG:boto:Method: GET 2024-04-12T03:10:43.464 DEBUG:boto:Path: /dgbpxg-20/?versions&max-keys=999 2024-04-12T03:10:43.464 DEBUG:boto:Data: 2024-04-12T03:10:43.464 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.464 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.464 DEBUG:boto:Params: {} 2024-04-12T03:10:43.464 DEBUG:boto:Token: None 2024-04-12T03:10:43.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/?versions 2024-04-12T03:10:43.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4= 2024-04-12T03:10:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0cs1uBLxe2+0uZWbbRD/MyoNv4=', 'Content-Length': '0'} 2024-04-12T03:10:43.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687c1675b35e51e1-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.466 DEBUG:boto:b'dgbpxg-20999falseobjnulltrue2024-04-12T03:10:17.881Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:10:43.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:10:43.467 DEBUG:boto:path=/dgbpxg-20/obj 2024-04-12T03:10:43.467 DEBUG:boto:auth_path=/dgbpxg-20/obj 2024-04-12T03:10:43.467 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.467 DEBUG:boto:Path: /dgbpxg-20/obj 2024-04-12T03:10:43.467 DEBUG:boto:Data: 2024-04-12T03:10:43.467 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.468 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.468 DEBUG:boto:Params: {} 2024-04-12T03:10:43.468 DEBUG:boto:Token: None 2024-04-12T03:10:43.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/obj 2024-04-12T03:10:43.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UgUIjbjapE9xErR/qHm7cojqEqQ= 2024-04-12T03:10:43.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UgUIjbjapE9xErR/qHm7cojqEqQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.469 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000517f4c08b2b5ae25-006618a633-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.470 DEBUG:boto:path=/dgbpxg-20/obj 2024-04-12T03:10:43.470 DEBUG:boto:auth_path=/dgbpxg-20/obj 2024-04-12T03:10:43.470 DEBUG:boto:Method: HEAD 2024-04-12T03:10:43.470 DEBUG:boto:Path: /dgbpxg-20/obj 2024-04-12T03:10:43.470 DEBUG:boto:Data: 2024-04-12T03:10:43.470 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.470 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.470 DEBUG:boto:Params: {} 2024-04-12T03:10:43.471 DEBUG:boto:Token: None 2024-04-12T03:10:43.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-20/obj 2024-04-12T03:10:43.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UgUIjbjapE9xErR/qHm7cojqEqQ= 2024-04-12T03:10:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UgUIjbjapE9xErR/qHm7cojqEqQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.472 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bf2123b234db7a56-006618a633-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-20 zones={a2, a1} 2024-04-12T03:10:43.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-12T03:10:43.473 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-21 2024-04-12T03:10:43.473 DEBUG:boto:path=/dgbpxg-21/ 2024-04-12T03:10:43.474 DEBUG:boto:auth_path=/dgbpxg-21/ 2024-04-12T03:10:43.474 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.474 DEBUG:boto:Path: /dgbpxg-21/ 2024-04-12T03:10:43.474 DEBUG:boto:Data: 2024-04-12T03:10:43.474 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.474 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.474 DEBUG:boto:Params: {} 2024-04-12T03:10:43.474 DEBUG:boto:Token: None 2024-04-12T03:10:43.475 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-21/ 2024-04-12T03:10:43.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q6Had51HwR8NQzR5PKnDq6uWZ8A= 2024-04-12T03:10:43.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q6Had51HwR8NQzR5PKnDq6uWZ8A=', 'Content-Length': '0'} 2024-04-12T03:10:43.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c2ab28b1db56dc7-006618a633-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.487 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-22 2024-04-12T03:10:43.487 DEBUG:boto:path=/dgbpxg-22/ 2024-04-12T03:10:43.487 DEBUG:boto:auth_path=/dgbpxg-22/ 2024-04-12T03:10:43.487 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.487 DEBUG:boto:Path: /dgbpxg-22/ 2024-04-12T03:10:43.488 DEBUG:boto:Data: 2024-04-12T03:10:43.488 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.488 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.488 DEBUG:boto:Params: {} 2024-04-12T03:10:43.488 DEBUG:boto:Token: None 2024-04-12T03:10:43.488 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-22/ 2024-04-12T03:10:43.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WCxXt/hou7PcxWBi47dId9ABBM0= 2024-04-12T03:10:43.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WCxXt/hou7PcxWBi47dId9ABBM0=', 'Content-Length': '0'} 2024-04-12T03:10:43.508 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cd5651be86de380-006618a633-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.509 DEBUG:boto:Using access key provided by client. 2024-04-12T03:10:43.509 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:10:43.510 DEBUG:boto:Using access key provided by client. 2024-04-12T03:10:43.510 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:10:43.510 DEBUG:boto:Using access key provided by client. 2024-04-12T03:10:43.510 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:10:43.511 DEBUG:boto:Using access key provided by client. 2024-04-12T03:10:43.511 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:10:43.511 INFO:rgw_multi.tests:create bucket zone=b1 name=dgbpxg-23 2024-04-12T03:10:43.511 DEBUG:boto:path=/dgbpxg-23/ 2024-04-12T03:10:43.511 DEBUG:boto:auth_path=/dgbpxg-23/ 2024-04-12T03:10:43.511 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.511 DEBUG:boto:Path: /dgbpxg-23/ 2024-04-12T03:10:43.511 DEBUG:boto:Data: 2024-04-12T03:10:43.512 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.512 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.512 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.512 DEBUG:boto:Params: {} 2024-04-12T03:10:43.512 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:10:43.512 DEBUG:boto:Token: None 2024-04-12T03:10:43.512 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-23/ 2024-04-12T03:10:43.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gESn1kPVaksNMGC2NHZEmOicqdo= 2024-04-12T03:10:43.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gESn1kPVaksNMGC2NHZEmOicqdo=', 'Content-Length': '0'} 2024-04-12T03:10:43.573 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003690296f8ae0c3fe-006618a633-10fd-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.574 INFO:rgw_multi.tests:create bucket zone=b2 name=dgbpxg-24 2024-04-12T03:10:43.574 DEBUG:boto:path=/dgbpxg-24/ 2024-04-12T03:10:43.574 DEBUG:boto:auth_path=/dgbpxg-24/ 2024-04-12T03:10:43.574 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.574 DEBUG:boto:Path: /dgbpxg-24/ 2024-04-12T03:10:43.574 DEBUG:boto:Data: 2024-04-12T03:10:43.574 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.574 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.575 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.575 DEBUG:boto:Params: {} 2024-04-12T03:10:43.575 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:10:43.575 DEBUG:boto:Token: None 2024-04-12T03:10:43.575 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-24/ 2024-04-12T03:10:43.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2xlu8JWB9aQnXGf9o+kmMufbkB0= 2024-04-12T03:10:43.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2xlu8JWB9aQnXGf9o+kmMufbkB0=', 'Content-Length': '0'} 2024-04-12T03:10:43.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000177899c9ff8f2c1d-006618a633-1103-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.635 DEBUG:boto:path=/dgbpxg-21/ 2024-04-12T03:10:43.635 DEBUG:boto:auth_path=/dgbpxg-21/ 2024-04-12T03:10:43.635 DEBUG:boto:path=/dgbpxg-21/?versioning 2024-04-12T03:10:43.635 DEBUG:boto:auth_path=/dgbpxg-21/?versioning 2024-04-12T03:10:43.635 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.635 DEBUG:boto:Path: /dgbpxg-21/?versioning 2024-04-12T03:10:43.635 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:43.635 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.636 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.636 DEBUG:boto:Params: {} 2024-04-12T03:10:43.636 DEBUG:boto:Token: None 2024-04-12T03:10:43.636 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-21/?versioning 2024-04-12T03:10:43.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3gR9+FrimYy+01HmvsASt2cp9i0= 2024-04-12T03:10:43.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3gR9+FrimYy+01HmvsASt2cp9i0=', 'Content-Length': '228'} 2024-04-12T03:10:43.641 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008f59e13c6e3f6f9-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.641 DEBUG:boto:path=/dgbpxg-21/ 2024-04-12T03:10:43.641 DEBUG:boto:auth_path=/dgbpxg-21/ 2024-04-12T03:10:43.642 DEBUG:boto:path=/dgbpxg-21/?versioning 2024-04-12T03:10:43.642 DEBUG:boto:auth_path=/dgbpxg-21/?versioning 2024-04-12T03:10:43.642 DEBUG:boto:Method: GET 2024-04-12T03:10:43.642 DEBUG:boto:Path: /dgbpxg-21/?versioning 2024-04-12T03:10:43.642 DEBUG:boto:Data: 2024-04-12T03:10:43.642 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.642 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.642 DEBUG:boto:Params: {} 2024-04-12T03:10:43.643 DEBUG:boto:Token: None 2024-04-12T03:10:43.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-21/?versioning 2024-04-12T03:10:43.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qmJNGGs+oJKjR33fIHhYCM3swNQ= 2024-04-12T03:10:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qmJNGGs+oJKjR33fIHhYCM3swNQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.644 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000fb2460f436280bb-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.686 DEBUG:boto:EnabledDisabled 2024-04-12T03:10:43.687 DEBUG:boto:path=/dgbpxg-22/ 2024-04-12T03:10:43.687 DEBUG:boto:auth_path=/dgbpxg-22/ 2024-04-12T03:10:43.687 DEBUG:boto:path=/dgbpxg-22/?versioning 2024-04-12T03:10:43.687 DEBUG:boto:auth_path=/dgbpxg-22/?versioning 2024-04-12T03:10:43.688 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.688 DEBUG:boto:Path: /dgbpxg-22/?versioning 2024-04-12T03:10:43.688 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:43.688 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.688 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.688 DEBUG:boto:Params: {} 2024-04-12T03:10:43.689 DEBUG:boto:Token: None 2024-04-12T03:10:43.689 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-22/?versioning 2024-04-12T03:10:43.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:arx6WWLFuQCYxXNxc5ggWJ3s+mY= 2024-04-12T03:10:43.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:arx6WWLFuQCYxXNxc5ggWJ3s+mY=', 'Content-Length': '228'} 2024-04-12T03:10:43.697 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000032b1348cd51e0f0d-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.697 DEBUG:boto:path=/dgbpxg-22/ 2024-04-12T03:10:43.697 DEBUG:boto:auth_path=/dgbpxg-22/ 2024-04-12T03:10:43.697 DEBUG:boto:path=/dgbpxg-22/?versioning 2024-04-12T03:10:43.698 DEBUG:boto:auth_path=/dgbpxg-22/?versioning 2024-04-12T03:10:43.698 DEBUG:boto:Method: GET 2024-04-12T03:10:43.698 DEBUG:boto:Path: /dgbpxg-22/?versioning 2024-04-12T03:10:43.698 DEBUG:boto:Data: 2024-04-12T03:10:43.698 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.699 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.699 DEBUG:boto:Params: {} 2024-04-12T03:10:43.699 DEBUG:boto:Token: None 2024-04-12T03:10:43.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-22/?versioning 2024-04-12T03:10:43.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QNITuZzaEdQXDU0jQtQNKT6uurc= 2024-04-12T03:10:43.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QNITuZzaEdQXDU0jQtQNKT6uurc=', 'Content-Length': '0'} 2024-04-12T03:10:43.700 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bda580328284e44-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.742 DEBUG:boto:EnabledDisabled 2024-04-12T03:10:43.743 DEBUG:boto:path=/dgbpxg-23/ 2024-04-12T03:10:43.743 DEBUG:boto:auth_path=/dgbpxg-23/ 2024-04-12T03:10:43.743 DEBUG:boto:path=/dgbpxg-23/?versioning 2024-04-12T03:10:43.743 DEBUG:boto:auth_path=/dgbpxg-23/?versioning 2024-04-12T03:10:43.743 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.743 DEBUG:boto:Path: /dgbpxg-23/?versioning 2024-04-12T03:10:43.743 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:43.743 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.743 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.744 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.744 DEBUG:boto:Params: {} 2024-04-12T03:10:43.744 DEBUG:boto:Token: None 2024-04-12T03:10:43.744 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-23/?versioning 2024-04-12T03:10:43.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CB9PqaD2Ukws7p5Vo3wgUEiaa0U= 2024-04-12T03:10:43.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CB9PqaD2Ukws7p5Vo3wgUEiaa0U=', 'Content-Length': '228'} 2024-04-12T03:10:43.753 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000952e147633ccf251-006618a633-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.753 DEBUG:boto:path=/dgbpxg-23/ 2024-04-12T03:10:43.753 DEBUG:boto:auth_path=/dgbpxg-23/ 2024-04-12T03:10:43.753 DEBUG:boto:path=/dgbpxg-23/?versioning 2024-04-12T03:10:43.753 DEBUG:boto:auth_path=/dgbpxg-23/?versioning 2024-04-12T03:10:43.753 DEBUG:boto:Method: GET 2024-04-12T03:10:43.753 DEBUG:boto:Path: /dgbpxg-23/?versioning 2024-04-12T03:10:43.753 DEBUG:boto:Data: 2024-04-12T03:10:43.753 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.753 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.754 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.754 DEBUG:boto:Params: {} 2024-04-12T03:10:43.754 DEBUG:boto:Token: None 2024-04-12T03:10:43.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-23/?versioning 2024-04-12T03:10:43.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K1jS/zgp1odA7S7WQJ48td68QRU= 2024-04-12T03:10:43.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K1jS/zgp1odA7S7WQJ48td68QRU=', 'Content-Length': '0'} 2024-04-12T03:10:43.755 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007728a383bd6d92a0-006618a633-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.755 DEBUG:boto:EnabledDisabled 2024-04-12T03:10:43.756 DEBUG:boto:path=/dgbpxg-24/ 2024-04-12T03:10:43.756 DEBUG:boto:auth_path=/dgbpxg-24/ 2024-04-12T03:10:43.756 DEBUG:boto:path=/dgbpxg-24/?versioning 2024-04-12T03:10:43.756 DEBUG:boto:auth_path=/dgbpxg-24/?versioning 2024-04-12T03:10:43.756 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.756 DEBUG:boto:Path: /dgbpxg-24/?versioning 2024-04-12T03:10:43.756 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:10:43.756 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.756 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.756 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.757 DEBUG:boto:Params: {} 2024-04-12T03:10:43.757 DEBUG:boto:Token: None 2024-04-12T03:10:43.757 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-24/?versioning 2024-04-12T03:10:43.757 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ylNzi1XZaMm2vndbmHvKKhBVnDQ= 2024-04-12T03:10:43.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ylNzi1XZaMm2vndbmHvKKhBVnDQ=', 'Content-Length': '228'} 2024-04-12T03:10:43.765 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005f094de4bb8e1b46-006618a633-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.765 DEBUG:boto:path=/dgbpxg-24/ 2024-04-12T03:10:43.765 DEBUG:boto:auth_path=/dgbpxg-24/ 2024-04-12T03:10:43.765 DEBUG:boto:path=/dgbpxg-24/?versioning 2024-04-12T03:10:43.765 DEBUG:boto:auth_path=/dgbpxg-24/?versioning 2024-04-12T03:10:43.765 DEBUG:boto:Method: GET 2024-04-12T03:10:43.766 DEBUG:boto:Path: /dgbpxg-24/?versioning 2024-04-12T03:10:43.766 DEBUG:boto:Data: 2024-04-12T03:10:43.766 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.766 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.766 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.766 DEBUG:boto:Params: {} 2024-04-12T03:10:43.766 DEBUG:boto:Token: None 2024-04-12T03:10:43.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-24/?versioning 2024-04-12T03:10:43.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxduU3haxKkbiYqf3nHInJrrmr4= 2024-04-12T03:10:43.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxduU3haxKkbiYqf3nHInJrrmr4=', 'Content-Length': '0'} 2024-04-12T03:10:43.767 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072431a96de02b218-006618a633-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.768 DEBUG:boto:EnabledDisabled 2024-04-12T03:10:43.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-12T03:10:43.768 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-25 2024-04-12T03:10:43.769 DEBUG:boto:path=/dgbpxg-25/ 2024-04-12T03:10:43.769 DEBUG:boto:auth_path=/dgbpxg-25/ 2024-04-12T03:10:43.769 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.769 DEBUG:boto:Path: /dgbpxg-25/ 2024-04-12T03:10:43.769 DEBUG:boto:Data: 2024-04-12T03:10:43.769 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.769 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.769 DEBUG:boto:Params: {} 2024-04-12T03:10:43.770 DEBUG:boto:Token: None 2024-04-12T03:10:43.770 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-25/ 2024-04-12T03:10:43.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Si3TUd0hzcvIryZHgR1JFNeg53E= 2024-04-12T03:10:43.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Si3TUd0hzcvIryZHgR1JFNeg53E=', 'Content-Length': '0'} 2024-04-12T03:10:43.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f7e1239e5c5237b6-006618a633-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.783 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-26 2024-04-12T03:10:43.783 DEBUG:boto:path=/dgbpxg-26/ 2024-04-12T03:10:43.783 DEBUG:boto:auth_path=/dgbpxg-26/ 2024-04-12T03:10:43.783 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.784 DEBUG:boto:Path: /dgbpxg-26/ 2024-04-12T03:10:43.784 DEBUG:boto:Data: 2024-04-12T03:10:43.784 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.784 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.784 DEBUG:boto:Params: {} 2024-04-12T03:10:43.784 DEBUG:boto:Token: None 2024-04-12T03:10:43.784 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-26/ 2024-04-12T03:10:43.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:29GQwI+uwa/YccZjTVB0aHMbGFc= 2024-04-12T03:10:43.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:29GQwI+uwa/YccZjTVB0aHMbGFc=', 'Content-Length': '0'} 2024-04-12T03:10:43.805 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca71921d229c7386-006618a633-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.805 INFO:rgw_multi.tests:create bucket zone=b1 name=dgbpxg-27 2024-04-12T03:10:43.806 DEBUG:boto:path=/dgbpxg-27/ 2024-04-12T03:10:43.806 DEBUG:boto:auth_path=/dgbpxg-27/ 2024-04-12T03:10:43.806 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.806 DEBUG:boto:Path: /dgbpxg-27/ 2024-04-12T03:10:43.806 DEBUG:boto:Data: 2024-04-12T03:10:43.806 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.806 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.806 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.807 DEBUG:boto:Params: {} 2024-04-12T03:10:43.807 DEBUG:boto:Token: None 2024-04-12T03:10:43.807 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-27/ 2024-04-12T03:10:43.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UExcIjAK9B4IEs/djM71Z7mS0dI= 2024-04-12T03:10:43.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UExcIjAK9B4IEs/djM71Z7mS0dI=', 'Content-Length': '0'} 2024-04-12T03:10:43.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c22ba7f477933ce-006618a633-10fd-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.834 INFO:rgw_multi.tests:create bucket zone=b2 name=dgbpxg-28 2024-04-12T03:10:43.834 DEBUG:boto:path=/dgbpxg-28/ 2024-04-12T03:10:43.834 DEBUG:boto:auth_path=/dgbpxg-28/ 2024-04-12T03:10:43.834 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.834 DEBUG:boto:Path: /dgbpxg-28/ 2024-04-12T03:10:43.835 DEBUG:boto:Data: 2024-04-12T03:10:43.835 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.835 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.835 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.835 DEBUG:boto:Params: {} 2024-04-12T03:10:43.835 DEBUG:boto:Token: None 2024-04-12T03:10:43.835 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-28/ 2024-04-12T03:10:43.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cvXe1k35Cvynfsts+cfJgi8NXRA= 2024-04-12T03:10:43.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cvXe1k35Cvynfsts+cfJgi8NXRA=', 'Content-Length': '0'} 2024-04-12T03:10:43.859 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ace2113ba82d2c38-006618a633-1103-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.860 DEBUG:boto:path=/dgbpxg-25/ 2024-04-12T03:10:43.860 DEBUG:boto:auth_path=/dgbpxg-25/ 2024-04-12T03:10:43.860 DEBUG:boto:path=/dgbpxg-25/?acl 2024-04-12T03:10:43.860 DEBUG:boto:auth_path=/dgbpxg-25/?acl 2024-04-12T03:10:43.860 DEBUG:boto:Method: GET 2024-04-12T03:10:43.860 DEBUG:boto:Path: /dgbpxg-25/?acl 2024-04-12T03:10:43.860 DEBUG:boto:Data: 2024-04-12T03:10:43.860 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.861 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.861 DEBUG:boto:Params: {} 2024-04-12T03:10:43.861 DEBUG:boto:Token: None 2024-04-12T03:10:43.861 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-25/?acl 2024-04-12T03:10:43.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QAhiAqO35MWGRKE/KEPPv0u0DgI= 2024-04-12T03:10:43.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QAhiAqO35MWGRKE/KEPPv0u0DgI=', 'Content-Length': '0'} 2024-04-12T03:10:43.862 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042db8d94aceb3599-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.863 DEBUG:boto:path=/dgbpxg-25/ 2024-04-12T03:10:43.863 DEBUG:boto:auth_path=/dgbpxg-25/ 2024-04-12T03:10:43.863 DEBUG:boto:path=/dgbpxg-25/?acl 2024-04-12T03:10:43.863 DEBUG:boto:auth_path=/dgbpxg-25/?acl 2024-04-12T03:10:43.863 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.863 DEBUG:boto:Path: /dgbpxg-25/?acl 2024-04-12T03:10:43.863 DEBUG:boto:Data: 2024-04-12T03:10:43.864 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:10:43.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.864 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.864 DEBUG:boto:Params: {} 2024-04-12T03:10:43.864 DEBUG:boto:Token: None 2024-04-12T03:10:43.864 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT x-amz-acl:public-read /dgbpxg-25/?acl 2024-04-12T03:10:43.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:chDtdys4ij4yEz0vPYL4TXgml2A= 2024-04-12T03:10:43.864 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:chDtdys4ij4yEz0vPYL4TXgml2A=', 'Content-Length': '0'} 2024-04-12T03:10:43.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4fb644e55019ef6-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.871 DEBUG:boto:path=/dgbpxg-25/ 2024-04-12T03:10:43.871 DEBUG:boto:auth_path=/dgbpxg-25/ 2024-04-12T03:10:43.871 DEBUG:boto:path=/dgbpxg-25/?acl 2024-04-12T03:10:43.871 DEBUG:boto:auth_path=/dgbpxg-25/?acl 2024-04-12T03:10:43.872 DEBUG:boto:Method: GET 2024-04-12T03:10:43.872 DEBUG:boto:Path: /dgbpxg-25/?acl 2024-04-12T03:10:43.872 DEBUG:boto:Data: 2024-04-12T03:10:43.872 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.872 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.872 DEBUG:boto:Params: {} 2024-04-12T03:10:43.872 DEBUG:boto:Token: None 2024-04-12T03:10:43.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-25/?acl 2024-04-12T03:10:43.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QAhiAqO35MWGRKE/KEPPv0u0DgI= 2024-04-12T03:10:43.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QAhiAqO35MWGRKE/KEPPv0u0DgI=', 'Content-Length': '0'} 2024-04-12T03:10:43.874 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000958f8cadb07e1940-006618a633-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.875 DEBUG:boto:path=/dgbpxg-26/ 2024-04-12T03:10:43.875 DEBUG:boto:auth_path=/dgbpxg-26/ 2024-04-12T03:10:43.875 DEBUG:boto:path=/dgbpxg-26/?acl 2024-04-12T03:10:43.875 DEBUG:boto:auth_path=/dgbpxg-26/?acl 2024-04-12T03:10:43.875 DEBUG:boto:Method: GET 2024-04-12T03:10:43.875 DEBUG:boto:Path: /dgbpxg-26/?acl 2024-04-12T03:10:43.875 DEBUG:boto:Data: 2024-04-12T03:10:43.875 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.875 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.875 DEBUG:boto:Params: {} 2024-04-12T03:10:43.876 DEBUG:boto:Token: None 2024-04-12T03:10:43.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-26/?acl 2024-04-12T03:10:43.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1qHRtGWx9IO20T8StyAf6mS9XDU= 2024-04-12T03:10:43.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1qHRtGWx9IO20T8StyAf6mS9XDU=', 'Content-Length': '0'} 2024-04-12T03:10:43.877 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5651623c98febb9-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.878 DEBUG:boto:path=/dgbpxg-26/ 2024-04-12T03:10:43.878 DEBUG:boto:auth_path=/dgbpxg-26/ 2024-04-12T03:10:43.878 DEBUG:boto:path=/dgbpxg-26/?acl 2024-04-12T03:10:43.878 DEBUG:boto:auth_path=/dgbpxg-26/?acl 2024-04-12T03:10:43.878 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.879 DEBUG:boto:Path: /dgbpxg-26/?acl 2024-04-12T03:10:43.879 DEBUG:boto:Data: 2024-04-12T03:10:43.879 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:10:43.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.879 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.879 DEBUG:boto:Params: {} 2024-04-12T03:10:43.879 DEBUG:boto:Token: None 2024-04-12T03:10:43.879 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT x-amz-acl:public-read /dgbpxg-26/?acl 2024-04-12T03:10:43.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z0DhoksNmE8o5vNQ7SNNoB433e0= 2024-04-12T03:10:43.880 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z0DhoksNmE8o5vNQ7SNNoB433e0=', 'Content-Length': '0'} 2024-04-12T03:10:43.889 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000020e4e0d6a137a83-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.889 DEBUG:boto:path=/dgbpxg-26/ 2024-04-12T03:10:43.889 DEBUG:boto:auth_path=/dgbpxg-26/ 2024-04-12T03:10:43.889 DEBUG:boto:path=/dgbpxg-26/?acl 2024-04-12T03:10:43.890 DEBUG:boto:auth_path=/dgbpxg-26/?acl 2024-04-12T03:10:43.890 DEBUG:boto:Method: GET 2024-04-12T03:10:43.890 DEBUG:boto:Path: /dgbpxg-26/?acl 2024-04-12T03:10:43.890 DEBUG:boto:Data: 2024-04-12T03:10:43.890 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.890 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.890 DEBUG:boto:Params: {} 2024-04-12T03:10:43.890 DEBUG:boto:Token: None 2024-04-12T03:10:43.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-26/?acl 2024-04-12T03:10:43.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1qHRtGWx9IO20T8StyAf6mS9XDU= 2024-04-12T03:10:43.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1qHRtGWx9IO20T8StyAf6mS9XDU=', 'Content-Length': '0'} 2024-04-12T03:10:43.892 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a83a81b0914c5a6-006618a633-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.892 DEBUG:boto:path=/dgbpxg-27/ 2024-04-12T03:10:43.892 DEBUG:boto:auth_path=/dgbpxg-27/ 2024-04-12T03:10:43.892 DEBUG:boto:path=/dgbpxg-27/?acl 2024-04-12T03:10:43.892 DEBUG:boto:auth_path=/dgbpxg-27/?acl 2024-04-12T03:10:43.892 DEBUG:boto:Method: GET 2024-04-12T03:10:43.892 DEBUG:boto:Path: /dgbpxg-27/?acl 2024-04-12T03:10:43.893 DEBUG:boto:Data: 2024-04-12T03:10:43.893 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.893 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.893 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.893 DEBUG:boto:Params: {} 2024-04-12T03:10:43.893 DEBUG:boto:Token: None 2024-04-12T03:10:43.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-27/?acl 2024-04-12T03:10:43.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbuJ5GHDF684sVLpOAqYopzkp+g= 2024-04-12T03:10:43.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbuJ5GHDF684sVLpOAqYopzkp+g=', 'Content-Length': '0'} 2024-04-12T03:10:43.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c0f21df9ab56ad05-006618a633-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.895 DEBUG:boto:path=/dgbpxg-27/ 2024-04-12T03:10:43.895 DEBUG:boto:auth_path=/dgbpxg-27/ 2024-04-12T03:10:43.895 DEBUG:boto:path=/dgbpxg-27/?acl 2024-04-12T03:10:43.895 DEBUG:boto:auth_path=/dgbpxg-27/?acl 2024-04-12T03:10:43.895 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.896 DEBUG:boto:Path: /dgbpxg-27/?acl 2024-04-12T03:10:43.896 DEBUG:boto:Data: 2024-04-12T03:10:43.896 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:10:43.896 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.896 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.896 DEBUG:boto:Params: {} 2024-04-12T03:10:43.896 DEBUG:boto:Token: None 2024-04-12T03:10:43.896 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT x-amz-acl:public-read /dgbpxg-27/?acl 2024-04-12T03:10:43.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kdglSkDf31XoChkeB8uiebNRPjQ= 2024-04-12T03:10:43.897 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kdglSkDf31XoChkeB8uiebNRPjQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.904 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c92de482a3363411-006618a633-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.904 DEBUG:boto:path=/dgbpxg-27/ 2024-04-12T03:10:43.904 DEBUG:boto:auth_path=/dgbpxg-27/ 2024-04-12T03:10:43.905 DEBUG:boto:path=/dgbpxg-27/?acl 2024-04-12T03:10:43.905 DEBUG:boto:auth_path=/dgbpxg-27/?acl 2024-04-12T03:10:43.905 DEBUG:boto:Method: GET 2024-04-12T03:10:43.905 DEBUG:boto:Path: /dgbpxg-27/?acl 2024-04-12T03:10:43.905 DEBUG:boto:Data: 2024-04-12T03:10:43.905 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.905 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:43.905 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.905 DEBUG:boto:Params: {} 2024-04-12T03:10:43.906 DEBUG:boto:Token: None 2024-04-12T03:10:43.906 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-27/?acl 2024-04-12T03:10:43.906 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbuJ5GHDF684sVLpOAqYopzkp+g= 2024-04-12T03:10:43.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbuJ5GHDF684sVLpOAqYopzkp+g=', 'Content-Length': '0'} 2024-04-12T03:10:43.907 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000459f1f040cdf512a-006618a633-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.951 DEBUG:boto:path=/dgbpxg-28/ 2024-04-12T03:10:43.951 DEBUG:boto:auth_path=/dgbpxg-28/ 2024-04-12T03:10:43.951 DEBUG:boto:path=/dgbpxg-28/?acl 2024-04-12T03:10:43.951 DEBUG:boto:auth_path=/dgbpxg-28/?acl 2024-04-12T03:10:43.951 DEBUG:boto:Method: GET 2024-04-12T03:10:43.951 DEBUG:boto:Path: /dgbpxg-28/?acl 2024-04-12T03:10:43.951 DEBUG:boto:Data: 2024-04-12T03:10:43.951 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.952 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.952 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.952 DEBUG:boto:Params: {} 2024-04-12T03:10:43.952 DEBUG:boto:Token: None 2024-04-12T03:10:43.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-28/?acl 2024-04-12T03:10:43.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hOAlSMcBtpq/jtgbDO2YAhYlJRQ= 2024-04-12T03:10:43.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hOAlSMcBtpq/jtgbDO2YAhYlJRQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000729b9134bc47d61f-006618a633-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.954 DEBUG:boto:path=/dgbpxg-28/ 2024-04-12T03:10:43.955 DEBUG:boto:auth_path=/dgbpxg-28/ 2024-04-12T03:10:43.955 DEBUG:boto:path=/dgbpxg-28/?acl 2024-04-12T03:10:43.955 DEBUG:boto:auth_path=/dgbpxg-28/?acl 2024-04-12T03:10:43.955 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.955 DEBUG:boto:Path: /dgbpxg-28/?acl 2024-04-12T03:10:43.955 DEBUG:boto:Data: 2024-04-12T03:10:43.955 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:10:43.956 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.956 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.956 DEBUG:boto:Params: {} 2024-04-12T03:10:43.956 DEBUG:boto:Token: None 2024-04-12T03:10:43.956 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT x-amz-acl:public-read /dgbpxg-28/?acl 2024-04-12T03:10:43.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f+K+NTa7Lrygjv0a2qCIRSGQm9U= 2024-04-12T03:10:43.957 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f+K+NTa7Lrygjv0a2qCIRSGQm9U=', 'Content-Length': '0'} 2024-04-12T03:10:43.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1f104e74530a521-006618a633-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.965 DEBUG:boto:path=/dgbpxg-28/ 2024-04-12T03:10:43.965 DEBUG:boto:auth_path=/dgbpxg-28/ 2024-04-12T03:10:43.965 DEBUG:boto:path=/dgbpxg-28/?acl 2024-04-12T03:10:43.965 DEBUG:boto:auth_path=/dgbpxg-28/?acl 2024-04-12T03:10:43.965 DEBUG:boto:Method: GET 2024-04-12T03:10:43.965 DEBUG:boto:Path: /dgbpxg-28/?acl 2024-04-12T03:10:43.966 DEBUG:boto:Data: 2024-04-12T03:10:43.966 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.966 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:43.966 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.966 DEBUG:boto:Params: {} 2024-04-12T03:10:43.966 DEBUG:boto:Token: None 2024-04-12T03:10:43.966 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-28/?acl 2024-04-12T03:10:43.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hOAlSMcBtpq/jtgbDO2YAhYlJRQ= 2024-04-12T03:10:43.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hOAlSMcBtpq/jtgbDO2YAhYlJRQ=', 'Content-Length': '0'} 2024-04-12T03:10:43.967 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ade6050cf915dc52-006618a633-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-12T03:10:43.968 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-29 2024-04-12T03:10:43.968 DEBUG:boto:path=/dgbpxg-29/ 2024-04-12T03:10:43.969 DEBUG:boto:auth_path=/dgbpxg-29/ 2024-04-12T03:10:43.969 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.969 DEBUG:boto:Path: /dgbpxg-29/ 2024-04-12T03:10:43.969 DEBUG:boto:Data: 2024-04-12T03:10:43.969 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:43.969 DEBUG:boto:Port: 8000 2024-04-12T03:10:43.969 DEBUG:boto:Params: {} 2024-04-12T03:10:43.969 DEBUG:boto:Token: None 2024-04-12T03:10:43.969 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-29/ 2024-04-12T03:10:43.970 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1UGzuyG6hiDplOA1S+Q4mrQDm/o= 2024-04-12T03:10:43.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1UGzuyG6hiDplOA1S+Q4mrQDm/o=', 'Content-Length': '0'} 2024-04-12T03:10:43.982 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a5e63506c1c4a0f-006618a633-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:43.982 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-30 2024-04-12T03:10:43.982 DEBUG:boto:path=/dgbpxg-30/ 2024-04-12T03:10:43.982 DEBUG:boto:auth_path=/dgbpxg-30/ 2024-04-12T03:10:43.982 DEBUG:boto:Method: PUT 2024-04-12T03:10:43.982 DEBUG:boto:Path: /dgbpxg-30/ 2024-04-12T03:10:43.982 DEBUG:boto:Data: 2024-04-12T03:10:43.983 DEBUG:boto:Headers: {} 2024-04-12T03:10:43.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:43.983 DEBUG:boto:Port: 8001 2024-04-12T03:10:43.983 DEBUG:boto:Params: {} 2024-04-12T03:10:43.983 DEBUG:boto:Token: None 2024-04-12T03:10:43.983 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:43 GMT /dgbpxg-30/ 2024-04-12T03:10:43.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tZKvLYpjQx0oORwKfXSKEtXm1nE= 2024-04-12T03:10:43.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tZKvLYpjQx0oORwKfXSKEtXm1nE=', 'Content-Length': '0'} 2024-04-12T03:10:44.004 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2c0b519e863be1f-006618a633-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:44.005 INFO:rgw_multi.tests:create bucket zone=b1 name=dgbpxg-31 2024-04-12T03:10:44.005 DEBUG:boto:path=/dgbpxg-31/ 2024-04-12T03:10:44.005 DEBUG:boto:auth_path=/dgbpxg-31/ 2024-04-12T03:10:44.005 DEBUG:boto:Method: PUT 2024-04-12T03:10:44.005 DEBUG:boto:Path: /dgbpxg-31/ 2024-04-12T03:10:44.005 DEBUG:boto:Data: 2024-04-12T03:10:44.005 DEBUG:boto:Headers: {} 2024-04-12T03:10:44.005 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:44.005 DEBUG:boto:Port: 8000 2024-04-12T03:10:44.005 DEBUG:boto:Params: {} 2024-04-12T03:10:44.006 DEBUG:boto:Token: None 2024-04-12T03:10:44.006 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:44 GMT /dgbpxg-31/ 2024-04-12T03:10:44.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ocKOUyJMIfz7UQO1lZ1pv6gnglQ= 2024-04-12T03:10:44.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ocKOUyJMIfz7UQO1lZ1pv6gnglQ=', 'Content-Length': '0'} 2024-04-12T03:10:44.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005bae89e4d1056fd5-006618a634-10fd-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:44.031 INFO:rgw_multi.tests:create bucket zone=b2 name=dgbpxg-32 2024-04-12T03:10:44.031 DEBUG:boto:path=/dgbpxg-32/ 2024-04-12T03:10:44.031 DEBUG:boto:auth_path=/dgbpxg-32/ 2024-04-12T03:10:44.031 DEBUG:boto:Method: PUT 2024-04-12T03:10:44.032 DEBUG:boto:Path: /dgbpxg-32/ 2024-04-12T03:10:44.032 DEBUG:boto:Data: 2024-04-12T03:10:44.032 DEBUG:boto:Headers: {} 2024-04-12T03:10:44.032 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:44.032 DEBUG:boto:Port: 8001 2024-04-12T03:10:44.032 DEBUG:boto:Params: {} 2024-04-12T03:10:44.032 DEBUG:boto:Token: None 2024-04-12T03:10:44.032 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:44 GMT /dgbpxg-32/ 2024-04-12T03:10:44.033 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LemaEaIxcKDNxhaBrsHilZl6yKg= 2024-04-12T03:10:44.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LemaEaIxcKDNxhaBrsHilZl6yKg=', 'Content-Length': '0'} 2024-04-12T03:10:44.056 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000426a0a3e75e5faa3-006618a634-1103-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:44.056 DEBUG:boto:path=/dgbpxg-29/ 2024-04-12T03:10:44.056 DEBUG:boto:auth_path=/dgbpxg-29/ 2024-04-12T03:10:44.057 DEBUG:boto:path=/dgbpxg-29/?cors 2024-04-12T03:10:44.057 DEBUG:boto:auth_path=/dgbpxg-29/?cors 2024-04-12T03:10:44.057 DEBUG:boto:Method: PUT 2024-04-12T03:10:44.057 DEBUG:boto:Path: /dgbpxg-29/?cors 2024-04-12T03:10:44.057 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:10:44.057 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:10:44.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:44.058 DEBUG:boto:Port: 8000 2024-04-12T03:10:44.058 DEBUG:boto:Params: {} 2024-04-12T03:10:44.058 DEBUG:boto:Token: None 2024-04-12T03:10:44.058 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:10:44 GMT /dgbpxg-29/?cors 2024-04-12T03:10:44.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YnIFrbHLk4eGmJIMnwdp12eRfDk= 2024-04-12T03:10:44.059 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YnIFrbHLk4eGmJIMnwdp12eRfDk=', 'Content-Length': '218'} 2024-04-12T03:10:44.063 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8cc85e8b3a80d29-006618a634-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:44.063 DEBUG:boto:path=/dgbpxg-29/ 2024-04-12T03:10:44.063 DEBUG:boto:auth_path=/dgbpxg-29/ 2024-04-12T03:10:44.064 DEBUG:boto:path=/dgbpxg-29/?cors 2024-04-12T03:10:44.064 DEBUG:boto:auth_path=/dgbpxg-29/?cors 2024-04-12T03:10:44.064 DEBUG:boto:Method: GET 2024-04-12T03:10:44.064 DEBUG:boto:Path: /dgbpxg-29/?cors 2024-04-12T03:10:44.064 DEBUG:boto:Data: 2024-04-12T03:10:44.064 DEBUG:boto:Headers: {} 2024-04-12T03:10:44.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:44.064 DEBUG:boto:Port: 8000 2024-04-12T03:10:44.064 DEBUG:boto:Params: {} 2024-04-12T03:10:44.065 DEBUG:boto:Token: None 2024-04-12T03:10:44.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:44 GMT /dgbpxg-29/?cors 2024-04-12T03:10:44.065 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ufgmKA00Pd6reIjdgnveQPe570= 2024-04-12T03:10:44.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ufgmKA00Pd6reIjdgnveQPe570=', 'Content-Length': '0'} 2024-04-12T03:10:44.066 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006aa188977b9bcbb-006618a634-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:44.106 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:10:44.107 DEBUG:boto:path=/dgbpxg-30/ 2024-04-12T03:10:44.107 DEBUG:boto:auth_path=/dgbpxg-30/ 2024-04-12T03:10:44.107 DEBUG:boto:path=/dgbpxg-30/?cors 2024-04-12T03:10:44.107 DEBUG:boto:auth_path=/dgbpxg-30/?cors 2024-04-12T03:10:44.107 DEBUG:boto:Method: PUT 2024-04-12T03:10:44.107 DEBUG:boto:Path: /dgbpxg-30/?cors 2024-04-12T03:10:44.107 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:10:44.108 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:10:44.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:44.108 DEBUG:boto:Port: 8001 2024-04-12T03:10:44.108 DEBUG:boto:Params: {} 2024-04-12T03:10:44.108 DEBUG:boto:Token: None 2024-04-12T03:10:44.108 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:10:44 GMT /dgbpxg-30/?cors 2024-04-12T03:10:44.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gi0YAP4Fr3B3URN4UGapb7phpNI= 2024-04-12T03:10:44.108 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gi0YAP4Fr3B3URN4UGapb7phpNI=', 'Content-Length': '218'} 2024-04-12T03:10:45.117 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be56da9776f986b8-006618a634-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:45.118 DEBUG:boto:path=/dgbpxg-30/ 2024-04-12T03:10:45.118 DEBUG:boto:auth_path=/dgbpxg-30/ 2024-04-12T03:10:45.118 DEBUG:boto:path=/dgbpxg-30/?cors 2024-04-12T03:10:45.118 DEBUG:boto:auth_path=/dgbpxg-30/?cors 2024-04-12T03:10:45.118 DEBUG:boto:Method: GET 2024-04-12T03:10:45.118 DEBUG:boto:Path: /dgbpxg-30/?cors 2024-04-12T03:10:45.118 DEBUG:boto:Data: 2024-04-12T03:10:45.118 DEBUG:boto:Headers: {} 2024-04-12T03:10:45.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:45.119 DEBUG:boto:Port: 8001 2024-04-12T03:10:45.119 DEBUG:boto:Params: {} 2024-04-12T03:10:45.119 DEBUG:boto:Token: None 2024-04-12T03:10:45.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:45 GMT /dgbpxg-30/?cors 2024-04-12T03:10:45.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QeVWAYiQTYpgWyWGgOWeFhllBHg= 2024-04-12T03:10:45.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QeVWAYiQTYpgWyWGgOWeFhllBHg=', 'Content-Length': '0'} 2024-04-12T03:10:45.120 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004590f92dcb6b9d9a-006618a635-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:45.120 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:10:45.121 DEBUG:boto:path=/dgbpxg-31/ 2024-04-12T03:10:45.121 DEBUG:boto:auth_path=/dgbpxg-31/ 2024-04-12T03:10:45.121 DEBUG:boto:path=/dgbpxg-31/?cors 2024-04-12T03:10:45.121 DEBUG:boto:auth_path=/dgbpxg-31/?cors 2024-04-12T03:10:45.121 DEBUG:boto:Method: PUT 2024-04-12T03:10:45.121 DEBUG:boto:Path: /dgbpxg-31/?cors 2024-04-12T03:10:45.121 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:10:45.121 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:10:45.122 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:45.122 DEBUG:boto:Port: 8000 2024-04-12T03:10:45.122 DEBUG:boto:Params: {} 2024-04-12T03:10:45.122 DEBUG:boto:Token: None 2024-04-12T03:10:45.122 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:10:45 GMT /dgbpxg-31/?cors 2024-04-12T03:10:45.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GxIP/HpEqV4lP0sG5lqub64NlvQ= 2024-04-12T03:10:45.122 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GxIP/HpEqV4lP0sG5lqub64NlvQ=', 'Content-Length': '218'} 2024-04-12T03:10:46.132 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8bbaa8f4e8a3b31-006618a635-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:46.132 DEBUG:boto:path=/dgbpxg-31/ 2024-04-12T03:10:46.133 DEBUG:boto:auth_path=/dgbpxg-31/ 2024-04-12T03:10:46.133 DEBUG:boto:path=/dgbpxg-31/?cors 2024-04-12T03:10:46.133 DEBUG:boto:auth_path=/dgbpxg-31/?cors 2024-04-12T03:10:46.133 DEBUG:boto:Method: GET 2024-04-12T03:10:46.133 DEBUG:boto:Path: /dgbpxg-31/?cors 2024-04-12T03:10:46.133 DEBUG:boto:Data: 2024-04-12T03:10:46.133 DEBUG:boto:Headers: {} 2024-04-12T03:10:46.133 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:10:46.133 DEBUG:boto:Port: 8000 2024-04-12T03:10:46.133 DEBUG:boto:Params: {} 2024-04-12T03:10:46.134 DEBUG:boto:Token: None 2024-04-12T03:10:46.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:46 GMT /dgbpxg-31/?cors 2024-04-12T03:10:46.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q19hrxu/JlW7DC19izOuSYpGCJI= 2024-04-12T03:10:46.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q19hrxu/JlW7DC19izOuSYpGCJI=', 'Content-Length': '0'} 2024-04-12T03:10:46.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001928284b523aa3c7-006618a636-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:46.135 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:10:46.136 DEBUG:boto:path=/dgbpxg-32/ 2024-04-12T03:10:46.136 DEBUG:boto:auth_path=/dgbpxg-32/ 2024-04-12T03:10:46.136 DEBUG:boto:path=/dgbpxg-32/?cors 2024-04-12T03:10:46.136 DEBUG:boto:auth_path=/dgbpxg-32/?cors 2024-04-12T03:10:46.136 DEBUG:boto:Method: PUT 2024-04-12T03:10:46.136 DEBUG:boto:Path: /dgbpxg-32/?cors 2024-04-12T03:10:46.136 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:10:46.137 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:10:46.137 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:46.137 DEBUG:boto:Port: 8001 2024-04-12T03:10:46.137 DEBUG:boto:Params: {} 2024-04-12T03:10:46.137 DEBUG:boto:Token: None 2024-04-12T03:10:46.137 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:10:46 GMT /dgbpxg-32/?cors 2024-04-12T03:10:46.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RHjxuOJA+/bowZyWpO3nXm8jkiQ= 2024-04-12T03:10:46.137 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RHjxuOJA+/bowZyWpO3nXm8jkiQ=', 'Content-Length': '218'} 2024-04-12T03:10:47.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f036a252e4df2d86-006618a636-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:47.148 DEBUG:boto:path=/dgbpxg-32/ 2024-04-12T03:10:47.148 DEBUG:boto:auth_path=/dgbpxg-32/ 2024-04-12T03:10:47.149 DEBUG:boto:path=/dgbpxg-32/?cors 2024-04-12T03:10:47.149 DEBUG:boto:auth_path=/dgbpxg-32/?cors 2024-04-12T03:10:47.149 DEBUG:boto:Method: GET 2024-04-12T03:10:47.149 DEBUG:boto:Path: /dgbpxg-32/?cors 2024-04-12T03:10:47.149 DEBUG:boto:Data: 2024-04-12T03:10:47.149 DEBUG:boto:Headers: {} 2024-04-12T03:10:47.149 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:10:47.149 DEBUG:boto:Port: 8001 2024-04-12T03:10:47.149 DEBUG:boto:Params: {} 2024-04-12T03:10:47.149 DEBUG:boto:Token: None 2024-04-12T03:10:47.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:10:47 GMT /dgbpxg-32/?cors 2024-04-12T03:10:47.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w8wRS8Wi1drQP+IDbaSGuLEtZTo= 2024-04-12T03:10:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w8wRS8Wi1drQP+IDbaSGuLEtZTo=', 'Content-Length': '0'} 2024-04-12T03:10:47.151 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000043e6cf73bce15e21-006618a637-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:47.194 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:10:47.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-12T03:10:47.195 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-33 2024-04-12T03:10:47.195 DEBUG:boto:path=/dgbpxg-33/ 2024-04-12T03:10:47.196 DEBUG:boto:auth_path=/dgbpxg-33/ 2024-04-12T03:10:47.196 DEBUG:boto:Method: PUT 2024-04-12T03:10:47.196 DEBUG:boto:Path: /dgbpxg-33/ 2024-04-12T03:10:47.196 DEBUG:boto:Data: 2024-04-12T03:10:47.196 DEBUG:boto:Headers: {} 2024-04-12T03:10:47.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:10:47.196 DEBUG:boto:Port: 8000 2024-04-12T03:10:47.196 DEBUG:boto:Params: {} 2024-04-12T03:10:47.196 DEBUG:boto:Token: None 2024-04-12T03:10:47.196 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:47 GMT /dgbpxg-33/ 2024-04-12T03:10:47.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1O1+atcPhqAKrfwBKs1jrqkWwaU= 2024-04-12T03:10:47.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1O1+atcPhqAKrfwBKs1jrqkWwaU=', 'Content-Length': '0'} 2024-04-12T03:10:47.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4b872f4402770cd-006618a637-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:47.212 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-34 2024-04-12T03:10:47.212 DEBUG:boto:path=/dgbpxg-34/ 2024-04-12T03:10:47.212 DEBUG:boto:auth_path=/dgbpxg-34/ 2024-04-12T03:10:47.213 DEBUG:boto:Method: PUT 2024-04-12T03:10:47.213 DEBUG:boto:Path: /dgbpxg-34/ 2024-04-12T03:10:47.213 DEBUG:boto:Data: 2024-04-12T03:10:47.213 DEBUG:boto:Headers: {} 2024-04-12T03:10:47.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:10:47.213 DEBUG:boto:Port: 8001 2024-04-12T03:10:47.214 DEBUG:boto:Params: {} 2024-04-12T03:10:47.214 DEBUG:boto:Token: None 2024-04-12T03:10:47.214 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:10:47 GMT /dgbpxg-34/ 2024-04-12T03:10:47.214 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lTOIjBKvxVw4jww2xxGkXRu1dEg= 2024-04-12T03:10:47.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lTOIjBKvxVw4jww2xxGkXRu1dEg=', 'Content-Length': '0'} 2024-04-12T03:10:47.239 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c94475fdb62e32b6-006618a637-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:10:47.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:47.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:47.239 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:47.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:47.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:47.320 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:10:47.332 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:10:47.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:47.333 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:10:47.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:47.334 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:10:47.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:47.335 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:10:47.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.336 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891447.229209_425.1", 2024-04-12T03:10:47.336 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:10:47.229209Z" 2024-04-12T03:10:47.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.336 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:10:47.338 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:10:47.338 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:10:47.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:47.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:47.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:47.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:47.387 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:47.418+0000 7f4b08a69780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:47.419+0000 7f4b08a69780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:47.434 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:10:47.434 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.435 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:47.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:10:47.437 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:47.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:47.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:10:47Z" 2024-04-12T03:10:47.441 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:10:47.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:10:47Z" } 2024-04-12T03:10:47.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:10:47.443 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:10:47.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:10:47.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:10:52.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:52.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:52.740 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:52.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:52.823+0000 7fb8964fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:52.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:52.824+0000 7fb8964fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:10:52.837 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:10:52.838 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:52.839 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:52.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:52.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:10:52.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:10:52.843 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:10:52.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:52.843 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:10:52Z" 2024-04-12T03:10:52.843 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:10:52.844 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:10:52Z" } 2024-04-12T03:10:52.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:10:52.844 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:10:52.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:10:52.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:10:57.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:57.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:10:57.846 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:10:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:57.968+0000 7fe9c279a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:10:57.969+0000 7fe9c279a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:10:57.982 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:10:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:10:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:10:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:10:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:10:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:10:57Z" 2024-04-12T03:10:57.989 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:10:57.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:10:57Z" } 2024-04-12T03:10:57.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:10:57.990 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:10:57.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:10:57.990 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:11:02.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:02.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:02.992 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:03.070+0000 7f873c7b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:03.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:03.071+0000 7f873c7b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:03.086 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:03.087 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:03.088 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:03.089 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:03.090 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:03.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:03.090 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:03Z" 2024-04-12T03:11:03.090 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:03.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:03Z" } 2024-04-12T03:11:03.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:03.091 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:03.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:03.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:08.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:08.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:08.093 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:08.147 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:08.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:08.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:08.182+0000 7f4d1bf46780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:08.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:08.183+0000 7f4d1bf46780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:08.196 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:08.196 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:08.197 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:08.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:08.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:08.200 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:08.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:08.202 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:08Z" 2024-04-12T03:11:08.203 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:08.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:08Z" } 2024-04-12T03:11:08.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:08.204 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:08.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:08.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:13.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:13.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:13.206 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:13.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:13.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:13.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:13.310+0000 7fd4af38f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:13.311+0000 7fd4af38f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:13.324 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:13.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:13.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:13.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:13.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:13.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:13.330 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:13.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:13.330 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:13Z" 2024-04-12T03:11:13.330 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:13.331 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:13Z" } 2024-04-12T03:11:13.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:11:13.331 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:13.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:11:13.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.494539_350.1 2024-04-12T03:11:18.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:18.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:18.332 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:18.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:18.425+0000 7f1fa8cf4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:18.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:18.426+0000 7f1fa8cf4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:18.440 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:18.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:18.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:18.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:11:18.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:18.445 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:18.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:18.446 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:18Z" 2024-04-12T03:11:18.446 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:18.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:18Z" } 2024-04-12T03:11:18.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:11:18.447 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:18.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:11:18.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:11:23.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:23.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:23.469 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:23.525 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:23.525 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:23.553+0000 7ff77c236780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:23.555+0000 7ff77c236780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:23.576 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:23.576 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:23.576 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:23.577 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:23.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:23.579 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:23.580 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:23.581 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:23.582 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:23.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:23.583 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:23Z" 2024-04-12T03:11:23.583 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:23.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:23Z" } 2024-04-12T03:11:23.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:23.584 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:23.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:23.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:28.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:28.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:28.586 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:28.641 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:28.670+0000 7f5ee3e30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:28.671+0000 7f5ee3e30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:28.684 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:28.684 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:28.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:28.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:28.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:28.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:28.689 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:28.690 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:28.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:28.691 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:28Z" 2024-04-12T03:11:28.691 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:28.692 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:28Z" } 2024-04-12T03:11:28.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:28.692 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:28.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:28.692 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:33.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:33.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:33.694 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:33.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:33.750 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:33.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:33.780+0000 7fb5f5999780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:33.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:33.781+0000 7fb5f5999780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:33.794 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:33.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.558140_357.1", 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.799 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:33.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:33.800 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:33.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:33.800 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:33Z" 2024-04-12T03:11:33.800 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:33.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.558140_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.558140Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:33Z" } 2024-04-12T03:11:33.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T03:11:33.801 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:33.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T03:11:33.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T03:11:38.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:38.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:38.803 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:38.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:38.882+0000 7f432dd27780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:38.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:38.883+0000 7f432dd27780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:38.896 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:38.896 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:38.896 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:38.897 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:38.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:38.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:38.901 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.902 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:38.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:38.903 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:38.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:38.903 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:38Z" 2024-04-12T03:11:38.903 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:38.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:38Z" } 2024-04-12T03:11:38.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:38.904 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:38.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:38.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:43.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:43.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:43.906 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:43.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:43.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:43.983+0000 7fc56a993780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:43.984+0000 7fc56a993780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:43.997 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:43.997 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:43.997 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:43.998 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:43.999 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:44.000 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:44.001 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:44.002 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:43Z" 2024-04-12T03:11:44.003 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:44.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:43Z" } 2024-04-12T03:11:44.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:11:44.004 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:44.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:11:44.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:11:49.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:49.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:49.005 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:49.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:49.095+0000 7fa501a8a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:49.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:49.096+0000 7fa501a8a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:49.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:49.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:49.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:49.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:49.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:49.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:49.117 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:49Z" 2024-04-12T03:11:49.117 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:49.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:49Z" } 2024-04-12T03:11:49.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:11:49.118 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:49.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:11:49.118 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:11:54.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:54.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:54.120 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:54.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:54.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:54.202+0000 7f4b7f78a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:54.203+0000 7f4b7f78a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:54.216 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:54Z" 2024-04-12T03:11:54.223 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:54.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:54Z" } 2024-04-12T03:11:54.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:11:54.225 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:54.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:11:54.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:11:59.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:59.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:11:59.227 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:11:59.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:59.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:59.310+0000 7f2dc0a16780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:59.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:11:59.311+0000 7f2dc0a16780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:11:59.326 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:11:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:11:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:11:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:11:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:11:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:11:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:11:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:11:59Z" 2024-04-12T03:11:59.332 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:11:59.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:11:59Z" } 2024-04-12T03:11:59.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:11:59.333 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:11:59.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:11:59.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:12:04.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:04.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:04.335 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:04.418+0000 7f39f6057780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:04.419+0000 7f39f6057780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:04.434 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:04.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:04.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:04.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:04.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:04.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:04.442 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:04.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:04.442 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:04Z" 2024-04-12T03:12:04.442 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:04.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:04Z" } 2024-04-12T03:12:04.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:04.443 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:04.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:04.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:12:09.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:09.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:09.445 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:09.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:09.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:09.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:09.529+0000 7f78c895b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:09.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:09.530+0000 7f78c895b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:09.545 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:09.545 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:09Z" 2024-04-12T03:12:09.551 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:09.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:09Z" } 2024-04-12T03:12:09.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:12:09.552 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:09.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:12:09.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.494539_350.1 2024-04-12T03:12:14.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:14.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:14.555 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:14.618 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:14.619 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:14.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:14.663+0000 7f5a8f4d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:14.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:14.665+0000 7f5a8f4d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:14.680 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:14Z" 2024-04-12T03:12:14.686 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:14.687 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:14Z" } 2024-04-12T03:12:14.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:14.688 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:14.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:14.688 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:12:19.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:19.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:19.690 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:19.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:19.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:19.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:19.774+0000 7fa4002dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:19.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:19.775+0000 7fa4002dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:19.788 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:19.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:19.789 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:19.789 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:19.789 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:19.790 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:19.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:19.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.790 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:19.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:19.791 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:19.792 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:19.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:19.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:19.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.759416_368.1", 2024-04-12T03:12:19.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:19.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:19.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.759416Z", 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:19.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:19.798 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:19.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:19.798 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:19Z" 2024-04-12T03:12:19.798 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:19.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.759416_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.759416Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:19Z" } 2024-04-12T03:12:19.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.759416_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.759416Z', 'realm_epoch': 2}}] 2024-04-12T03:12:19.800 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:19.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.759416_368.1'} 2024-04-12T03:12:19.800 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.759416_368.1 2024-04-12T03:12:24.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:24.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:24.802 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:24.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:24.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:24.880+0000 7f1d99885780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:24.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:24.881+0000 7f1d99885780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:24.894 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:24.894 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:24.894 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:24.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:24.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:24.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:24.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:24.898 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.961337_393.1", 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:24.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.961337Z", 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:24Z" 2024-04-12T03:12:24.901 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:24.902 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.961337_393.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.961337Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:24Z" } 2024-04-12T03:12:24.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.961337_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.961337Z', 'realm_epoch': 2}}] 2024-04-12T03:12:24.902 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:24.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.961337_393.1'} 2024-04-12T03:12:24.902 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.961337_393.1 2024-04-12T03:12:29.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:29.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:29.904 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:29.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:29.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:29.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:29.991+0000 7fd1a0107780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:29.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:29.992+0000 7fd1a0107780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:30.005 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:30.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:30.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:30.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:12:30.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:29Z" 2024-04-12T03:12:30.011 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:30.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:29Z" } 2024-04-12T03:12:30.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:12:30.013 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:30.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:12:30.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:12:34.214 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:12:34.213+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:12:35.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:35.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:35.014 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:35.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:35.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:35.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:35.102+0000 7f90a0fd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:35.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:35.103+0000 7f90a0fd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:35.117 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:35.117 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:35.117 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:35.117 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:35.118 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:35.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:35.120 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:35.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:35.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:35.123 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:35.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:35.123 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:35Z" 2024-04-12T03:12:35.123 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:35.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:35Z" } 2024-04-12T03:12:35.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:35.124 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:35.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:35.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:12:40.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:40.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:40.126 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:40.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:40.211+0000 7f1e09c2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:40.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:40.212+0000 7f1e09c2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:40.226 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:40.227 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:40.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:40.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:40.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:40.231 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:40Z" 2024-04-12T03:12:40.232 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:40.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:40Z" } 2024-04-12T03:12:40.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:40.233 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:40.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:40.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:12:45.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:45.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:45.235 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:45.288 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:45.288 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:45.315+0000 7fccd28e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:45.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:45.316+0000 7fccd28e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:45.330 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:45.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:45Z" 2024-04-12T03:12:45.336 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:45.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:45Z" } 2024-04-12T03:12:45.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:45.338 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:45.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:45.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:12:50.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:50.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:50.340 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:50.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:50.395 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:50.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:50.423+0000 7f2b319a5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:50.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:50.424+0000 7f2b319a5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:50.439 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:50.439 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:50Z" 2024-04-12T03:12:50.445 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:50.446 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:50Z" } 2024-04-12T03:12:50.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:12:50.447 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:50.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:12:50.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:12:55.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:55.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:55.448 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:55.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:55.533+0000 7f8951433780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:55.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:12:55.534+0000 7f8951433780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:55.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:12:55.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:55.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:55.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:55.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:12:55.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:12:55.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:12:55.555 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:12:55.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:12:55.555 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:12:55Z" 2024-04-12T03:12:55.555 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:12:55.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:12:55Z" } 2024-04-12T03:12:55.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:12:55.556 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:12:55.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:12:55.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:00.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:00.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:00.558 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:00.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:00.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:00.640+0000 7f3ffb8b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:00.641+0000 7f3ffb8b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:00.654 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:13:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:00Z" 2024-04-12T03:13:00.660 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:00.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:00Z" } 2024-04-12T03:13:00.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:13:00.662 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:00.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:13:00.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:13:05.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:05.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:05.664 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:05.720 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:05.720 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:05.749+0000 7f59b6086780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:05.750+0000 7f59b6086780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:05.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:05.765 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:05.766 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:05Z" 2024-04-12T03:13:05.770 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:05.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:05Z" } 2024-04-12T03:13:05.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:05.771 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:05.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:05.771 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:10.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:10.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:10.772 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:10.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:10.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:10.857+0000 7f10ad476780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:10.858+0000 7f10ad476780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:10.871 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:10Z" 2024-04-12T03:13:10.877 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:10.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:10Z" } 2024-04-12T03:13:10.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:10.879 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:10.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:10.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:15.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:15.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:15.881 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:15.967+0000 7fdc0f17c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:15.968+0000 7fdc0f17c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:15.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:15.983 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:15.984 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:15.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:13:15.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:15.987 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:15.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:15.988 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:15Z" 2024-04-12T03:13:15.988 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:15.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:15Z" } 2024-04-12T03:13:15.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:13:15.989 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:15.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:13:15.989 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:13:20.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:20.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:20.991 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:21.075+0000 7fa358aa3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:21.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:21.076+0000 7fa358aa3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:21.090 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:21.091 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:21.092 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:21.093 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:21.094 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.095 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:21Z" 2024-04-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:21.097 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:21Z" } 2024-04-12T03:13:21.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:21.097 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:21.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:21.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:26.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:26.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:26.099 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:26.154 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:26.155 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:26.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:26.183+0000 7fedb5208780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:26.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:26.184+0000 7fedb5208780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:26Z" 2024-04-12T03:13:26.206 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:26.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:26Z" } 2024-04-12T03:13:26.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:26.208 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:26.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:26.208 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:31.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:31.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:31.209 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:31.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:31.293+0000 7fbeb79cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:31.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:31.294+0000 7fbeb79cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:31.308 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:31.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:31.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:31.311 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:31.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:31.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:31.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:31.315 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:31Z" 2024-04-12T03:13:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:31.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:31Z" } 2024-04-12T03:13:31.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:31.316 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:31.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:31.316 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:36.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:36.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:36.318 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:36.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:36.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:36.403+0000 7fae1a659780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:36.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:36.404+0000 7fae1a659780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:36.417 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:36.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:36.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:36.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:36.422 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:36.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:36.423 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:36Z" 2024-04-12T03:13:36.423 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:36.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:36Z" } 2024-04-12T03:13:36.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:36.424 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:36.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:36.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:41.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:41.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:41.426 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:41.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:41.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:41.512+0000 7fa29c30f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:41.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:41.513+0000 7fa29c30f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:41.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:41.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:41.529 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:41.530 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:41.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:41.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:41Z" 2024-04-12T03:13:41.533 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:41.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:41Z" } 2024-04-12T03:13:41.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:41.535 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:41.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:41.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:46.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:46.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:46.537 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:46.592 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:46.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:46.623+0000 7fcfe6636780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:46.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:46.625+0000 7fcfe6636780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:46.639 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:46.640 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:46.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:46.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:46.644 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:46.645 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:46.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:46.646 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:46Z" 2024-04-12T03:13:46.646 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:46.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:46Z" } 2024-04-12T03:13:46.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:13:46.647 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:46.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:13:46.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:13:51.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:51.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:51.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:51.704 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:51.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:51.737+0000 7f3e82c12780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:51.738+0000 7f3e82c12780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:51.752 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:51.752 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:51.752 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:51.752 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:51.753 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:51.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:51Z" 2024-04-12T03:13:51.758 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:51.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:51Z" } 2024-04-12T03:13:51.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:13:51.759 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:51.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:13:51.760 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:13:56.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:56.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:56.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:56.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:56.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:56.844+0000 7f6e0fc1b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:56.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:13:56.846+0000 7f6e0fc1b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:56.858 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:13:56.858 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:13:56.858 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:13:56.858 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:56.859 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:13:56.860 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:56.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:13:56.862 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:13:56.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:13:56.864 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:13:56Z" 2024-04-12T03:13:56.865 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:13:56.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:13:56Z" } 2024-04-12T03:13:56.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:13:56.866 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:13:56.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:13:56.866 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:14:01.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:01.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:01.868 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:01.923 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:01.924 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:01.954+0000 7fac088c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:01.955+0000 7fac088c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:01.969 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:01.969 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:01.969 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:01.969 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:01.969 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:01.970 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:01.971 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:01.972 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:01.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:01.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:01Z" 2024-04-12T03:14:01.975 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:01.976 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:01Z" } 2024-04-12T03:14:01.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:01.976 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:01.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:01.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:06.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:06.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:06.979 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:07.064+0000 7f69994cc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:07.065+0000 7f69994cc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:07.079 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:07.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:07.081 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:07.082 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:07.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:07.085 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:07Z" 2024-04-12T03:14:07.086 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:07.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:07Z" } 2024-04-12T03:14:07.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:07.088 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:07.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:07.088 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:12.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:12.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:12.090 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:12.145 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:12.146 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:12.174+0000 7fd47cfe2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:12.175+0000 7fd47cfe2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:12.192 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:12.193 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:12.194 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:12.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:12.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:12Z" 2024-04-12T03:14:12.198 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:12.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:12Z" } 2024-04-12T03:14:12.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:14:12.200 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:12.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:14:12.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T03:14:17.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:17.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:17.201 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:17.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:17.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:17.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:17.293+0000 7f0c7c936780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:17.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:17.294+0000 7f0c7c936780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:17.307 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:17.307 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:17.307 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:17.308 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:17.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:17.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.558140_357.1", 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:17.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:17Z" 2024-04-12T03:14:17.313 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:17.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.558140_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.558140Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:17Z" } 2024-04-12T03:14:17.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T03:14:17.315 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:17.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T03:14:17.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T03:14:22.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:22.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:22.317 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:22.373 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:22.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:22.404+0000 7fb637d8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:22.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:22.405+0000 7fb637d8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:22.418 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.558140_357.1", 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:22Z" 2024-04-12T03:14:22.426 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:22.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.558140_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.558140Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:22Z" } 2024-04-12T03:14:22.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T03:14:22.428 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:22.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T03:14:22.428 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T03:14:27.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:27.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:27.429 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:27.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:27.516+0000 7f4550679780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:27.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:27.517+0000 7f4550679780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:27.530 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:27.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:27.534 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:27.535 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:27Z" 2024-04-12T03:14:27.536 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:27.537 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:27Z" } 2024-04-12T03:14:27.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:27.538 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:27.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:27.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:32.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:32.539 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:32.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:32.630+0000 7f83e0f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:32.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:32.632+0000 7f83e0f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:32Z" 2024-04-12T03:14:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:32.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:32Z" } 2024-04-12T03:14:32.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:32.655 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:32.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:32.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:37.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:37.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:37.657 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:37.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:37.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:37.750+0000 7fe9cb720780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:37.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:37.751+0000 7fe9cb720780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:37.765 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:37.765 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:37.765 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:37.765 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:37.765 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:37.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:37.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:37.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:37.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:37.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:37.771 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:37Z" 2024-04-12T03:14:37.772 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:37.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:37Z" } 2024-04-12T03:14:37.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:37.774 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:37.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:37.774 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:42.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:42.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:42.776 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:42.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:42.862+0000 7f684fd7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:42.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:42.863+0000 7f684fd7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:42.877 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:42.877 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:42.878 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:42.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:42.880 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:42.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:42.882 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:42.883 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:42.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:42.884 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:42Z" 2024-04-12T03:14:42.884 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:42.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:42Z" } 2024-04-12T03:14:42.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:42.885 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:42.885 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:42.885 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:47.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:47.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:47.887 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:47.942 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:47.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:47.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:47.971+0000 7f5ae9cef780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:47.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:47.972+0000 7f5ae9cef780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:47.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:47.987 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.988 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:47.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:47.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:47.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:47.993 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:47.994 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:47Z" 2024-04-12T03:14:47.995 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:47.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:47Z" } 2024-04-12T03:14:47.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:47.996 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:47.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:47.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:52.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.998 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:53.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:53.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:53.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:53.078+0000 7f9797686780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:53.079+0000 7f9797686780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:53.094 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:53.095 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:53.096 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:53Z" 2024-04-12T03:14:53.100 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:53.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:53Z" } 2024-04-12T03:14:53.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:14:53.101 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:53.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:14:53.101 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:14:58.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.103 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.159 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.159 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:58.189+0000 7f1823119780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:14:58.190+0000 7f1823119780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.205 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:14:58.205 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:14:58.205 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:58.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:58.207 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:58.208 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:14:58.209 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.637900_362.1", 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.637900Z", 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:14:58.210 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T03:14:58.211 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:14:58.212 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.637900_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.637900Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:58Z" } 2024-04-12T03:14:58.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.637900_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.637900Z', 'realm_epoch': 2}}] 2024-04-12T03:14:58.212 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:14:58.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.637900_362.1'} 2024-04-12T03:14:58.213 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.637900_362.1 2024-04-12T03:15:03.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:03.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:03.214 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:03.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:03.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:03.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:03.300+0000 7f1d3d902780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:03.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:03.301+0000 7f1d3d902780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:03.315 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:03.317 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:03.320 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:03.321 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:03Z" 2024-04-12T03:15:03.322 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:03.324 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:03Z" } 2024-04-12T03:15:03.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:03.324 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:03.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:03.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:08.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:08.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:08.326 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:08.381 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:08.382 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:08.412+0000 7fc4e27ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:08.413+0000 7fc4e27ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:08.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:08.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:08.429 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:08.430 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:08.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:08.432 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:08.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:08.433 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:08Z" 2024-04-12T03:15:08.433 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:08.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:08Z" } 2024-04-12T03:15:08.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:08.434 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:08.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:08.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:13.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:13.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:13.436 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:13.492 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:13.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:13.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:13.523+0000 7f38de8be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:13.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:13.524+0000 7f38de8be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:13.538 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:13.538 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:13.538 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:13.538 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:13.538 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:13.539 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:13.540 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:13.541 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:13.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:13.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:13Z" 2024-04-12T03:15:13.544 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:13.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:13Z" } 2024-04-12T03:15:13.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:13.545 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:13.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:13.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:18.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:18.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:18.546 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:18.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:18.634+0000 7fe8a7e92780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:18.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:18.635+0000 7fe8a7e92780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:18.649 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:18.649 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:18.649 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:18.649 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:18.650 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:18.651 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:18.652 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:18.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:18.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:18Z" 2024-04-12T03:15:18.655 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:18.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:18Z" } 2024-04-12T03:15:18.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:18.656 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:18.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:18.656 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:23.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:23.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:23.658 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:23.744+0000 7f743365c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:23.745+0000 7f743365c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:23.759 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:23.759 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:23.759 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:23.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:23.759 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:23.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:23.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:23.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:23.763 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:23.764 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:23Z" 2024-04-12T03:15:23.766 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:23.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:23Z" } 2024-04-12T03:15:23.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:23.768 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:23.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:23.768 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:28.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:28.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:28.770 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:28.857+0000 7fe9d8861780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:28.858+0000 7fe9d8861780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:28.872 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:28Z" 2024-04-12T03:15:28.878 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:28.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:28Z" } 2024-04-12T03:15:28.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:28.879 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:28.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:28.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:33.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:33.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:33.882 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:33.942 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:33.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:33.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:33.971+0000 7f04d1255780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:33.972+0000 7f04d1255780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:33.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:33.987 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:33.987 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:33.987 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:33.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:33.989 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:33Z" 2024-04-12T03:15:33.993 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:33.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:33Z" } 2024-04-12T03:15:33.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:33.994 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:33.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:33.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:38.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:38.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:38.996 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:39.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:39.081+0000 7fd4e2a70780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:39.082+0000 7fd4e2a70780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:39Z" 2024-04-12T03:15:39.104 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:39.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:39Z" } 2024-04-12T03:15:39.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:39.105 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:39.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:39.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:44.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:44.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:44.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:44.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:44.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:44.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:44.195+0000 7f0ce5dd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:44.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:44.196+0000 7f0ce5dd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.211 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:44.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:44.213 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:44.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:44Z" 2024-04-12T03:15:44.216 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:44.217 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:44Z" } 2024-04-12T03:15:44.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:44.218 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:44.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:44.218 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:49.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:49.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:49.219 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:49.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:49.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:49.308+0000 7f6839bd5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:49.309+0000 7f6839bd5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:49.326 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:49.326 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:49.326 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:49.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:49.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:49.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:49.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:49Z" 2024-04-12T03:15:49.332 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:49.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:49Z" } 2024-04-12T03:15:49.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:15:49.333 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T03:15:49.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:15:49.333 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T03:15:54.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-04-12T03:15:54.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T03:15:54.339 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-35 2024-04-12T03:15:54.339 DEBUG:boto:path=/dgbpxg-35/ 2024-04-12T03:15:54.339 DEBUG:boto:auth_path=/dgbpxg-35/ 2024-04-12T03:15:54.340 DEBUG:boto:Method: PUT 2024-04-12T03:15:54.340 DEBUG:boto:Path: /dgbpxg-35/ 2024-04-12T03:15:54.340 DEBUG:boto:Data: 2024-04-12T03:15:54.340 DEBUG:boto:Headers: {} 2024-04-12T03:15:54.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:15:54.340 DEBUG:boto:Port: 8000 2024-04-12T03:15:54.340 DEBUG:boto:Params: {} 2024-04-12T03:15:54.341 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:15:54.341 DEBUG:boto:Token: None 2024-04-12T03:15:54.341 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-35/ 2024-04-12T03:15:54.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:84PkzjT2im0qstSP109FuY8l928= 2024-04-12T03:15:54.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:84PkzjT2im0qstSP109FuY8l928=', 'Content-Length': '0'} 2024-04-12T03:15:54.385 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d678c193c9fa1d43-006618a76a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.385 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-36 2024-04-12T03:15:54.386 DEBUG:boto:path=/dgbpxg-36/ 2024-04-12T03:15:54.386 DEBUG:boto:auth_path=/dgbpxg-36/ 2024-04-12T03:15:54.386 DEBUG:boto:Method: PUT 2024-04-12T03:15:54.386 DEBUG:boto:Path: /dgbpxg-36/ 2024-04-12T03:15:54.386 DEBUG:boto:Data: 2024-04-12T03:15:54.386 DEBUG:boto:Headers: {} 2024-04-12T03:15:54.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:15:54.386 DEBUG:boto:Port: 8001 2024-04-12T03:15:54.386 DEBUG:boto:Params: {} 2024-04-12T03:15:54.387 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:15:54.387 DEBUG:boto:Token: None 2024-04-12T03:15:54.387 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-36/ 2024-04-12T03:15:54.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aWK5uJMBnMQqflXWXh2SasgxLZg= 2024-04-12T03:15:54.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aWK5uJMBnMQqflXWXh2SasgxLZg=', 'Content-Length': '0'} 2024-04-12T03:15:54.443 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000caab7bb155e0f62f-006618a76a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.444 DEBUG:boto:path=/dgbpxg-35/ 2024-04-12T03:15:54.444 DEBUG:boto:auth_path=/dgbpxg-35/ 2024-04-12T03:15:54.444 DEBUG:boto:Method: HEAD 2024-04-12T03:15:54.444 DEBUG:boto:Path: /dgbpxg-35/ 2024-04-12T03:15:54.444 DEBUG:boto:Data: 2024-04-12T03:15:54.444 DEBUG:boto:Headers: {} 2024-04-12T03:15:54.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:15:54.444 DEBUG:boto:Port: 8000 2024-04-12T03:15:54.445 DEBUG:boto:Params: {} 2024-04-12T03:15:54.445 DEBUG:boto:Token: None 2024-04-12T03:15:54.445 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-35/ 2024-04-12T03:15:54.445 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ptVlmH46P4ctydhvvsra4wagkQ4= 2024-04-12T03:15:54.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ptVlmH46P4ctydhvvsra4wagkQ4=', 'Content-Length': '0'} 2024-04-12T03:15:54.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afd562a8852d7fc1-006618a76a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.447 DEBUG:boto:path=/dgbpxg-35/key 2024-04-12T03:15:54.448 DEBUG:boto:auth_path=/dgbpxg-35/key 2024-04-12T03:15:54.448 DEBUG:boto:Method: PUT 2024-04-12T03:15:54.448 DEBUG:boto:Path: /dgbpxg-35/key 2024-04-12T03:15:54.448 DEBUG:boto:Data: 2024-04-12T03:15:54.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T03:15:54.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:15:54.448 DEBUG:boto:Port: 8000 2024-04-12T03:15:54.448 DEBUG:boto:Params: {} 2024-04-12T03:15:54.448 DEBUG:boto:Token: None 2024-04-12T03:15:54.449 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-35/key 2024-04-12T03:15:54.449 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TAA4tTy3BjfJyCsDJZy9xQVRZtk= 2024-04-12T03:15:54.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TAA4tTy3BjfJyCsDJZy9xQVRZtk='} 2024-04-12T03:15:54.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a8e9d3de7b1f3e0-006618a76a-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.455 DEBUG:boto:path=/dgbpxg-36/ 2024-04-12T03:15:54.455 DEBUG:boto:auth_path=/dgbpxg-36/ 2024-04-12T03:15:54.455 DEBUG:boto:Method: HEAD 2024-04-12T03:15:54.455 DEBUG:boto:Path: /dgbpxg-36/ 2024-04-12T03:15:54.455 DEBUG:boto:Data: 2024-04-12T03:15:54.455 DEBUG:boto:Headers: {} 2024-04-12T03:15:54.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:15:54.455 DEBUG:boto:Port: 8001 2024-04-12T03:15:54.455 DEBUG:boto:Params: {} 2024-04-12T03:15:54.456 DEBUG:boto:Token: None 2024-04-12T03:15:54.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-36/ 2024-04-12T03:15:54.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L8dZDokjU4DFNl4JmRSYk7C9HRQ= 2024-04-12T03:15:54.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L8dZDokjU4DFNl4JmRSYk7C9HRQ=', 'Content-Length': '0'} 2024-04-12T03:15:54.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1f61640be882376-006618a76a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.458 DEBUG:boto:path=/dgbpxg-36/key 2024-04-12T03:15:54.459 DEBUG:boto:auth_path=/dgbpxg-36/key 2024-04-12T03:15:54.459 DEBUG:boto:Method: PUT 2024-04-12T03:15:54.459 DEBUG:boto:Path: /dgbpxg-36/key 2024-04-12T03:15:54.459 DEBUG:boto:Data: 2024-04-12T03:15:54.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T03:15:54.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:15:54.460 DEBUG:boto:Port: 8001 2024-04-12T03:15:54.460 DEBUG:boto:Params: {} 2024-04-12T03:15:54.460 DEBUG:boto:Token: None 2024-04-12T03:15:54.460 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 12 Apr 2024 03:15:54 GMT /dgbpxg-36/key 2024-04-12T03:15:54.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IVmhKlSbH5xeYm8DUN0A82xSaVs= 2024-04-12T03:15:54.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IVmhKlSbH5xeYm8DUN0A82xSaVs='} 2024-04-12T03:15:54.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072746f72aab18c3d-006618a76a-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:54.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:54.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:54.466 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:54.514 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:54.515 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:15:54.555 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:15:54.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:54.557 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891754.433386_468.1", 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:15:54.433386Z" 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.558 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:15:54.559 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:15:54.560 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:15:54.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:54.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:54.560 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:54.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:54.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:54.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:54.647+0000 7f76e4f59780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:54.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:54.649+0000 7f76e4f59780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:54.662 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:54Z" 2024-04-12T03:15:54.669 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:54.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:54Z" } 2024-04-12T03:15:54.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:15:54.670 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:15:54.670 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:15:54.670 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:15:59.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:59.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:59.672 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:59.725 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:59.725 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:59.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:59.754+0000 7f01b62ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:59.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:15:59.755+0000 7f01b62ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:59.769 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:15:59.769 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:15:59.769 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:15:59.769 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:15:59.769 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:59.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:59.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:59.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:15:59.774 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:15:59Z" 2024-04-12T03:15:59.775 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:15:59.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:59Z" } 2024-04-12T03:15:59.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:15:59.777 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:15:59.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:15:59.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:04.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:04.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:04.778 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:04.834 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:04.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:04.865+0000 7fefa4845780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:04.866+0000 7fefa4845780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:04.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:04.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:04.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:04.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:04Z" 2024-04-12T03:16:04.885 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:04.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:04Z" } 2024-04-12T03:16:04.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:16:04.886 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:04.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:16:04.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:09.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:09.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:09.888 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:09.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:09.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:09.970+0000 7fd024634780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:09.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:09.971+0000 7fd024634780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:09.984 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:09.984 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:09.985 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:09.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:16:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:09Z" 2024-04-12T03:16:09.990 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:09.991 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:09Z" } 2024-04-12T03:16:09.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:16:09.992 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:09.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:16:09.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:14.223 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:16:14.221+0000 7fdb33d4a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:16:14.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:14.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:14.993 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:15.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:15.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:15.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:15.078+0000 7fe71ebcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:15.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:15.079+0000 7fe71ebcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:15.100 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:15.100 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:15.100 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:15.100 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:15.101 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:15.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:15.103 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:15.104 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.105 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:15.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:15.106 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:15.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:15.106 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:15Z" 2024-04-12T03:16:15.106 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:15.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:15Z" } 2024-04-12T03:16:15.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:16:15.107 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:15.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:16:15.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:15.373 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:16:15.371+0000 7fdb29d36700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:16:20.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:20.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:20.109 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:20.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:20.197+0000 7f3f46045780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:20.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:20.199+0000 7f3f46045780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:20.214 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:20.214 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:20.214 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:20.214 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:20.214 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:20.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:20.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:20.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:20.219 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:20.220 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.221 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:20.222 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:20.222 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:20.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:20.222 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:20Z" 2024-04-12T03:16:20.222 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:20.223 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:20Z" } 2024-04-12T03:16:20.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:16:20.224 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:20.224 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:16:20.224 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:25.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:25.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:25.225 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:25.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:25.312+0000 7f93a132e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:25.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:25.313+0000 7f93a132e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:25Z" 2024-04-12T03:16:25.335 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:25.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:25Z" } 2024-04-12T03:16:25.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:16:25.337 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:25.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:16:25.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:16:30.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:30.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:30.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:30.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:30.395 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:30.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:30.426+0000 7fa3c71a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:30.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:30.427+0000 7fa3c71a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:30.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:30.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:16:30.445 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:30.446 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:30.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:30.447 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:30Z" 2024-04-12T03:16:30.447 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:30.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:30Z" } 2024-04-12T03:16:30.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:16:30.448 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:30.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:16:30.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:16:35.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:35.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:35.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:35.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:35.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:35.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:35.533+0000 7f35abe52780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:35.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:35.534+0000 7f35abe52780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:35.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:35.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:35.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:35.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:35.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:35Z" 2024-04-12T03:16:35.554 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:35.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:35Z" } 2024-04-12T03:16:35.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:16:35.555 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:35.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:16:35.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:16:40.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:40.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:40.558 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:40.641+0000 7f5d3f7f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:40.643+0000 7f5d3f7f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:40.656 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:40.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:40.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:40.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:40.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:40Z" 2024-04-12T03:16:40.662 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:40.663 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:40Z" } 2024-04-12T03:16:40.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:16:40.664 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:40.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:16:40.664 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:16:45.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:45.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:45.665 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:45.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:45.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:45.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:45.752+0000 7f2d64423780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:45.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:45.753+0000 7f2d64423780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:45Z" 2024-04-12T03:16:45.772 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:45.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:45Z" } 2024-04-12T03:16:45.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:16:45.774 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:45.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:16:45.774 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:16:50.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:50.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:50.776 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:50.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:50.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:50.862+0000 7f9fa10d9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:50.863+0000 7f9fa10d9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:50.876 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:50Z" 2024-04-12T03:16:50.882 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:50.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:50Z" } 2024-04-12T03:16:50.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:16:50.884 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:50.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:16:50.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:16:55.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:55.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:16:55.885 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:16:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:16:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:16:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:55.971+0000 7fc1a338c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:16:55.972+0000 7fc1a338c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:16:55.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:16:55.987 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:16:55.987 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:16:55.987 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:16:55.987 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:55.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:16:55.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:16:55.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:16:55.994 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:16:55Z" 2024-04-12T03:16:55.994 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:16:55.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:16:55Z" } 2024-04-12T03:16:55.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:16:55.995 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:16:55.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:16:55.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T03:17:00.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:00.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:00.998 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:01.083+0000 7fb961563780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:01.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:01.084+0000 7fb961563780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:01.099 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:01.099 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:01.100 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:01.101 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:01.102 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:01.103 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:01.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:01Z" 2024-04-12T03:17:01.105 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:01.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:01Z" } 2024-04-12T03:17:01.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:17:01.107 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:01.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:17:01.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T03:17:06.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:06.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:06.108 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:06.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:06.194+0000 7fce28ac0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:06.195+0000 7fce28ac0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:06Z" 2024-04-12T03:17:06.215 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:06.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:06Z" } 2024-04-12T03:17:06.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:06.216 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:06.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:06.216 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:17:11.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:11.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:11.218 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:11.268 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:11.296+0000 7fa618e97780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:11.297+0000 7fa618e97780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:11.310 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:11.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:11.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:11.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:11.317 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:11Z" 2024-04-12T03:17:11.317 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:11.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:11Z" } 2024-04-12T03:17:11.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:11.318 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:11.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:11.318 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:17:16.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:16.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:16.320 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:16.404+0000 7f30a9bc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:16.405+0000 7f30a9bc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:16.419 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:16.419 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:16.419 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:16.419 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:16.419 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:16.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:16.422 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:16.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:16.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:16Z" 2024-04-12T03:17:16.425 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:16.426 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:16Z" } 2024-04-12T03:17:16.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:16.426 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:16.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:16.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:17:21.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:21.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:21.430 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:21.485 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:21.485 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:21.514+0000 7fe7859e5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:21.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:21.516+0000 7fe7859e5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:21.529 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:21.529 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:21.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:21.533 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:21.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:21.535 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:17:21.536 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.537 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:21.538 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:21.538 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:21.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:21.538 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:21Z" 2024-04-12T03:17:21.538 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:21.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:21Z" } 2024-04-12T03:17:21.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:17:21.540 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:21.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:17:21.540 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:17:26.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:26.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:26.542 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:26.598 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:26.629+0000 7f197b5b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:26.630+0000 7f197b5b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:26.644 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:26.644 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:26.644 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:26.645 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:26.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:26.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:26.649 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:26.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:26.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:26.652 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:26.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:26.652 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:26Z" 2024-04-12T03:17:26.652 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:26.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:26Z" } 2024-04-12T03:17:26.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:17:26.653 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:26.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:17:26.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:17:31.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:31.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:31.655 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:31.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:31.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:31.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:31.742+0000 7f3884e08780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:31.743+0000 7f3884e08780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:31Z" 2024-04-12T03:17:31.763 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:31.764 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:31Z" } 2024-04-12T03:17:31.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:31.764 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:31.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:31.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:17:36.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:36.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:36.766 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:36.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:36.883+0000 7fcfe4b87780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:36.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:36.884+0000 7fcfe4b87780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.558140_357.1", 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:36.905 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:36Z" 2024-04-12T03:17:36.905 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:36.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.558140_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.558140Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:36Z" } 2024-04-12T03:17:36.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T03:17:36.906 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:36.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T03:17:36.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.558140_357.1 2024-04-12T03:17:41.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:41.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:41.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:41.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:41.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:41.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:41.991+0000 7f5b5d18d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:41.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:41.992+0000 7f5b5d18d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:41Z" 2024-04-12T03:17:42.013 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:42.013 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:41Z" } 2024-04-12T03:17:42.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:17:42.014 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:42.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:17:42.014 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:17:47.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:47.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:47.016 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:47.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:47.170+0000 7fe4a6849780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:47.171+0000 7fe4a6849780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:47.185 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:47.185 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:47.185 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:47.185 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:47.185 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:47.186 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:47.187 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:47.188 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:47.189 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:47.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:47Z" 2024-04-12T03:17:47.191 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:47.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:47Z" } 2024-04-12T03:17:47.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:47.193 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:47.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:47.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:17:48.180 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:17:48.179+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:17:52.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:52.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:52.194 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:52.280+0000 7efd2d522780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:52.282+0000 7efd2d522780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:52.296 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:52.296 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:17:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:52.307 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:52.307 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:52.307 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:17:52.307 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:52.307 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:52.308 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:52.308 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:52.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.308 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:52.308 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:52.309 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:52.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:52.309 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:52Z" 2024-04-12T03:17:52.309 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:52.311 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:52Z" } 2024-04-12T03:17:52.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:17:52.311 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:52.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:17:52.312 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T03:17:57.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:57.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:17:57.314 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:17:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:17:57.370 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:17:57.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:57.401+0000 7fbc0b078780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:57.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:17:57.402+0000 7fbc0b078780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:17:57.416 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:57.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:57.418 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:57.419 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:17:57.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:17:57.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:17:57Z" 2024-04-12T03:17:57.422 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:17:57.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:17:57Z" } 2024-04-12T03:17:57.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:17:57.424 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:17:57.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:17:57.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:02.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:02.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:02.426 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:02.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:02.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:02.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:02.523+0000 7f6586502780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:02.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:02.524+0000 7f6586502780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:02.538 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:02.538 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:02.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:02.540 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:02.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:02.542 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:02.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:02.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:02.545 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:02Z" 2024-04-12T03:18:02.545 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:02.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:02Z" } 2024-04-12T03:18:02.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:02.546 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:02.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:02.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:07.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:07.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:07.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:07.606 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:07.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:07.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:07.637+0000 7fedb86c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:07.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:07.638+0000 7fedb86c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:07.652 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:07.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:07.654 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:07.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:07.656 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:07.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:07.658 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:07Z" 2024-04-12T03:18:07.659 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:07.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:07Z" } 2024-04-12T03:18:07.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:18:07.660 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:07.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:18:07.660 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:18:12.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:12.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:12.662 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:12.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:12.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:12.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:12.749+0000 7fa297883780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:12.750+0000 7fa297883780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:12.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:12.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:12.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:12.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:12.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:12.770 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:12Z" 2024-04-12T03:18:12.770 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:12.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:12Z" } 2024-04-12T03:18:12.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:12.771 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:12.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:12.771 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:17.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:17.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:17.773 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:17.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:17.825 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:17.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:17.853+0000 7f572e122780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:17.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:17.854+0000 7f572e122780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:17.867 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:17.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:17.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:17.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:17.872 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:17Z" 2024-04-12T03:18:17.873 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:17.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:17Z" } 2024-04-12T03:18:17.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:17.875 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:17.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:17.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:22.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:22.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:22.878 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:22.934 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:22.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:22.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:22.966+0000 7f196c882780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:22.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:22.967+0000 7f196c882780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:22.982 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:22.982 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:22.983 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:22.984 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:22.985 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:22.987 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:22Z" 2024-04-12T03:18:22.988 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:22.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:22Z" } 2024-04-12T03:18:22.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:22.990 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:22.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:22.990 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:27.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:27.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:27.992 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:28.047 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:28.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:28.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:28.077+0000 7fdbfd0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:28.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:28.078+0000 7fdbfd0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:28.094 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:28.097 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:28.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:28.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:28.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:28.101 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:28.101 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:28.101 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:28.101 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:28.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:28.103 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:28.103 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:28.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:28.103 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:28.103 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:28.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:28.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:28.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:28.107 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:28.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:28.107 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:28Z" 2024-04-12T03:18:28.107 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:28.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:28Z" } 2024-04-12T03:18:28.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:28.109 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:28.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:28.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:33.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:33.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:33.111 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:33.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:33.196+0000 7f7fd82c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:33.197+0000 7f7fd82c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:33.211 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:33.212 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:33.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:33.214 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:33.215 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:33.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:33.217 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:33Z" 2024-04-12T03:18:33.217 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:33.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:33Z" } 2024-04-12T03:18:33.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:33.218 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:33.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:33.218 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:38.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:38.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:38.412 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:38.463 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:38.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:38.493+0000 7ff6e6ae4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:38.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:38.494+0000 7ff6e6ae4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:38.507 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:38.508 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:38.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:38.510 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:38.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:38.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:38.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:38.514 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:38Z" 2024-04-12T03:18:38.514 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:38.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:38Z" } 2024-04-12T03:18:38.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:18:38.515 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:38.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:18:38.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:18:43.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:43.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:43.516 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:43.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:43.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:43.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:43.608+0000 7fa2f6065780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:43.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:43.609+0000 7fa2f6065780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:43.626 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:43.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:43.628 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:43.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:43Z" 2024-04-12T03:18:43.630 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:43.631 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:43Z" } 2024-04-12T03:18:43.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:18:43.632 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:43.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:18:43.632 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T03:18:48.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:48.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:48.633 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:48.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:48.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:48.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:48.720+0000 7f96797e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:48.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:48.721+0000 7f96797e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:48.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:48.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:48.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:48.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:48.739 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:48.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:48.741 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:48Z" 2024-04-12T03:18:48.741 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:48.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:48Z" } 2024-04-12T03:18:48.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:18:48.742 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:48.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:18:48.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:18:53.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:53.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:53.744 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:53.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:53.830+0000 7ff261ed4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:53.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:53.831+0000 7ff261ed4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:53.846 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:53.846 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:53.846 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:53.846 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.494539_350.1", 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:53Z" 2024-04-12T03:18:53.853 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:53.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.494539_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.494539Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:53Z" } 2024-04-12T03:18:53.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T03:18:53.854 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:53.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T03:18:53.854 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T03:18:58.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:58.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:58.856 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:58.917 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:58.917 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:58.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:58.956+0000 7f11be899780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:58.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:18:58.957+0000 7f11be899780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:58.971 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:18:58.971 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:18:58.971 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:18:58.971 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:18:58.972 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:58.974 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:18:58.975 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.558140_357.1", 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.976 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:18:58.977 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:18:58.977 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:18:58.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:18:58.977 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:18:58Z" 2024-04-12T03:18:58.977 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:18:58.978 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.558140_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.558140Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:58Z" } 2024-04-12T03:18:58.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T03:18:58.978 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:18:58.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T03:18:58.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.558140_357.1 2024-04-12T03:19:03.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:03.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:03.980 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:04.036 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:04.036 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:04.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:04.065+0000 7fafb4289780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:04.066+0000 7fafb4289780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:04.081 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:04.082 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:04.083 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:04.084 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:04.085 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:04.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:04Z" 2024-04-12T03:19:04.087 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:04.088 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:04Z" } 2024-04-12T03:19:04.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:04.088 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:04.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:04.089 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:09.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:09.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:09.090 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:09.146 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:09.146 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:09.176+0000 7fc455fb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:09.177+0000 7fc455fb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:09Z" 2024-04-12T03:19:09.199 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:09.200 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:09Z" } 2024-04-12T03:19:09.200 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:09.200 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:09.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:09.201 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:14.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:14.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:14.202 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:14.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:14.285+0000 7f269975f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:14.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:14.286+0000 7f269975f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:14.301 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:14.301 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:14.301 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:14.301 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:14.301 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:14.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:14.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:14.304 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:14Z" 2024-04-12T03:19:14.307 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:14.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:14Z" } 2024-04-12T03:19:14.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:14.308 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:14.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:14.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:19.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:19.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:19.310 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:19.366 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:19.366 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:19.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:19.396+0000 7f826a26a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:19.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:19.397+0000 7f826a26a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:19Z" 2024-04-12T03:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:19.418 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:19Z" } 2024-04-12T03:19:19.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:19.419 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:19.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:19.419 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:24.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:24.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:24.421 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:24.508+0000 7fdf887db780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:24.509+0000 7fdf887db780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.523 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:24.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:24.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:24.528 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:24Z" 2024-04-12T03:19:24.529 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:24.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:24Z" } 2024-04-12T03:19:24.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:24.531 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:24.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:24.531 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:29.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:29.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:29.532 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:29.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:29.621+0000 7f8f67d30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:29.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:29.622+0000 7f8f67d30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:29.636 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:29Z" 2024-04-12T03:19:29.644 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:29.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:29Z" } 2024-04-12T03:19:29.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:29.645 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:29.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:29.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:34.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:34.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:34.648 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:34.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:34.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:34.758+0000 7f332ecff780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:34.759+0000 7f332ecff780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:34.772 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:34.772 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:34.772 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:34.773 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:34.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:34.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:34.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:34.774 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:34.775 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:34.776 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:34Z" 2024-04-12T03:19:34.779 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:34.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:34Z" } 2024-04-12T03:19:34.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:34.781 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:34.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:34.781 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:39.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:39.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:39.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:39.870+0000 7f161e841780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:39.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:39.870+0000 7f161e841780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:39.885 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:39.886 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:39.887 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:39.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:39.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:39.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:39.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:39Z" 2024-04-12T03:19:39.892 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:39.893 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:39Z" } 2024-04-12T03:19:39.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:39.894 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:39.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:39.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:44.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:44.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:44.896 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:44.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:44.976+0000 7f741eb18780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:44.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:44.977+0000 7f741eb18780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:44.992 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:44.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:44.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:44.997 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:44.997 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:44.997 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:44.997 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:44.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:44.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:44.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:44.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:44.998 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:44.999 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:44.999 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:44.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:44.999 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:44.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:44Z" 2024-04-12T03:19:45.002 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:45.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:44Z" } 2024-04-12T03:19:45.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:45.004 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:45.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:45.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:50.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:50.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:50.006 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:50.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:50.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:50.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:50.093+0000 7f9c0a36c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:50.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:50.094+0000 7f9c0a36c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:50.109 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:50Z" 2024-04-12T03:19:50.119 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:50.121 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:50Z" } 2024-04-12T03:19:50.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:50.122 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:50.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:50.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:19:55.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:55.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:55.124 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:55.177 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:55.178 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:55.208+0000 7f8a7c689780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:55.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:19:55.209+0000 7f8a7c689780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:55.224 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:19:55.224 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:19:55.224 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:19:55.224 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:19:55.224 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:55.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:55.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:55.227 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:55.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:55.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:19:55Z" 2024-04-12T03:19:55.231 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:19:55.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:55Z" } 2024-04-12T03:19:55.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:19:55.233 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:19:55.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:19:55.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:00.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:00.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:00.234 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:00.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:00.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:00.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:00.320+0000 7f734fb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:00.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:00.321+0000 7f734fb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.336 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:00.337 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:00.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:00.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:00Z" 2024-04-12T03:20:00.341 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:00.342 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:00Z" } 2024-04-12T03:20:00.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:00.343 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:00.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:00.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:05.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:05.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:05.344 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:05.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:05.427+0000 7fd576122780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:05.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:05.428+0000 7fd576122780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:05.442 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:05.442 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:05.442 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:05.442 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:05.442 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:05.445 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:05.446 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:05Z" 2024-04-12T03:20:05.448 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:05.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:05Z" } 2024-04-12T03:20:05.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:05.449 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:05.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:05.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:10.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:10.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:10.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:10.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:10.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:10.535+0000 7fda943d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:10.536+0000 7fda943d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:10.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:10.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:10.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:10.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:10.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:10.556 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:10.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:10.557 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:10Z" 2024-04-12T03:20:10.557 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:10.558 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:10Z" } 2024-04-12T03:20:10.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:10.558 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:10.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:10.559 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:15.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:15.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:15.560 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:15.612 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:15.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:15.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:15.640+0000 7f2ef83cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:15.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:15.641+0000 7f2ef83cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:15.653 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:15Z" 2024-04-12T03:20:15.659 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:15.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:15Z" } 2024-04-12T03:20:15.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:20:15.661 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:15.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:20:15.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:20:20.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:20.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:20.662 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:20.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:20.751+0000 7f0a4b592780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:20.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:20.752+0000 7f0a4b592780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:20.766 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:20.766 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:20.766 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:20.766 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:20.766 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:20.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:20.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:20.769 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:20.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:20.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:20.772 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:20.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:20.772 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:20Z" 2024-04-12T03:20:20.772 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:20.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:20Z" } 2024-04-12T03:20:20.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:20.773 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:20.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:20.773 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:25.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:25.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:25.775 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:25.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:25.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:25.863+0000 7f7f09091780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:25.864+0000 7f7f09091780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.879 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:25.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:25.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:25.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:25.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:25Z" 2024-04-12T03:20:25.884 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:25.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:25Z" } 2024-04-12T03:20:25.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:20:25.886 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:25.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:20:25.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:20:30.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:30.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:30.887 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:30.974+0000 7f208067b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:30.975+0000 7f208067b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:30.991 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:30.991 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:30.991 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:30.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:30.992 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:30.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:30.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.552617_356.1", 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:30.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:30.997 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:30.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:30.998 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:30Z" 2024-04-12T03:20:30.998 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:30.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.552617_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.552617Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:30Z" } 2024-04-12T03:20:30.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.552617_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.552617Z', 'realm_epoch': 2}}] 2024-04-12T03:20:30.999 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:30.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.552617_356.1'} 2024-04-12T03:20:30.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.552617_356.1 2024-04-12T03:20:36.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:36.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:36.001 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:36.086+0000 7f51fb361780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:36.088+0000 7f51fb361780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:36.103 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:36.103 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:36.103 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:36.103 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:36.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:36.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:36.106 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:36.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:36.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:36.109 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.552617_356.1", 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:36.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:36.111 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:36Z" 2024-04-12T03:20:36.112 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:36.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.552617_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.552617Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:36Z" } 2024-04-12T03:20:36.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.552617_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.552617Z', 'realm_epoch': 2}}] 2024-04-12T03:20:36.113 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:36.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.552617_356.1'} 2024-04-12T03:20:36.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.552617_356.1 2024-04-12T03:20:41.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:41.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:41.115 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:41.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:41.199+0000 7fe83dfbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:41.200+0000 7fe83dfbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:41.214 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:41.214 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:41.214 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:41.214 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:41.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:41.216 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:41.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:41.218 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:41.219 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.612433_358.1", 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.220 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:41.221 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:41.221 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:41.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:41.221 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:41Z" 2024-04-12T03:20:41.221 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:41.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.612433_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.612433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:41Z" } 2024-04-12T03:20:41.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T03:20:41.223 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:41.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T03:20:41.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T03:20:46.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:46.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:46.224 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:46.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:46.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:46.313+0000 7ff03f263780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:46.314+0000 7ff03f263780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:46.328 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:46.328 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:46.328 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:46.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:46.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:46.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:46.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:46.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:46.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:46.335 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:46Z" 2024-04-12T03:20:46.336 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:46.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:46Z" } 2024-04-12T03:20:46.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:46.337 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:46.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:46.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:51.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:51.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:51.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:51.428+0000 7fd154327780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:51.429+0000 7fd154327780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:51.445 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:51Z" 2024-04-12T03:20:51.451 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:51.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:51Z" } 2024-04-12T03:20:51.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:51.452 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:51.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:51.452 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:20:56.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:56.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:20:56.454 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:20:56.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:20:56.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:20:56.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:56.535+0000 7f1d0ce07780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:56.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:20:56.536+0000 7f1d0ce07780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:20:56.549 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:56.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:56.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:56.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712891443.549294_354.1", 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T03:20:56.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:20:56Z" 2024-04-12T03:20:56.555 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:20:56.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891443.549294_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:10:43.549294Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:20:56Z" } 2024-04-12T03:20:56.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T03:20:56.557 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T03:20:56.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T03:20:56.557 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T03:21:01.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-04-12T03:21:01.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:21:01.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:21:01.563 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:21:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:21:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:21:01.641 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:21:01.641 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:21:01.641 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:21:01.642 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:21:01.643 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.644 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:21:01.645 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:21:01.645 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:21:01.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:21:01.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:21:01.646 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:21:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:21:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:21:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T03:21:01.761 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:21:01.761 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:21:01.761 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5, 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:21:01.762 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.763 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:21:01.764 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:21:01.765 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.766 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:21:01.767 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.768 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.769 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:21:01.770 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:21:01.773 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:21:01.774 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:21:01.776 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:21:01.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:21:01.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:21:01.779 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:21:01.780 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:21:01.781 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:21:01.782 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:21:01.783 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2024-04-12T03:22:01.784 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-37 2024-04-12T03:22:01.785 DEBUG:boto:path=/dgbpxg-37/ 2024-04-12T03:22:01.785 DEBUG:boto:auth_path=/dgbpxg-37/ 2024-04-12T03:22:01.786 DEBUG:boto:Method: PUT 2024-04-12T03:22:01.786 DEBUG:boto:Path: /dgbpxg-37/ 2024-04-12T03:22:01.786 DEBUG:boto:Data: 2024-04-12T03:22:01.786 DEBUG:boto:Headers: {} 2024-04-12T03:22:01.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:22:01.786 DEBUG:boto:Port: 8000 2024-04-12T03:22:01.786 DEBUG:boto:Params: {} 2024-04-12T03:22:01.787 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:22:01.787 DEBUG:boto:Token: None 2024-04-12T03:22:01.787 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:22:01 GMT /dgbpxg-37/ 2024-04-12T03:22:01.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qXUV+RC9w1IrlIBNqgkmg63JBhY= 2024-04-12T03:22:01.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:22:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qXUV+RC9w1IrlIBNqgkmg63JBhY=', 'Content-Length': '0'} 2024-04-12T03:22:01.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007261640766587e3c-006618a8d9-1845-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:22:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:22:01.835 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:22:01.835 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:22:01.835 DEBUG:boto:Method: PUT 2024-04-12T03:22:01.835 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:22:01.835 DEBUG:boto:Data: 2024-04-12T03:22:01.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:22:01.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:22:01.836 DEBUG:boto:Port: 8000 2024-04-12T03:22:01.836 DEBUG:boto:Params: {} 2024-04-12T03:22:01.836 DEBUG:boto:Token: None 2024-04-12T03:22:01.836 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:22:01 GMT /dgbpxg-37/testredirect 2024-04-12T03:22:01.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bH1BzkB9JGjlBIqloPSJDf5XLww= 2024-04-12T03:22:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:22:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bH1BzkB9JGjlBIqloPSJDf5XLww='} 2024-04-12T03:22:01.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f0a5ce126885a4d-006618a8d9-1845-a1'), ('Date', 'Fri, 12 Apr 2024 03:22:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:22:01.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:22:01.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:22:01.847 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:22:01.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:22:01.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:22:01.932 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:01.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:22:01.832537Z" 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:01.946 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:22:01.947 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:22:01.948 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:22:01.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:22:01.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:22:01.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:22:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:22:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:22:02.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:22:02.025+0000 7ff0ddcc6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:22:02.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:22:02.026+0000 7ff0ddcc6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:22:02.040 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:22:02.040 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.041 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:22:02.043 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:22:02.044 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:22:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:22:02.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:22:02.047 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:22:02.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.047 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:22:02Z" 2024-04-12T03:22:02.047 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:22:02.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:22:02Z" } 2024-04-12T03:22:02.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:22:02.048 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:22:02.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:22:02.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:22:02.049 DEBUG:boto:path=/dgbpxg-37/ 2024-04-12T03:22:02.049 DEBUG:boto:auth_path=/dgbpxg-37/ 2024-04-12T03:22:02.049 DEBUG:boto:Method: HEAD 2024-04-12T03:22:02.049 DEBUG:boto:Path: /dgbpxg-37/ 2024-04-12T03:22:02.049 DEBUG:boto:Data: 2024-04-12T03:22:02.049 DEBUG:boto:Headers: {} 2024-04-12T03:22:02.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:22:02.049 DEBUG:boto:Port: 8001 2024-04-12T03:22:02.049 DEBUG:boto:Params: {} 2024-04-12T03:22:02.050 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:22:02.050 DEBUG:boto:Token: None 2024-04-12T03:22:02.050 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:22:02 GMT /dgbpxg-37/ 2024-04-12T03:22:02.050 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K9Xrbaw+ifZsbvqzfk6IjuL3YJQ= 2024-04-12T03:22:02.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:22:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K9Xrbaw+ifZsbvqzfk6IjuL3YJQ=', 'Content-Length': '0'} 2024-04-12T03:22:02.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008856cae952ee39bd-006618a8da-1847-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:22:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:22:02.066 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:22:02.066 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:22:02.066 DEBUG:boto:Method: HEAD 2024-04-12T03:22:02.066 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:22:02.066 DEBUG:boto:Data: 2024-04-12T03:22:02.066 DEBUG:boto:Headers: {} 2024-04-12T03:22:02.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:22:02.066 DEBUG:boto:Port: 8001 2024-04-12T03:22:02.066 DEBUG:boto:Params: {} 2024-04-12T03:22:02.067 DEBUG:boto:Token: None 2024-04-12T03:22:02.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:22:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:22:02.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y0uoSdCkczz8yZHJX83Ap9XLmTo= 2024-04-12T03:22:02.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:22:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y0uoSdCkczz8yZHJX83Ap9XLmTo=', 'Content-Length': '0'} 2024-04-12T03:22:02.068 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e89f816c5d596df6-006618a8da-1847-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:22:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:22:02.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=b7d2416c-9d6f-4780-9e14-1f66614caba8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:22:02.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=b7d2416c-9d6f-4780-9e14-1f66614caba8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:22:02.069 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=b7d2416c-9d6f-4780-9e14-1f66614caba8 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:22:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:22:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:22:02.135 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.136 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:22:02.138 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:22:02.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:22:02.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:22:02.140 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:22:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:22:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:22:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T03:22:02.253 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:22:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:22:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6, 2024-04-12T03:22:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:22:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:22:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:22:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:22:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:22:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:22:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:22:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:22:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:22:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:22:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:22:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:22:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:22:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:22:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:22:02.275 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:22:02.275 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T03:23:02.277 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.277 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.278 DEBUG:boto:Method: HEAD 2024-04-12T03:23:02.278 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.278 DEBUG:boto:Data: 2024-04-12T03:23:02.278 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.278 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.278 DEBUG:boto:Params: {} 2024-04-12T03:23:02.279 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.279 DEBUG:boto:Token: None 2024-04-12T03:23:02.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F/MZD9nIJHJpEOOocxjx02fPu1Q= 2024-04-12T03:23:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F/MZD9nIJHJpEOOocxjx02fPu1Q=', 'Content-Length': '0'} 2024-04-12T03:23:02.312 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx00000e3c7ba718c27b516-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.312 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.312 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2024-04-12T03:23:02.313 DEBUG:boto:Token: None 2024-04-12T03:23:02.313 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F/MZD9nIJHJpEOOocxjx02fPu1Q= 2024-04-12T03:23:02.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F/MZD9nIJHJpEOOocxjx02fPu1Q=', 'Content-Length': '0'} 2024-04-12T03:23:02.329 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:22:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000054a2d9a64c747b0a-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.329 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.329 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.330 DEBUG:boto:Method: GET 2024-04-12T03:23:02.330 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.330 DEBUG:boto:Data: 2024-04-12T03:23:02.330 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.330 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.330 DEBUG:boto:Params: {} 2024-04-12T03:23:02.330 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.330 DEBUG:boto:Token: None 2024-04-12T03:23:02.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.343 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx0000062da53984be49f93-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.343 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.343 DEBUG:boto:Token: None 2024-04-12T03:23:02.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.346 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:22:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000009563ea9ccff28d29-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.346 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.346 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.346 DEBUG:boto:Method: PUT 2024-04-12T03:23:02.347 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.347 DEBUG:boto:Data: 2024-04-12T03:23:02.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:23:02.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:23:02.347 DEBUG:boto:Port: 8000 2024-04-12T03:23:02.347 DEBUG:boto:Params: {} 2024-04-12T03:23:02.347 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:23:02.347 DEBUG:boto:Token: None 2024-04-12T03:23:02.347 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uw3Fjm0VAz0gHWZHj6HUKT+6O6Q= 2024-04-12T03:23:02.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uw3Fjm0VAz0gHWZHj6HUKT+6O6Q='} 2024-04-12T03:23:02.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c636bf7ccfd36b4a-006618a916-1871-a1'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.368 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.368 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.368 DEBUG:boto:Method: GET 2024-04-12T03:23:02.368 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.368 DEBUG:boto:Data: 2024-04-12T03:23:02.368 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.368 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.369 DEBUG:boto:Params: {} 2024-04-12T03:23:02.369 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.369 DEBUG:boto:Token: None 2024-04-12T03:23:02.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.381 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx0000090086a9e71004c99-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.381 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.381 DEBUG:boto:Token: None 2024-04-12T03:23:02.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.383 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx0000093e08f5328434a37-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.384 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.384 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.384 DEBUG:boto:Method: PUT 2024-04-12T03:23:02.384 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.384 DEBUG:boto:Data: 2024-04-12T03:23:02.384 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:23:02.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:23:02.384 DEBUG:boto:Port: 8000 2024-04-12T03:23:02.385 DEBUG:boto:Params: {} 2024-04-12T03:23:02.385 DEBUG:boto:Token: None 2024-04-12T03:23:02.385 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.385 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fK/fHDA9rOjVTAOV+EDbNO+Bc0s= 2024-04-12T03:23:02.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fK/fHDA9rOjVTAOV+EDbNO+Bc0s='} 2024-04-12T03:23:02.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000422f5fd6dfbdc1fc-006618a916-1871-a1'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.391 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.391 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.391 DEBUG:boto:Method: GET 2024-04-12T03:23:02.391 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.391 DEBUG:boto:Data: 2024-04-12T03:23:02.391 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.392 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.392 DEBUG:boto:Params: {} 2024-04-12T03:23:02.392 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.392 DEBUG:boto:Token: None 2024-04-12T03:23:02.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.420 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx0000033a9192a76c2be12-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.421 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.421 DEBUG:boto:Token: None 2024-04-12T03:23:02.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.423 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000c49ed0211a6c66b5-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.424 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.424 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.424 DEBUG:boto:Method: PUT 2024-04-12T03:23:02.424 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.424 DEBUG:boto:Data: 2024-04-12T03:23:02.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:23:02.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:23:02.424 DEBUG:boto:Port: 8000 2024-04-12T03:23:02.424 DEBUG:boto:Params: {} 2024-04-12T03:23:02.424 DEBUG:boto:Token: None 2024-04-12T03:23:02.425 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7p8dV7RQ3q2Yye9gQWiSRlQroIg= 2024-04-12T03:23:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7p8dV7RQ3q2Yye9gQWiSRlQroIg='} 2024-04-12T03:23:02.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d541edf361d5ce1f-006618a916-1871-a1'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.431 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.431 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.431 DEBUG:boto:Method: GET 2024-04-12T03:23:02.431 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.431 DEBUG:boto:Data: 2024-04-12T03:23:02.431 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.431 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.431 DEBUG:boto:Params: {} 2024-04-12T03:23:02.431 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.431 DEBUG:boto:Token: None 2024-04-12T03:23:02.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.443 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx000000ff9bd1f69f81ba4-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.443 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.443 DEBUG:boto:Token: None 2024-04-12T03:23:02.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.445 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000feab5c824de35c95-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.486 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.487 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.487 DEBUG:boto:Method: PUT 2024-04-12T03:23:02.487 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.487 DEBUG:boto:Data: 2024-04-12T03:23:02.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:23:02.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:23:02.487 DEBUG:boto:Port: 8000 2024-04-12T03:23:02.487 DEBUG:boto:Params: {} 2024-04-12T03:23:02.487 DEBUG:boto:Token: None 2024-04-12T03:23:02.487 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/vCzh4vu2GPS4vsa68bB4RPabLI= 2024-04-12T03:23:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/vCzh4vu2GPS4vsa68bB4RPabLI='} 2024-04-12T03:23:02.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e3b7039879e3a8e-006618a916-1871-a1'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.493 DEBUG:boto:path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.494 DEBUG:boto:auth_path=/dgbpxg-37/testredirect 2024-04-12T03:23:02.494 DEBUG:boto:Method: GET 2024-04-12T03:23:02.494 DEBUG:boto:Path: /dgbpxg-37/testredirect 2024-04-12T03:23:02.494 DEBUG:boto:Data: 2024-04-12T03:23:02.494 DEBUG:boto:Headers: {} 2024-04-12T03:23:02.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:23:02.494 DEBUG:boto:Port: 8001 2024-04-12T03:23:02.494 DEBUG:boto:Params: {} 2024-04-12T03:23:02.494 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:23:02.494 DEBUG:boto:Token: None 2024-04-12T03:23:02.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.506 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/dgbpxg-37/testredirect?'), ('x-amz-request-id', 'tx00000a19edbf9027eb3f2-006618a916-186d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.507 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/dgbpxg-37/testredirect 2024-04-12T03:23:02.507 DEBUG:boto:Token: None 2024-04-12T03:23:02.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:23:02 GMT /dgbpxg-37/testredirect 2024-04-12T03:23:02.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo= 2024-04-12T03:23:02.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:23:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TPm/MPgyu8KE9qAqxMJ9S4R6Wo=', 'Content-Length': '0'} 2024-04-12T03:23:02.509 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000bafc696336e72e40-006618a916-1871-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:23:02.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:23:02.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:23:02.510 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:23:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:23:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:23:02.585 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:23:02.586 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T03:23:02.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:23:02.588 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:23:02.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:23:02.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:23:02.590 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:23:02.638 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:23:02.638 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:23:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T03:23:02.703 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:23:02.703 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:23:02.703 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7, 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:23:02.704 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:23:02.705 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:23:02.706 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:23:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:23:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:23:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:23:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:23:02.715 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:23:02.716 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:23:02.717 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:23:02.718 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:23:02.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.720 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:23:02.721 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:23:02.722 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:23:02.723 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:23:02.724 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:23:02.725 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2024-04-12T03:23:02.945 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:23:02.943+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:24:02.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:24:02.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:24:02.727 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:24:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:02.790 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:02.815 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:24:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:24:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:24:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:24:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:24:02.819 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:24:02.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:24:02.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:24:02.820 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:24:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8, 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:24:02.949 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.950 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:24:02.951 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:24:02.952 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:24:02.953 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:24:02.954 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:24:02.955 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.956 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.957 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:24:02.959 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:24:02.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:24:02.961 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:24:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:24:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:24:02.965 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:24:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:24:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:24:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:24:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.971 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:24:02.971 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:24:02.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.972 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:24:02.972 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:24:02.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:24:02.973 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:24:02.974 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.975 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:24:02.976 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:24:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:24:02.978 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:24:02.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:24:02.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:24:02.978 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:24:02.979 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:24:02.979 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:24:02.979 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:24:02.979 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2024-04-12T03:25:02.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-04-12T03:25:02.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:02.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:02.984 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:25:03.083 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:22:01.832537Z" 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.085 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:25:03.086 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:03.086 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:25:03.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:03.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:03.087 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:03.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:03.165+0000 7f5478a00780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:03.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:03.166+0000 7f5478a00780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:03.179 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:03.179 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:03.179 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:03.179 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:03.179 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:03.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:03.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:03.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:03.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:03.184 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:03.185 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:03.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.185 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:03Z" 2024-04-12T03:25:03.185 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:03.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:03Z" } 2024-04-12T03:25:03.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:03.186 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:03.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:03.186 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:25:03.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:03.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RCLIPQLTT1I9NFE9', '--secret', '8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:03.187 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key RCLIPQLTT1I9NFE9 --secret 8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:03.251 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=ff072c13-de01-44da-babe-80b386c4a5cf 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "cbb9cfea-0874-45a6-8092-1afdd399573f", 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "remove.rgw.control", 2024-04-12T03:25:03.265 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "remove.rgw.log", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-12T03:25:03.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "remove.rgw.otp", 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RCLIPQLTT1I9NFE9", 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "8TfnveyiBcem61uV4ZL1lHgMEC8k1cGh" 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:25:03.267 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:03.268 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-12T03:25:03.269 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:03.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:03.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:03.270 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:03.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:03.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:03.328+0000 7ff074dbc780 0 period (3ee804bf-b01f-4af0-895f-822e923b43dd does not have zone cbb9cfea-0874-45a6-8092-1afdd399573f configured 2024-04-12T03:25:09.638 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b7d2416c-9d6f-4780-9e14-1f66614caba8 2024-04-12T03:25:09.665 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:09.665 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:09.665 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9, 2024-04-12T03:25:09.665 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:09.666 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.667 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.668 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "cbb9cfea-0874-45a6-8092-1afdd399573f", 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.669 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.671 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.672 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:09.673 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:09.674 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:25:09.675 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.676 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.677 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.678 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.679 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.680 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "cbb9cfea-0874-45a6-8092-1afdd399573f", 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 132998596 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.682 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:09.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:09.684 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:09.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:25:09.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:09.688 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:09.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.688 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:09.746 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:09.774 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:09.775 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:09.782 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:09.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.784 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:09.853 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:zone name remove not found in zonegroup a 2024-04-12T03:25:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-12T03:25:09.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:09.899 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:09.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:09.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.030 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:10.123+0000 7f31d35c4780 0 WARNING: can't generate connection for zone cbb9cfea-0874-45a6-8092-1afdd399573f id remove: no endpoints defined 2024-04-12T03:25:10.194 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:10.194 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10, 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:25:10.195 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.196 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.197 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:10.198 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.199 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:10.200 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.201 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.202 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.221 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.222 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.224 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:25:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:25:10.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:10.230 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:10.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-12T03:25:10.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.294 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.294 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.325 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:10.325 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "ba061b05-09b3-4941-b8fc-84d9347d6358", 2024-04-12T03:25:10.325 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.326 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [] 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.327 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.328 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:10.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.329 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.384 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.384 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:10.428+0000 7f46c3aad780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-12T03:25:10.434 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:10.434 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "6a497e9d-8dc0-4bc5-a87d-578db70ba032", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "remove.rgw.control", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "remove.rgw.log", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-12T03:25:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "remove.rgw.otp", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2024-04-12T03:25:10.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.438 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.438 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-12T03:25:10.438 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:10.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.439 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.630 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:10.630 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:10.630 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11, 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:25:10.631 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.632 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.633 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.634 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.636 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:25:10.637 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.638 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:10.640 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.643 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "ba061b05-09b3-4941-b8fc-84d9347d6358", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "6a497e9d-8dc0-4bc5-a87d-578db70ba032", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "6a497e9d-8dc0-4bc5-a87d-578db70ba032", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2024-04-12T03:25:10.644 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:10.645 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:10.647 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "6a497e9d-8dc0-4bc5-a87d-578db70ba032", 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 393967482 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:10.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:10.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.650 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:25:10.650 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:10.650 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:25:10.657 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.658 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:25:10.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:10.661 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:10.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.662 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:10.788+0000 7fa061835780 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-12T03:25:10.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:10.794+0000 7fa061835780 0 Removed zone from zonegroup remove_zg 2024-04-12T03:25:10.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.812 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:10.918+0000 7f4c85b99780 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-12T03:25:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-12T03:25:10.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:10.925 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:11.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:11.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:25:11.015 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:25:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12, 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T03:25:11.158 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:11.159 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:11.160 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:11.162 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.163 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:11.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.165 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T03:25:11.166 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:11.167 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:11.168 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:11.168 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T03:25:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T03:25:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T03:25:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T03:25:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T03:25:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:11.192 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:11.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-12T03:25:11.194 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-38 2024-04-12T03:25:11.195 DEBUG:boto:path=/dgbpxg-38/ 2024-04-12T03:25:11.195 DEBUG:boto:auth_path=/dgbpxg-38/ 2024-04-12T03:25:11.195 DEBUG:boto:Method: PUT 2024-04-12T03:25:11.195 DEBUG:boto:Path: /dgbpxg-38/ 2024-04-12T03:25:11.195 DEBUG:boto:Data: 2024-04-12T03:25:11.195 DEBUG:boto:Headers: {} 2024-04-12T03:25:11.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:11.195 DEBUG:boto:Port: 8000 2024-04-12T03:25:11.196 DEBUG:boto:Params: {} 2024-04-12T03:25:11.196 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:25:11.196 DEBUG:boto:Token: None 2024-04-12T03:25:11.197 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:11 GMT /dgbpxg-38/ 2024-04-12T03:25:11.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:52pet/W5uv7bOT1vGyyAT8m1Fa0= 2024-04-12T03:25:11.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:52pet/W5uv7bOT1vGyyAT8m1Fa0=', 'Content-Length': '0'} 2024-04-12T03:25:11.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e704975a20ca8d1d-006618a997-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:11.255 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-39 2024-04-12T03:25:11.255 DEBUG:boto:path=/dgbpxg-39/ 2024-04-12T03:25:11.256 DEBUG:boto:auth_path=/dgbpxg-39/ 2024-04-12T03:25:11.256 DEBUG:boto:Method: PUT 2024-04-12T03:25:11.256 DEBUG:boto:Path: /dgbpxg-39/ 2024-04-12T03:25:11.256 DEBUG:boto:Data: 2024-04-12T03:25:11.256 DEBUG:boto:Headers: {} 2024-04-12T03:25:11.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:25:11.256 DEBUG:boto:Port: 8001 2024-04-12T03:25:11.256 DEBUG:boto:Params: {} 2024-04-12T03:25:11.257 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:25:11.257 DEBUG:boto:Token: None 2024-04-12T03:25:11.257 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:11 GMT /dgbpxg-39/ 2024-04-12T03:25:11.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9pf07W1RZYybavOA6HPt1CRrjS4= 2024-04-12T03:25:11.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9pf07W1RZYybavOA6HPt1CRrjS4=', 'Content-Length': '0'} 2024-04-12T03:25:11.469 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042b62abd00885ca8-006618a997-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:11.470 INFO:rgw_multi.tests:create bucket zone=b1 name=dgbpxg-40 2024-04-12T03:25:11.470 DEBUG:boto:path=/dgbpxg-40/ 2024-04-12T03:25:11.470 DEBUG:boto:auth_path=/dgbpxg-40/ 2024-04-12T03:25:11.470 DEBUG:boto:Method: PUT 2024-04-12T03:25:11.470 DEBUG:boto:Path: /dgbpxg-40/ 2024-04-12T03:25:11.470 DEBUG:boto:Data: 2024-04-12T03:25:11.470 DEBUG:boto:Headers: {} 2024-04-12T03:25:11.470 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:25:11.470 DEBUG:boto:Port: 8000 2024-04-12T03:25:11.471 DEBUG:boto:Params: {} 2024-04-12T03:25:11.471 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:25:11.471 DEBUG:boto:Token: None 2024-04-12T03:25:11.471 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:11 GMT /dgbpxg-40/ 2024-04-12T03:25:11.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zrN0kDlBFAKMALrZttzk3mRLPL0= 2024-04-12T03:25:11.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:11 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zrN0kDlBFAKMALrZttzk3mRLPL0=', 'Content-Length': '0'} 2024-04-12T03:25:12.703 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bac5baf72bf1654-006618a998-115d-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.704 INFO:rgw_multi.tests:create bucket zone=b2 name=dgbpxg-41 2024-04-12T03:25:12.704 DEBUG:boto:path=/dgbpxg-41/ 2024-04-12T03:25:12.704 DEBUG:boto:auth_path=/dgbpxg-41/ 2024-04-12T03:25:12.704 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.704 DEBUG:boto:Path: /dgbpxg-41/ 2024-04-12T03:25:12.704 DEBUG:boto:Data: 2024-04-12T03:25:12.704 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.704 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:25:12.705 DEBUG:boto:Port: 8001 2024-04-12T03:25:12.705 DEBUG:boto:Params: {} 2024-04-12T03:25:12.705 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:25:12.705 DEBUG:boto:Token: None 2024-04-12T03:25:12.705 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-41/ 2024-04-12T03:25:12.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kK70LNU2KFWRy888sX+jpJW1y3o= 2024-04-12T03:25:12.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kK70LNU2KFWRy888sX+jpJW1y3o=', 'Content-Length': '0'} 2024-04-12T03:25:12.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085c9cad46f21b3b3-006618a998-1159-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.759 DEBUG:boto:path=/dgbpxg-38/ 2024-04-12T03:25:12.759 DEBUG:boto:auth_path=/dgbpxg-38/ 2024-04-12T03:25:12.759 DEBUG:boto:path=/dgbpxg-38/?website 2024-04-12T03:25:12.759 DEBUG:boto:auth_path=/dgbpxg-38/?website 2024-04-12T03:25:12.759 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.759 DEBUG:boto:Path: /dgbpxg-38/?website 2024-04-12T03:25:12.759 DEBUG:boto:Data: index.htmlerror.html 2024-04-12T03:25:12.759 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:12.759 DEBUG:boto:Port: 8000 2024-04-12T03:25:12.759 DEBUG:boto:Params: {} 2024-04-12T03:25:12.760 DEBUG:boto:Token: None 2024-04-12T03:25:12.760 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-38/?website 2024-04-12T03:25:12.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFBCiPT/Xdip/rpsfWYTDzTANU0= 2024-04-12T03:25:12.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFBCiPT/Xdip/rpsfWYTDzTANU0=', 'Content-Length': '241'} 2024-04-12T03:25:12.761 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx0000040c49846d2c160e8-006618a998-1921-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-12T03:25:12.762 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-42 2024-04-12T03:25:12.762 DEBUG:boto:path=/dgbpxg-42/ 2024-04-12T03:25:12.763 DEBUG:boto:auth_path=/dgbpxg-42/ 2024-04-12T03:25:12.763 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.763 DEBUG:boto:Path: /dgbpxg-42/ 2024-04-12T03:25:12.763 DEBUG:boto:Data: 2024-04-12T03:25:12.763 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:12.763 DEBUG:boto:Port: 8000 2024-04-12T03:25:12.763 DEBUG:boto:Params: {} 2024-04-12T03:25:12.763 DEBUG:boto:Token: None 2024-04-12T03:25:12.763 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-42/ 2024-04-12T03:25:12.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aFxx98v4699ZZkw7kFfGSDjNksE= 2024-04-12T03:25:12.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aFxx98v4699ZZkw7kFfGSDjNksE=', 'Content-Length': '0'} 2024-04-12T03:25:12.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008962470d80c6cadd-006618a998-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.782 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-43 2024-04-12T03:25:12.782 DEBUG:boto:path=/dgbpxg-43/ 2024-04-12T03:25:12.782 DEBUG:boto:auth_path=/dgbpxg-43/ 2024-04-12T03:25:12.782 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.782 DEBUG:boto:Path: /dgbpxg-43/ 2024-04-12T03:25:12.782 DEBUG:boto:Data: 2024-04-12T03:25:12.782 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:25:12.782 DEBUG:boto:Port: 8001 2024-04-12T03:25:12.782 DEBUG:boto:Params: {} 2024-04-12T03:25:12.783 DEBUG:boto:Token: None 2024-04-12T03:25:12.783 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-43/ 2024-04-12T03:25:12.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y5bKV8XnviTRpW2rKOt39bU8Jh4= 2024-04-12T03:25:12.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y5bKV8XnviTRpW2rKOt39bU8Jh4=', 'Content-Length': '0'} 2024-04-12T03:25:12.811 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000daeb4015a9d1c9e-006618a998-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.811 INFO:rgw_multi.tests:create bucket zone=b1 name=dgbpxg-44 2024-04-12T03:25:12.811 DEBUG:boto:path=/dgbpxg-44/ 2024-04-12T03:25:12.811 DEBUG:boto:auth_path=/dgbpxg-44/ 2024-04-12T03:25:12.811 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.811 DEBUG:boto:Path: /dgbpxg-44/ 2024-04-12T03:25:12.812 DEBUG:boto:Data: 2024-04-12T03:25:12.812 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.812 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:25:12.812 DEBUG:boto:Port: 8000 2024-04-12T03:25:12.812 DEBUG:boto:Params: {} 2024-04-12T03:25:12.812 DEBUG:boto:Token: None 2024-04-12T03:25:12.812 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-44/ 2024-04-12T03:25:12.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IkVANpYwtDMxQLlcZUZPPmdz2s8= 2024-04-12T03:25:12.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IkVANpYwtDMxQLlcZUZPPmdz2s8=', 'Content-Length': '0'} 2024-04-12T03:25:12.870 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002bd4084f2b8b8416-006618a998-115d-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.871 INFO:rgw_multi.tests:create bucket zone=b2 name=dgbpxg-45 2024-04-12T03:25:12.871 DEBUG:boto:path=/dgbpxg-45/ 2024-04-12T03:25:12.871 DEBUG:boto:auth_path=/dgbpxg-45/ 2024-04-12T03:25:12.871 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.871 DEBUG:boto:Path: /dgbpxg-45/ 2024-04-12T03:25:12.871 DEBUG:boto:Data: 2024-04-12T03:25:12.872 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.872 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:25:12.872 DEBUG:boto:Port: 8001 2024-04-12T03:25:12.872 DEBUG:boto:Params: {} 2024-04-12T03:25:12.872 DEBUG:boto:Token: None 2024-04-12T03:25:12.872 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-45/ 2024-04-12T03:25:12.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nSllIOI1SNOdwz+uKktHJJcbVRM= 2024-04-12T03:25:12.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nSllIOI1SNOdwz+uKktHJJcbVRM=', 'Content-Length': '0'} 2024-04-12T03:25:12.913 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e7a23130f42e81d-006618a998-1159-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.913 DEBUG:boto:path=/dgbpxg-42/ 2024-04-12T03:25:12.913 DEBUG:boto:auth_path=/dgbpxg-42/ 2024-04-12T03:25:12.913 DEBUG:boto:path=/dgbpxg-42/?policy 2024-04-12T03:25:12.914 DEBUG:boto:auth_path=/dgbpxg-42/?policy 2024-04-12T03:25:12.914 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.914 DEBUG:boto:Path: /dgbpxg-42/?policy 2024-04-12T03:25:12.914 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:25:12.914 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:12.914 DEBUG:boto:Port: 8000 2024-04-12T03:25:12.914 DEBUG:boto:Params: {} 2024-04-12T03:25:12.914 DEBUG:boto:Token: None 2024-04-12T03:25:12.914 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-42/?policy 2024-04-12T03:25:12.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Kvjf64BzLIRUKr+FzTdDmehm8IU= 2024-04-12T03:25:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Kvjf64BzLIRUKr+FzTdDmehm8IU=', 'Content-Length': '97'} 2024-04-12T03:25:12.920 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e0f8c69a55a3bb8-006618a998-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.920 DEBUG:boto:path=/dgbpxg-42/ 2024-04-12T03:25:12.920 DEBUG:boto:auth_path=/dgbpxg-42/ 2024-04-12T03:25:12.920 DEBUG:boto:path=/dgbpxg-42/?policy 2024-04-12T03:25:12.920 DEBUG:boto:auth_path=/dgbpxg-42/?policy 2024-04-12T03:25:12.920 DEBUG:boto:Method: GET 2024-04-12T03:25:12.920 DEBUG:boto:Path: /dgbpxg-42/?policy 2024-04-12T03:25:12.920 DEBUG:boto:Data: 2024-04-12T03:25:12.920 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:12.921 DEBUG:boto:Port: 8000 2024-04-12T03:25:12.921 DEBUG:boto:Params: {} 2024-04-12T03:25:12.921 DEBUG:boto:Token: None 2024-04-12T03:25:12.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-42/?policy 2024-04-12T03:25:12.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GKFB84p7QiRzJZvCU7FqSAq7/lY= 2024-04-12T03:25:12.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GKFB84p7QiRzJZvCU7FqSAq7/lY=', 'Content-Length': '0'} 2024-04-12T03:25:12.922 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000727b2e322040ae1f-006618a998-1921-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:25:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:12.922 DEBUG:boto:path=/dgbpxg-43/ 2024-04-12T03:25:12.922 DEBUG:boto:auth_path=/dgbpxg-43/ 2024-04-12T03:25:12.923 DEBUG:boto:path=/dgbpxg-43/?policy 2024-04-12T03:25:12.923 DEBUG:boto:auth_path=/dgbpxg-43/?policy 2024-04-12T03:25:12.923 DEBUG:boto:Method: PUT 2024-04-12T03:25:12.923 DEBUG:boto:Path: /dgbpxg-43/?policy 2024-04-12T03:25:12.923 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:25:12.923 DEBUG:boto:Headers: {} 2024-04-12T03:25:12.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:25:12.923 DEBUG:boto:Port: 8001 2024-04-12T03:25:12.923 DEBUG:boto:Params: {} 2024-04-12T03:25:12.923 DEBUG:boto:Token: None 2024-04-12T03:25:12.923 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:12 GMT /dgbpxg-43/?policy 2024-04-12T03:25:12.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sys5P6jnYXIHnrcm3h7DYolahNc= 2024-04-12T03:25:12.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:12 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sys5P6jnYXIHnrcm3h7DYolahNc=', 'Content-Length': '97'} 2024-04-12T03:25:13.934 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001fdf324c6d4211f4-006618a998-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:13.934 DEBUG:boto:path=/dgbpxg-43/ 2024-04-12T03:25:13.935 DEBUG:boto:auth_path=/dgbpxg-43/ 2024-04-12T03:25:13.935 DEBUG:boto:path=/dgbpxg-43/?policy 2024-04-12T03:25:13.935 DEBUG:boto:auth_path=/dgbpxg-43/?policy 2024-04-12T03:25:13.935 DEBUG:boto:Method: GET 2024-04-12T03:25:13.935 DEBUG:boto:Path: /dgbpxg-43/?policy 2024-04-12T03:25:13.935 DEBUG:boto:Data: 2024-04-12T03:25:13.935 DEBUG:boto:Headers: {} 2024-04-12T03:25:13.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:25:13.935 DEBUG:boto:Port: 8001 2024-04-12T03:25:13.935 DEBUG:boto:Params: {} 2024-04-12T03:25:13.935 DEBUG:boto:Token: None 2024-04-12T03:25:13.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:13 GMT /dgbpxg-43/?policy 2024-04-12T03:25:13.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eRZ6o+O3jcGvlOJON+k4ubC+1VM= 2024-04-12T03:25:13.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eRZ6o+O3jcGvlOJON+k4ubC+1VM=', 'Content-Length': '0'} 2024-04-12T03:25:13.937 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040eaef72fc72596d-006618a999-192d-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:25:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:13.937 DEBUG:boto:path=/dgbpxg-44/ 2024-04-12T03:25:13.937 DEBUG:boto:auth_path=/dgbpxg-44/ 2024-04-12T03:25:13.937 DEBUG:boto:path=/dgbpxg-44/?policy 2024-04-12T03:25:13.938 DEBUG:boto:auth_path=/dgbpxg-44/?policy 2024-04-12T03:25:13.938 DEBUG:boto:Method: PUT 2024-04-12T03:25:13.938 DEBUG:boto:Path: /dgbpxg-44/?policy 2024-04-12T03:25:13.938 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:25:13.938 DEBUG:boto:Headers: {} 2024-04-12T03:25:13.938 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:25:13.938 DEBUG:boto:Port: 8000 2024-04-12T03:25:13.938 DEBUG:boto:Params: {} 2024-04-12T03:25:13.938 DEBUG:boto:Token: None 2024-04-12T03:25:13.938 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:13 GMT /dgbpxg-44/?policy 2024-04-12T03:25:13.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:px2E6e26Rw0YH6qWwDVPWzd0qqE= 2024-04-12T03:25:13.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:px2E6e26Rw0YH6qWwDVPWzd0qqE=', 'Content-Length': '97'} 2024-04-12T03:25:14.950 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014db827edb3b0cc1-006618a999-115d-b1'), ('Date', 'Fri, 12 Apr 2024 03:25:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:14.951 DEBUG:boto:path=/dgbpxg-44/ 2024-04-12T03:25:14.951 DEBUG:boto:auth_path=/dgbpxg-44/ 2024-04-12T03:25:14.951 DEBUG:boto:path=/dgbpxg-44/?policy 2024-04-12T03:25:14.951 DEBUG:boto:auth_path=/dgbpxg-44/?policy 2024-04-12T03:25:14.951 DEBUG:boto:Method: GET 2024-04-12T03:25:14.951 DEBUG:boto:Path: /dgbpxg-44/?policy 2024-04-12T03:25:14.951 DEBUG:boto:Data: 2024-04-12T03:25:14.951 DEBUG:boto:Headers: {} 2024-04-12T03:25:14.951 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8000 2024-04-12T03:25:14.951 DEBUG:boto:Port: 8000 2024-04-12T03:25:14.951 DEBUG:boto:Params: {} 2024-04-12T03:25:14.952 DEBUG:boto:Token: None 2024-04-12T03:25:14.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:14 GMT /dgbpxg-44/?policy 2024-04-12T03:25:14.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YAYfL6xzdc+Ku2elVECNGE23zAI= 2024-04-12T03:25:14.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YAYfL6xzdc+Ku2elVECNGE23zAI=', 'Content-Length': '0'} 2024-04-12T03:25:14.953 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d426547e7054922a-006618a99a-115d-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:25:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:14.954 DEBUG:boto:path=/dgbpxg-45/ 2024-04-12T03:25:14.954 DEBUG:boto:auth_path=/dgbpxg-45/ 2024-04-12T03:25:14.954 DEBUG:boto:path=/dgbpxg-45/?policy 2024-04-12T03:25:14.954 DEBUG:boto:auth_path=/dgbpxg-45/?policy 2024-04-12T03:25:14.954 DEBUG:boto:Method: PUT 2024-04-12T03:25:14.954 DEBUG:boto:Path: /dgbpxg-45/?policy 2024-04-12T03:25:14.954 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:25:14.954 DEBUG:boto:Headers: {} 2024-04-12T03:25:14.954 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:25:14.954 DEBUG:boto:Port: 8001 2024-04-12T03:25:14.955 DEBUG:boto:Params: {} 2024-04-12T03:25:14.955 DEBUG:boto:Token: None 2024-04-12T03:25:14.955 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:14 GMT /dgbpxg-45/?policy 2024-04-12T03:25:14.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WhE1ApIhQnJz1KwqqcNG0zuJXKw= 2024-04-12T03:25:14.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WhE1ApIhQnJz1KwqqcNG0zuJXKw=', 'Content-Length': '97'} 2024-04-12T03:25:15.966 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0e58bf378fb860f-006618a99a-1159-b2'), ('Date', 'Fri, 12 Apr 2024 03:25:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:15.966 DEBUG:boto:path=/dgbpxg-45/ 2024-04-12T03:25:15.966 DEBUG:boto:auth_path=/dgbpxg-45/ 2024-04-12T03:25:15.967 DEBUG:boto:path=/dgbpxg-45/?policy 2024-04-12T03:25:15.967 DEBUG:boto:auth_path=/dgbpxg-45/?policy 2024-04-12T03:25:15.967 DEBUG:boto:Method: GET 2024-04-12T03:25:15.967 DEBUG:boto:Path: /dgbpxg-45/?policy 2024-04-12T03:25:15.967 DEBUG:boto:Data: 2024-04-12T03:25:15.967 DEBUG:boto:Headers: {} 2024-04-12T03:25:15.967 DEBUG:boto:Host: smithi181.front.sepia.ceph.com:8001 2024-04-12T03:25:15.967 DEBUG:boto:Port: 8001 2024-04-12T03:25:15.967 DEBUG:boto:Params: {} 2024-04-12T03:25:15.967 DEBUG:boto:Token: None 2024-04-12T03:25:15.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:15 GMT /dgbpxg-45/?policy 2024-04-12T03:25:15.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UyGnYQlvh47OsFiQqI8j+qzdi/E= 2024-04-12T03:25:15.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UyGnYQlvh47OsFiQqI8j+qzdi/E=', 'Content-Length': '0'} 2024-04-12T03:25:15.969 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000767a5140d6f1818f-006618a99b-1159-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:25:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:15.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-12T03:25:15.970 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-46 2024-04-12T03:25:15.970 DEBUG:boto:path=/dgbpxg-46/ 2024-04-12T03:25:15.970 DEBUG:boto:auth_path=/dgbpxg-46/ 2024-04-12T03:25:15.970 DEBUG:boto:Method: PUT 2024-04-12T03:25:15.970 DEBUG:boto:Path: /dgbpxg-46/ 2024-04-12T03:25:15.971 DEBUG:boto:Data: 2024-04-12T03:25:15.971 DEBUG:boto:Headers: {} 2024-04-12T03:25:15.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:25:15.971 DEBUG:boto:Port: 8000 2024-04-12T03:25:15.971 DEBUG:boto:Params: {} 2024-04-12T03:25:15.971 DEBUG:boto:Token: None 2024-04-12T03:25:15.971 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:15 GMT /dgbpxg-46/ 2024-04-12T03:25:15.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NYLQxXRnUCT6Alqp8YSjgRioIxo= 2024-04-12T03:25:15.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NYLQxXRnUCT6Alqp8YSjgRioIxo=', 'Content-Length': '0'} 2024-04-12T03:25:15.988 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000288a17d9e2201fe5-006618a99b-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:15.989 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-47 2024-04-12T03:25:15.989 DEBUG:boto:path=/dgbpxg-47/ 2024-04-12T03:25:15.989 DEBUG:boto:auth_path=/dgbpxg-47/ 2024-04-12T03:25:15.989 DEBUG:boto:Method: PUT 2024-04-12T03:25:15.989 DEBUG:boto:Path: /dgbpxg-47/ 2024-04-12T03:25:15.989 DEBUG:boto:Data: 2024-04-12T03:25:15.989 DEBUG:boto:Headers: {} 2024-04-12T03:25:15.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:25:15.989 DEBUG:boto:Port: 8001 2024-04-12T03:25:15.990 DEBUG:boto:Params: {} 2024-04-12T03:25:15.990 DEBUG:boto:Token: None 2024-04-12T03:25:15.990 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:15 GMT /dgbpxg-47/ 2024-04-12T03:25:15.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ozw9H4lnw7zstBSLvaHZN73iKE= 2024-04-12T03:25:15.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ozw9H4lnw7zstBSLvaHZN73iKE=', 'Content-Length': '0'} 2024-04-12T03:25:16.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eba3825dc8fd4b5d-006618a99b-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:16.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:16.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:16.020 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:16.100 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:25:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892316.006867_853.1", 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:25:16.006867Z" 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.114 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:25:16.115 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:16.115 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:25:16.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:16.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:16.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:16.192+0000 7fd728a30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:16.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:16.193+0000 7fd728a30780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:16.206 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:16.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:16.211 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:16Z" 2024-04-12T03:25:16.212 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:16.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:16Z" } 2024-04-12T03:25:16.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:16.214 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:16.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:16.214 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:21.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:21.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:21.216 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:21.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:21.304+0000 7f7336044780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:21.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:21.305+0000 7f7336044780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:21Z" 2024-04-12T03:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:21.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:21Z" } 2024-04-12T03:25:21.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:21.328 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:21.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:21.329 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:26.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:26.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:26.330 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:26.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:26.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:26.417+0000 7fdd08686780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:26.418+0000 7fdd08686780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:26.435 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:26.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:26.437 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:26.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:26.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:26.441 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:26Z" 2024-04-12T03:25:26.442 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:26.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:26Z" } 2024-04-12T03:25:26.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:26.443 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:26.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:26.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:31.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:31.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:31.445 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:31.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:31.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:31.532+0000 7f72d25e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:31.532+0000 7f72d25e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:31.548 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:31.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:31.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:31.551 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:31.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:31.553 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:31.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:31.554 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:31Z" 2024-04-12T03:25:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:31.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:31Z" } 2024-04-12T03:25:31.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:31.555 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:31.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:31.555 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:36.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:36.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:36.556 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:36.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:36.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:36.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:36.645+0000 7efd790ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:36.646+0000 7efd790ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:36.662 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:36.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:36.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:36Z" 2024-04-12T03:25:36.665 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:36.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:36Z" } 2024-04-12T03:25:36.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:36.667 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:36.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:36.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:41.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:41.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:41.669 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:41.723 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:41.723 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:41.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:41.754+0000 7fdd48c53780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:41.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:41.755+0000 7fdd48c53780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:41.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:41.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:41.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:41.772 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:41.773 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:41.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:41.775 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:41.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:41.775 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:41Z" 2024-04-12T03:25:41.775 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:41.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:41Z" } 2024-04-12T03:25:41.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:41.776 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:41.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:41.776 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:46.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:46.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:46.778 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:46.866+0000 7f6ad0cdf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:46.867+0000 7f6ad0cdf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:46.883 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:46.883 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:46.883 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.884 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:46.885 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:46.886 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:46.887 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:46.888 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:46.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:46.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:46.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:46.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:46.890 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:46.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:46.890 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:46.890 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:46.890 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:46.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:46Z" 2024-04-12T03:25:46.892 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:46.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:46Z" } 2024-04-12T03:25:46.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:46.894 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:46.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:46.895 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:51.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:51.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:51.896 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:51.984+0000 7fbdada65780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:51.985+0000 7fbdada65780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:51.999 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:52.000 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:52.001 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:52.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:52.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:51Z" 2024-04-12T03:25:52.005 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:52.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:51Z" } 2024-04-12T03:25:52.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:52.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:52.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:52.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:25:57.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:57.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:25:57.008 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:25:57.063 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:57.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:57.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:57.095+0000 7f12e20f2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:57.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:25:57.096+0000 7f12e20f2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:25:57.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:57.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:57.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:57.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:25:57.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:25:57.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:25:57.116 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:25:57Z" 2024-04-12T03:25:57.117 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:25:57.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:25:57Z" } 2024-04-12T03:25:57.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:25:57.118 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:25:57.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:25:57.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:02.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:02.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:02.120 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:02.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:02.208+0000 7efdcda46780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:02.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:02.209+0000 7efdcda46780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:02.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:02.225 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:02.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:02.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:02.228 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:02.229 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:02.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:02.230 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:02Z" 2024-04-12T03:26:02.230 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:02.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:02Z" } 2024-04-12T03:26:02.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:02.231 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:02.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:02.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:07.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.233 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:07.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:07.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:07.320+0000 7feb0ab38780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:07.321+0000 7feb0ab38780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:07.336 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:07.336 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:07Z" 2024-04-12T03:26:07.342 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:07.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:07Z" } 2024-04-12T03:26:07.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:07.344 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:07.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:07.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:12.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:12.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:12.345 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:12.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:12.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:12.435 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:26:12.433+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:26:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:12.441+0000 7f88b3963780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:12.442+0000 7f88b3963780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:12.473 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:12Z" 2024-04-12T03:26:12.479 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:12.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:12Z" } 2024-04-12T03:26:12.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:12.480 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:12.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:12.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:17.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:17.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:17.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:17.571+0000 7fb3b19b8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:17.572+0000 7fb3b19b8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:17.587 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:17.587 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:17.587 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:17.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:17.589 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:17.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:17.592 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:17Z" 2024-04-12T03:26:17.593 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:17.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:17Z" } 2024-04-12T03:26:17.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:17.595 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:17.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:17.595 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:22.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:22.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:22.597 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:22.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:22.684+0000 7f24cb2cd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:22.685+0000 7f24cb2cd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:22.699 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:22.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:22.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:22.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:22.704 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:22Z" 2024-04-12T03:26:22.705 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:22.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:22Z" } 2024-04-12T03:26:22.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:22.707 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:22.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:22.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:27.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:27.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:27.709 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:27.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:27.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:27.801+0000 7ffabe771780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:27.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:27.802+0000 7ffabe771780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:27.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:27.819 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:27.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:27.821 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:27.822 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:27.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:27.824 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:27Z" 2024-04-12T03:26:27.824 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:27.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:27Z" } 2024-04-12T03:26:27.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:27.825 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:27.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:27.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:32.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:32.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:32.827 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:32.917+0000 7f0c81b97780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:32.918+0000 7f0c81b97780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:32.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:32.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:32.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:32.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:32.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:32Z" 2024-04-12T03:26:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:32.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:32Z" } 2024-04-12T03:26:32.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:32.940 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:32.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:32.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:37.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:37.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:37.942 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:38.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:38.031+0000 7f304677f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:38.032+0000 7f304677f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:38.046 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:38Z" 2024-04-12T03:26:38.052 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:38.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:38Z" } 2024-04-12T03:26:38.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:38.053 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:38.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:38.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:43.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:43.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:43.055 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:43.111 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:43.111 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:43.142+0000 7fe6052a4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:43.143+0000 7fe6052a4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:43.157 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:43.157 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:43.157 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:43.157 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:43.157 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:43.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:43.159 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:43.160 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:43.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:43.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:43Z" 2024-04-12T03:26:43.163 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:43.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:43Z" } 2024-04-12T03:26:43.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:43.164 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:43.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:43.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:48.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:48.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:48.167 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:48.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:48.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:48.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:48.253+0000 7fd4fb392780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:48.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:48.254+0000 7fd4fb392780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:48.269 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:48.269 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.270 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:48.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:48.273 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:48.274 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:48.275 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:48.276 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:48.277 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:48.278 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:48.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:48.278 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:48Z" 2024-04-12T03:26:48.278 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:48.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:48Z" } 2024-04-12T03:26:48.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:48.279 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:48.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:48.280 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:53.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:53.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:53.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:53.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:53.368+0000 7f615e987780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:53.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:53.369+0000 7f615e987780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:53Z" 2024-04-12T03:26:53.391 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:53.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:53Z" } 2024-04-12T03:26:53.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:53.392 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:53.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:53.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:26:58.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:58.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:58.393 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:58.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:58.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:58.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:58.481+0000 7f97a8553780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:26:58.482+0000 7f97a8553780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:58.497 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:58.498 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:58.500 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:58.501 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:26:58.502 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:26:58Z" 2024-04-12T03:26:58.503 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:26:58.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:58Z" } 2024-04-12T03:26:58.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:26:58.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:26:58.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:26:58.505 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:27:03.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:03.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:03.506 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:03.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:03.596+0000 7fd262bde780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:03.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:03.597+0000 7fd262bde780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:03.612 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.613 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:03.614 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:03.615 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:03.616 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:03.617 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:03Z" 2024-04-12T03:27:03.619 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:03.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:03Z" } 2024-04-12T03:27:03.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:27:03.621 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:03.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:27:03.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:27:08.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:08.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:08.622 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:08.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:08.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:08.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:08.712+0000 7f74b3587780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:08.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:08.713+0000 7f74b3587780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:08.727 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:08.727 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:08.728 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:08.729 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:08.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892121.832537_549.1", 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:22:01.832537Z", 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:08.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:08Z" 2024-04-12T03:27:08.733 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:08.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892121.832537_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:22:01.832537Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:08Z" } 2024-04-12T03:27:08.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892121.832537_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:22:01.832537Z', 'realm_epoch': 2}}] 2024-04-12T03:27:08.734 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:08.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892121.832537_549.1'} 2024-04-12T03:27:08.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892121.832537_549.1 2024-04-12T03:27:13.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:13.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:13.736 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:13.786 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:13.786 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:13.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:13.815+0000 7f535fd25780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:13.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:13.816+0000 7f535fd25780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:13.829 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:13.829 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:13.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:13.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:13.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892311.237083_777.1", 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:11.237083Z", 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:13.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:13.835 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:13Z" 2024-04-12T03:27:13.836 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:13.836 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892311.237083_777.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:11.237083Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:13Z" } 2024-04-12T03:27:13.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892311.237083_777.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:11.237083Z', 'realm_epoch': 2}}] 2024-04-12T03:27:13.837 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:13.837 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892311.237083_777.1'} 2024-04-12T03:27:13.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892311.237083_777.1 2024-04-12T03:27:18.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:18.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:18.838 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:18.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:18.916+0000 7ff1c1fd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:18.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:18.917+0000 7ff1c1fd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:18.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:18.933 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:18.934 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.634568_815.1", 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.634568Z", 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:18Z" 2024-04-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:18.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.634568_815.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.634568Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:18Z" } 2024-04-12T03:27:18.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.634568_815.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.634568Z', 'realm_epoch': 2}}] 2024-04-12T03:27:18.939 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:18.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.634568_815.1'} 2024-04-12T03:27:18.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.634568_815.1 2024-04-12T03:27:23.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:23.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:23.940 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:24.032+0000 7f56150c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:24.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:24.033+0000 7f56150c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:24.047 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:24.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:24.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:24.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:24.051 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:24.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:24.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:24.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:24.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:24.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:24.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:24.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:24.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:24.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:24.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.630571_812.1", 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.630571Z", 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:24.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:24Z" 2024-04-12T03:27:24.056 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:24.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.630571_812.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.630571Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:24Z" } 2024-04-12T03:27:24.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.630571_812.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.630571Z', 'realm_epoch': 2}}] 2024-04-12T03:27:24.058 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:24.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.630571_812.1'} 2024-04-12T03:27:24.058 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.630571_812.1 2024-04-12T03:27:29.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:29.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:29.059 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:29.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:29.135+0000 7f0b8a9ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:29.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:29.136+0000 7f0b8a9ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.150 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:29.151 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:29.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:29.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:29.154 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.630571_812.1", 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.630571Z", 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.155 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:29.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:29.156 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:29.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:29.156 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:29Z" 2024-04-12T03:27:29.156 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:29.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.630571_812.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.630571Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:29Z" } 2024-04-12T03:27:29.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.630571_812.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.630571Z', 'realm_epoch': 2}}] 2024-04-12T03:27:29.157 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:29.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.630571_812.1'} 2024-04-12T03:27:29.157 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.630571_812.1 2024-04-12T03:27:34.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:34.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:34.159 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:34.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:34.252+0000 7fea60e98780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:34.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:34.253+0000 7fea60e98780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:34.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:34.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.634568_815.1", 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.634568Z", 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:34.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:34Z" 2024-04-12T03:27:34.273 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:34.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.634568_815.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.634568Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:34Z" } 2024-04-12T03:27:34.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.634568_815.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.634568Z', 'realm_epoch': 2}}] 2024-04-12T03:27:34.274 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:34.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.634568_815.1'} 2024-04-12T03:27:34.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.634568_815.1 2024-04-12T03:27:39.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:39.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:39.276 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:39.328 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:39.328 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:39.355+0000 7f1266536780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:39.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:39.356+0000 7f1266536780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:39.373 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.374 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:39.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:39.376 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:39.377 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.726907_817.1", 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:39.378 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.726907Z", 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:39.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:39.380 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:39Z" 2024-04-12T03:27:39.380 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:39.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.726907_817.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.726907Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:39Z" } 2024-04-12T03:27:39.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.726907_817.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.726907Z', 'realm_epoch': 2}}] 2024-04-12T03:27:39.381 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:39.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.726907_817.1'} 2024-04-12T03:27:39.381 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.726907_817.1 2024-04-12T03:27:44.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:44.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:44.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:44.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:44.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:44.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:44.461+0000 7f25d01ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:44.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:44.462+0000 7f25d01ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:44.487 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:44.487 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:44.487 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:44.487 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:44.487 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:44.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:44.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.490 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:44.491 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.637244_816.1", 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:44.492 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.637244Z", 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:44.493 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:44Z" 2024-04-12T03:27:44.494 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:44.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.637244_816.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.637244Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:44Z" } 2024-04-12T03:27:44.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.637244_816.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.637244Z', 'realm_epoch': 2}}] 2024-04-12T03:27:44.495 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:44.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.637244_816.1'} 2024-04-12T03:27:44.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.637244_816.1 2024-04-12T03:27:49.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:49.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:49.497 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:49.558 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:49.558 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:49.597+0000 7fef9cdc1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:49.599+0000 7fef9cdc1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892312.630571_812.1", 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:12.630571Z", 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:49Z" 2024-04-12T03:27:49.620 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:49.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892312.630571_812.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:12.630571Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:49Z" } 2024-04-12T03:27:49.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892312.630571_812.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:12.630571Z', 'realm_epoch': 2}}] 2024-04-12T03:27:49.621 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:49.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892312.630571_812.1'} 2024-04-12T03:27:49.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892316.006867_853.1 target=1_1712892312.630571_812.1 2024-04-12T03:27:54.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:54.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:54.623 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:54.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:54.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:54.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:54.709+0000 7f0450fbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:54.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:27:54.711+0000 7f0450fbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.725 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:54.726 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:54.727 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:54.728 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:54.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892316.006867_853.1", 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:54.730 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:25:16.006867Z", 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:27:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:27:54.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:54.732 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:54Z" 2024-04-12T03:27:54.732 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:54.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892316.006867_853.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:25:16.006867Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:27:54Z" } 2024-04-12T03:27:54.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892316.006867_853.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:25:16.006867Z', 'realm_epoch': 2}}] 2024-04-12T03:27:54.733 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:54.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892316.006867_853.1'} 2024-04-12T03:27:54.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:27:54.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:27:54.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:27:54.734 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:27:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:54.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:27:54.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:27:54.849 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:27:54.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:54.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:54.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:54.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:54.964 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:55.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:55.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:55.052 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:27:55.052 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.29", 2024-04-12T03:27:55.052 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000078", 2024-04-12T03:27:55.052 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.history", 2024-04-12T03:27:55.052 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:0", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:0", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.23", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000070", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:6", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000104", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:2", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000026", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:6", 2024-04-12T03:27:55.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:6", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:7", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:9", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:2", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.8", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000028", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:3", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:5", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.30", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000040", 2024-04-12T03:27:55.054 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.6", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:0", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000015", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000069", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000095", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000003", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000047", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1.qd/YlHxmik2.0", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000035", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:5", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000037", 2024-04-12T03:27:55.055 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0.tsDG7AtrmED.0", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:8", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000024", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000000", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:2", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000031", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:5", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:3", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:9", 2024-04-12T03:27:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:1", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:3", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:6", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:7", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:6", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "data_loggenerations_metadata", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:6", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000076", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:0", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000113", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000005", 2024-04-12T03:27:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:5", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000011", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:3", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:10", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000058", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000052", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:10", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.21", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:8", 2024-04-12T03:27:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:2", 2024-04-12T03:27:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:6", 2024-04-12T03:27:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:2", 2024-04-12T03:27:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:1", 2024-04-12T03:27:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:10", 2024-04-12T03:27:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:0", 2024-04-12T03:27:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:1", 2024-04-12T03:27:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000088", 2024-04-12T03:27:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000080", 2024-04-12T03:27:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000090", 2024-04-12T03:27:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000110", 2024-04-12T03:27:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3.qNvwCEIQDgq.0", 2024-04-12T03:27:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.0", 2024-04-12T03:27:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:2", 2024-04-12T03:27:55.062 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000096", 2024-04-12T03:27:55.062 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000087", 2024-04-12T03:27:55.062 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:1", 2024-04-12T03:27:55.062 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000008", 2024-04-12T03:27:55.062 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:3", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:10", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000006", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000029", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:9", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000089", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000022", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:7", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog.trim", 2024-04-12T03:27:55.063 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000016", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:7", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:0", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:7", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000041", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:2", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000018", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000092", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:9", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:0", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:0", 2024-04-12T03:27:55.064 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.27", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.4", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:9", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:7", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:0", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000014", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:8", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:6", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:2", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000112", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000007", 2024-04-12T03:27:55.065 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000021", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.16", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000064", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000071", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000074", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:4", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:9", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:8", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:10", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:9", 2024-04-12T03:27:55.066 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.3.retry", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000081", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000009", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000121", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:6", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:4", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:5", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.17", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:3", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000125", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000082", 2024-04-12T03:27:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000105", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000059", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:9", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:10", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:5", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:2", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:8", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000077", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000032", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:7", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:1", 2024-04-12T03:27:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:1", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:8", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.24", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:4", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:2", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:0", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:9", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000053", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000091", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:7", 2024-04-12T03:27:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000065", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:7", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000083", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000010", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2.CLnALvpp4dE.0", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.7", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.3", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:3", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:8", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:9", 2024-04-12T03:27:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:5", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:5", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000045", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:1", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:8", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000002", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:2", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.12", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:2", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000116", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.25", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000034", 2024-04-12T03:27:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:9", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:4", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000101", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.13", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:4", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:1", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:4", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:3", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:3", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:3", 2024-04-12T03:27:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000079", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000049", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000117", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000044", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:6", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:6", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.14", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:4", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000066", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:5", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:0", 2024-04-12T03:27:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:8", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:10", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000068", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000085", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:7", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000073", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000038", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:8", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:0", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.10", 2024-04-12T03:27:55.074 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000118", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:0", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000036", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000103", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000119", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:7", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000098", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.1.retry", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:1", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:7", 2024-04-12T03:27:55.075 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000027", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:6", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000019", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:10", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000039", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:7", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:3", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:4", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000100", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.3", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000093", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:9", 2024-04-12T03:27:55.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:1", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:9", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.2", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000004", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000063", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000122", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:9", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000057", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000054", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000114", 2024-04-12T03:27:55.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:10", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.2", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:6", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:8", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:0", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:4", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000012", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:5", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:10", 2024-04-12T03:27:55.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:10", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000084", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000043", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:10", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.11", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:1", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:10", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000111", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000048", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000020", 2024-04-12T03:27:55.079 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.2", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:2", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:7", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000099", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000056", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:8", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:2", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:4", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.19", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.28", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15", 2024-04-12T03:27:55.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:2", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:4", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.1", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000072", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:3", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000062", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:8", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000115", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.1", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000033", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:7", 2024-04-12T03:27:55.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000025", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:5", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000123", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:10", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:0", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.31", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.18", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000108", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:3", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000094", 2024-04-12T03:27:55.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:2", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000060", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:0", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000109", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.15", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:3", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000013", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:7", 2024-04-12T03:27:55.083 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.0", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:1", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:5", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000051", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:10", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:6", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000106", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000017", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:5", 2024-04-12T03:27:55.084 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.0.retry", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:1", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000023", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:10", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:9", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:4", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:5", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000107", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000046", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.26", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.22", 2024-04-12T03:27:55.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:9", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000067", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:0", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:1", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:4", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.9", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000086", 2024-04-12T03:27:55.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:6", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.5", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:9", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:4", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.0", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.20", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:4", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:2", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:5", 2024-04-12T03:27:55.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:10", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000050", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:8", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000102", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:4", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:6", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:7", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:8", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:6", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:8", 2024-04-12T03:27:55.088 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000124", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000001", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:3", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000126", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000055", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000061", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:1", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000097", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:5", 2024-04-12T03:27:55.089 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000042", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:1", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:3", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:8", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:5", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:3", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000075", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000120", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000030", 2024-04-12T03:27:55.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:4", 2024-04-12T03:27:55.091 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:1", 2024-04-12T03:27:55.091 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.2.retry", 2024-04-12T03:27:55.091 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.d800b3e1-d3eb-4250-a3c6-cce8e78e25b6.3" 2024-04-12T03:27:55.091 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:27:55.091 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T03:27:55.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.092 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:55.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:55.138 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.29", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000078", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.history", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:7", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.23", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000070", 2024-04-12T03:27:55.181 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000104", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000026", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.8", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000028", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:3", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:6", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:2", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.30", 2024-04-12T03:27:55.182 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:5", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:3", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:6", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000040", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.6", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:7", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:10", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000015", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:1", 2024-04-12T03:27:55.183 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000069", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000095", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000003", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000047", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:3", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000035", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000037", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:1", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:0", 2024-04-12T03:27:55.184 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:6", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000024", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:8", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000000", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000031", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:4", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:8", 2024-04-12T03:27:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:10", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:7", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:4", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "data_loggenerations_metadata", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000076", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000113", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000005", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000011", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:6", 2024-04-12T03:27:55.186 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000058", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000052", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.21", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:1", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:9", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:3", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000088", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000080", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000090", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000110", 2024-04-12T03:27:55.187 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:2", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:7", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:4", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.0", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:5", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000096", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:5", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:6", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:5", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000087", 2024-04-12T03:27:55.188 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000008", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:1", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:2", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000006", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000029", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000089", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.0.retry", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:10", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000022", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:5", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0.Bkdc65T6fKr.0", 2024-04-12T03:27:55.189 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:8", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog.trim", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:2", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000016", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000041", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:10", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:5", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:3", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000018", 2024-04-12T03:27:55.190 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000092", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:7", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.27", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.4", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:2", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000014", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000112", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000007", 2024-04-12T03:27:55.191 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000021", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:5", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.16", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000064", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000071", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:8", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000074", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:1", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:4", 2024-04-12T03:27:55.192 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:0", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.1.retry", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000081", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000009", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:3", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:0", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000121", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:10", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:8", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.17", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000125", 2024-04-12T03:27:55.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000082", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.1", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:8", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000105", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000059", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:1", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:2", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:9", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:5", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000077", 2024-04-12T03:27:55.194 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:10", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000032", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:8", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:9", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:0", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:9", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:6", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:6", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11", 2024-04-12T03:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.24", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:6", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000053", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000091", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:7", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:6", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:9", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000065", 2024-04-12T03:27:55.196 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000083", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000010", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.7", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.3", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:7", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:10", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:2", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:1", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.2", 2024-04-12T03:27:55.197 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.2", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:9", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000045", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:2", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:10", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000002", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.12", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000116", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:6", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.25", 2024-04-12T03:27:55.198 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:1", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:4", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000034", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:5", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000101", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.13", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:6", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.1", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:10", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:3", 2024-04-12T03:27:55.199 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000079", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000049", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:9", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000117", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000044", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:0", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:0", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.14", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000066", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:5", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:3", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:3", 2024-04-12T03:27:55.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:0", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:3", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000068", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000085", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000073", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.2.retry", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000038", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:2", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.10", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000118", 2024-04-12T03:27:55.201 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000036", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000103", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:9", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:1", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.3", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:0", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000119", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:9", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000098", 2024-04-12T03:27:55.202 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:9", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:0", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:4", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.0", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000027", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000019", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000039", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:8", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:2", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:4", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:7", 2024-04-12T03:27:55.203 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000100", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.3", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000093", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:8", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:0", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000004", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000063", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000122", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000057", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:1", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000054", 2024-04-12T03:27:55.204 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000114", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:4", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:4", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.2", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:2", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:8", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:4", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:4", 2024-04-12T03:27:55.205 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000012", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000084", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000043", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:0", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.11", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000111", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000048", 2024-04-12T03:27:55.206 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:9", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000020", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.2", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.3.retry", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:8", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000099", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000056", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:5", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:6", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.19", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.28", 2024-04-12T03:27:55.207 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:0", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:3", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:10", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000072", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:3", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000062", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:7", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000115", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.1", 2024-04-12T03:27:55.208 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000033", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:4", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:5", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:10", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:9", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:6", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000025", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000123", 2024-04-12T03:27:55.209 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.31", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:8", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:9", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.18", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000108", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:0", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000094", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:7", 2024-04-12T03:27:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:6", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000060", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000109", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.15", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:2", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:0", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000013", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:5", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.3ee804bf-b01f-4af0-895f-822e923b43dd.0", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:3", 2024-04-12T03:27:55.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:7", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:9", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:2", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:1", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000051", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:10", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000106", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2.ZbSNcF0EWa1.0", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:8", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:7", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000017", 2024-04-12T03:27:55.212 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000023", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:2", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:9", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000107", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:4", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-12:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.12:9", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:7", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000046", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.26", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:1", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.22", 2024-04-12T03:27:55.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18:0", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:2", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:1", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:10", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:10", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000067", 2024-04-12T03:27:55.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:5", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-37:b7d2416c-9d6f-4780-9e14-1f66614caba8.6217.1:6", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-18:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.18", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-20:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.20:4", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:4", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.9", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000086", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.5", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:9", 2024-04-12T03:27:55.215 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3.kn8UxPV7ULc.0", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:0", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:6", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.20", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:7", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-35:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.35:8", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:3", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:3", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.0", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1.RLXDdSnnW/r.0", 2024-04-12T03:27:55.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-19:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.19:8", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000050", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000102", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:6", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:4", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:4", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16:2", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:1", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:1", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:7", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000124", 2024-04-12T03:27:55.217 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000001", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:3", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000126", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:10", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000055", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000061", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:5", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:8", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-11:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.11:7", 2024-04-12T03:27:55.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-10:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.10:5", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-17:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.17:1", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000097", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:0", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-13:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.13:8", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-16:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.16", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000042", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-14:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.14:7", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b7d2416c-9d6f-4780-9e14-1f66614caba8.3", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-7:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.7:5", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-8:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.8:3", 2024-04-12T03:27:55.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:10", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000075", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-36:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.36:1", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-9:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.9:2", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-15:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.15:10", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000120", 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000030" 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:27:55.220 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-04-12T03:27:55.221 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:27:55.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.222 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:55.268 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:55.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:55.315 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:27:55.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003608", 2024-04-12T03:27:55.315 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:27:54.947249Z" 2024-04-12T03:27:55.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:27:54.944369Z" 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006705", 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:27:54.945315Z" 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005860", 2024-04-12T03:27:55.317 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:27:54.946357Z" 2024-04-12T03:27:55.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.317 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:27:55.318 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:27:55.318 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:27:55.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:27:55.318 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:27:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:27:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:27:55.413 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:55.414 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:55.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:55.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:16:01.302775Z" 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:27:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:23:02.806655Z" 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:27:55.418 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:27:55Z" 2024-04-12T03:27:55.419 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:27:55.420 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:16:01.302775Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:23:02.806655Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:27:55Z" } 2024-04-12T03:27:55.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:16:01.302775Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:23:02.806655Z'}}] 2024-04-12T03:27:55.421 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:27:55.421 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:27:55.421 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003608 target=00000000000000000000:00000000000000002764, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:28:00.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:00.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:00.423 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.532 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:00.534 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:16:01.302775Z" 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:00.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:23:02.806655Z" 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:00.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:00.537 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:00.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:00.537 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:00Z" 2024-04-12T03:28:00.537 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:00.538 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:16:01.302775Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:23:02.806655Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:00Z" } 2024-04-12T03:28:00.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:16:01.302775Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:23:02.806655Z'}}] 2024-04-12T03:28:00.538 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:28:00.538 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:28:00.538 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003608 target=00000000000000000000:00000000000000002764, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:28:05.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:05.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:05.539 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:05.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:05.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:05.643 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:05.643 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:16:01.302775Z" 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:23:02.806655Z" 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:05.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:05.649 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:05Z" 2024-04-12T03:28:05.649 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:05.649 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:16:01.302775Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:23:02.806655Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:05Z" } 2024-04-12T03:28:05.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:16:01.302775Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:23:02.806655Z'}}] 2024-04-12T03:28:05.650 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:28:05.650 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:28:05.650 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003608 target=00000000000000000000:00000000000000002764, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:28:10.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:10.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:10.651 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:10.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:10.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:10.751 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:10.751 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:10.751 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:10.751 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:10.751 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:10.752 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:33.309883Z" 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:10.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.304100Z" 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:10.754 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:16:01.302775Z" 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:10.755 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:23:02.806655Z" 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:10Z" 2024-04-12T03:28:10.756 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:10.757 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:33.309883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.304100Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:16:01.302775Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:23:02.806655Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:10Z" } 2024-04-12T03:28:10.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:33.309883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.304100Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:16:01.302775Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:23:02.806655Z'}}] 2024-04-12T03:28:10.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:28:10.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:28:10.758 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003608 target=00000000000000000000:00000000000000002764, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:28:15.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.759 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:15.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:15.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:15.864 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003608", 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:27:54.947249Z" 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:27:54.944369Z" 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006705", 2024-04-12T03:28:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:27:54.945315Z" 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005860", 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:27:54.946357Z" 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:15Z" 2024-04-12T03:28:15.869 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:15.870 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003608", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:27:54.947249Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002061", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:27:54.944369Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006705", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:27:54.945315Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005860", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:27:54.946357Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:15Z" } 2024-04-12T03:28:15.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003608', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:27:54.947249Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002061', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:27:54.944369Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006705', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:27:54.945315Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005860', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:27:54.946357Z'}}] 2024-04-12T03:28:15.871 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:28:15.871 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006705', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:28:15.871 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:28:15.871 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:28:15.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.871 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003467", 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:10.831514Z" 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:10.828351Z" 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006564", 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:10.829411Z" 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:15.969 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:28:15.970 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:15.970 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:28:15.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:15.971 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:16.062 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:16.062 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:16.062 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:16.062 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:16.062 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:16.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.506001Z" 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:16.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.828351Z" 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006564", 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:16.065 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.829411Z" 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:16.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:16Z" 2024-04-12T03:28:16.067 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:16.068 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.506001Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002061", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.828351Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006564", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.829411Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005157", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.830455Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:16Z" } 2024-04-12T03:28:16.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.506001Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002061', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.828351Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.829411Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005157', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.830455Z'}}] 2024-04-12T03:28:16.069 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:16.069 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:16.069 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002622 2024-04-12T03:28:21.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:21.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:21.070 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:21.175 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:21.176 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:21.177 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:21.177 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:21.177 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:21.177 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:01:11.506001Z" 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:28:21.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.828351Z" 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006564", 2024-04-12T03:28:21.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.829411Z" 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:28:21.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:21.181 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:21Z" 2024-04-12T03:28:21.182 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:21.182 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:01:11.506001Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002061", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.828351Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006564", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.829411Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005157", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.830455Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:21Z" } 2024-04-12T03:28:21.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:01:11.506001Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002061', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.828351Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.829411Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005157', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.830455Z'}}] 2024-04-12T03:28:21.183 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:21.183 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:21.183 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002622 2024-04-12T03:28:26.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.184 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003467", 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.831514Z" 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.828351Z" 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006564", 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.829411Z" 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:26Z" 2024-04-12T03:28:26.294 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:26.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003467", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.831514Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002061", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.828351Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006564", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.829411Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005157", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.830455Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:26Z" } 2024-04-12T03:28:26.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.831514Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002061', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.828351Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.829411Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005157', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.830455Z'}}] 2024-04-12T03:28:26.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:26.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006564', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:28:26.296 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:28:26.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2024-04-12T03:28:26.297 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-48 2024-04-12T03:28:26.298 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:26.298 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:26.298 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.298 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:26.298 DEBUG:boto:Data: 2024-04-12T03:28:26.298 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.298 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.298 DEBUG:boto:Params: {} 2024-04-12T03:28:26.299 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:28:26.299 DEBUG:boto:Token: None 2024-04-12T03:28:26.299 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/ 2024-04-12T03:28:26.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xe9UQxHCyH2yXCu7/LFLKYrfv+o= 2024-04-12T03:28:26.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xe9UQxHCyH2yXCu7/LFLKYrfv+o=', 'Content-Length': '0'} 2024-04-12T03:28:26.327 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fb292c96abce317-006618aa5a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.327 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-49 2024-04-12T03:28:26.328 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:26.328 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:26.328 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.328 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:26.328 DEBUG:boto:Data: 2024-04-12T03:28:26.328 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.328 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.328 DEBUG:boto:Params: {} 2024-04-12T03:28:26.328 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:28:26.329 DEBUG:boto:Token: None 2024-04-12T03:28:26.329 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/ 2024-04-12T03:28:26.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FQnGDzVQcVP+yRkNjH10eeHZ97U= 2024-04-12T03:28:26.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FQnGDzVQcVP+yRkNjH10eeHZ97U=', 'Content-Length': '0'} 2024-04-12T03:28:26.366 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001648e0043bb3674f-006618aa5a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.366 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:26.367 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:26.367 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.367 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:26.367 DEBUG:boto:Data: 2024-04-12T03:28:26.367 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.367 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.367 DEBUG:boto:Params: {} 2024-04-12T03:28:26.367 DEBUG:boto:Token: None 2024-04-12T03:28:26.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/ 2024-04-12T03:28:26.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8= 2024-04-12T03:28:26.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8=', 'Content-Length': '0'} 2024-04-12T03:28:26.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd90296430dd863f-006618aa5a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.370 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:26.370 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:26.371 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.371 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:28:26.371 DEBUG:boto:Data: 2024-04-12T03:28:26.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.371 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.371 DEBUG:boto:Params: {} 2024-04-12T03:28:26.371 DEBUG:boto:Token: None 2024-04-12T03:28:26.371 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/obj1 2024-04-12T03:28:26.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:du9H+mGqn533gOIAi7V1fFHMMPk= 2024-04-12T03:28:26.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:du9H+mGqn533gOIAi7V1fFHMMPk='} 2024-04-12T03:28:26.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084fa0631e17372bf-006618aa5a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.378 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:26.378 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:26.378 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.378 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:26.378 DEBUG:boto:Data: 2024-04-12T03:28:26.378 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.378 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.378 DEBUG:boto:Params: {} 2024-04-12T03:28:26.379 DEBUG:boto:Token: None 2024-04-12T03:28:26.379 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/ 2024-04-12T03:28:26.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8= 2024-04-12T03:28:26.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8=', 'Content-Length': '0'} 2024-04-12T03:28:26.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045b9c096353fbb79-006618aa5a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.381 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:26.381 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:26.381 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.381 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:28:26.381 DEBUG:boto:Data: 2024-04-12T03:28:26.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.382 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.382 DEBUG:boto:Params: {} 2024-04-12T03:28:26.382 DEBUG:boto:Token: None 2024-04-12T03:28:26.382 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/obj2 2024-04-12T03:28:26.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vk4ony0OZ9vu5AS8Mwfun8ptUV4= 2024-04-12T03:28:26.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vk4ony0OZ9vu5AS8Mwfun8ptUV4='} 2024-04-12T03:28:26.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff771818c9c59a67-006618aa5a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.387 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:26.387 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:26.388 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.388 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:26.388 DEBUG:boto:Data: 2024-04-12T03:28:26.388 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.388 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.388 DEBUG:boto:Params: {} 2024-04-12T03:28:26.388 DEBUG:boto:Token: None 2024-04-12T03:28:26.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/ 2024-04-12T03:28:26.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8= 2024-04-12T03:28:26.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8=', 'Content-Length': '0'} 2024-04-12T03:28:26.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000538fcdd9bde96b53-006618aa5a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.390 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:26.391 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:26.391 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.391 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:28:26.391 DEBUG:boto:Data: 2024-04-12T03:28:26.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.391 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.391 DEBUG:boto:Params: {} 2024-04-12T03:28:26.391 DEBUG:boto:Token: None 2024-04-12T03:28:26.391 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/obj3 2024-04-12T03:28:26.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xqZn21GhyJ7u3yLja/z26B/Q7hM= 2024-04-12T03:28:26.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xqZn21GhyJ7u3yLja/z26B/Q7hM='} 2024-04-12T03:28:26.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007962f9b93c67e0e0-006618aa5a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.439 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:26.439 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:26.439 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.439 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:26.439 DEBUG:boto:Data: 2024-04-12T03:28:26.439 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.439 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.440 DEBUG:boto:Params: {} 2024-04-12T03:28:26.440 DEBUG:boto:Token: None 2024-04-12T03:28:26.440 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/ 2024-04-12T03:28:26.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8= 2024-04-12T03:28:26.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k9SJWOHKReNGcfJhfYN81icJ2h8=', 'Content-Length': '0'} 2024-04-12T03:28:26.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b91587ad84a9cbb0-006618aa5a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.442 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:26.442 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:26.442 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.442 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:28:26.443 DEBUG:boto:Data: 2024-04-12T03:28:26.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:26.443 DEBUG:boto:Port: 8000 2024-04-12T03:28:26.443 DEBUG:boto:Params: {} 2024-04-12T03:28:26.443 DEBUG:boto:Token: None 2024-04-12T03:28:26.443 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-48/obj4 2024-04-12T03:28:26.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PAxw+tCj5sb8RVwoYFirTFxgKgw= 2024-04-12T03:28:26.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PAxw+tCj5sb8RVwoYFirTFxgKgw='} 2024-04-12T03:28:26.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001de4ab29a7b05874-006618aa5a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.448 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:26.448 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:26.448 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.448 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:26.448 DEBUG:boto:Data: 2024-04-12T03:28:26.449 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.449 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.449 DEBUG:boto:Params: {} 2024-04-12T03:28:26.449 DEBUG:boto:Token: None 2024-04-12T03:28:26.449 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/ 2024-04-12T03:28:26.449 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4= 2024-04-12T03:28:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4=', 'Content-Length': '0'} 2024-04-12T03:28:26.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036cdef3538f0417a-006618aa5a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.451 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:26.452 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:26.452 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.452 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:28:26.452 DEBUG:boto:Data: 2024-04-12T03:28:26.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.452 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.452 DEBUG:boto:Params: {} 2024-04-12T03:28:26.452 DEBUG:boto:Token: None 2024-04-12T03:28:26.453 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/obj1 2024-04-12T03:28:26.453 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0E21XF/iNIbXKU/IA9eegDra6gw= 2024-04-12T03:28:26.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0E21XF/iNIbXKU/IA9eegDra6gw='} 2024-04-12T03:28:26.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000588efcef44c50d2a-006618aa5a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.458 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:26.458 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:26.459 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.459 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:26.459 DEBUG:boto:Data: 2024-04-12T03:28:26.459 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.459 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.459 DEBUG:boto:Params: {} 2024-04-12T03:28:26.459 DEBUG:boto:Token: None 2024-04-12T03:28:26.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/ 2024-04-12T03:28:26.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4= 2024-04-12T03:28:26.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4=', 'Content-Length': '0'} 2024-04-12T03:28:26.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047dc4440fc9a4e18-006618aa5a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.462 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:26.462 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:26.462 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.462 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:28:26.462 DEBUG:boto:Data: 2024-04-12T03:28:26.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.462 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.462 DEBUG:boto:Params: {} 2024-04-12T03:28:26.463 DEBUG:boto:Token: None 2024-04-12T03:28:26.463 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/obj2 2024-04-12T03:28:26.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Aki6Y5E9y7/rb+dzUPXKtNGrzI= 2024-04-12T03:28:26.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Aki6Y5E9y7/rb+dzUPXKtNGrzI='} 2024-04-12T03:28:26.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b75c4bf93b916be4-006618aa5a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.471 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:26.471 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:26.471 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.471 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:26.471 DEBUG:boto:Data: 2024-04-12T03:28:26.471 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.471 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.471 DEBUG:boto:Params: {} 2024-04-12T03:28:26.472 DEBUG:boto:Token: None 2024-04-12T03:28:26.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/ 2024-04-12T03:28:26.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4= 2024-04-12T03:28:26.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4=', 'Content-Length': '0'} 2024-04-12T03:28:26.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223296c7c5c8528a-006618aa5a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.474 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:26.474 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:26.474 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.474 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:28:26.474 DEBUG:boto:Data: 2024-04-12T03:28:26.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.475 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.475 DEBUG:boto:Params: {} 2024-04-12T03:28:26.475 DEBUG:boto:Token: None 2024-04-12T03:28:26.475 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/obj3 2024-04-12T03:28:26.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c6q7q4Aqf0cNomm2AqeQPRoXk6E= 2024-04-12T03:28:26.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c6q7q4Aqf0cNomm2AqeQPRoXk6E='} 2024-04-12T03:28:26.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a7592034fb562e5-006618aa5a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.523 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:26.523 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:26.523 DEBUG:boto:Method: HEAD 2024-04-12T03:28:26.523 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:26.523 DEBUG:boto:Data: 2024-04-12T03:28:26.523 DEBUG:boto:Headers: {} 2024-04-12T03:28:26.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.524 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.524 DEBUG:boto:Params: {} 2024-04-12T03:28:26.524 DEBUG:boto:Token: None 2024-04-12T03:28:26.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/ 2024-04-12T03:28:26.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4= 2024-04-12T03:28:26.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpTlAwEZJAg13RlpWG7P9GjPTp4=', 'Content-Length': '0'} 2024-04-12T03:28:26.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d13fe60e32f6b6-006618aa5a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.527 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:26.527 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:26.527 DEBUG:boto:Method: PUT 2024-04-12T03:28:26.527 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:28:26.527 DEBUG:boto:Data: 2024-04-12T03:28:26.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:26.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:26.528 DEBUG:boto:Port: 8001 2024-04-12T03:28:26.528 DEBUG:boto:Params: {} 2024-04-12T03:28:26.528 DEBUG:boto:Token: None 2024-04-12T03:28:26.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:26 GMT /dgbpxg-49/obj4 2024-04-12T03:28:26.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZWBbm5ZQ9WpLLEmEJx3YX/gksZs= 2024-04-12T03:28:26.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZWBbm5ZQ9WpLLEmEJx3YX/gksZs='} 2024-04-12T03:28:26.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c27ccf6e963cd96c-006618aa5a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:26.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.534 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:26.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:26.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:28:26.623 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:28:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892506.354990_883.1", 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:26.354990Z" 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.625 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:28:26.626 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892506.354990_883.1'} 2024-04-12T03:28:26.626 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:28:26.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:26.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:26.703+0000 7fd38779a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:26.704+0000 7fd38779a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:26.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:26.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:26.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:26.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892506.319630_876.1", 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:26.319630Z", 2024-04-12T03:28:26.722 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:26Z" 2024-04-12T03:28:26.723 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:26.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892506.319630_876.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:26.319630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:28:26Z" } 2024-04-12T03:28:26.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892506.319630_876.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:26.319630Z', 'realm_epoch': 2}}] 2024-04-12T03:28:26.724 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892506.354990_883.1'} 2024-04-12T03:28:26.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892506.319630_876.1'} 2024-04-12T03:28:26.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892506.354990_883.1 target=1_1712892506.319630_876.1 2024-04-12T03:28:31.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:31.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:31.726 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:31.813+0000 7f7e7f8b6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:31.814+0000 7f7e7f8b6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:31.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:31.830 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:31.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892506.354990_883.1", 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:28:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:26.354990Z", 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:28:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:28:31Z" 2024-04-12T03:28:31.835 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:28:31.836 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892506.354990_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:26.354990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:28:31Z" } 2024-04-12T03:28:31.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892506.354990_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:26.354990Z', 'realm_epoch': 2}}] 2024-04-12T03:28:31.836 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892506.354990_883.1'} 2024-04-12T03:28:31.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892506.354990_883.1'} 2024-04-12T03:28:31.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:28:31.836 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-48 2024-04-12T03:28:31.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:31.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:31.837 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:31.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:31.927+0000 7effe8a88780 1 waiting to reach incremental sync.. 2024-04-12T03:28:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:36.929+0000 7effe8a88780 1 bucket sync caught up with source: 2024-04-12T03:28:36.930 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.39.6, , 00000000001.48.6, , 00000000001.27.6, , , 00000000001.25.6, , , ] 2024-04-12T03:28:36.930 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.39.6, , 00000000001.48.6, , 00000000001.27.6, , , 00000000001.25.6, , , ] 2024-04-12T03:28:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:36.929+0000 7effe8a88780 0 bucket checkpoint complete 2024-04-12T03:28:36.947 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-48 2024-04-12T03:28:36.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:36.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:36.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:37.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:37.044+0000 7f7085af4780 1 bucket sync caught up with source: 2024-04-12T03:28:37.045 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.2171.6, , 00000000001.1318.6, , 00000000001.415.6, , , 00000000001.2461.6, , , ] 2024-04-12T03:28:37.045 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.2171.6, , 00000000001.1318.6, , 00000000001.415.6, , , 00000000001.2461.6, , , ] 2024-04-12T03:28:37.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:37.044+0000 7f7085af4780 0 bucket checkpoint complete 2024-04-12T03:28:37.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-48 zones={a2, a1} 2024-04-12T03:28:37.058 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.058 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.058 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.059 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:37.059 DEBUG:boto:Data: 2024-04-12T03:28:37.059 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.059 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.059 DEBUG:boto:Params: {} 2024-04-12T03:28:37.059 DEBUG:boto:Token: None 2024-04-12T03:28:37.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpNO5LeXr/2+QtenKD+nHWtdy6s= 2024-04-12T03:28:37.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpNO5LeXr/2+QtenKD+nHWtdy6s=', 'Content-Length': '0'} 2024-04-12T03:28:37.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068ec6f0a512beac1-006618aa65-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.062 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.062 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.062 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.062 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:37.062 DEBUG:boto:Data: 2024-04-12T03:28:37.062 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.062 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.063 DEBUG:boto:Params: {} 2024-04-12T03:28:37.063 DEBUG:boto:Token: None 2024-04-12T03:28:37.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IpNO5LeXr/2+QtenKD+nHWtdy6s= 2024-04-12T03:28:37.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IpNO5LeXr/2+QtenKD+nHWtdy6s=', 'Content-Length': '0'} 2024-04-12T03:28:37.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fbcfcfea2186363-006618aa65-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:28:37.066 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.066 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.066 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.066 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.066 DEBUG:boto:Method: GET 2024-04-12T03:28:37.066 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.066 DEBUG:boto:Data: 2024-04-12T03:28:37.066 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.067 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.067 DEBUG:boto:Params: {} 2024-04-12T03:28:37.067 DEBUG:boto:Token: None 2024-04-12T03:28:37.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/?versions 2024-04-12T03:28:37.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg= 2024-04-12T03:28:37.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg=', 'Content-Length': '0'} 2024-04-12T03:28:37.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a9cba8de4d6227-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.069 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:28:37.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:28:37.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:28:37.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:28:37.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:28:37.070 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.071 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.071 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.071 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.071 DEBUG:boto:Method: GET 2024-04-12T03:28:37.071 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.071 DEBUG:boto:Data: 2024-04-12T03:28:37.071 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.071 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.071 DEBUG:boto:Params: {} 2024-04-12T03:28:37.071 DEBUG:boto:Token: None 2024-04-12T03:28:37.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/?versions 2024-04-12T03:28:37.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg= 2024-04-12T03:28:37.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg=', 'Content-Length': '0'} 2024-04-12T03:28:37.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b2fda3df88402d-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.074 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:28:37.075 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.075 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.075 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.075 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.075 DEBUG:boto:Method: GET 2024-04-12T03:28:37.075 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.075 DEBUG:boto:Data: 2024-04-12T03:28:37.075 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.075 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.075 DEBUG:boto:Params: {} 2024-04-12T03:28:37.076 DEBUG:boto:Token: None 2024-04-12T03:28:37.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/?versions 2024-04-12T03:28:37.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg= 2024-04-12T03:28:37.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg=', 'Content-Length': '0'} 2024-04-12T03:28:37.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f602905c6e1e73-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.078 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.079 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.079 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.079 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.079 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.079 DEBUG:boto:Method: GET 2024-04-12T03:28:37.079 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:28:37.079 DEBUG:boto:Data: 2024-04-12T03:28:37.079 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.079 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.080 DEBUG:boto:Params: {} 2024-04-12T03:28:37.080 DEBUG:boto:Token: None 2024-04-12T03:28:37.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/?versions 2024-04-12T03:28:37.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg= 2024-04-12T03:28:37.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIzeD2iulnGeiI9ewAhfH6jGsFg=', 'Content-Length': '0'} 2024-04-12T03:28:37.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000790f564cd79a1dad-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.082 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:37.083 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.083 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.083 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.083 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.083 DEBUG:boto:Method: GET 2024-04-12T03:28:37.083 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.083 DEBUG:boto:Data: 2024-04-12T03:28:37.083 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.083 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.083 DEBUG:boto:Params: {} 2024-04-12T03:28:37.084 DEBUG:boto:Token: None 2024-04-12T03:28:37.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno= 2024-04-12T03:28:37.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno=', 'Content-Length': '0'} 2024-04-12T03:28:37.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd272a64780721e2-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.086 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.086 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.086 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.086 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.086 DEBUG:boto:Method: GET 2024-04-12T03:28:37.086 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.086 DEBUG:boto:Data: 2024-04-12T03:28:37.086 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.086 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.086 DEBUG:boto:Params: {} 2024-04-12T03:28:37.087 DEBUG:boto:Token: None 2024-04-12T03:28:37.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno= 2024-04-12T03:28:37.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno=', 'Content-Length': '0'} 2024-04-12T03:28:37.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f185ce9bc96b56f8-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.089 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.089 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.089 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.089 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.089 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.089 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.089 DEBUG:boto:Data: 2024-04-12T03:28:37.089 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.090 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.090 DEBUG:boto:Params: {} 2024-04-12T03:28:37.090 DEBUG:boto:Token: None 2024-04-12T03:28:37.090 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AbEvAgBI/OsIi6wykqEvJ3t4Be4= 2024-04-12T03:28:37.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AbEvAgBI/OsIi6wykqEvJ3t4Be4=', 'Content-Length': '0'} 2024-04-12T03:28:37.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec2f6ab3895276c9-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.092 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.092 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.092 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.092 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.092 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.092 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.092 DEBUG:boto:Data: 2024-04-12T03:28:37.092 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.092 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.093 DEBUG:boto:Params: {} 2024-04-12T03:28:37.093 DEBUG:boto:Token: None 2024-04-12T03:28:37.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AbEvAgBI/OsIi6wykqEvJ3t4Be4= 2024-04-12T03:28:37.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AbEvAgBI/OsIi6wykqEvJ3t4Be4=', 'Content-Length': '0'} 2024-04-12T03:28:37.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5ee5ffd923dd18-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:37.095 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.095 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.095 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.095 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.095 DEBUG:boto:Method: GET 2024-04-12T03:28:37.095 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.095 DEBUG:boto:Data: 2024-04-12T03:28:37.095 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.096 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.096 DEBUG:boto:Params: {} 2024-04-12T03:28:37.096 DEBUG:boto:Token: None 2024-04-12T03:28:37.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno= 2024-04-12T03:28:37.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno=', 'Content-Length': '0'} 2024-04-12T03:28:37.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c195a1fa07271752-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.098 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.098 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.098 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.098 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.098 DEBUG:boto:Method: GET 2024-04-12T03:28:37.098 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.099 DEBUG:boto:Data: 2024-04-12T03:28:37.099 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.099 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.099 DEBUG:boto:Params: {} 2024-04-12T03:28:37.099 DEBUG:boto:Token: None 2024-04-12T03:28:37.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:28:37.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno= 2024-04-12T03:28:37.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GnZFd5p83RmL7sDX7b9S5TReSno=', 'Content-Length': '0'} 2024-04-12T03:28:37.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000826746b80b0cea7d-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.101 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.101 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.101 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.101 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.102 DEBUG:boto:Method: GET 2024-04-12T03:28:37.102 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.102 DEBUG:boto:Data: 2024-04-12T03:28:37.102 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.102 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.102 DEBUG:boto:Params: {} 2024-04-12T03:28:37.102 DEBUG:boto:Token: None 2024-04-12T03:28:37.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084c3c0220be9438d-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.105 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.105 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.105 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.105 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.105 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.105 DEBUG:boto:Method: GET 2024-04-12T03:28:37.106 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.106 DEBUG:boto:Data: 2024-04-12T03:28:37.106 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.106 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.106 DEBUG:boto:Params: {} 2024-04-12T03:28:37.106 DEBUG:boto:Token: None 2024-04-12T03:28:37.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057633802085896cf-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.108 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.109 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.109 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.109 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.109 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:28:37.109 DEBUG:boto:Data: 2024-04-12T03:28:37.109 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.109 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.109 DEBUG:boto:Params: {} 2024-04-12T03:28:37.110 DEBUG:boto:Token: None 2024-04-12T03:28:37.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1 2024-04-12T03:28:37.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ONlOhSn8y7Vf1+Ldem/thTnwLbg= 2024-04-12T03:28:37.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ONlOhSn8y7Vf1+Ldem/thTnwLbg=', 'Content-Length': '0'} 2024-04-12T03:28:37.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb34f2a8d584285-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.111 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.112 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.112 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.112 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:28:37.112 DEBUG:boto:Data: 2024-04-12T03:28:37.112 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.112 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.112 DEBUG:boto:Params: {} 2024-04-12T03:28:37.112 DEBUG:boto:Token: None 2024-04-12T03:28:37.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1 2024-04-12T03:28:37.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ONlOhSn8y7Vf1+Ldem/thTnwLbg= 2024-04-12T03:28:37.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ONlOhSn8y7Vf1+Ldem/thTnwLbg=', 'Content-Length': '0'} 2024-04-12T03:28:37.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009343f569583c4f98-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:37.114 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.114 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.115 DEBUG:boto:Method: GET 2024-04-12T03:28:37.115 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:28:37.115 DEBUG:boto:Data: 2024-04-12T03:28:37.115 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.115 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.115 DEBUG:boto:Params: {} 2024-04-12T03:28:37.115 DEBUG:boto:Token: None 2024-04-12T03:28:37.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1 2024-04-12T03:28:37.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O8K8bS3RdHoQ67lyGw8ZqOk2xcM= 2024-04-12T03:28:37.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O8K8bS3RdHoQ67lyGw8ZqOk2xcM=', 'Content-Length': '0'} 2024-04-12T03:28:37.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003781361baa0ab453-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.117 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:28:37.117 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:28:37.117 DEBUG:boto:Method: GET 2024-04-12T03:28:37.118 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:28:37.118 DEBUG:boto:Data: 2024-04-12T03:28:37.118 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.118 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.118 DEBUG:boto:Params: {} 2024-04-12T03:28:37.118 DEBUG:boto:Token: None 2024-04-12T03:28:37.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj1 2024-04-12T03:28:37.118 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O8K8bS3RdHoQ67lyGw8ZqOk2xcM= 2024-04-12T03:28:37.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O8K8bS3RdHoQ67lyGw8ZqOk2xcM=', 'Content-Length': '0'} 2024-04-12T03:28:37.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000772f49d3a67495c7-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.120 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.120 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.120 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.120 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.121 DEBUG:boto:Method: GET 2024-04-12T03:28:37.121 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.121 DEBUG:boto:Data: 2024-04-12T03:28:37.121 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.121 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.121 DEBUG:boto:Params: {} 2024-04-12T03:28:37.121 DEBUG:boto:Token: None 2024-04-12T03:28:37.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be3cd3261dae8d3-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.123 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.124 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.124 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.124 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.124 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.124 DEBUG:boto:Method: GET 2024-04-12T03:28:37.124 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:28:37.124 DEBUG:boto:Data: 2024-04-12T03:28:37.124 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.125 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.125 DEBUG:boto:Params: {} 2024-04-12T03:28:37.125 DEBUG:boto:Token: None 2024-04-12T03:28:37.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e76fa2a986858106-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.127 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:37.128 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.128 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.128 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.128 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.128 DEBUG:boto:Method: GET 2024-04-12T03:28:37.128 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.128 DEBUG:boto:Data: 2024-04-12T03:28:37.128 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.128 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.129 DEBUG:boto:Params: {} 2024-04-12T03:28:37.129 DEBUG:boto:Token: None 2024-04-12T03:28:37.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4= 2024-04-12T03:28:37.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4=', 'Content-Length': '0'} 2024-04-12T03:28:37.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008901f63b9b93f7c6-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.174 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.174 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.174 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.175 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.175 DEBUG:boto:Method: GET 2024-04-12T03:28:37.175 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.175 DEBUG:boto:Data: 2024-04-12T03:28:37.175 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.175 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.175 DEBUG:boto:Params: {} 2024-04-12T03:28:37.175 DEBUG:boto:Token: None 2024-04-12T03:28:37.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4= 2024-04-12T03:28:37.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4=', 'Content-Length': '0'} 2024-04-12T03:28:37.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e1e1a009c036e6d-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.177 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.177 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.178 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.178 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.178 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.178 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.178 DEBUG:boto:Data: 2024-04-12T03:28:37.178 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.178 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.178 DEBUG:boto:Params: {} 2024-04-12T03:28:37.178 DEBUG:boto:Token: None 2024-04-12T03:28:37.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6BMA4J7IH9lzUIkVfNMS2pLA2I= 2024-04-12T03:28:37.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6BMA4J7IH9lzUIkVfNMS2pLA2I=', 'Content-Length': '0'} 2024-04-12T03:28:37.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd3bbb41641a0870-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.180 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.180 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.180 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.180 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.181 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.181 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.181 DEBUG:boto:Data: 2024-04-12T03:28:37.181 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.181 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.181 DEBUG:boto:Params: {} 2024-04-12T03:28:37.181 DEBUG:boto:Token: None 2024-04-12T03:28:37.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6BMA4J7IH9lzUIkVfNMS2pLA2I= 2024-04-12T03:28:37.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6BMA4J7IH9lzUIkVfNMS2pLA2I=', 'Content-Length': '0'} 2024-04-12T03:28:37.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033182abfa2dd118f-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:37.183 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.183 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.183 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.183 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.184 DEBUG:boto:Method: GET 2024-04-12T03:28:37.184 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.184 DEBUG:boto:Data: 2024-04-12T03:28:37.184 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.184 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.184 DEBUG:boto:Params: {} 2024-04-12T03:28:37.184 DEBUG:boto:Token: None 2024-04-12T03:28:37.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4= 2024-04-12T03:28:37.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4=', 'Content-Length': '0'} 2024-04-12T03:28:37.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001faf755d71c727ad-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.186 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.186 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.186 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.186 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.187 DEBUG:boto:Method: GET 2024-04-12T03:28:37.187 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.187 DEBUG:boto:Data: 2024-04-12T03:28:37.187 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.187 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.187 DEBUG:boto:Params: {} 2024-04-12T03:28:37.187 DEBUG:boto:Token: None 2024-04-12T03:28:37.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:28:37.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4= 2024-04-12T03:28:37.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KByyBj1t6UyhUFGYf9Bxcakh2k4=', 'Content-Length': '0'} 2024-04-12T03:28:37.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cd512bdbe2eb216-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.230 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.231 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.231 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.231 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.231 DEBUG:boto:Method: GET 2024-04-12T03:28:37.231 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.231 DEBUG:boto:Data: 2024-04-12T03:28:37.231 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.231 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.231 DEBUG:boto:Params: {} 2024-04-12T03:28:37.231 DEBUG:boto:Token: None 2024-04-12T03:28:37.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000295daf6665d6d4db-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.234 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.234 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.234 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.234 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.234 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.234 DEBUG:boto:Method: GET 2024-04-12T03:28:37.234 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.235 DEBUG:boto:Data: 2024-04-12T03:28:37.235 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.235 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.235 DEBUG:boto:Params: {} 2024-04-12T03:28:37.235 DEBUG:boto:Token: None 2024-04-12T03:28:37.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18f28c435416baf-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.238 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.238 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.238 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.238 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.238 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:28:37.238 DEBUG:boto:Data: 2024-04-12T03:28:37.238 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.238 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.239 DEBUG:boto:Params: {} 2024-04-12T03:28:37.239 DEBUG:boto:Token: None 2024-04-12T03:28:37.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2 2024-04-12T03:28:37.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jsi6iRTec8tHT9EfcWlrLe9e6fg= 2024-04-12T03:28:37.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jsi6iRTec8tHT9EfcWlrLe9e6fg=', 'Content-Length': '0'} 2024-04-12T03:28:37.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003af12e52746c5b1a-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.241 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.241 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.241 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.241 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:28:37.241 DEBUG:boto:Data: 2024-04-12T03:28:37.241 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.241 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.241 DEBUG:boto:Params: {} 2024-04-12T03:28:37.241 DEBUG:boto:Token: None 2024-04-12T03:28:37.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2 2024-04-12T03:28:37.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jsi6iRTec8tHT9EfcWlrLe9e6fg= 2024-04-12T03:28:37.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jsi6iRTec8tHT9EfcWlrLe9e6fg=', 'Content-Length': '0'} 2024-04-12T03:28:37.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005080c6f6ab0affec-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:37.244 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.244 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.244 DEBUG:boto:Method: GET 2024-04-12T03:28:37.244 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:28:37.244 DEBUG:boto:Data: 2024-04-12T03:28:37.244 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.244 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.244 DEBUG:boto:Params: {} 2024-04-12T03:28:37.244 DEBUG:boto:Token: None 2024-04-12T03:28:37.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2 2024-04-12T03:28:37.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7gj1DSe7tTcXTZ0lvCKgz/YkCHA= 2024-04-12T03:28:37.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7gj1DSe7tTcXTZ0lvCKgz/YkCHA=', 'Content-Length': '0'} 2024-04-12T03:28:37.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcbb58c2ec964025-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.246 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:28:37.246 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:28:37.246 DEBUG:boto:Method: GET 2024-04-12T03:28:37.246 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:28:37.247 DEBUG:boto:Data: 2024-04-12T03:28:37.247 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.247 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.247 DEBUG:boto:Params: {} 2024-04-12T03:28:37.247 DEBUG:boto:Token: None 2024-04-12T03:28:37.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj2 2024-04-12T03:28:37.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7gj1DSe7tTcXTZ0lvCKgz/YkCHA= 2024-04-12T03:28:37.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7gj1DSe7tTcXTZ0lvCKgz/YkCHA=', 'Content-Length': '0'} 2024-04-12T03:28:37.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d59fe0208ffd8ed-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.249 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.249 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.249 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.249 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.249 DEBUG:boto:Method: GET 2024-04-12T03:28:37.250 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.250 DEBUG:boto:Data: 2024-04-12T03:28:37.250 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.250 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.250 DEBUG:boto:Params: {} 2024-04-12T03:28:37.250 DEBUG:boto:Token: None 2024-04-12T03:28:37.250 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.250 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df7a25021a4dfa1-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.252 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.253 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.253 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.253 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.253 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.253 DEBUG:boto:Method: GET 2024-04-12T03:28:37.253 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:28:37.253 DEBUG:boto:Data: 2024-04-12T03:28:37.253 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.253 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.253 DEBUG:boto:Params: {} 2024-04-12T03:28:37.253 DEBUG:boto:Token: None 2024-04-12T03:28:37.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029965c55f6b760d0-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.256 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:37.256 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.257 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.257 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.257 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.257 DEBUG:boto:Method: GET 2024-04-12T03:28:37.257 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.257 DEBUG:boto:Data: 2024-04-12T03:28:37.257 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.257 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.257 DEBUG:boto:Params: {} 2024-04-12T03:28:37.257 DEBUG:boto:Token: None 2024-04-12T03:28:37.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.258 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0= 2024-04-12T03:28:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0=', 'Content-Length': '0'} 2024-04-12T03:28:37.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c341378bd049c997-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.260 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.260 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.260 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.260 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.260 DEBUG:boto:Method: GET 2024-04-12T03:28:37.260 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.260 DEBUG:boto:Data: 2024-04-12T03:28:37.260 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.260 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.260 DEBUG:boto:Params: {} 2024-04-12T03:28:37.261 DEBUG:boto:Token: None 2024-04-12T03:28:37.261 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.261 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0= 2024-04-12T03:28:37.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0=', 'Content-Length': '0'} 2024-04-12T03:28:37.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081b751834db5e976-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.262 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.263 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.263 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.263 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.263 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.263 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.263 DEBUG:boto:Data: 2024-04-12T03:28:37.263 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.263 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.263 DEBUG:boto:Params: {} 2024-04-12T03:28:37.263 DEBUG:boto:Token: None 2024-04-12T03:28:37.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tgh2MFZmX7ekFn/wLzisef9J43s= 2024-04-12T03:28:37.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tgh2MFZmX7ekFn/wLzisef9J43s=', 'Content-Length': '0'} 2024-04-12T03:28:37.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000393c5fbbf0719e59-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.265 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.265 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.265 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.266 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.266 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.266 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.266 DEBUG:boto:Data: 2024-04-12T03:28:37.266 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.266 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.266 DEBUG:boto:Params: {} 2024-04-12T03:28:37.266 DEBUG:boto:Token: None 2024-04-12T03:28:37.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tgh2MFZmX7ekFn/wLzisef9J43s= 2024-04-12T03:28:37.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tgh2MFZmX7ekFn/wLzisef9J43s=', 'Content-Length': '0'} 2024-04-12T03:28:37.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007890ae0962e2ef28-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:37.269 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.269 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.269 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.269 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.269 DEBUG:boto:Method: GET 2024-04-12T03:28:37.269 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.269 DEBUG:boto:Data: 2024-04-12T03:28:37.269 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.269 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.269 DEBUG:boto:Params: {} 2024-04-12T03:28:37.269 DEBUG:boto:Token: None 2024-04-12T03:28:37.270 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.270 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0= 2024-04-12T03:28:37.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0=', 'Content-Length': '0'} 2024-04-12T03:28:37.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071f1c18525eabc90-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.272 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.272 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.272 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.272 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.272 DEBUG:boto:Method: GET 2024-04-12T03:28:37.272 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.272 DEBUG:boto:Data: 2024-04-12T03:28:37.272 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.272 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.273 DEBUG:boto:Params: {} 2024-04-12T03:28:37.273 DEBUG:boto:Token: None 2024-04-12T03:28:37.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:28:37.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0= 2024-04-12T03:28:37.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p+jpkJ3pnPCT1N1HwqyyyDMIUF0=', 'Content-Length': '0'} 2024-04-12T03:28:37.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001be704a748296d3-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.275 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.275 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.275 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.275 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.275 DEBUG:boto:Method: GET 2024-04-12T03:28:37.275 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.275 DEBUG:boto:Data: 2024-04-12T03:28:37.275 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.275 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.275 DEBUG:boto:Params: {} 2024-04-12T03:28:37.276 DEBUG:boto:Token: None 2024-04-12T03:28:37.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e71219660e3e48a3-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.278 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.278 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.278 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.278 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.278 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.278 DEBUG:boto:Method: GET 2024-04-12T03:28:37.279 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.279 DEBUG:boto:Data: 2024-04-12T03:28:37.279 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.279 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.279 DEBUG:boto:Params: {} 2024-04-12T03:28:37.279 DEBUG:boto:Token: None 2024-04-12T03:28:37.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e136d46a288f5dc9-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.322 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.323 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.323 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.323 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.323 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:28:37.323 DEBUG:boto:Data: 2024-04-12T03:28:37.323 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.323 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.323 DEBUG:boto:Params: {} 2024-04-12T03:28:37.323 DEBUG:boto:Token: None 2024-04-12T03:28:37.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3 2024-04-12T03:28:37.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KTdCGSap4KWaYsyTUXkbAvxh0iM= 2024-04-12T03:28:37.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KTdCGSap4KWaYsyTUXkbAvxh0iM=', 'Content-Length': '0'} 2024-04-12T03:28:37.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c81c08da30cbf20-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.326 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.326 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.326 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.326 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:28:37.326 DEBUG:boto:Data: 2024-04-12T03:28:37.326 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.326 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.326 DEBUG:boto:Params: {} 2024-04-12T03:28:37.327 DEBUG:boto:Token: None 2024-04-12T03:28:37.327 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3 2024-04-12T03:28:37.327 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KTdCGSap4KWaYsyTUXkbAvxh0iM= 2024-04-12T03:28:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KTdCGSap4KWaYsyTUXkbAvxh0iM=', 'Content-Length': '0'} 2024-04-12T03:28:37.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001425903d0bac873-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:37.329 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.329 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.329 DEBUG:boto:Method: GET 2024-04-12T03:28:37.329 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:28:37.329 DEBUG:boto:Data: 2024-04-12T03:28:37.329 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.330 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.330 DEBUG:boto:Params: {} 2024-04-12T03:28:37.330 DEBUG:boto:Token: None 2024-04-12T03:28:37.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3 2024-04-12T03:28:37.330 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+RTGLtJrOCGPfGrE2LTNYyHpoqE= 2024-04-12T03:28:37.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+RTGLtJrOCGPfGrE2LTNYyHpoqE=', 'Content-Length': '0'} 2024-04-12T03:28:37.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002be931a605d0fc52-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.332 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:28:37.332 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:28:37.332 DEBUG:boto:Method: GET 2024-04-12T03:28:37.332 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:28:37.332 DEBUG:boto:Data: 2024-04-12T03:28:37.333 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.333 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.333 DEBUG:boto:Params: {} 2024-04-12T03:28:37.333 DEBUG:boto:Token: None 2024-04-12T03:28:37.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj3 2024-04-12T03:28:37.333 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+RTGLtJrOCGPfGrE2LTNYyHpoqE= 2024-04-12T03:28:37.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+RTGLtJrOCGPfGrE2LTNYyHpoqE=', 'Content-Length': '0'} 2024-04-12T03:28:37.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a27ffa50763be5c-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.335 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.335 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.335 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.335 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.335 DEBUG:boto:Method: GET 2024-04-12T03:28:37.335 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.335 DEBUG:boto:Data: 2024-04-12T03:28:37.336 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.336 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.336 DEBUG:boto:Params: {} 2024-04-12T03:28:37.336 DEBUG:boto:Token: None 2024-04-12T03:28:37.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1cd771b095b08f7-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.338 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.339 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.339 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.339 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.339 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.339 DEBUG:boto:Method: GET 2024-04-12T03:28:37.339 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:28:37.339 DEBUG:boto:Data: 2024-04-12T03:28:37.339 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.339 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.340 DEBUG:boto:Params: {} 2024-04-12T03:28:37.340 DEBUG:boto:Token: None 2024-04-12T03:28:37.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aba94b4140971df-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.342 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:37.342 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.342 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.343 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.343 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.343 DEBUG:boto:Method: GET 2024-04-12T03:28:37.343 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.343 DEBUG:boto:Data: 2024-04-12T03:28:37.343 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.343 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.343 DEBUG:boto:Params: {} 2024-04-12T03:28:37.343 DEBUG:boto:Token: None 2024-04-12T03:28:37.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE= 2024-04-12T03:28:37.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE=', 'Content-Length': '0'} 2024-04-12T03:28:37.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0843844af071a24-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.346 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.346 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.346 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.346 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.346 DEBUG:boto:Method: GET 2024-04-12T03:28:37.346 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.346 DEBUG:boto:Data: 2024-04-12T03:28:37.346 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.346 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.347 DEBUG:boto:Params: {} 2024-04-12T03:28:37.347 DEBUG:boto:Token: None 2024-04-12T03:28:37.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE= 2024-04-12T03:28:37.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE=', 'Content-Length': '0'} 2024-04-12T03:28:37.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b184d18047a50d9-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.349 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.349 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.349 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.349 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.349 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.349 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.349 DEBUG:boto:Data: 2024-04-12T03:28:37.349 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.350 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.350 DEBUG:boto:Params: {} 2024-04-12T03:28:37.350 DEBUG:boto:Token: None 2024-04-12T03:28:37.350 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.350 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AIlIhze4VlsAyYI+DdEFy4IJHRI= 2024-04-12T03:28:37.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AIlIhze4VlsAyYI+DdEFy4IJHRI=', 'Content-Length': '0'} 2024-04-12T03:28:37.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ce71db69f04a1ca-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.352 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.352 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.352 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.353 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.353 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.353 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.353 DEBUG:boto:Data: 2024-04-12T03:28:37.353 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.353 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.353 DEBUG:boto:Params: {} 2024-04-12T03:28:37.353 DEBUG:boto:Token: None 2024-04-12T03:28:37.353 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.353 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AIlIhze4VlsAyYI+DdEFy4IJHRI= 2024-04-12T03:28:37.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AIlIhze4VlsAyYI+DdEFy4IJHRI=', 'Content-Length': '0'} 2024-04-12T03:28:37.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e5e107f69a36abf-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:37.355 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.355 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.356 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.356 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.356 DEBUG:boto:Method: GET 2024-04-12T03:28:37.356 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.356 DEBUG:boto:Data: 2024-04-12T03:28:37.356 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.356 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.356 DEBUG:boto:Params: {} 2024-04-12T03:28:37.356 DEBUG:boto:Token: None 2024-04-12T03:28:37.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.357 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE= 2024-04-12T03:28:37.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE=', 'Content-Length': '0'} 2024-04-12T03:28:37.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb1896eeaa5bc1c8-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.359 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.359 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.359 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.359 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.359 DEBUG:boto:Method: GET 2024-04-12T03:28:37.359 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.360 DEBUG:boto:Data: 2024-04-12T03:28:37.360 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.360 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.360 DEBUG:boto:Params: {} 2024-04-12T03:28:37.360 DEBUG:boto:Token: None 2024-04-12T03:28:37.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:28:37.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE= 2024-04-12T03:28:37.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uoz+YRsnJK/cXHXE4ohZZZS9NNE=', 'Content-Length': '0'} 2024-04-12T03:28:37.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca18267ac6af2eb1-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.402 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.402 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.402 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.403 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.403 DEBUG:boto:Method: GET 2024-04-12T03:28:37.403 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.403 DEBUG:boto:Data: 2024-04-12T03:28:37.403 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.403 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.403 DEBUG:boto:Params: {} 2024-04-12T03:28:37.403 DEBUG:boto:Token: None 2024-04-12T03:28:37.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae0c8401b72daca6-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.406 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.406 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.406 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.406 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.406 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.406 DEBUG:boto:Method: GET 2024-04-12T03:28:37.406 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.407 DEBUG:boto:Data: 2024-04-12T03:28:37.407 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.407 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.407 DEBUG:boto:Params: {} 2024-04-12T03:28:37.407 DEBUG:boto:Token: None 2024-04-12T03:28:37.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008772e53e7287960b-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.409 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.409 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.410 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.410 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.410 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:28:37.410 DEBUG:boto:Data: 2024-04-12T03:28:37.410 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.410 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.410 DEBUG:boto:Params: {} 2024-04-12T03:28:37.410 DEBUG:boto:Token: None 2024-04-12T03:28:37.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4 2024-04-12T03:28:37.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEwuC+bftXgFd7l9tYImZmy6wrI= 2024-04-12T03:28:37.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEwuC+bftXgFd7l9tYImZmy6wrI=', 'Content-Length': '0'} 2024-04-12T03:28:37.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eda925c64145798-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.412 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.412 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.412 DEBUG:boto:Method: HEAD 2024-04-12T03:28:37.413 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:28:37.413 DEBUG:boto:Data: 2024-04-12T03:28:37.413 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.413 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.413 DEBUG:boto:Params: {} 2024-04-12T03:28:37.413 DEBUG:boto:Token: None 2024-04-12T03:28:37.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4 2024-04-12T03:28:37.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEwuC+bftXgFd7l9tYImZmy6wrI= 2024-04-12T03:28:37.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEwuC+bftXgFd7l9tYImZmy6wrI=', 'Content-Length': '0'} 2024-04-12T03:28:37.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005f9212335f8c4e4-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:37.415 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.415 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.415 DEBUG:boto:Method: GET 2024-04-12T03:28:37.415 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:28:37.415 DEBUG:boto:Data: 2024-04-12T03:28:37.415 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.416 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.416 DEBUG:boto:Params: {} 2024-04-12T03:28:37.416 DEBUG:boto:Token: None 2024-04-12T03:28:37.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4 2024-04-12T03:28:37.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hAxurURwZ+RUCSLx0yNR477L3AM= 2024-04-12T03:28:37.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hAxurURwZ+RUCSLx0yNR477L3AM=', 'Content-Length': '0'} 2024-04-12T03:28:37.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002c2b778f02bf6cf-006618aa65-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.418 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:28:37.418 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:28:37.418 DEBUG:boto:Method: GET 2024-04-12T03:28:37.418 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:28:37.418 DEBUG:boto:Data: 2024-04-12T03:28:37.418 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.419 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.419 DEBUG:boto:Params: {} 2024-04-12T03:28:37.421 DEBUG:boto:Token: None 2024-04-12T03:28:37.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/obj4 2024-04-12T03:28:37.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hAxurURwZ+RUCSLx0yNR477L3AM= 2024-04-12T03:28:37.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hAxurURwZ+RUCSLx0yNR477L3AM=', 'Content-Length': '0'} 2024-04-12T03:28:37.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbc081d2c8174dc4-006618aa65-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.424 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.424 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.424 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.424 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.424 DEBUG:boto:Method: GET 2024-04-12T03:28:37.424 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.424 DEBUG:boto:Data: 2024-04-12T03:28:37.424 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:37.425 DEBUG:boto:Port: 8001 2024-04-12T03:28:37.425 DEBUG:boto:Params: {} 2024-04-12T03:28:37.425 DEBUG:boto:Token: None 2024-04-12T03:28:37.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822133c11b1709cf-006618aa65-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.427 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.428 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:37.428 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:37.428 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.428 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.428 DEBUG:boto:Method: GET 2024-04-12T03:28:37.428 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:28:37.428 DEBUG:boto:Data: 2024-04-12T03:28:37.428 DEBUG:boto:Headers: {} 2024-04-12T03:28:37.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:37.429 DEBUG:boto:Port: 8000 2024-04-12T03:28:37.429 DEBUG:boto:Params: {} 2024-04-12T03:28:37.429 DEBUG:boto:Token: None 2024-04-12T03:28:37.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:37 GMT /dgbpxg-48/ 2024-04-12T03:28:37.429 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg= 2024-04-12T03:28:37.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3GH0qoieHI/OvBUVCb8gSXgZXg=', 'Content-Length': '0'} 2024-04-12T03:28:37.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cc81633d533b843-006618aa65-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:37.432 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:37.432 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-48 zones={a2, a1} 2024-04-12T03:28:37.432 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-49 2024-04-12T03:28:37.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:37.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:37.433 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:37.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:37.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:37.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:37.523+0000 7fd985357780 1 waiting to reach incremental sync.. 2024-04-12T03:28:42.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:42.524+0000 7fd985357780 1 waiting to reach incremental sync.. 2024-04-12T03:28:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:47.524+0000 7fd985357780 1 waiting to reach incremental sync.. 2024-04-12T03:28:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:52.525+0000 7fd985357780 1 waiting to reach incremental sync.. 2024-04-12T03:28:57.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:57.525+0000 7fd985357780 1 bucket sync caught up with source: 2024-04-12T03:28:57.527 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.28.6, , 00000000001.44.6, , 00000000001.38.6, , , 00000000001.34.6, , , ] 2024-04-12T03:28:57.527 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.28.6, , 00000000001.44.6, , 00000000001.38.6, , , 00000000001.34.6, , , ] 2024-04-12T03:28:57.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:57.525+0000 7fd985357780 0 bucket checkpoint complete 2024-04-12T03:28:57.545 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-49 2024-04-12T03:28:57.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:57.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:57.546 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:57.608 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:57.609 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:57.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:57.648+0000 7f88dda0d780 1 bucket sync caught up with source: 2024-04-12T03:28:57.649 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.3209.6, , 00000000001.412.6, , 00000000001.187.6, , , 00000000001.765.6, , , ] 2024-04-12T03:28:57.649 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.3209.6, , 00000000001.412.6, , 00000000001.187.6, , , 00000000001.765.6, , , ] 2024-04-12T03:28:57.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:57.648+0000 7f88dda0d780 0 bucket checkpoint complete 2024-04-12T03:28:57.662 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-49 zones={a2, a1} 2024-04-12T03:28:57.663 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.663 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.663 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.663 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:57.663 DEBUG:boto:Data: 2024-04-12T03:28:57.663 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.663 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.663 DEBUG:boto:Params: {} 2024-04-12T03:28:57.663 DEBUG:boto:Token: None 2024-04-12T03:28:57.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oo9BJaBBk+jUWpWnY8p3jNc9M0A= 2024-04-12T03:28:57.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oo9BJaBBk+jUWpWnY8p3jNc9M0A=', 'Content-Length': '0'} 2024-04-12T03:28:57.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5b1cf24e2694bca-006618aa79-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.666 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.667 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.667 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.667 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:57.667 DEBUG:boto:Data: 2024-04-12T03:28:57.667 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.667 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.667 DEBUG:boto:Params: {} 2024-04-12T03:28:57.667 DEBUG:boto:Token: None 2024-04-12T03:28:57.667 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oo9BJaBBk+jUWpWnY8p3jNc9M0A= 2024-04-12T03:28:57.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oo9BJaBBk+jUWpWnY8p3jNc9M0A=', 'Content-Length': '0'} 2024-04-12T03:28:57.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff313d93f6f8ffc-006618aa79-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.670 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:28:57.670 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.670 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.670 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.670 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.670 DEBUG:boto:Method: GET 2024-04-12T03:28:57.670 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.671 DEBUG:boto:Data: 2024-04-12T03:28:57.671 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.671 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.671 DEBUG:boto:Params: {} 2024-04-12T03:28:57.671 DEBUG:boto:Token: None 2024-04-12T03:28:57.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/?versions 2024-04-12T03:28:57.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8= 2024-04-12T03:28:57.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8=', 'Content-Length': '0'} 2024-04-12T03:28:57.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e009065761fc00-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.673 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:28:57.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:28:57.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:28:57.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:28:57.674 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:28:57.674 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.675 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.675 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.675 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.675 DEBUG:boto:Method: GET 2024-04-12T03:28:57.675 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.675 DEBUG:boto:Data: 2024-04-12T03:28:57.675 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.675 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.675 DEBUG:boto:Params: {} 2024-04-12T03:28:57.675 DEBUG:boto:Token: None 2024-04-12T03:28:57.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/?versions 2024-04-12T03:28:57.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8= 2024-04-12T03:28:57.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8=', 'Content-Length': '0'} 2024-04-12T03:28:57.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0a64524c5950e3-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.678 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:28:57.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:28:57.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:28:57.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:28:57.679 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.679 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.679 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.679 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.679 DEBUG:boto:Method: GET 2024-04-12T03:28:57.680 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.680 DEBUG:boto:Data: 2024-04-12T03:28:57.680 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.680 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.680 DEBUG:boto:Params: {} 2024-04-12T03:28:57.680 DEBUG:boto:Token: None 2024-04-12T03:28:57.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/?versions 2024-04-12T03:28:57.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8= 2024-04-12T03:28:57.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8=', 'Content-Length': '0'} 2024-04-12T03:28:57.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf5049f463104f9a-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.683 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.683 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.683 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.683 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.683 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.683 DEBUG:boto:Method: GET 2024-04-12T03:28:57.683 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:28:57.684 DEBUG:boto:Data: 2024-04-12T03:28:57.684 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.684 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.684 DEBUG:boto:Params: {} 2024-04-12T03:28:57.684 DEBUG:boto:Token: None 2024-04-12T03:28:57.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/?versions 2024-04-12T03:28:57.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8= 2024-04-12T03:28:57.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xY9CobdS1eH/LRmqFZ5Ex6+tuu8=', 'Content-Length': '0'} 2024-04-12T03:28:57.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa34d7ebddc80100-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.686 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:57.687 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.687 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.687 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.687 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.687 DEBUG:boto:Method: GET 2024-04-12T03:28:57.688 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.688 DEBUG:boto:Data: 2024-04-12T03:28:57.688 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.688 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.688 DEBUG:boto:Params: {} 2024-04-12T03:28:57.688 DEBUG:boto:Token: None 2024-04-12T03:28:57.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8= 2024-04-12T03:28:57.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8=', 'Content-Length': '0'} 2024-04-12T03:28:57.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ed6125c357a2e6-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.730 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.730 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.730 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.731 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.731 DEBUG:boto:Method: GET 2024-04-12T03:28:57.731 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.731 DEBUG:boto:Data: 2024-04-12T03:28:57.731 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.731 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.731 DEBUG:boto:Params: {} 2024-04-12T03:28:57.731 DEBUG:boto:Token: None 2024-04-12T03:28:57.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8= 2024-04-12T03:28:57.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8=', 'Content-Length': '0'} 2024-04-12T03:28:57.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093c4cdb16c27acec-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.734 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.734 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.734 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.734 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.734 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.734 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.734 DEBUG:boto:Data: 2024-04-12T03:28:57.735 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.735 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.735 DEBUG:boto:Params: {} 2024-04-12T03:28:57.735 DEBUG:boto:Token: None 2024-04-12T03:28:57.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:35fMJ0LbpHLTpxopk60Xa+UjGC8= 2024-04-12T03:28:57.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35fMJ0LbpHLTpxopk60Xa+UjGC8=', 'Content-Length': '0'} 2024-04-12T03:28:57.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009eec2ae1ca3879e2-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.737 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.737 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.737 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.737 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.738 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.738 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.738 DEBUG:boto:Data: 2024-04-12T03:28:57.738 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.738 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.738 DEBUG:boto:Params: {} 2024-04-12T03:28:57.738 DEBUG:boto:Token: None 2024-04-12T03:28:57.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:35fMJ0LbpHLTpxopk60Xa+UjGC8= 2024-04-12T03:28:57.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35fMJ0LbpHLTpxopk60Xa+UjGC8=', 'Content-Length': '0'} 2024-04-12T03:28:57.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc4985cc4cbbc50f-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:57.741 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.741 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.741 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.741 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.741 DEBUG:boto:Method: GET 2024-04-12T03:28:57.741 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.741 DEBUG:boto:Data: 2024-04-12T03:28:57.741 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.742 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.742 DEBUG:boto:Params: {} 2024-04-12T03:28:57.742 DEBUG:boto:Token: None 2024-04-12T03:28:57.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8= 2024-04-12T03:28:57.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8=', 'Content-Length': '0'} 2024-04-12T03:28:57.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000444c9b5f71a814c2-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.744 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.744 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.744 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.744 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.744 DEBUG:boto:Method: GET 2024-04-12T03:28:57.744 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.744 DEBUG:boto:Data: 2024-04-12T03:28:57.744 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.745 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.745 DEBUG:boto:Params: {} 2024-04-12T03:28:57.745 DEBUG:boto:Token: None 2024-04-12T03:28:57.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:28:57.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8= 2024-04-12T03:28:57.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6bKf+6lXF2jZxgaM8GT08lYGqL8=', 'Content-Length': '0'} 2024-04-12T03:28:57.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008815a30a68b2de4-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.747 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.747 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.747 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.747 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.747 DEBUG:boto:Method: GET 2024-04-12T03:28:57.747 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.748 DEBUG:boto:Data: 2024-04-12T03:28:57.748 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.748 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.748 DEBUG:boto:Params: {} 2024-04-12T03:28:57.748 DEBUG:boto:Token: None 2024-04-12T03:28:57.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd052795c6c2026f-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.750 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.751 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.751 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.751 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.751 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.751 DEBUG:boto:Method: GET 2024-04-12T03:28:57.751 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.751 DEBUG:boto:Data: 2024-04-12T03:28:57.752 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.752 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.752 DEBUG:boto:Params: {} 2024-04-12T03:28:57.752 DEBUG:boto:Token: None 2024-04-12T03:28:57.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005015d3f351c637a1-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.798 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.799 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.799 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.799 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.799 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:28:57.799 DEBUG:boto:Data: 2024-04-12T03:28:57.799 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.799 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.799 DEBUG:boto:Params: {} 2024-04-12T03:28:57.799 DEBUG:boto:Token: None 2024-04-12T03:28:57.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1 2024-04-12T03:28:57.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C9lJ/mLLMwLzFbrK2EDckfHIfHM= 2024-04-12T03:28:57.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C9lJ/mLLMwLzFbrK2EDckfHIfHM=', 'Content-Length': '0'} 2024-04-12T03:28:57.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009703f2caac653b1-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.801 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.801 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.802 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.802 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:28:57.802 DEBUG:boto:Data: 2024-04-12T03:28:57.802 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.802 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.802 DEBUG:boto:Params: {} 2024-04-12T03:28:57.802 DEBUG:boto:Token: None 2024-04-12T03:28:57.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1 2024-04-12T03:28:57.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C9lJ/mLLMwLzFbrK2EDckfHIfHM= 2024-04-12T03:28:57.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C9lJ/mLLMwLzFbrK2EDckfHIfHM=', 'Content-Length': '0'} 2024-04-12T03:28:57.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a19478172e612e93-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:28:57.805 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.805 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.805 DEBUG:boto:Method: GET 2024-04-12T03:28:57.805 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:28:57.805 DEBUG:boto:Data: 2024-04-12T03:28:57.805 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.805 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.805 DEBUG:boto:Params: {} 2024-04-12T03:28:57.806 DEBUG:boto:Token: None 2024-04-12T03:28:57.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1 2024-04-12T03:28:57.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h2IaSdSzZGnFPwDeI3GZUatHxRA= 2024-04-12T03:28:57.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h2IaSdSzZGnFPwDeI3GZUatHxRA=', 'Content-Length': '0'} 2024-04-12T03:28:57.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091412723d3df1cf1-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.807 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:28:57.807 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:28:57.808 DEBUG:boto:Method: GET 2024-04-12T03:28:57.808 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:28:57.808 DEBUG:boto:Data: 2024-04-12T03:28:57.808 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.808 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.808 DEBUG:boto:Params: {} 2024-04-12T03:28:57.809 DEBUG:boto:Token: None 2024-04-12T03:28:57.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj1 2024-04-12T03:28:57.809 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h2IaSdSzZGnFPwDeI3GZUatHxRA= 2024-04-12T03:28:57.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h2IaSdSzZGnFPwDeI3GZUatHxRA=', 'Content-Length': '0'} 2024-04-12T03:28:57.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c75ac49012d5de99-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.811 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.811 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.811 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.811 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.811 DEBUG:boto:Method: GET 2024-04-12T03:28:57.811 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.811 DEBUG:boto:Data: 2024-04-12T03:28:57.811 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.811 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.812 DEBUG:boto:Params: {} 2024-04-12T03:28:57.812 DEBUG:boto:Token: None 2024-04-12T03:28:57.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f833b30333905a3d-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.814 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.814 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.815 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.815 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.815 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.815 DEBUG:boto:Method: GET 2024-04-12T03:28:57.815 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:28:57.815 DEBUG:boto:Data: 2024-04-12T03:28:57.815 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.815 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.815 DEBUG:boto:Params: {} 2024-04-12T03:28:57.815 DEBUG:boto:Token: None 2024-04-12T03:28:57.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9cdae86f5c653cc-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.818 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:57.818 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.819 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.819 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.819 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.819 DEBUG:boto:Method: GET 2024-04-12T03:28:57.819 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.819 DEBUG:boto:Data: 2024-04-12T03:28:57.819 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.819 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.819 DEBUG:boto:Params: {} 2024-04-12T03:28:57.819 DEBUG:boto:Token: None 2024-04-12T03:28:57.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek= 2024-04-12T03:28:57.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek=', 'Content-Length': '0'} 2024-04-12T03:28:57.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e4c5879d3596896-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.822 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.822 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.822 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.822 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.822 DEBUG:boto:Method: GET 2024-04-12T03:28:57.822 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.822 DEBUG:boto:Data: 2024-04-12T03:28:57.823 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.823 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.823 DEBUG:boto:Params: {} 2024-04-12T03:28:57.823 DEBUG:boto:Token: None 2024-04-12T03:28:57.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek= 2024-04-12T03:28:57.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek=', 'Content-Length': '0'} 2024-04-12T03:28:57.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006165c3453a586d17-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.825 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.825 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.825 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.825 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.825 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.826 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.826 DEBUG:boto:Data: 2024-04-12T03:28:57.826 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.826 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.826 DEBUG:boto:Params: {} 2024-04-12T03:28:57.826 DEBUG:boto:Token: None 2024-04-12T03:28:57.826 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0XnQBLJu7Vj1IC6UhbzRpZU/TWQ= 2024-04-12T03:28:57.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0XnQBLJu7Vj1IC6UhbzRpZU/TWQ=', 'Content-Length': '0'} 2024-04-12T03:28:57.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000134c489e9d366ec2-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.828 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.828 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.828 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.828 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.828 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.829 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.829 DEBUG:boto:Data: 2024-04-12T03:28:57.829 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.829 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.829 DEBUG:boto:Params: {} 2024-04-12T03:28:57.829 DEBUG:boto:Token: None 2024-04-12T03:28:57.829 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0XnQBLJu7Vj1IC6UhbzRpZU/TWQ= 2024-04-12T03:28:57.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0XnQBLJu7Vj1IC6UhbzRpZU/TWQ=', 'Content-Length': '0'} 2024-04-12T03:28:57.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046a0a0a89ca5f93f-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:57.831 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.831 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.831 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.831 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.831 DEBUG:boto:Method: GET 2024-04-12T03:28:57.831 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.832 DEBUG:boto:Data: 2024-04-12T03:28:57.832 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.832 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.832 DEBUG:boto:Params: {} 2024-04-12T03:28:57.832 DEBUG:boto:Token: None 2024-04-12T03:28:57.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek= 2024-04-12T03:28:57.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek=', 'Content-Length': '0'} 2024-04-12T03:28:57.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0175d6b5ba7d32-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.834 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.834 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.834 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.834 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.835 DEBUG:boto:Method: GET 2024-04-12T03:28:57.835 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.835 DEBUG:boto:Data: 2024-04-12T03:28:57.835 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.835 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.835 DEBUG:boto:Params: {} 2024-04-12T03:28:57.835 DEBUG:boto:Token: None 2024-04-12T03:28:57.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:28:57.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek= 2024-04-12T03:28:57.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M6mP0d5yIX7TQQMkVho1o6vYaek=', 'Content-Length': '0'} 2024-04-12T03:28:57.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096e44c68f11a37ab-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.878 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.878 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.878 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.879 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.879 DEBUG:boto:Method: GET 2024-04-12T03:28:57.879 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.879 DEBUG:boto:Data: 2024-04-12T03:28:57.879 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.879 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.879 DEBUG:boto:Params: {} 2024-04-12T03:28:57.879 DEBUG:boto:Token: None 2024-04-12T03:28:57.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ca79826ab5f0ae-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.882 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.883 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.883 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.883 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.883 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.884 DEBUG:boto:Method: GET 2024-04-12T03:28:57.884 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.884 DEBUG:boto:Data: 2024-04-12T03:28:57.884 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.885 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.885 DEBUG:boto:Params: {} 2024-04-12T03:28:57.885 DEBUG:boto:Token: None 2024-04-12T03:28:57.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.885 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dad452e1e7cd8f9-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.888 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.889 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.889 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.889 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.889 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:28:57.889 DEBUG:boto:Data: 2024-04-12T03:28:57.889 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.890 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.890 DEBUG:boto:Params: {} 2024-04-12T03:28:57.890 DEBUG:boto:Token: None 2024-04-12T03:28:57.890 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2 2024-04-12T03:28:57.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiVh1aE7sLOYAn0w227ybetnBRE= 2024-04-12T03:28:57.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiVh1aE7sLOYAn0w227ybetnBRE=', 'Content-Length': '0'} 2024-04-12T03:28:57.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdf9648e888f0e64-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.892 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.892 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.892 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.892 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:28:57.892 DEBUG:boto:Data: 2024-04-12T03:28:57.893 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.893 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.893 DEBUG:boto:Params: {} 2024-04-12T03:28:57.893 DEBUG:boto:Token: None 2024-04-12T03:28:57.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2 2024-04-12T03:28:57.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiVh1aE7sLOYAn0w227ybetnBRE= 2024-04-12T03:28:57.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiVh1aE7sLOYAn0w227ybetnBRE=', 'Content-Length': '0'} 2024-04-12T03:28:57.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006589e172e83bfce-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:28:57.895 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.895 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.895 DEBUG:boto:Method: GET 2024-04-12T03:28:57.895 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:28:57.895 DEBUG:boto:Data: 2024-04-12T03:28:57.896 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.896 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.896 DEBUG:boto:Params: {} 2024-04-12T03:28:57.896 DEBUG:boto:Token: None 2024-04-12T03:28:57.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2 2024-04-12T03:28:57.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eCkYCft52ljy1GSsmKfIq+qUZoE= 2024-04-12T03:28:57.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eCkYCft52ljy1GSsmKfIq+qUZoE=', 'Content-Length': '0'} 2024-04-12T03:28:57.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8e9ee5308961771-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.898 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:28:57.898 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:28:57.898 DEBUG:boto:Method: GET 2024-04-12T03:28:57.899 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:28:57.899 DEBUG:boto:Data: 2024-04-12T03:28:57.899 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.899 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.899 DEBUG:boto:Params: {} 2024-04-12T03:28:57.899 DEBUG:boto:Token: None 2024-04-12T03:28:57.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj2 2024-04-12T03:28:57.899 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eCkYCft52ljy1GSsmKfIq+qUZoE= 2024-04-12T03:28:57.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eCkYCft52ljy1GSsmKfIq+qUZoE=', 'Content-Length': '0'} 2024-04-12T03:28:57.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093e5498294e76a77-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.901 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.901 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.901 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.902 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.902 DEBUG:boto:Method: GET 2024-04-12T03:28:57.902 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.902 DEBUG:boto:Data: 2024-04-12T03:28:57.902 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.902 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.902 DEBUG:boto:Params: {} 2024-04-12T03:28:57.902 DEBUG:boto:Token: None 2024-04-12T03:28:57.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e7d143211a8c17-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.905 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.905 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.905 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.906 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.906 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.906 DEBUG:boto:Method: GET 2024-04-12T03:28:57.906 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:28:57.906 DEBUG:boto:Data: 2024-04-12T03:28:57.906 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.906 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.906 DEBUG:boto:Params: {} 2024-04-12T03:28:57.906 DEBUG:boto:Token: None 2024-04-12T03:28:57.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c913d1a9449502-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.909 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:57.909 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.910 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.910 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.910 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.910 DEBUG:boto:Method: GET 2024-04-12T03:28:57.910 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.910 DEBUG:boto:Data: 2024-04-12T03:28:57.910 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.910 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.910 DEBUG:boto:Params: {} 2024-04-12T03:28:57.910 DEBUG:boto:Token: None 2024-04-12T03:28:57.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA= 2024-04-12T03:28:57.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA=', 'Content-Length': '0'} 2024-04-12T03:28:57.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000193fb559e637d590-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.913 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.913 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.913 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.913 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.913 DEBUG:boto:Method: GET 2024-04-12T03:28:57.913 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.913 DEBUG:boto:Data: 2024-04-12T03:28:57.913 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.914 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.914 DEBUG:boto:Params: {} 2024-04-12T03:28:57.914 DEBUG:boto:Token: None 2024-04-12T03:28:57.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.914 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA= 2024-04-12T03:28:57.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA=', 'Content-Length': '0'} 2024-04-12T03:28:57.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfbd6affda84bfdd-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.916 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.916 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.916 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.916 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.916 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.917 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.917 DEBUG:boto:Data: 2024-04-12T03:28:57.917 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.917 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.917 DEBUG:boto:Params: {} 2024-04-12T03:28:57.917 DEBUG:boto:Token: None 2024-04-12T03:28:57.917 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0VBx/LRTe8yGASzlJJnhFnUW/CQ= 2024-04-12T03:28:57.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0VBx/LRTe8yGASzlJJnhFnUW/CQ=', 'Content-Length': '0'} 2024-04-12T03:28:57.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055e0e64668da0290-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.919 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.919 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.919 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.919 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.920 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.920 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.920 DEBUG:boto:Data: 2024-04-12T03:28:57.920 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.920 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.920 DEBUG:boto:Params: {} 2024-04-12T03:28:57.920 DEBUG:boto:Token: None 2024-04-12T03:28:57.920 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0VBx/LRTe8yGASzlJJnhFnUW/CQ= 2024-04-12T03:28:57.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0VBx/LRTe8yGASzlJJnhFnUW/CQ=', 'Content-Length': '0'} 2024-04-12T03:28:57.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c040b228ade18ee5-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:57.922 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.923 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.923 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.923 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.923 DEBUG:boto:Method: GET 2024-04-12T03:28:57.923 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.923 DEBUG:boto:Data: 2024-04-12T03:28:57.923 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.923 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.923 DEBUG:boto:Params: {} 2024-04-12T03:28:57.923 DEBUG:boto:Token: None 2024-04-12T03:28:57.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA= 2024-04-12T03:28:57.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA=', 'Content-Length': '0'} 2024-04-12T03:28:57.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c53d138ec4aced7e-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.926 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.926 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.926 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.926 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.926 DEBUG:boto:Method: GET 2024-04-12T03:28:57.926 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.926 DEBUG:boto:Data: 2024-04-12T03:28:57.926 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.927 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.927 DEBUG:boto:Params: {} 2024-04-12T03:28:57.927 DEBUG:boto:Token: None 2024-04-12T03:28:57.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:28:57.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA= 2024-04-12T03:28:57.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WyzzbyYU4RRTLYe8jS0xyFwQwrA=', 'Content-Length': '0'} 2024-04-12T03:28:57.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b159a70ca4687237-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.929 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.929 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.929 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.930 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.930 DEBUG:boto:Method: GET 2024-04-12T03:28:57.930 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.930 DEBUG:boto:Data: 2024-04-12T03:28:57.930 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.930 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.930 DEBUG:boto:Params: {} 2024-04-12T03:28:57.930 DEBUG:boto:Token: None 2024-04-12T03:28:57.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cf6fb2f6202dde-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.933 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.933 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.933 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.933 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.933 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.934 DEBUG:boto:Method: GET 2024-04-12T03:28:57.934 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.934 DEBUG:boto:Data: 2024-04-12T03:28:57.934 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.934 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.934 DEBUG:boto:Params: {} 2024-04-12T03:28:57.934 DEBUG:boto:Token: None 2024-04-12T03:28:57.934 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.934 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d749f222124ad125-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.937 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.937 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.937 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.937 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.937 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:28:57.937 DEBUG:boto:Data: 2024-04-12T03:28:57.937 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.938 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.938 DEBUG:boto:Params: {} 2024-04-12T03:28:57.938 DEBUG:boto:Token: None 2024-04-12T03:28:57.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3 2024-04-12T03:28:57.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rw0hgKFgpcbqH/yCUCNoq/XpCAE= 2024-04-12T03:28:57.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rw0hgKFgpcbqH/yCUCNoq/XpCAE=', 'Content-Length': '0'} 2024-04-12T03:28:57.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000246030973cabf941-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.940 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.940 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.940 DEBUG:boto:Method: HEAD 2024-04-12T03:28:57.940 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:28:57.940 DEBUG:boto:Data: 2024-04-12T03:28:57.940 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.941 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.941 DEBUG:boto:Params: {} 2024-04-12T03:28:57.941 DEBUG:boto:Token: None 2024-04-12T03:28:57.941 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3 2024-04-12T03:28:57.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rw0hgKFgpcbqH/yCUCNoq/XpCAE= 2024-04-12T03:28:57.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rw0hgKFgpcbqH/yCUCNoq/XpCAE=', 'Content-Length': '0'} 2024-04-12T03:28:57.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b01fb8a124c1a4e-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:28:57.943 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.943 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.943 DEBUG:boto:Method: GET 2024-04-12T03:28:57.943 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:28:57.944 DEBUG:boto:Data: 2024-04-12T03:28:57.944 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.944 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.944 DEBUG:boto:Params: {} 2024-04-12T03:28:57.944 DEBUG:boto:Token: None 2024-04-12T03:28:57.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3 2024-04-12T03:28:57.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7vSz775xS7qo+ax3mOOE49B+ETk= 2024-04-12T03:28:57.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7vSz775xS7qo+ax3mOOE49B+ETk=', 'Content-Length': '0'} 2024-04-12T03:28:57.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e9ac16009ecaa2-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.946 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:28:57.946 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:28:57.946 DEBUG:boto:Method: GET 2024-04-12T03:28:57.946 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:28:57.947 DEBUG:boto:Data: 2024-04-12T03:28:57.947 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.947 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.947 DEBUG:boto:Params: {} 2024-04-12T03:28:57.947 DEBUG:boto:Token: None 2024-04-12T03:28:57.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj3 2024-04-12T03:28:57.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7vSz775xS7qo+ax3mOOE49B+ETk= 2024-04-12T03:28:57.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7vSz775xS7qo+ax3mOOE49B+ETk=', 'Content-Length': '0'} 2024-04-12T03:28:57.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4cc83efcaa467f5-006618aa79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.990 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.990 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.990 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.991 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.991 DEBUG:boto:Method: GET 2024-04-12T03:28:57.991 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.991 DEBUG:boto:Data: 2024-04-12T03:28:57.991 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.991 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.991 DEBUG:boto:Params: {} 2024-04-12T03:28:57.992 DEBUG:boto:Token: None 2024-04-12T03:28:57.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128b2a0792bc5156-006618aa79-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.994 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.994 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:57.994 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:57.995 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.995 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.995 DEBUG:boto:Method: GET 2024-04-12T03:28:57.995 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:28:57.995 DEBUG:boto:Data: 2024-04-12T03:28:57.995 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:57.995 DEBUG:boto:Port: 8000 2024-04-12T03:28:57.995 DEBUG:boto:Params: {} 2024-04-12T03:28:57.995 DEBUG:boto:Token: None 2024-04-12T03:28:57.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/ 2024-04-12T03:28:57.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8= 2024-04-12T03:28:57.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYnKf52glsUIjDYF5JyAOGmlD+8=', 'Content-Length': '0'} 2024-04-12T03:28:57.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8bb3a20b4e181cf-006618aa79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:57.998 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:57.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:57.998 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:57.998 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:57.999 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:57.999 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:57.999 DEBUG:boto:Method: GET 2024-04-12T03:28:57.999 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:57.999 DEBUG:boto:Data: 2024-04-12T03:28:57.999 DEBUG:boto:Headers: {} 2024-04-12T03:28:57.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:57.999 DEBUG:boto:Port: 8001 2024-04-12T03:28:57.999 DEBUG:boto:Params: {} 2024-04-12T03:28:57.999 DEBUG:boto:Token: None 2024-04-12T03:28:57.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:57 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q2jq9S4gt8RtxMoYXRccs9ajnAw= 2024-04-12T03:28:58.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q2jq9S4gt8RtxMoYXRccs9ajnAw=', 'Content-Length': '0'} 2024-04-12T03:28:58.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000571cb0ebdcd5482b-006618aa79-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.002 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.002 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.002 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.002 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.002 DEBUG:boto:Method: GET 2024-04-12T03:28:58.002 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.002 DEBUG:boto:Data: 2024-04-12T03:28:58.002 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.002 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.003 DEBUG:boto:Params: {} 2024-04-12T03:28:58.003 DEBUG:boto:Token: None 2024-04-12T03:28:58.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y= 2024-04-12T03:28:58.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y=', 'Content-Length': '0'} 2024-04-12T03:28:58.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dd7ab481cfe0d21-006618aa7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.005 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.005 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.005 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.005 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.005 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.005 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.005 DEBUG:boto:Data: 2024-04-12T03:28:58.005 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.006 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.006 DEBUG:boto:Params: {} 2024-04-12T03:28:58.006 DEBUG:boto:Token: None 2024-04-12T03:28:58.006 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pv3Grebi4U5gPY+sVIUA5j2gbzw= 2024-04-12T03:28:58.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pv3Grebi4U5gPY+sVIUA5j2gbzw=', 'Content-Length': '0'} 2024-04-12T03:28:58.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bd4ef30e1f59223-006618aa7a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.008 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.008 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.008 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.008 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.008 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.008 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.008 DEBUG:boto:Data: 2024-04-12T03:28:58.009 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.009 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.009 DEBUG:boto:Params: {} 2024-04-12T03:28:58.009 DEBUG:boto:Token: None 2024-04-12T03:28:58.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pv3Grebi4U5gPY+sVIUA5j2gbzw= 2024-04-12T03:28:58.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pv3Grebi4U5gPY+sVIUA5j2gbzw=', 'Content-Length': '0'} 2024-04-12T03:28:58.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f77e8a454b595897-006618aa7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:58.011 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.011 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.011 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.011 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.011 DEBUG:boto:Method: GET 2024-04-12T03:28:58.012 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.012 DEBUG:boto:Data: 2024-04-12T03:28:58.012 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.012 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.012 DEBUG:boto:Params: {} 2024-04-12T03:28:58.012 DEBUG:boto:Token: None 2024-04-12T03:28:58.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y= 2024-04-12T03:28:58.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y=', 'Content-Length': '0'} 2024-04-12T03:28:58.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000813fef7914087418-006618aa7a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.014 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.014 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.014 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.015 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.015 DEBUG:boto:Method: GET 2024-04-12T03:28:58.015 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.015 DEBUG:boto:Data: 2024-04-12T03:28:58.015 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.015 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.015 DEBUG:boto:Params: {} 2024-04-12T03:28:58.015 DEBUG:boto:Token: None 2024-04-12T03:28:58.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:28:58.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y= 2024-04-12T03:28:58.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fdIFNAVEMo8RWRXQrL/pTmHl76Y=', 'Content-Length': '0'} 2024-04-12T03:28:58.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6496ebdc4e90baf-006618aa7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.018 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.018 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.018 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.018 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.018 DEBUG:boto:Method: GET 2024-04-12T03:28:58.018 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.018 DEBUG:boto:Data: 2024-04-12T03:28:58.018 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.019 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.019 DEBUG:boto:Params: {} 2024-04-12T03:28:58.019 DEBUG:boto:Token: None 2024-04-12T03:28:58.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms= 2024-04-12T03:28:58.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms=', 'Content-Length': '0'} 2024-04-12T03:28:58.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c52940251c731c-006618aa7a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.021 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:58.022 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.022 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.022 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.022 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.022 DEBUG:boto:Method: GET 2024-04-12T03:28:58.022 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.022 DEBUG:boto:Data: 2024-04-12T03:28:58.022 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.022 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.023 DEBUG:boto:Params: {} 2024-04-12T03:28:58.023 DEBUG:boto:Token: None 2024-04-12T03:28:58.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms= 2024-04-12T03:28:58.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms=', 'Content-Length': '0'} 2024-04-12T03:28:58.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554720d00c5b4a31-006618aa7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.025 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:58.025 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.026 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.026 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.026 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:28:58.026 DEBUG:boto:Data: 2024-04-12T03:28:58.026 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.026 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.026 DEBUG:boto:Params: {} 2024-04-12T03:28:58.026 DEBUG:boto:Token: None 2024-04-12T03:28:58.027 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4 2024-04-12T03:28:58.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OpHZm8NpfokGeV16Vbu3gUW/obo= 2024-04-12T03:28:58.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OpHZm8NpfokGeV16Vbu3gUW/obo=', 'Content-Length': '0'} 2024-04-12T03:28:58.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eba8108d1f10b39d-006618aa7a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.028 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.028 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.029 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.029 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:28:58.029 DEBUG:boto:Data: 2024-04-12T03:28:58.029 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.029 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.029 DEBUG:boto:Params: {} 2024-04-12T03:28:58.029 DEBUG:boto:Token: None 2024-04-12T03:28:58.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4 2024-04-12T03:28:58.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OpHZm8NpfokGeV16Vbu3gUW/obo= 2024-04-12T03:28:58.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OpHZm8NpfokGeV16Vbu3gUW/obo=', 'Content-Length': '0'} 2024-04-12T03:28:58.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092fda54d48ead338-006618aa7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:28:58.031 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.031 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.031 DEBUG:boto:Method: GET 2024-04-12T03:28:58.032 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:28:58.032 DEBUG:boto:Data: 2024-04-12T03:28:58.032 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.032 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.032 DEBUG:boto:Params: {} 2024-04-12T03:28:58.032 DEBUG:boto:Token: None 2024-04-12T03:28:58.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4 2024-04-12T03:28:58.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9G7PjkfOgjsFV1nisBjbUYBlhpM= 2024-04-12T03:28:58.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9G7PjkfOgjsFV1nisBjbUYBlhpM=', 'Content-Length': '0'} 2024-04-12T03:28:58.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e60fa543021acc7c-006618aa7a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.034 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:28:58.034 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:28:58.034 DEBUG:boto:Method: GET 2024-04-12T03:28:58.035 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:28:58.035 DEBUG:boto:Data: 2024-04-12T03:28:58.035 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.035 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.035 DEBUG:boto:Params: {} 2024-04-12T03:28:58.035 DEBUG:boto:Token: None 2024-04-12T03:28:58.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj4 2024-04-12T03:28:58.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9G7PjkfOgjsFV1nisBjbUYBlhpM= 2024-04-12T03:28:58.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9G7PjkfOgjsFV1nisBjbUYBlhpM=', 'Content-Length': '0'} 2024-04-12T03:28:58.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000179baeb37d66b440-006618aa7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.082 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.082 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.082 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.083 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.083 DEBUG:boto:Method: GET 2024-04-12T03:28:58.083 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.083 DEBUG:boto:Data: 2024-04-12T03:28:58.083 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.083 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.083 DEBUG:boto:Params: {} 2024-04-12T03:28:58.083 DEBUG:boto:Token: None 2024-04-12T03:28:58.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms= 2024-04-12T03:28:58.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms=', 'Content-Length': '0'} 2024-04-12T03:28:58.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000263a7f247bf9188f-006618aa7a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.086 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:58.086 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.086 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.087 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.087 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.087 DEBUG:boto:Method: GET 2024-04-12T03:28:58.087 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:28:58.087 DEBUG:boto:Data: 2024-04-12T03:28:58.087 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.087 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.087 DEBUG:boto:Params: {} 2024-04-12T03:28:58.087 DEBUG:boto:Token: None 2024-04-12T03:28:58.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms= 2024-04-12T03:28:58.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flKIJ0d9MwmKzgLvpuSRjTcvIms=', 'Content-Length': '0'} 2024-04-12T03:28:58.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d899fa9467bfdb17-006618aa7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.090 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:58.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-49 zones={a2, a1} 2024-04-12T03:28:58.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.091 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:28:58.140 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:58.140 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:58.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dgbpxg-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dgbpxg-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.207 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dgbpxg-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:28:58.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:58.256 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:58.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.321 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:28:58.373 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:58.373 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:58.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dgbpxg-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dgbpxg-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:28:58.439 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dgbpxg-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:28:58.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:58.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:58.557 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:58.557 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:58.557 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.557 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:58.557 DEBUG:boto:Data: 2024-04-12T03:28:58.557 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.558 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.558 DEBUG:boto:Params: {} 2024-04-12T03:28:58.558 DEBUG:boto:Token: None 2024-04-12T03:28:58.558 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/ 2024-04-12T03:28:58.558 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk= 2024-04-12T03:28:58.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk=', 'Content-Length': '0'} 2024-04-12T03:28:58.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f14b730b5bc7fb3-006618aa7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.561 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:28:58.561 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:28:58.561 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.561 DEBUG:boto:Path: /dgbpxg-48/obj5 2024-04-12T03:28:58.561 DEBUG:boto:Data: 2024-04-12T03:28:58.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.561 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.562 DEBUG:boto:Params: {} 2024-04-12T03:28:58.562 DEBUG:boto:Token: None 2024-04-12T03:28:58.562 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/obj5 2024-04-12T03:28:58.562 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0gOI7VMpA0weXPCJHHRhsb+I160= 2024-04-12T03:28:58.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0gOI7VMpA0weXPCJHHRhsb+I160='} 2024-04-12T03:28:58.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089d61331e3b67b61-006618aa7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.567 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:58.567 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:58.567 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.567 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:58.568 DEBUG:boto:Data: 2024-04-12T03:28:58.568 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.568 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.568 DEBUG:boto:Params: {} 2024-04-12T03:28:58.568 DEBUG:boto:Token: None 2024-04-12T03:28:58.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/ 2024-04-12T03:28:58.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk= 2024-04-12T03:28:58.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk=', 'Content-Length': '0'} 2024-04-12T03:28:58.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f79f976f3ea3a4dd-006618aa7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.570 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:28:58.571 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:28:58.571 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.571 DEBUG:boto:Path: /dgbpxg-48/obj6 2024-04-12T03:28:58.571 DEBUG:boto:Data: 2024-04-12T03:28:58.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.571 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.571 DEBUG:boto:Params: {} 2024-04-12T03:28:58.571 DEBUG:boto:Token: None 2024-04-12T03:28:58.572 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/obj6 2024-04-12T03:28:58.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:16EChbd6ONFLdKZEmczXkEZQ6wU= 2024-04-12T03:28:58.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:16EChbd6ONFLdKZEmczXkEZQ6wU='} 2024-04-12T03:28:58.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d8bed7a44f9cbdf-006618aa7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.577 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:58.577 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:58.577 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.577 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:58.577 DEBUG:boto:Data: 2024-04-12T03:28:58.577 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.577 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.577 DEBUG:boto:Params: {} 2024-04-12T03:28:58.577 DEBUG:boto:Token: None 2024-04-12T03:28:58.578 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/ 2024-04-12T03:28:58.578 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk= 2024-04-12T03:28:58.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk=', 'Content-Length': '0'} 2024-04-12T03:28:58.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9e29c36b1751f5a-006618aa7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.580 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:28:58.580 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:28:58.580 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.580 DEBUG:boto:Path: /dgbpxg-48/obj7 2024-04-12T03:28:58.580 DEBUG:boto:Data: 2024-04-12T03:28:58.580 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.581 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.581 DEBUG:boto:Params: {} 2024-04-12T03:28:58.581 DEBUG:boto:Token: None 2024-04-12T03:28:58.581 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/obj7 2024-04-12T03:28:58.581 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OYl/4Sl109jD6FQ7KjlMArwr7ek= 2024-04-12T03:28:58.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OYl/4Sl109jD6FQ7KjlMArwr7ek='} 2024-04-12T03:28:58.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c00bb6c081eaaa9f-006618aa7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.627 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:28:58.627 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:28:58.627 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.627 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:28:58.627 DEBUG:boto:Data: 2024-04-12T03:28:58.627 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.627 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.627 DEBUG:boto:Params: {} 2024-04-12T03:28:58.628 DEBUG:boto:Token: None 2024-04-12T03:28:58.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/ 2024-04-12T03:28:58.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk= 2024-04-12T03:28:58.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxjAvULwcg27sqAztvVs+09hYlk=', 'Content-Length': '0'} 2024-04-12T03:28:58.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bedbb123972e925-006618aa7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.631 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:28:58.631 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:28:58.631 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.631 DEBUG:boto:Path: /dgbpxg-48/obj8 2024-04-12T03:28:58.631 DEBUG:boto:Data: 2024-04-12T03:28:58.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:28:58.631 DEBUG:boto:Port: 8000 2024-04-12T03:28:58.631 DEBUG:boto:Params: {} 2024-04-12T03:28:58.631 DEBUG:boto:Token: None 2024-04-12T03:28:58.632 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-48/obj8 2024-04-12T03:28:58.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pX98p5d9buKIhb6YfLKOKAhdTn0= 2024-04-12T03:28:58.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pX98p5d9buKIhb6YfLKOKAhdTn0='} 2024-04-12T03:28:58.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003516378d36da3127-006618aa7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.637 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.637 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.637 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.637 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:58.637 DEBUG:boto:Data: 2024-04-12T03:28:58.637 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.637 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.637 DEBUG:boto:Params: {} 2024-04-12T03:28:58.638 DEBUG:boto:Token: None 2024-04-12T03:28:58.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.638 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM= 2024-04-12T03:28:58.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM=', 'Content-Length': '0'} 2024-04-12T03:28:58.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000658373b79c3809e9-006618aa7a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.640 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:28:58.640 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:28:58.641 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.641 DEBUG:boto:Path: /dgbpxg-49/obj5 2024-04-12T03:28:58.641 DEBUG:boto:Data: 2024-04-12T03:28:58.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.641 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.641 DEBUG:boto:Params: {} 2024-04-12T03:28:58.641 DEBUG:boto:Token: None 2024-04-12T03:28:58.641 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj5 2024-04-12T03:28:58.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k94ToOzwMvD1Du6SkCtX1ThkwOY= 2024-04-12T03:28:58.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k94ToOzwMvD1Du6SkCtX1ThkwOY='} 2024-04-12T03:28:58.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007888188f7bea4bdb-006618aa7a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.647 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.647 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.647 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.647 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:58.647 DEBUG:boto:Data: 2024-04-12T03:28:58.647 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.648 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.648 DEBUG:boto:Params: {} 2024-04-12T03:28:58.648 DEBUG:boto:Token: None 2024-04-12T03:28:58.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM= 2024-04-12T03:28:58.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM=', 'Content-Length': '0'} 2024-04-12T03:28:58.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e53f30b99a47ac3-006618aa7a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.651 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:28:58.651 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:28:58.651 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.651 DEBUG:boto:Path: /dgbpxg-49/obj6 2024-04-12T03:28:58.651 DEBUG:boto:Data: 2024-04-12T03:28:58.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.651 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.652 DEBUG:boto:Params: {} 2024-04-12T03:28:58.652 DEBUG:boto:Token: None 2024-04-12T03:28:58.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj6 2024-04-12T03:28:58.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dv5Plc9sSHwO+b6VTtvLoXrN8ug= 2024-04-12T03:28:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dv5Plc9sSHwO+b6VTtvLoXrN8ug='} 2024-04-12T03:28:58.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027eec4833994831f-006618aa7a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.695 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.695 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.695 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.695 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:58.695 DEBUG:boto:Data: 2024-04-12T03:28:58.695 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.696 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.696 DEBUG:boto:Params: {} 2024-04-12T03:28:58.696 DEBUG:boto:Token: None 2024-04-12T03:28:58.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM= 2024-04-12T03:28:58.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM=', 'Content-Length': '0'} 2024-04-12T03:28:58.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000249c5e66998265fa-006618aa7a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.698 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:28:58.698 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:28:58.699 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.699 DEBUG:boto:Path: /dgbpxg-49/obj7 2024-04-12T03:28:58.699 DEBUG:boto:Data: 2024-04-12T03:28:58.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.699 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.699 DEBUG:boto:Params: {} 2024-04-12T03:28:58.699 DEBUG:boto:Token: None 2024-04-12T03:28:58.699 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj7 2024-04-12T03:28:58.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PGLwuAHMAiNYU5F9RQStwDGERGc= 2024-04-12T03:28:58.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PGLwuAHMAiNYU5F9RQStwDGERGc='} 2024-04-12T03:28:58.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faae4740a4daaf0f-006618aa7a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.704 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:28:58.704 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:28:58.705 DEBUG:boto:Method: HEAD 2024-04-12T03:28:58.705 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:28:58.705 DEBUG:boto:Data: 2024-04-12T03:28:58.705 DEBUG:boto:Headers: {} 2024-04-12T03:28:58.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.705 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.705 DEBUG:boto:Params: {} 2024-04-12T03:28:58.705 DEBUG:boto:Token: None 2024-04-12T03:28:58.705 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/ 2024-04-12T03:28:58.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM= 2024-04-12T03:28:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FelAuMY+4jyko/GO3rsWyFLnOQM=', 'Content-Length': '0'} 2024-04-12T03:28:58.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c5cabc5deb1c10a-006618aa7a-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.708 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:28:58.708 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:28:58.708 DEBUG:boto:Method: PUT 2024-04-12T03:28:58.708 DEBUG:boto:Path: /dgbpxg-49/obj8 2024-04-12T03:28:58.708 DEBUG:boto:Data: 2024-04-12T03:28:58.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:28:58.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:28:58.708 DEBUG:boto:Port: 8001 2024-04-12T03:28:58.708 DEBUG:boto:Params: {} 2024-04-12T03:28:58.709 DEBUG:boto:Token: None 2024-04-12T03:28:58.709 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:28:58 GMT /dgbpxg-49/obj8 2024-04-12T03:28:58.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HDsx3QX0j8NNecIP37hF3LD0Ry8= 2024-04-12T03:28:58.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HDsx3QX0j8NNecIP37hF3LD0Ry8='} 2024-04-12T03:28:58.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be7e6f084360428b-006618aa7a-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:58.751 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-48 2024-04-12T03:28:58.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:58.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:58.751 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:58.806 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:58.806 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:58.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:28:58.840+0000 7f627dfbc780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:58.842 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.39.6, , 00000000001.48.6, , 00000000001.27.6, , , 00000000001.25.6, , , ] 2024-04-12T03:28:58.842 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.43.6, 00000000002.55.1, 00000000004.52.6, 00000000002.46.1, 00000000004.31.6, 00000000002.11.1, 00000000002.46.1, 00000000004.29.6, 00000000002.16.1, 00000000002.29.1, 00000000002.32.1] 2024-04-12T03:29:03.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:03.841+0000 7f627dfbc780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:03.843 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.39.6, , 00000000001.48.6, , 00000000001.27.6, , , 00000000001.25.6, , , ] 2024-04-12T03:29:03.844 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.43.6, 00000000002.55.1, 00000000004.52.6, 00000000002.46.1, 00000000004.31.6, 00000000002.11.1, 00000000002.46.1, 00000000004.29.6, 00000000002.16.1, 00000000002.29.1, 00000000002.32.1] 2024-04-12T03:29:08.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:08.844+0000 7f627dfbc780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:08.845 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.39.6, , 00000000001.48.6, , 00000000001.27.6, , , 00000000001.25.6, , , ] 2024-04-12T03:29:08.845 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.43.6, 00000000002.55.1, 00000000004.52.6, 00000000002.46.1, 00000000004.31.6, 00000000002.11.1, 00000000002.46.1, 00000000004.29.6, 00000000002.16.1, 00000000002.29.1, 00000000002.32.1] 2024-04-12T03:29:13.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:13.845+0000 7f627dfbc780 1 bucket sync caught up with source: 2024-04-12T03:29:13.846 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.43.6, 00000000002.55.1, 00000000004.52.6, 00000000002.46.1, 00000000004.31.6, 00000000002.11.1, 00000000002.46.1, 00000000004.29.6, 00000000002.16.1, 00000000002.29.1, 00000000002.32.1] 2024-04-12T03:29:13.846 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.43.6, 00000000002.55.1, 00000000004.52.6, 00000000002.46.1, 00000000004.31.6, 00000000002.11.1, 00000000002.46.1, 00000000004.29.6, 00000000002.16.1, 00000000002.29.1, 00000000002.32.1] 2024-04-12T03:29:13.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:13.845+0000 7f627dfbc780 0 bucket checkpoint complete 2024-04-12T03:29:13.860 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-48 2024-04-12T03:29:13.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:13.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:13.861 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:13.928 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:13.928 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:13.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:13.962+0000 7fc3b1bd3780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:13.963 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.2171.6, , 00000000001.1318.6, , 00000000001.415.6, , , 00000000001.2461.6, , , ] 2024-04-12T03:29:13.963 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.2175.6, , 00000000002.1322.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:18.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:18.963+0000 7fc3b1bd3780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:18.964 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.2175.6, , 00000000001.1318.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:18.964 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.2175.6, , 00000000002.1322.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:23.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:23.965+0000 7fc3b1bd3780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:23.966 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.2175.6, , 00000000001.1318.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:23.966 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.2175.6, , 00000000002.1322.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:28.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:28.966+0000 7fc3b1bd3780 1 bucket sync caught up with source: 2024-04-12T03:29:28.967 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.2175.6, , 00000000002.1322.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:28.967 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.2175.6, , 00000000002.1322.6, , 00000000002.419.6, , , 00000000002.2465.6, , , ] 2024-04-12T03:29:28.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:28.966+0000 7fc3b1bd3780 0 bucket checkpoint complete 2024-04-12T03:29:28.983 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-48 zones={a2, a1} 2024-04-12T03:29:28.983 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:28.983 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:28.984 DEBUG:boto:Method: HEAD 2024-04-12T03:29:28.984 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:29:28.984 DEBUG:boto:Data: 2024-04-12T03:29:28.984 DEBUG:boto:Headers: {} 2024-04-12T03:29:28.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:28.984 DEBUG:boto:Port: 8001 2024-04-12T03:29:28.984 DEBUG:boto:Params: {} 2024-04-12T03:29:28.984 DEBUG:boto:Token: None 2024-04-12T03:29:28.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:28 GMT /dgbpxg-48/ 2024-04-12T03:29:28.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pmrh5shSZ5txHPNHRtY1JTp97/Q= 2024-04-12T03:29:28.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:28 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pmrh5shSZ5txHPNHRtY1JTp97/Q=', 'Content-Length': '0'} 2024-04-12T03:29:28.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf92d6f2e4a4fc5-006618aa98-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:28.988 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:28.988 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:28.988 DEBUG:boto:Method: HEAD 2024-04-12T03:29:28.988 DEBUG:boto:Path: /dgbpxg-48/ 2024-04-12T03:29:28.988 DEBUG:boto:Data: 2024-04-12T03:29:28.988 DEBUG:boto:Headers: {} 2024-04-12T03:29:28.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:28.988 DEBUG:boto:Port: 8000 2024-04-12T03:29:28.988 DEBUG:boto:Params: {} 2024-04-12T03:29:28.989 DEBUG:boto:Token: None 2024-04-12T03:29:28.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:28 GMT /dgbpxg-48/ 2024-04-12T03:29:28.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pmrh5shSZ5txHPNHRtY1JTp97/Q= 2024-04-12T03:29:28.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:28 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pmrh5shSZ5txHPNHRtY1JTp97/Q=', 'Content-Length': '0'} 2024-04-12T03:29:28.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1d7a7e74344c307-006618aa98-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:28.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:29:28.992 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:28.992 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:28.992 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.992 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.992 DEBUG:boto:Method: GET 2024-04-12T03:29:28.992 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.992 DEBUG:boto:Data: 2024-04-12T03:29:28.992 DEBUG:boto:Headers: {} 2024-04-12T03:29:28.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:28.993 DEBUG:boto:Port: 8001 2024-04-12T03:29:28.993 DEBUG:boto:Params: {} 2024-04-12T03:29:28.993 DEBUG:boto:Token: None 2024-04-12T03:29:28.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:28 GMT /dgbpxg-48/?versions 2024-04-12T03:29:28.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jnlr03+TQLfGudCkcevEO4lkDlQ= 2024-04-12T03:29:28.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:28 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jnlr03+TQLfGudCkcevEO4lkDlQ=', 'Content-Length': '0'} 2024-04-12T03:29:28.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec7160185d23375-006618aa98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:28.995 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:28.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:28.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:28.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:28.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:28.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:29:28.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:29:28.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:29:28.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:29:28.997 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:29:28.997 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:28.997 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:28.997 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.997 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.997 DEBUG:boto:Method: GET 2024-04-12T03:29:28.997 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:28.998 DEBUG:boto:Data: 2024-04-12T03:29:28.998 DEBUG:boto:Headers: {} 2024-04-12T03:29:28.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:28.998 DEBUG:boto:Port: 8000 2024-04-12T03:29:28.998 DEBUG:boto:Params: {} 2024-04-12T03:29:28.998 DEBUG:boto:Token: None 2024-04-12T03:29:28.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:28 GMT /dgbpxg-48/?versions 2024-04-12T03:29:28.998 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jnlr03+TQLfGudCkcevEO4lkDlQ= 2024-04-12T03:29:28.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:28 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jnlr03+TQLfGudCkcevEO4lkDlQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002cb7847ed5d9fae-006618aa98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.001 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:29:29.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:29:29.002 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.003 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.003 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.003 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.003 DEBUG:boto:Method: GET 2024-04-12T03:29:29.003 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.003 DEBUG:boto:Data: 2024-04-12T03:29:29.003 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.003 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.003 DEBUG:boto:Params: {} 2024-04-12T03:29:29.003 DEBUG:boto:Token: None 2024-04-12T03:29:29.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/?versions 2024-04-12T03:29:29.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:75PykpROmUmIQUbdiqJgCI2lsW8= 2024-04-12T03:29:29.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:75PykpROmUmIQUbdiqJgCI2lsW8=', 'Content-Length': '0'} 2024-04-12T03:29:29.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b74e67821be2cd-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.006 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.007 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.007 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.007 DEBUG:boto:path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.007 DEBUG:boto:auth_path=/dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.007 DEBUG:boto:Method: GET 2024-04-12T03:29:29.007 DEBUG:boto:Path: /dgbpxg-48/?versions&max-keys=999 2024-04-12T03:29:29.007 DEBUG:boto:Data: 2024-04-12T03:29:29.007 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.008 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.008 DEBUG:boto:Params: {} 2024-04-12T03:29:29.008 DEBUG:boto:Token: None 2024-04-12T03:29:29.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/?versions 2024-04-12T03:29:29.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:75PykpROmUmIQUbdiqJgCI2lsW8= 2024-04-12T03:29:29.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:75PykpROmUmIQUbdiqJgCI2lsW8=', 'Content-Length': '0'} 2024-04-12T03:29:29.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d35c5bbadba69e58-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.010 DEBUG:boto:b'dgbpxg-48999falseobj1nulltrue2024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:29.011 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.011 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.011 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.011 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.011 DEBUG:boto:Method: GET 2024-04-12T03:29:29.011 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.011 DEBUG:boto:Data: 2024-04-12T03:29:29.012 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.012 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.012 DEBUG:boto:Params: {} 2024-04-12T03:29:29.012 DEBUG:boto:Token: None 2024-04-12T03:29:29.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI= 2024-04-12T03:29:29.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI=', 'Content-Length': '0'} 2024-04-12T03:29:29.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9a6cea5e5fa6fc9-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.014 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.014 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.014 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.015 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.015 DEBUG:boto:Method: GET 2024-04-12T03:29:29.015 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.015 DEBUG:boto:Data: 2024-04-12T03:29:29.015 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.015 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.015 DEBUG:boto:Params: {} 2024-04-12T03:29:29.015 DEBUG:boto:Token: None 2024-04-12T03:29:29.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI= 2024-04-12T03:29:29.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI=', 'Content-Length': '0'} 2024-04-12T03:29:29.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d43d20c8a1757da5-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.058 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.058 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.058 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.059 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.059 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.059 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.059 DEBUG:boto:Data: 2024-04-12T03:29:29.059 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.059 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.059 DEBUG:boto:Params: {} 2024-04-12T03:29:29.059 DEBUG:boto:Token: None 2024-04-12T03:29:29.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwFXvCfh9i2kMZxq3G3hlbP+g4w= 2024-04-12T03:29:29.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwFXvCfh9i2kMZxq3G3hlbP+g4w=', 'Content-Length': '0'} 2024-04-12T03:29:29.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4533e703906642d-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.062 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.062 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.062 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.062 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.062 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.062 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.062 DEBUG:boto:Data: 2024-04-12T03:29:29.062 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.062 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.063 DEBUG:boto:Params: {} 2024-04-12T03:29:29.063 DEBUG:boto:Token: None 2024-04-12T03:29:29.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwFXvCfh9i2kMZxq3G3hlbP+g4w= 2024-04-12T03:29:29.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwFXvCfh9i2kMZxq3G3hlbP+g4w=', 'Content-Length': '0'} 2024-04-12T03:29:29.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b508bd75cf2dcb-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:29.065 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.065 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.065 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.065 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.065 DEBUG:boto:Method: GET 2024-04-12T03:29:29.065 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.066 DEBUG:boto:Data: 2024-04-12T03:29:29.066 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.066 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.066 DEBUG:boto:Params: {} 2024-04-12T03:29:29.066 DEBUG:boto:Token: None 2024-04-12T03:29:29.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI= 2024-04-12T03:29:29.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI=', 'Content-Length': '0'} 2024-04-12T03:29:29.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014fc81f3477d8210-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.068 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.068 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.068 DEBUG:boto:path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.068 DEBUG:boto:auth_path=/dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.068 DEBUG:boto:Method: GET 2024-04-12T03:29:29.069 DEBUG:boto:Path: /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.069 DEBUG:boto:Data: 2024-04-12T03:29:29.069 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.069 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.069 DEBUG:boto:Params: {} 2024-04-12T03:29:29.069 DEBUG:boto:Token: None 2024-04-12T03:29:29.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1?versionId=null 2024-04-12T03:29:29.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI= 2024-04-12T03:29:29.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1cBUV252elk9dtuC05Ev2gdSRmI=', 'Content-Length': '0'} 2024-04-12T03:29:29.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000727b4c75afc5f822-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.071 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.071 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.072 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.072 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.072 DEBUG:boto:Method: GET 2024-04-12T03:29:29.072 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.072 DEBUG:boto:Data: 2024-04-12T03:29:29.072 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.072 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.072 DEBUG:boto:Params: {} 2024-04-12T03:29:29.072 DEBUG:boto:Token: None 2024-04-12T03:29:29.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16ecef01deff846-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.075 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.076 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.076 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.076 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.076 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.076 DEBUG:boto:Method: GET 2024-04-12T03:29:29.076 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.076 DEBUG:boto:Data: 2024-04-12T03:29:29.077 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.077 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.077 DEBUG:boto:Params: {} 2024-04-12T03:29:29.077 DEBUG:boto:Token: None 2024-04-12T03:29:29.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.077 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009127c7e0d0ca4a79-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.080 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.080 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.080 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.080 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.080 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:29:29.080 DEBUG:boto:Data: 2024-04-12T03:29:29.080 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.081 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.081 DEBUG:boto:Params: {} 2024-04-12T03:29:29.081 DEBUG:boto:Token: None 2024-04-12T03:29:29.081 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1 2024-04-12T03:29:29.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2hJZU5xJ2E8gIUzpz31rB6gX6iw= 2024-04-12T03:29:29.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2hJZU5xJ2E8gIUzpz31rB6gX6iw=', 'Content-Length': '0'} 2024-04-12T03:29:29.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b90f7a3a6a1e5ed-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.083 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.083 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.083 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.083 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:29:29.083 DEBUG:boto:Data: 2024-04-12T03:29:29.083 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.083 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.084 DEBUG:boto:Params: {} 2024-04-12T03:29:29.084 DEBUG:boto:Token: None 2024-04-12T03:29:29.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1 2024-04-12T03:29:29.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2hJZU5xJ2E8gIUzpz31rB6gX6iw= 2024-04-12T03:29:29.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2hJZU5xJ2E8gIUzpz31rB6gX6iw=', 'Content-Length': '0'} 2024-04-12T03:29:29.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fda573d8fc264a8-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:29.086 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.086 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.086 DEBUG:boto:Method: GET 2024-04-12T03:29:29.086 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:29:29.087 DEBUG:boto:Data: 2024-04-12T03:29:29.087 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.087 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.087 DEBUG:boto:Params: {} 2024-04-12T03:29:29.087 DEBUG:boto:Token: None 2024-04-12T03:29:29.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1 2024-04-12T03:29:29.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zx3IC+BmeLg0vp9mc6XYqu0107c= 2024-04-12T03:29:29.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zx3IC+BmeLg0vp9mc6XYqu0107c=', 'Content-Length': '0'} 2024-04-12T03:29:29.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005403514dfb91c75-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.130 DEBUG:boto:path=/dgbpxg-48/obj1 2024-04-12T03:29:29.130 DEBUG:boto:auth_path=/dgbpxg-48/obj1 2024-04-12T03:29:29.131 DEBUG:boto:Method: GET 2024-04-12T03:29:29.131 DEBUG:boto:Path: /dgbpxg-48/obj1 2024-04-12T03:29:29.131 DEBUG:boto:Data: 2024-04-12T03:29:29.131 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.131 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.131 DEBUG:boto:Params: {} 2024-04-12T03:29:29.131 DEBUG:boto:Token: None 2024-04-12T03:29:29.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj1 2024-04-12T03:29:29.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zx3IC+BmeLg0vp9mc6XYqu0107c= 2024-04-12T03:29:29.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zx3IC+BmeLg0vp9mc6XYqu0107c=', 'Content-Length': '0'} 2024-04-12T03:29:29.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006151176fa07080cb-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.134 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.134 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.134 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.134 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.134 DEBUG:boto:Method: GET 2024-04-12T03:29:29.134 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.134 DEBUG:boto:Data: 2024-04-12T03:29:29.134 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.135 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.135 DEBUG:boto:Params: {} 2024-04-12T03:29:29.135 DEBUG:boto:Token: None 2024-04-12T03:29:29.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e26e5061e72f525-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.137 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.138 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.138 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.138 DEBUG:boto:path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.138 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.138 DEBUG:boto:Method: GET 2024-04-12T03:29:29.138 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj1 2024-04-12T03:29:29.139 DEBUG:boto:Data: 2024-04-12T03:29:29.139 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.139 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.139 DEBUG:boto:Params: {} 2024-04-12T03:29:29.139 DEBUG:boto:Token: None 2024-04-12T03:29:29.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5cbff8930c5928-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.141 DEBUG:boto:b'dgbpxg-48obj11000falseobj12024-04-12T03:28:26.373Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:29.142 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.142 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.142 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.142 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.142 DEBUG:boto:Method: GET 2024-04-12T03:29:29.142 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.143 DEBUG:boto:Data: 2024-04-12T03:29:29.143 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.143 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.143 DEBUG:boto:Params: {} 2024-04-12T03:29:29.143 DEBUG:boto:Token: None 2024-04-12T03:29:29.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U= 2024-04-12T03:29:29.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U=', 'Content-Length': '0'} 2024-04-12T03:29:29.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f3ce467307d6b4c-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.145 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.146 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.146 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.146 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.146 DEBUG:boto:Method: GET 2024-04-12T03:29:29.146 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.146 DEBUG:boto:Data: 2024-04-12T03:29:29.146 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.146 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.146 DEBUG:boto:Params: {} 2024-04-12T03:29:29.147 DEBUG:boto:Token: None 2024-04-12T03:29:29.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U= 2024-04-12T03:29:29.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U=', 'Content-Length': '0'} 2024-04-12T03:29:29.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005050126eb095386a-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.149 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.149 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.149 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.149 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.149 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.149 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.150 DEBUG:boto:Data: 2024-04-12T03:29:29.150 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.150 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.150 DEBUG:boto:Params: {} 2024-04-12T03:29:29.150 DEBUG:boto:Token: None 2024-04-12T03:29:29.150 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ka5PFbERzfm6IeJl4amShaOcCbI= 2024-04-12T03:29:29.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ka5PFbERzfm6IeJl4amShaOcCbI=', 'Content-Length': '0'} 2024-04-12T03:29:29.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076058712f1b985cb-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.152 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.153 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.153 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.153 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.153 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.153 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.153 DEBUG:boto:Data: 2024-04-12T03:29:29.153 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.153 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.153 DEBUG:boto:Params: {} 2024-04-12T03:29:29.154 DEBUG:boto:Token: None 2024-04-12T03:29:29.154 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ka5PFbERzfm6IeJl4amShaOcCbI= 2024-04-12T03:29:29.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ka5PFbERzfm6IeJl4amShaOcCbI=', 'Content-Length': '0'} 2024-04-12T03:29:29.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010c8332a4c0ddca6-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:29.156 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.156 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.156 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.156 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.157 DEBUG:boto:Method: GET 2024-04-12T03:29:29.157 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.157 DEBUG:boto:Data: 2024-04-12T03:29:29.157 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.157 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.157 DEBUG:boto:Params: {} 2024-04-12T03:29:29.157 DEBUG:boto:Token: None 2024-04-12T03:29:29.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U= 2024-04-12T03:29:29.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U=', 'Content-Length': '0'} 2024-04-12T03:29:29.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000babe53915fca694c-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.160 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.160 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.160 DEBUG:boto:path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.160 DEBUG:boto:auth_path=/dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.160 DEBUG:boto:Method: GET 2024-04-12T03:29:29.160 DEBUG:boto:Path: /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.160 DEBUG:boto:Data: 2024-04-12T03:29:29.160 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.161 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.161 DEBUG:boto:Params: {} 2024-04-12T03:29:29.161 DEBUG:boto:Token: None 2024-04-12T03:29:29.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2?versionId=null 2024-04-12T03:29:29.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U= 2024-04-12T03:29:29.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZJNo3yrvdlXs08ag0M5PiwH/M9U=', 'Content-Length': '0'} 2024-04-12T03:29:29.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a976b69abbdae29-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.163 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.163 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.163 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.163 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.164 DEBUG:boto:Method: GET 2024-04-12T03:29:29.164 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.164 DEBUG:boto:Data: 2024-04-12T03:29:29.164 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.164 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.164 DEBUG:boto:Params: {} 2024-04-12T03:29:29.164 DEBUG:boto:Token: None 2024-04-12T03:29:29.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004160b534e2af8854-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.167 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.167 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.167 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.167 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.167 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.167 DEBUG:boto:Method: GET 2024-04-12T03:29:29.168 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.168 DEBUG:boto:Data: 2024-04-12T03:29:29.168 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.168 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.168 DEBUG:boto:Params: {} 2024-04-12T03:29:29.168 DEBUG:boto:Token: None 2024-04-12T03:29:29.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0715ceba354b475-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.170 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.171 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.171 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.171 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.171 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:29:29.171 DEBUG:boto:Data: 2024-04-12T03:29:29.171 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.171 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.171 DEBUG:boto:Params: {} 2024-04-12T03:29:29.172 DEBUG:boto:Token: None 2024-04-12T03:29:29.172 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2 2024-04-12T03:29:29.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/x1FPJiM9ZE4Y15QG1C7kEnlbsw= 2024-04-12T03:29:29.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/x1FPJiM9ZE4Y15QG1C7kEnlbsw=', 'Content-Length': '0'} 2024-04-12T03:29:29.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000412936626cbf27c2-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.173 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.174 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.174 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.174 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:29:29.174 DEBUG:boto:Data: 2024-04-12T03:29:29.174 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.174 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.174 DEBUG:boto:Params: {} 2024-04-12T03:29:29.174 DEBUG:boto:Token: None 2024-04-12T03:29:29.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2 2024-04-12T03:29:29.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/x1FPJiM9ZE4Y15QG1C7kEnlbsw= 2024-04-12T03:29:29.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/x1FPJiM9ZE4Y15QG1C7kEnlbsw=', 'Content-Length': '0'} 2024-04-12T03:29:29.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002db9ba64c21200bf-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:29.176 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.177 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.177 DEBUG:boto:Method: GET 2024-04-12T03:29:29.177 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:29:29.177 DEBUG:boto:Data: 2024-04-12T03:29:29.177 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.177 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.177 DEBUG:boto:Params: {} 2024-04-12T03:29:29.177 DEBUG:boto:Token: None 2024-04-12T03:29:29.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2 2024-04-12T03:29:29.177 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HfIrqjB/gpZv3pYqu2EKMpEvG9k= 2024-04-12T03:29:29.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HfIrqjB/gpZv3pYqu2EKMpEvG9k=', 'Content-Length': '0'} 2024-04-12T03:29:29.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000521f39b93791f6f4-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.222 DEBUG:boto:path=/dgbpxg-48/obj2 2024-04-12T03:29:29.222 DEBUG:boto:auth_path=/dgbpxg-48/obj2 2024-04-12T03:29:29.223 DEBUG:boto:Method: GET 2024-04-12T03:29:29.223 DEBUG:boto:Path: /dgbpxg-48/obj2 2024-04-12T03:29:29.223 DEBUG:boto:Data: 2024-04-12T03:29:29.223 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.223 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.223 DEBUG:boto:Params: {} 2024-04-12T03:29:29.223 DEBUG:boto:Token: None 2024-04-12T03:29:29.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj2 2024-04-12T03:29:29.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HfIrqjB/gpZv3pYqu2EKMpEvG9k= 2024-04-12T03:29:29.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HfIrqjB/gpZv3pYqu2EKMpEvG9k=', 'Content-Length': '0'} 2024-04-12T03:29:29.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e05cede5e89ced1c-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.226 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.226 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.226 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.226 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.226 DEBUG:boto:Method: GET 2024-04-12T03:29:29.226 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.226 DEBUG:boto:Data: 2024-04-12T03:29:29.226 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.226 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.227 DEBUG:boto:Params: {} 2024-04-12T03:29:29.227 DEBUG:boto:Token: None 2024-04-12T03:29:29.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546fc8bd1c0a1ad6-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.230 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.230 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.230 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.230 DEBUG:boto:path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.230 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.230 DEBUG:boto:Method: GET 2024-04-12T03:29:29.230 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj2 2024-04-12T03:29:29.231 DEBUG:boto:Data: 2024-04-12T03:29:29.231 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.231 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.231 DEBUG:boto:Params: {} 2024-04-12T03:29:29.231 DEBUG:boto:Token: None 2024-04-12T03:29:29.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090aee1edd02c78bf-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.234 DEBUG:boto:b'dgbpxg-48obj21000falseobj22024-04-12T03:28:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:29.234 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.235 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.235 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.235 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.235 DEBUG:boto:Method: GET 2024-04-12T03:29:29.235 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.235 DEBUG:boto:Data: 2024-04-12T03:29:29.235 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.235 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.235 DEBUG:boto:Params: {} 2024-04-12T03:29:29.235 DEBUG:boto:Token: None 2024-04-12T03:29:29.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI= 2024-04-12T03:29:29.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI=', 'Content-Length': '0'} 2024-04-12T03:29:29.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000588eae401c6a7fee-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.238 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.238 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.238 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.238 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.238 DEBUG:boto:Method: GET 2024-04-12T03:29:29.238 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.238 DEBUG:boto:Data: 2024-04-12T03:29:29.238 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.239 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.239 DEBUG:boto:Params: {} 2024-04-12T03:29:29.239 DEBUG:boto:Token: None 2024-04-12T03:29:29.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI= 2024-04-12T03:29:29.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI=', 'Content-Length': '0'} 2024-04-12T03:29:29.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008082ed38253e898c-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.241 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.241 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.241 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.241 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.241 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.241 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.241 DEBUG:boto:Data: 2024-04-12T03:29:29.241 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.242 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.242 DEBUG:boto:Params: {} 2024-04-12T03:29:29.242 DEBUG:boto:Token: None 2024-04-12T03:29:29.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q4Oe9aN1FXoAp0WMw9xiK4NqQ8A= 2024-04-12T03:29:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q4Oe9aN1FXoAp0WMw9xiK4NqQ8A=', 'Content-Length': '0'} 2024-04-12T03:29:29.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084a6fc2fcb5387d6-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.244 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.244 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.244 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.244 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.244 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.244 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.244 DEBUG:boto:Data: 2024-04-12T03:29:29.244 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.245 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.245 DEBUG:boto:Params: {} 2024-04-12T03:29:29.245 DEBUG:boto:Token: None 2024-04-12T03:29:29.245 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q4Oe9aN1FXoAp0WMw9xiK4NqQ8A= 2024-04-12T03:29:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q4Oe9aN1FXoAp0WMw9xiK4NqQ8A=', 'Content-Length': '0'} 2024-04-12T03:29:29.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4ab5d15ea2ef776-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:29.247 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.247 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.247 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.247 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.247 DEBUG:boto:Method: GET 2024-04-12T03:29:29.247 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.247 DEBUG:boto:Data: 2024-04-12T03:29:29.247 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.247 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.247 DEBUG:boto:Params: {} 2024-04-12T03:29:29.248 DEBUG:boto:Token: None 2024-04-12T03:29:29.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI= 2024-04-12T03:29:29.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI=', 'Content-Length': '0'} 2024-04-12T03:29:29.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df1e8c5520a7c691-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.250 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.250 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.250 DEBUG:boto:path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.250 DEBUG:boto:auth_path=/dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.250 DEBUG:boto:Method: GET 2024-04-12T03:29:29.250 DEBUG:boto:Path: /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.250 DEBUG:boto:Data: 2024-04-12T03:29:29.250 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.250 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.250 DEBUG:boto:Params: {} 2024-04-12T03:29:29.251 DEBUG:boto:Token: None 2024-04-12T03:29:29.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3?versionId=null 2024-04-12T03:29:29.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI= 2024-04-12T03:29:29.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Xxu2DUf1thlyeNCA9zhnQHO7pI=', 'Content-Length': '0'} 2024-04-12T03:29:29.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f346b3a50061285e-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.294 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.294 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.295 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.295 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.295 DEBUG:boto:Method: GET 2024-04-12T03:29:29.295 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.295 DEBUG:boto:Data: 2024-04-12T03:29:29.295 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.295 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.295 DEBUG:boto:Params: {} 2024-04-12T03:29:29.295 DEBUG:boto:Token: None 2024-04-12T03:29:29.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000489bb02c2828c1ca-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.298 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.298 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.298 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.299 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.299 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.299 DEBUG:boto:Method: GET 2024-04-12T03:29:29.299 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.299 DEBUG:boto:Data: 2024-04-12T03:29:29.299 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.299 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.299 DEBUG:boto:Params: {} 2024-04-12T03:29:29.299 DEBUG:boto:Token: None 2024-04-12T03:29:29.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000933ba87596753507-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.302 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.302 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.302 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.302 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.303 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:29:29.303 DEBUG:boto:Data: 2024-04-12T03:29:29.303 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.303 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.303 DEBUG:boto:Params: {} 2024-04-12T03:29:29.303 DEBUG:boto:Token: None 2024-04-12T03:29:29.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3 2024-04-12T03:29:29.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TlWN/Ny9TjQt9c55RKPSMny1/2U= 2024-04-12T03:29:29.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TlWN/Ny9TjQt9c55RKPSMny1/2U=', 'Content-Length': '0'} 2024-04-12T03:29:29.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcd9ba14012bdb46-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.305 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.305 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.305 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.306 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:29:29.306 DEBUG:boto:Data: 2024-04-12T03:29:29.306 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.306 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.306 DEBUG:boto:Params: {} 2024-04-12T03:29:29.306 DEBUG:boto:Token: None 2024-04-12T03:29:29.306 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3 2024-04-12T03:29:29.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TlWN/Ny9TjQt9c55RKPSMny1/2U= 2024-04-12T03:29:29.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TlWN/Ny9TjQt9c55RKPSMny1/2U=', 'Content-Length': '0'} 2024-04-12T03:29:29.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158893089cb831b1-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:29.308 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.308 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.309 DEBUG:boto:Method: GET 2024-04-12T03:29:29.309 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:29:29.309 DEBUG:boto:Data: 2024-04-12T03:29:29.309 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.309 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.309 DEBUG:boto:Params: {} 2024-04-12T03:29:29.309 DEBUG:boto:Token: None 2024-04-12T03:29:29.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3 2024-04-12T03:29:29.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:crQahmqcXFJPHAgOCtpQT3gOBgM= 2024-04-12T03:29:29.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:crQahmqcXFJPHAgOCtpQT3gOBgM=', 'Content-Length': '0'} 2024-04-12T03:29:29.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0da175358fd8d4b-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.311 DEBUG:boto:path=/dgbpxg-48/obj3 2024-04-12T03:29:29.311 DEBUG:boto:auth_path=/dgbpxg-48/obj3 2024-04-12T03:29:29.312 DEBUG:boto:Method: GET 2024-04-12T03:29:29.312 DEBUG:boto:Path: /dgbpxg-48/obj3 2024-04-12T03:29:29.312 DEBUG:boto:Data: 2024-04-12T03:29:29.312 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.312 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.312 DEBUG:boto:Params: {} 2024-04-12T03:29:29.312 DEBUG:boto:Token: None 2024-04-12T03:29:29.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj3 2024-04-12T03:29:29.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:crQahmqcXFJPHAgOCtpQT3gOBgM= 2024-04-12T03:29:29.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:crQahmqcXFJPHAgOCtpQT3gOBgM=', 'Content-Length': '0'} 2024-04-12T03:29:29.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e65586fa58d4134-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.314 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.314 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.314 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.314 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.315 DEBUG:boto:Method: GET 2024-04-12T03:29:29.315 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.315 DEBUG:boto:Data: 2024-04-12T03:29:29.315 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.315 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.315 DEBUG:boto:Params: {} 2024-04-12T03:29:29.315 DEBUG:boto:Token: None 2024-04-12T03:29:29.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d90163367047e03-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.317 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.318 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.318 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.318 DEBUG:boto:path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.318 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.318 DEBUG:boto:Method: GET 2024-04-12T03:29:29.318 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj3 2024-04-12T03:29:29.318 DEBUG:boto:Data: 2024-04-12T03:29:29.319 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.319 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.319 DEBUG:boto:Params: {} 2024-04-12T03:29:29.319 DEBUG:boto:Token: None 2024-04-12T03:29:29.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663477de6482f13f-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.321 DEBUG:boto:b'dgbpxg-48obj31000falseobj32024-04-12T03:28:26.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:29.321 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.322 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.322 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.322 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.322 DEBUG:boto:Method: GET 2024-04-12T03:29:29.322 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.322 DEBUG:boto:Data: 2024-04-12T03:29:29.322 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.322 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.322 DEBUG:boto:Params: {} 2024-04-12T03:29:29.323 DEBUG:boto:Token: None 2024-04-12T03:29:29.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc= 2024-04-12T03:29:29.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc=', 'Content-Length': '0'} 2024-04-12T03:29:29.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a130f3b4a504348-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.325 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.325 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.325 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.325 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.325 DEBUG:boto:Method: GET 2024-04-12T03:29:29.325 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.326 DEBUG:boto:Data: 2024-04-12T03:29:29.326 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.326 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.326 DEBUG:boto:Params: {} 2024-04-12T03:29:29.326 DEBUG:boto:Token: None 2024-04-12T03:29:29.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc= 2024-04-12T03:29:29.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc=', 'Content-Length': '0'} 2024-04-12T03:29:29.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009991a35ceab438f2-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.328 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.328 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.328 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.328 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.329 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.329 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.329 DEBUG:boto:Data: 2024-04-12T03:29:29.329 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.329 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.329 DEBUG:boto:Params: {} 2024-04-12T03:29:29.329 DEBUG:boto:Token: None 2024-04-12T03:29:29.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/2oPdnHqNwWWcs8wurtfPl+37dY= 2024-04-12T03:29:29.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/2oPdnHqNwWWcs8wurtfPl+37dY=', 'Content-Length': '0'} 2024-04-12T03:29:29.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59a151a37591e91-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.333 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.333 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.333 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.333 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.334 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.334 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.334 DEBUG:boto:Data: 2024-04-12T03:29:29.334 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.334 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.334 DEBUG:boto:Params: {} 2024-04-12T03:29:29.334 DEBUG:boto:Token: None 2024-04-12T03:29:29.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/2oPdnHqNwWWcs8wurtfPl+37dY= 2024-04-12T03:29:29.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/2oPdnHqNwWWcs8wurtfPl+37dY=', 'Content-Length': '0'} 2024-04-12T03:29:29.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000066033eb9260ccc3-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:29.336 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.337 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.337 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.337 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.337 DEBUG:boto:Method: GET 2024-04-12T03:29:29.337 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.337 DEBUG:boto:Data: 2024-04-12T03:29:29.337 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.337 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.337 DEBUG:boto:Params: {} 2024-04-12T03:29:29.338 DEBUG:boto:Token: None 2024-04-12T03:29:29.338 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc= 2024-04-12T03:29:29.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc=', 'Content-Length': '0'} 2024-04-12T03:29:29.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030ed2bb301ebf039-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.382 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.383 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.383 DEBUG:boto:path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.383 DEBUG:boto:auth_path=/dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.383 DEBUG:boto:Method: GET 2024-04-12T03:29:29.383 DEBUG:boto:Path: /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.383 DEBUG:boto:Data: 2024-04-12T03:29:29.383 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.383 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.383 DEBUG:boto:Params: {} 2024-04-12T03:29:29.384 DEBUG:boto:Token: None 2024-04-12T03:29:29.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4?versionId=null 2024-04-12T03:29:29.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc= 2024-04-12T03:29:29.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPZm5rIuLKMoxzfariGAbfsP3Vc=', 'Content-Length': '0'} 2024-04-12T03:29:29.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f8b768d5bfafeff-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.386 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.386 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.386 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.386 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.386 DEBUG:boto:Method: GET 2024-04-12T03:29:29.386 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.386 DEBUG:boto:Data: 2024-04-12T03:29:29.386 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.387 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.387 DEBUG:boto:Params: {} 2024-04-12T03:29:29.387 DEBUG:boto:Token: None 2024-04-12T03:29:29.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da0cb6958932fa1-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.389 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.390 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.390 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.390 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.390 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.390 DEBUG:boto:Method: GET 2024-04-12T03:29:29.390 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.390 DEBUG:boto:Data: 2024-04-12T03:29:29.390 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.390 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.391 DEBUG:boto:Params: {} 2024-04-12T03:29:29.391 DEBUG:boto:Token: None 2024-04-12T03:29:29.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a61cfd07d9deb23e-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.393 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.393 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.394 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.394 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.394 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:29:29.394 DEBUG:boto:Data: 2024-04-12T03:29:29.394 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.394 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.394 DEBUG:boto:Params: {} 2024-04-12T03:29:29.394 DEBUG:boto:Token: None 2024-04-12T03:29:29.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4 2024-04-12T03:29:29.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:it99z1/5Q+xmxnYSJv1wWxwkCUc= 2024-04-12T03:29:29.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:it99z1/5Q+xmxnYSJv1wWxwkCUc=', 'Content-Length': '0'} 2024-04-12T03:29:29.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afec31673790be69-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.397 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.397 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.397 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.397 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:29:29.397 DEBUG:boto:Data: 2024-04-12T03:29:29.397 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.397 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.397 DEBUG:boto:Params: {} 2024-04-12T03:29:29.397 DEBUG:boto:Token: None 2024-04-12T03:29:29.398 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4 2024-04-12T03:29:29.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:it99z1/5Q+xmxnYSJv1wWxwkCUc= 2024-04-12T03:29:29.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:it99z1/5Q+xmxnYSJv1wWxwkCUc=', 'Content-Length': '0'} 2024-04-12T03:29:29.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075563ee90d212af3-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:29.400 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.400 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.400 DEBUG:boto:Method: GET 2024-04-12T03:29:29.400 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:29:29.400 DEBUG:boto:Data: 2024-04-12T03:29:29.400 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.400 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.400 DEBUG:boto:Params: {} 2024-04-12T03:29:29.400 DEBUG:boto:Token: None 2024-04-12T03:29:29.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4 2024-04-12T03:29:29.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uyjr+K2kqxFaItkXYx/+6Gl5Sok= 2024-04-12T03:29:29.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uyjr+K2kqxFaItkXYx/+6Gl5Sok=', 'Content-Length': '0'} 2024-04-12T03:29:29.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f260795f6e15b8d-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.402 DEBUG:boto:path=/dgbpxg-48/obj4 2024-04-12T03:29:29.403 DEBUG:boto:auth_path=/dgbpxg-48/obj4 2024-04-12T03:29:29.403 DEBUG:boto:Method: GET 2024-04-12T03:29:29.403 DEBUG:boto:Path: /dgbpxg-48/obj4 2024-04-12T03:29:29.403 DEBUG:boto:Data: 2024-04-12T03:29:29.403 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.403 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.403 DEBUG:boto:Params: {} 2024-04-12T03:29:29.403 DEBUG:boto:Token: None 2024-04-12T03:29:29.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj4 2024-04-12T03:29:29.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uyjr+K2kqxFaItkXYx/+6Gl5Sok= 2024-04-12T03:29:29.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uyjr+K2kqxFaItkXYx/+6Gl5Sok=', 'Content-Length': '0'} 2024-04-12T03:29:29.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8dcbbf063fd2fd4-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.405 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.405 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.406 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.406 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.406 DEBUG:boto:Method: GET 2024-04-12T03:29:29.406 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.406 DEBUG:boto:Data: 2024-04-12T03:29:29.406 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.406 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.406 DEBUG:boto:Params: {} 2024-04-12T03:29:29.406 DEBUG:boto:Token: None 2024-04-12T03:29:29.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d786443e01d9099-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.409 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.409 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.409 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.409 DEBUG:boto:path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.409 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.410 DEBUG:boto:Method: GET 2024-04-12T03:29:29.410 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj4 2024-04-12T03:29:29.410 DEBUG:boto:Data: 2024-04-12T03:29:29.410 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.410 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.410 DEBUG:boto:Params: {} 2024-04-12T03:29:29.410 DEBUG:boto:Token: None 2024-04-12T03:29:29.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bd5eac72331c6d-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.413 DEBUG:boto:b'dgbpxg-48obj41000falseobj42024-04-12T03:28:26.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:29.413 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.413 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.413 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.413 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.413 DEBUG:boto:Method: GET 2024-04-12T03:29:29.414 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.414 DEBUG:boto:Data: 2024-04-12T03:29:29.414 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.414 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.414 DEBUG:boto:Params: {} 2024-04-12T03:29:29.414 DEBUG:boto:Token: None 2024-04-12T03:29:29.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs= 2024-04-12T03:29:29.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs=', 'Content-Length': '0'} 2024-04-12T03:29:29.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000856a480df39a5270-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.416 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.417 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.417 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.417 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.417 DEBUG:boto:Method: GET 2024-04-12T03:29:29.417 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.417 DEBUG:boto:Data: 2024-04-12T03:29:29.417 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.417 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.417 DEBUG:boto:Params: {} 2024-04-12T03:29:29.417 DEBUG:boto:Token: None 2024-04-12T03:29:29.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs= 2024-04-12T03:29:29.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs=', 'Content-Length': '0'} 2024-04-12T03:29:29.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20afd8cb6ef87a3-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.420 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.420 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.420 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.420 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.420 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.420 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.420 DEBUG:boto:Data: 2024-04-12T03:29:29.420 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.421 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.421 DEBUG:boto:Params: {} 2024-04-12T03:29:29.421 DEBUG:boto:Token: None 2024-04-12T03:29:29.421 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:96bVwZOIelLgPdALhcYNpTwo2Lo= 2024-04-12T03:29:29.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:96bVwZOIelLgPdALhcYNpTwo2Lo=', 'Content-Length': '0'} 2024-04-12T03:29:29.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e89cb1936c2d0c9-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.423 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.423 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.423 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.423 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.423 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.423 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.423 DEBUG:boto:Data: 2024-04-12T03:29:29.423 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.424 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.424 DEBUG:boto:Params: {} 2024-04-12T03:29:29.424 DEBUG:boto:Token: None 2024-04-12T03:29:29.424 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:96bVwZOIelLgPdALhcYNpTwo2Lo= 2024-04-12T03:29:29.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:96bVwZOIelLgPdALhcYNpTwo2Lo=', 'Content-Length': '0'} 2024-04-12T03:29:29.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036337d768416a2a5-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:29.426 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.426 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.426 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.426 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.427 DEBUG:boto:Method: GET 2024-04-12T03:29:29.427 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.427 DEBUG:boto:Data: 2024-04-12T03:29:29.427 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.427 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.427 DEBUG:boto:Params: {} 2024-04-12T03:29:29.427 DEBUG:boto:Token: None 2024-04-12T03:29:29.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs= 2024-04-12T03:29:29.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs=', 'Content-Length': '0'} 2024-04-12T03:29:29.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079fbf4f63130b81f-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.429 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.429 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.429 DEBUG:boto:path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.429 DEBUG:boto:auth_path=/dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.429 DEBUG:boto:Method: GET 2024-04-12T03:29:29.430 DEBUG:boto:Path: /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.430 DEBUG:boto:Data: 2024-04-12T03:29:29.430 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.430 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.430 DEBUG:boto:Params: {} 2024-04-12T03:29:29.430 DEBUG:boto:Token: None 2024-04-12T03:29:29.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5?versionId=null 2024-04-12T03:29:29.430 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs= 2024-04-12T03:29:29.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D98PWjTED6TjtN3vTZbcpPkZuZs=', 'Content-Length': '0'} 2024-04-12T03:29:29.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3b63f4a1514e519-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.432 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.432 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.432 DEBUG:boto:path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.432 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.433 DEBUG:boto:Method: GET 2024-04-12T03:29:29.433 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.433 DEBUG:boto:Data: 2024-04-12T03:29:29.433 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.433 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.433 DEBUG:boto:Params: {} 2024-04-12T03:29:29.433 DEBUG:boto:Token: None 2024-04-12T03:29:29.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034675661871395bb-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.436 DEBUG:boto:b'dgbpxg-48obj51000falseobj52024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.436 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.436 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.436 DEBUG:boto:path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.436 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.436 DEBUG:boto:Method: GET 2024-04-12T03:29:29.436 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.436 DEBUG:boto:Data: 2024-04-12T03:29:29.437 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.437 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.437 DEBUG:boto:Params: {} 2024-04-12T03:29:29.437 DEBUG:boto:Token: None 2024-04-12T03:29:29.437 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f11efc488e11cfeb-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.440 DEBUG:boto:b'dgbpxg-48obj51000falseobj52024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.440 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.440 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.440 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.440 DEBUG:boto:Path: /dgbpxg-48/obj5 2024-04-12T03:29:29.440 DEBUG:boto:Data: 2024-04-12T03:29:29.440 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.441 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.441 DEBUG:boto:Params: {} 2024-04-12T03:29:29.441 DEBUG:boto:Token: None 2024-04-12T03:29:29.441 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5 2024-04-12T03:29:29.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iz6xBojqcHwApR8gaHxxOAdMCzA= 2024-04-12T03:29:29.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iz6xBojqcHwApR8gaHxxOAdMCzA=', 'Content-Length': '0'} 2024-04-12T03:29:29.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de58374869cd8903-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.443 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.443 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.443 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.443 DEBUG:boto:Path: /dgbpxg-48/obj5 2024-04-12T03:29:29.443 DEBUG:boto:Data: 2024-04-12T03:29:29.443 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.443 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.443 DEBUG:boto:Params: {} 2024-04-12T03:29:29.444 DEBUG:boto:Token: None 2024-04-12T03:29:29.444 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5 2024-04-12T03:29:29.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iz6xBojqcHwApR8gaHxxOAdMCzA= 2024-04-12T03:29:29.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iz6xBojqcHwApR8gaHxxOAdMCzA=', 'Content-Length': '0'} 2024-04-12T03:29:29.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f285a67cfed7e208-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:29.446 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.446 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.446 DEBUG:boto:Method: GET 2024-04-12T03:29:29.446 DEBUG:boto:Path: /dgbpxg-48/obj5 2024-04-12T03:29:29.446 DEBUG:boto:Data: 2024-04-12T03:29:29.446 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.446 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.446 DEBUG:boto:Params: {} 2024-04-12T03:29:29.447 DEBUG:boto:Token: None 2024-04-12T03:29:29.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5 2024-04-12T03:29:29.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HR3ZlfYF8bzLvFJFReuePMSehLs= 2024-04-12T03:29:29.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HR3ZlfYF8bzLvFJFReuePMSehLs=', 'Content-Length': '0'} 2024-04-12T03:29:29.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089aa06e9ccb1d4be-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.490 DEBUG:boto:path=/dgbpxg-48/obj5 2024-04-12T03:29:29.490 DEBUG:boto:auth_path=/dgbpxg-48/obj5 2024-04-12T03:29:29.491 DEBUG:boto:Method: GET 2024-04-12T03:29:29.491 DEBUG:boto:Path: /dgbpxg-48/obj5 2024-04-12T03:29:29.491 DEBUG:boto:Data: 2024-04-12T03:29:29.491 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.491 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.491 DEBUG:boto:Params: {} 2024-04-12T03:29:29.491 DEBUG:boto:Token: None 2024-04-12T03:29:29.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj5 2024-04-12T03:29:29.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HR3ZlfYF8bzLvFJFReuePMSehLs= 2024-04-12T03:29:29.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HR3ZlfYF8bzLvFJFReuePMSehLs=', 'Content-Length': '0'} 2024-04-12T03:29:29.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007775b57043e0c6cc-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.493 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.494 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.494 DEBUG:boto:path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.494 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.494 DEBUG:boto:Method: GET 2024-04-12T03:29:29.494 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.494 DEBUG:boto:Data: 2024-04-12T03:29:29.494 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.494 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.494 DEBUG:boto:Params: {} 2024-04-12T03:29:29.495 DEBUG:boto:Token: None 2024-04-12T03:29:29.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b494b3a94aee71ea-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.497 DEBUG:boto:b'dgbpxg-48obj51000falseobj52024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.497 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.497 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.498 DEBUG:boto:path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.498 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.498 DEBUG:boto:Method: GET 2024-04-12T03:29:29.498 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj5 2024-04-12T03:29:29.498 DEBUG:boto:Data: 2024-04-12T03:29:29.498 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.498 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.498 DEBUG:boto:Params: {} 2024-04-12T03:29:29.498 DEBUG:boto:Token: None 2024-04-12T03:29:29.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c9d41be7e2609d-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.501 DEBUG:boto:b'dgbpxg-48obj51000falseobj52024-04-12T03:28:58.564Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:29.501 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.501 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.502 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.502 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.502 DEBUG:boto:Method: GET 2024-04-12T03:29:29.502 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.502 DEBUG:boto:Data: 2024-04-12T03:29:29.502 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.502 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.502 DEBUG:boto:Params: {} 2024-04-12T03:29:29.502 DEBUG:boto:Token: None 2024-04-12T03:29:29.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ= 2024-04-12T03:29:29.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000788af4c3f36cd08b-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.505 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.505 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.505 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.505 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.505 DEBUG:boto:Method: GET 2024-04-12T03:29:29.505 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.505 DEBUG:boto:Data: 2024-04-12T03:29:29.505 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.506 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.506 DEBUG:boto:Params: {} 2024-04-12T03:29:29.506 DEBUG:boto:Token: None 2024-04-12T03:29:29.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ= 2024-04-12T03:29:29.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000081a8c11ed59fd2a-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.508 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.508 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.508 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.508 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.508 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.508 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.508 DEBUG:boto:Data: 2024-04-12T03:29:29.508 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.509 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.509 DEBUG:boto:Params: {} 2024-04-12T03:29:29.509 DEBUG:boto:Token: None 2024-04-12T03:29:29.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.509 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GtlWzHAOg1nNPIXVJmzaYBeJELY= 2024-04-12T03:29:29.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GtlWzHAOg1nNPIXVJmzaYBeJELY=', 'Content-Length': '0'} 2024-04-12T03:29:29.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d7bf7d635aa3076-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.511 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.511 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.511 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.511 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.511 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.511 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.511 DEBUG:boto:Data: 2024-04-12T03:29:29.511 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.512 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.512 DEBUG:boto:Params: {} 2024-04-12T03:29:29.512 DEBUG:boto:Token: None 2024-04-12T03:29:29.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GtlWzHAOg1nNPIXVJmzaYBeJELY= 2024-04-12T03:29:29.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GtlWzHAOg1nNPIXVJmzaYBeJELY=', 'Content-Length': '0'} 2024-04-12T03:29:29.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c0e2db59d7f201a-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:29.514 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.514 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.514 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.514 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.514 DEBUG:boto:Method: GET 2024-04-12T03:29:29.514 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.515 DEBUG:boto:Data: 2024-04-12T03:29:29.515 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.515 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.515 DEBUG:boto:Params: {} 2024-04-12T03:29:29.515 DEBUG:boto:Token: None 2024-04-12T03:29:29.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ= 2024-04-12T03:29:29.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c35940f92436866-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.517 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.517 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.517 DEBUG:boto:path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.518 DEBUG:boto:auth_path=/dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.518 DEBUG:boto:Method: GET 2024-04-12T03:29:29.518 DEBUG:boto:Path: /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.518 DEBUG:boto:Data: 2024-04-12T03:29:29.518 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.518 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.518 DEBUG:boto:Params: {} 2024-04-12T03:29:29.518 DEBUG:boto:Token: None 2024-04-12T03:29:29.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6?versionId=null 2024-04-12T03:29:29.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ= 2024-04-12T03:29:29.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9VO57DbUjeGHlhYrisZgO3oO4JQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f51fecb63172148-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.520 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.520 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.521 DEBUG:boto:path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.521 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.521 DEBUG:boto:Method: GET 2024-04-12T03:29:29.521 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.521 DEBUG:boto:Data: 2024-04-12T03:29:29.521 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.521 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.521 DEBUG:boto:Params: {} 2024-04-12T03:29:29.521 DEBUG:boto:Token: None 2024-04-12T03:29:29.521 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f065e422a1356231-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.524 DEBUG:boto:b'dgbpxg-48obj61000falseobj62024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.524 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.524 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.524 DEBUG:boto:path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.524 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.524 DEBUG:boto:Method: GET 2024-04-12T03:29:29.524 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.524 DEBUG:boto:Data: 2024-04-12T03:29:29.525 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.525 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.525 DEBUG:boto:Params: {} 2024-04-12T03:29:29.525 DEBUG:boto:Token: None 2024-04-12T03:29:29.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4dfca80cd535302-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.527 DEBUG:boto:b'dgbpxg-48obj61000falseobj62024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.528 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.528 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.528 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.528 DEBUG:boto:Path: /dgbpxg-48/obj6 2024-04-12T03:29:29.528 DEBUG:boto:Data: 2024-04-12T03:29:29.528 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.528 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.528 DEBUG:boto:Params: {} 2024-04-12T03:29:29.528 DEBUG:boto:Token: None 2024-04-12T03:29:29.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6 2024-04-12T03:29:29.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KCVCfQEfra7rtstCNcB2BT33NgE= 2024-04-12T03:29:29.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KCVCfQEfra7rtstCNcB2BT33NgE=', 'Content-Length': '0'} 2024-04-12T03:29:29.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d6f1959dc16c1db-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.530 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.530 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.530 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.531 DEBUG:boto:Path: /dgbpxg-48/obj6 2024-04-12T03:29:29.531 DEBUG:boto:Data: 2024-04-12T03:29:29.531 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.531 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.531 DEBUG:boto:Params: {} 2024-04-12T03:29:29.531 DEBUG:boto:Token: None 2024-04-12T03:29:29.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6 2024-04-12T03:29:29.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KCVCfQEfra7rtstCNcB2BT33NgE= 2024-04-12T03:29:29.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KCVCfQEfra7rtstCNcB2BT33NgE=', 'Content-Length': '0'} 2024-04-12T03:29:29.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6b85429a4736d7-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:29.533 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.533 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.533 DEBUG:boto:Method: GET 2024-04-12T03:29:29.533 DEBUG:boto:Path: /dgbpxg-48/obj6 2024-04-12T03:29:29.533 DEBUG:boto:Data: 2024-04-12T03:29:29.533 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.534 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.534 DEBUG:boto:Params: {} 2024-04-12T03:29:29.534 DEBUG:boto:Token: None 2024-04-12T03:29:29.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6 2024-04-12T03:29:29.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Ymf7ZVpT41ZilM+sMKoGxVB0XY= 2024-04-12T03:29:29.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Ymf7ZVpT41ZilM+sMKoGxVB0XY=', 'Content-Length': '0'} 2024-04-12T03:29:29.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067e46bedeb43af9b-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.536 DEBUG:boto:path=/dgbpxg-48/obj6 2024-04-12T03:29:29.536 DEBUG:boto:auth_path=/dgbpxg-48/obj6 2024-04-12T03:29:29.536 DEBUG:boto:Method: GET 2024-04-12T03:29:29.536 DEBUG:boto:Path: /dgbpxg-48/obj6 2024-04-12T03:29:29.536 DEBUG:boto:Data: 2024-04-12T03:29:29.536 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.537 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.537 DEBUG:boto:Params: {} 2024-04-12T03:29:29.537 DEBUG:boto:Token: None 2024-04-12T03:29:29.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj6 2024-04-12T03:29:29.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Ymf7ZVpT41ZilM+sMKoGxVB0XY= 2024-04-12T03:29:29.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Ymf7ZVpT41ZilM+sMKoGxVB0XY=', 'Content-Length': '0'} 2024-04-12T03:29:29.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb3b89f3f6d17ff4-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.582 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.583 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.583 DEBUG:boto:path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.583 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.583 DEBUG:boto:Method: GET 2024-04-12T03:29:29.583 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.583 DEBUG:boto:Data: 2024-04-12T03:29:29.583 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.583 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.583 DEBUG:boto:Params: {} 2024-04-12T03:29:29.583 DEBUG:boto:Token: None 2024-04-12T03:29:29.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fbb0c83e37b998a-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.586 DEBUG:boto:b'dgbpxg-48obj61000falseobj62024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.586 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.586 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.586 DEBUG:boto:path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.586 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.586 DEBUG:boto:Method: GET 2024-04-12T03:29:29.587 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj6 2024-04-12T03:29:29.587 DEBUG:boto:Data: 2024-04-12T03:29:29.587 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.587 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.587 DEBUG:boto:Params: {} 2024-04-12T03:29:29.587 DEBUG:boto:Token: None 2024-04-12T03:29:29.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e483f94598e2c7-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.590 DEBUG:boto:b'dgbpxg-48obj61000falseobj62024-04-12T03:28:58.573Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:29.590 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.590 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.590 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.591 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.591 DEBUG:boto:Method: GET 2024-04-12T03:29:29.591 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.591 DEBUG:boto:Data: 2024-04-12T03:29:29.591 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.591 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.591 DEBUG:boto:Params: {} 2024-04-12T03:29:29.591 DEBUG:boto:Token: None 2024-04-12T03:29:29.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ= 2024-04-12T03:29:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb0cef8cb0563968-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.594 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.594 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.594 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.594 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.595 DEBUG:boto:Method: GET 2024-04-12T03:29:29.595 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.595 DEBUG:boto:Data: 2024-04-12T03:29:29.595 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.595 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.595 DEBUG:boto:Params: {} 2024-04-12T03:29:29.595 DEBUG:boto:Token: None 2024-04-12T03:29:29.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ= 2024-04-12T03:29:29.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000298ae684161cdc10-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.597 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.597 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.597 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.598 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.598 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.598 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.598 DEBUG:boto:Data: 2024-04-12T03:29:29.598 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.598 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.598 DEBUG:boto:Params: {} 2024-04-12T03:29:29.598 DEBUG:boto:Token: None 2024-04-12T03:29:29.598 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BY6u479vzFzDu7rnIbCOONNRqaA= 2024-04-12T03:29:29.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BY6u479vzFzDu7rnIbCOONNRqaA=', 'Content-Length': '0'} 2024-04-12T03:29:29.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0842037c2b10c80-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.600 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.600 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.600 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.600 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.601 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.601 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.601 DEBUG:boto:Data: 2024-04-12T03:29:29.601 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.601 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.601 DEBUG:boto:Params: {} 2024-04-12T03:29:29.601 DEBUG:boto:Token: None 2024-04-12T03:29:29.601 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BY6u479vzFzDu7rnIbCOONNRqaA= 2024-04-12T03:29:29.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BY6u479vzFzDu7rnIbCOONNRqaA=', 'Content-Length': '0'} 2024-04-12T03:29:29.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011d35d73cfebb377-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:29.603 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.603 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.603 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.603 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.603 DEBUG:boto:Method: GET 2024-04-12T03:29:29.603 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.604 DEBUG:boto:Data: 2024-04-12T03:29:29.604 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.604 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.604 DEBUG:boto:Params: {} 2024-04-12T03:29:29.604 DEBUG:boto:Token: None 2024-04-12T03:29:29.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ= 2024-04-12T03:29:29.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038b7fc5f8b548f94-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.606 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.606 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.606 DEBUG:boto:path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.606 DEBUG:boto:auth_path=/dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.606 DEBUG:boto:Method: GET 2024-04-12T03:29:29.607 DEBUG:boto:Path: /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.607 DEBUG:boto:Data: 2024-04-12T03:29:29.607 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.607 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.607 DEBUG:boto:Params: {} 2024-04-12T03:29:29.607 DEBUG:boto:Token: None 2024-04-12T03:29:29.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7?versionId=null 2024-04-12T03:29:29.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ= 2024-04-12T03:29:29.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Q+nbK7Mo3kVWXDjy0ISK+3YbHQ=', 'Content-Length': '0'} 2024-04-12T03:29:29.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c07e05f8217ef79f-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.609 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.609 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.609 DEBUG:boto:path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.609 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.610 DEBUG:boto:Method: GET 2024-04-12T03:29:29.610 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.610 DEBUG:boto:Data: 2024-04-12T03:29:29.610 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.610 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.610 DEBUG:boto:Params: {} 2024-04-12T03:29:29.610 DEBUG:boto:Token: None 2024-04-12T03:29:29.610 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c857fe64f3124c-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.612 DEBUG:boto:b'dgbpxg-48obj71000falseobj72024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.613 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.613 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.613 DEBUG:boto:path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.613 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.613 DEBUG:boto:Method: GET 2024-04-12T03:29:29.613 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.613 DEBUG:boto:Data: 2024-04-12T03:29:29.613 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.613 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.613 DEBUG:boto:Params: {} 2024-04-12T03:29:29.614 DEBUG:boto:Token: None 2024-04-12T03:29:29.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0655ee2fcdeaace-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.616 DEBUG:boto:b'dgbpxg-48obj71000falseobj72024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.616 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.616 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.616 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.616 DEBUG:boto:Path: /dgbpxg-48/obj7 2024-04-12T03:29:29.617 DEBUG:boto:Data: 2024-04-12T03:29:29.617 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.617 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.617 DEBUG:boto:Params: {} 2024-04-12T03:29:29.617 DEBUG:boto:Token: None 2024-04-12T03:29:29.617 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7 2024-04-12T03:29:29.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NiERKuRe2wEMfwxCdsvKYrTLmH8= 2024-04-12T03:29:29.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NiERKuRe2wEMfwxCdsvKYrTLmH8=', 'Content-Length': '0'} 2024-04-12T03:29:29.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078900f55117eb17a-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.619 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.619 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.619 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.619 DEBUG:boto:Path: /dgbpxg-48/obj7 2024-04-12T03:29:29.619 DEBUG:boto:Data: 2024-04-12T03:29:29.619 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.619 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.620 DEBUG:boto:Params: {} 2024-04-12T03:29:29.620 DEBUG:boto:Token: None 2024-04-12T03:29:29.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7 2024-04-12T03:29:29.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NiERKuRe2wEMfwxCdsvKYrTLmH8= 2024-04-12T03:29:29.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NiERKuRe2wEMfwxCdsvKYrTLmH8=', 'Content-Length': '0'} 2024-04-12T03:29:29.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a61bc292b899ed6b-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:29.622 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.622 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.622 DEBUG:boto:Method: GET 2024-04-12T03:29:29.622 DEBUG:boto:Path: /dgbpxg-48/obj7 2024-04-12T03:29:29.622 DEBUG:boto:Data: 2024-04-12T03:29:29.622 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.623 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.623 DEBUG:boto:Params: {} 2024-04-12T03:29:29.623 DEBUG:boto:Token: None 2024-04-12T03:29:29.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7 2024-04-12T03:29:29.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z7z0iz2YkSo4GWvyU6jNXS4IaMY= 2024-04-12T03:29:29.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z7z0iz2YkSo4GWvyU6jNXS4IaMY=', 'Content-Length': '0'} 2024-04-12T03:29:29.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcbb0aa344869322-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.666 DEBUG:boto:path=/dgbpxg-48/obj7 2024-04-12T03:29:29.666 DEBUG:boto:auth_path=/dgbpxg-48/obj7 2024-04-12T03:29:29.667 DEBUG:boto:Method: GET 2024-04-12T03:29:29.667 DEBUG:boto:Path: /dgbpxg-48/obj7 2024-04-12T03:29:29.667 DEBUG:boto:Data: 2024-04-12T03:29:29.667 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.667 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.667 DEBUG:boto:Params: {} 2024-04-12T03:29:29.667 DEBUG:boto:Token: None 2024-04-12T03:29:29.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj7 2024-04-12T03:29:29.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z7z0iz2YkSo4GWvyU6jNXS4IaMY= 2024-04-12T03:29:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z7z0iz2YkSo4GWvyU6jNXS4IaMY=', 'Content-Length': '0'} 2024-04-12T03:29:29.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b8b9a28063ebe1e-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.670 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.670 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.670 DEBUG:boto:path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.670 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.670 DEBUG:boto:Method: GET 2024-04-12T03:29:29.670 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.670 DEBUG:boto:Data: 2024-04-12T03:29:29.670 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.671 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.671 DEBUG:boto:Params: {} 2024-04-12T03:29:29.671 DEBUG:boto:Token: None 2024-04-12T03:29:29.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840e48477b1fce2a-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.673 DEBUG:boto:b'dgbpxg-48obj71000falseobj72024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.674 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.674 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.674 DEBUG:boto:path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.674 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.674 DEBUG:boto:Method: GET 2024-04-12T03:29:29.674 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj7 2024-04-12T03:29:29.674 DEBUG:boto:Data: 2024-04-12T03:29:29.674 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.675 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.675 DEBUG:boto:Params: {} 2024-04-12T03:29:29.675 DEBUG:boto:Token: None 2024-04-12T03:29:29.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000433492bbd449a83a-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.677 DEBUG:boto:b'dgbpxg-48obj71000falseobj72024-04-12T03:28:58.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:29.678 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.678 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.678 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.678 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.678 DEBUG:boto:Method: GET 2024-04-12T03:29:29.679 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.679 DEBUG:boto:Data: 2024-04-12T03:29:29.679 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.679 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.679 DEBUG:boto:Params: {} 2024-04-12T03:29:29.679 DEBUG:boto:Token: None 2024-04-12T03:29:29.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo= 2024-04-12T03:29:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo=', 'Content-Length': '0'} 2024-04-12T03:29:29.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a67509fc01f38ec6-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.682 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.682 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.682 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.682 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.682 DEBUG:boto:Method: GET 2024-04-12T03:29:29.682 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.683 DEBUG:boto:Data: 2024-04-12T03:29:29.683 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.683 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.683 DEBUG:boto:Params: {} 2024-04-12T03:29:29.683 DEBUG:boto:Token: None 2024-04-12T03:29:29.683 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.683 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo= 2024-04-12T03:29:29.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo=', 'Content-Length': '0'} 2024-04-12T03:29:29.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb1247bdee01c3ff-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.685 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.685 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.685 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.685 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.686 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.686 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.686 DEBUG:boto:Data: 2024-04-12T03:29:29.686 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.686 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.686 DEBUG:boto:Params: {} 2024-04-12T03:29:29.686 DEBUG:boto:Token: None 2024-04-12T03:29:29.686 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.686 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vgYa/d77IfbmJm4NKSWossTUgek= 2024-04-12T03:29:29.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vgYa/d77IfbmJm4NKSWossTUgek=', 'Content-Length': '0'} 2024-04-12T03:29:29.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3327c34622a2dcc-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.688 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.688 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.688 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.688 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.689 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.689 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.689 DEBUG:boto:Data: 2024-04-12T03:29:29.689 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.689 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.689 DEBUG:boto:Params: {} 2024-04-12T03:29:29.689 DEBUG:boto:Token: None 2024-04-12T03:29:29.689 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vgYa/d77IfbmJm4NKSWossTUgek= 2024-04-12T03:29:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vgYa/d77IfbmJm4NKSWossTUgek=', 'Content-Length': '0'} 2024-04-12T03:29:29.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001ae3d1a25a23ae3-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:29.692 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.692 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.692 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.692 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.692 DEBUG:boto:Method: GET 2024-04-12T03:29:29.692 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.692 DEBUG:boto:Data: 2024-04-12T03:29:29.692 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.692 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.692 DEBUG:boto:Params: {} 2024-04-12T03:29:29.693 DEBUG:boto:Token: None 2024-04-12T03:29:29.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo= 2024-04-12T03:29:29.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo=', 'Content-Length': '0'} 2024-04-12T03:29:29.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d88388ade4f9fadf-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.695 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.695 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.695 DEBUG:boto:path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.695 DEBUG:boto:auth_path=/dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.695 DEBUG:boto:Method: GET 2024-04-12T03:29:29.695 DEBUG:boto:Path: /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.695 DEBUG:boto:Data: 2024-04-12T03:29:29.695 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.695 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.695 DEBUG:boto:Params: {} 2024-04-12T03:29:29.696 DEBUG:boto:Token: None 2024-04-12T03:29:29.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8?versionId=null 2024-04-12T03:29:29.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo= 2024-04-12T03:29:29.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xf/HWezcH2gzu8Y+ZE4vIsQ6fEo=', 'Content-Length': '0'} 2024-04-12T03:29:29.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a74490506fbfd69-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.738 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.738 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.739 DEBUG:boto:path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.739 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.739 DEBUG:boto:Method: GET 2024-04-12T03:29:29.739 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.739 DEBUG:boto:Data: 2024-04-12T03:29:29.739 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.739 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.739 DEBUG:boto:Params: {} 2024-04-12T03:29:29.739 DEBUG:boto:Token: None 2024-04-12T03:29:29.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061676f961601af83-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.743 DEBUG:boto:b'dgbpxg-48obj81000falseobj82024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.743 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.743 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.744 DEBUG:boto:path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.744 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.744 DEBUG:boto:Method: GET 2024-04-12T03:29:29.744 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.744 DEBUG:boto:Data: 2024-04-12T03:29:29.744 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.744 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.744 DEBUG:boto:Params: {} 2024-04-12T03:29:29.744 DEBUG:boto:Token: None 2024-04-12T03:29:29.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055ae664082bdeea0-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.747 DEBUG:boto:b'dgbpxg-48obj81000falseobj82024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.747 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.747 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.748 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.748 DEBUG:boto:Path: /dgbpxg-48/obj8 2024-04-12T03:29:29.748 DEBUG:boto:Data: 2024-04-12T03:29:29.748 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.748 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.748 DEBUG:boto:Params: {} 2024-04-12T03:29:29.748 DEBUG:boto:Token: None 2024-04-12T03:29:29.749 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8 2024-04-12T03:29:29.749 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HQObh0CPhWo7wIR1KBnsvolrjak= 2024-04-12T03:29:29.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HQObh0CPhWo7wIR1KBnsvolrjak=', 'Content-Length': '0'} 2024-04-12T03:29:29.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4e4ae13ef31e83b-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.750 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.751 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.751 DEBUG:boto:Method: HEAD 2024-04-12T03:29:29.751 DEBUG:boto:Path: /dgbpxg-48/obj8 2024-04-12T03:29:29.751 DEBUG:boto:Data: 2024-04-12T03:29:29.751 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.751 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.751 DEBUG:boto:Params: {} 2024-04-12T03:29:29.752 DEBUG:boto:Token: None 2024-04-12T03:29:29.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8 2024-04-12T03:29:29.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HQObh0CPhWo7wIR1KBnsvolrjak= 2024-04-12T03:29:29.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HQObh0CPhWo7wIR1KBnsvolrjak=', 'Content-Length': '0'} 2024-04-12T03:29:29.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b487de2b35f94171-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:29.754 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.754 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.754 DEBUG:boto:Method: GET 2024-04-12T03:29:29.754 DEBUG:boto:Path: /dgbpxg-48/obj8 2024-04-12T03:29:29.754 DEBUG:boto:Data: 2024-04-12T03:29:29.754 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.754 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.754 DEBUG:boto:Params: {} 2024-04-12T03:29:29.755 DEBUG:boto:Token: None 2024-04-12T03:29:29.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8 2024-04-12T03:29:29.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsp83QCMSXRa+AeolrmkQG3SFMM= 2024-04-12T03:29:29.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsp83QCMSXRa+AeolrmkQG3SFMM=', 'Content-Length': '0'} 2024-04-12T03:29:29.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-48:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c879bc25be7af176-006618aa99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.757 DEBUG:boto:path=/dgbpxg-48/obj8 2024-04-12T03:29:29.757 DEBUG:boto:auth_path=/dgbpxg-48/obj8 2024-04-12T03:29:29.757 DEBUG:boto:Method: GET 2024-04-12T03:29:29.757 DEBUG:boto:Path: /dgbpxg-48/obj8 2024-04-12T03:29:29.757 DEBUG:boto:Data: 2024-04-12T03:29:29.757 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.758 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.758 DEBUG:boto:Params: {} 2024-04-12T03:29:29.758 DEBUG:boto:Token: None 2024-04-12T03:29:29.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/obj8 2024-04-12T03:29:29.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsp83QCMSXRa+AeolrmkQG3SFMM= 2024-04-12T03:29:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsp83QCMSXRa+AeolrmkQG3SFMM=', 'Content-Length': '0'} 2024-04-12T03:29:29.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e6d909e1ce44bc-006618aa99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.760 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.760 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.760 DEBUG:boto:path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.760 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.760 DEBUG:boto:Method: GET 2024-04-12T03:29:29.761 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.761 DEBUG:boto:Data: 2024-04-12T03:29:29.761 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:29.761 DEBUG:boto:Port: 8001 2024-04-12T03:29:29.761 DEBUG:boto:Params: {} 2024-04-12T03:29:29.761 DEBUG:boto:Token: None 2024-04-12T03:29:29.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb8b45411ad261a-006618aa99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.763 DEBUG:boto:b'dgbpxg-48obj81000falseobj82024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.764 DEBUG:boto:path=/dgbpxg-48/ 2024-04-12T03:29:29.764 DEBUG:boto:auth_path=/dgbpxg-48/ 2024-04-12T03:29:29.764 DEBUG:boto:path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.764 DEBUG:boto:auth_path=/dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.764 DEBUG:boto:Method: GET 2024-04-12T03:29:29.764 DEBUG:boto:Path: /dgbpxg-48/?prefix=obj8 2024-04-12T03:29:29.764 DEBUG:boto:Data: 2024-04-12T03:29:29.764 DEBUG:boto:Headers: {} 2024-04-12T03:29:29.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:29.765 DEBUG:boto:Port: 8000 2024-04-12T03:29:29.765 DEBUG:boto:Params: {} 2024-04-12T03:29:29.765 DEBUG:boto:Token: None 2024-04-12T03:29:29.765 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:29 GMT /dgbpxg-48/ 2024-04-12T03:29:29.765 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc= 2024-04-12T03:29:29.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPTCgliCUU/oO8eE7bpW9AZiKQc=', 'Content-Length': '0'} 2024-04-12T03:29:29.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000583ee911b377038-006618aa99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:29.767 DEBUG:boto:b'dgbpxg-48obj81000falseobj82024-04-12T03:28:58.633Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:29.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-48 zones={a2, a1} 2024-04-12T03:29:29.767 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-49 2024-04-12T03:29:29.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:29.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:29.768 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:29.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:29.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:29.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:29.856+0000 7fa15e9c9780 1 waiting for incremental sync to catch up: 2024-04-12T03:29:29.857 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.31.1, 00000000002.49.1, 00000000004.52.6, 00000000002.29.1, 00000000003.40.1, 00000000002.20.1, 00000000002.18.1, 00000000003.36.1, 00000000002.20.1, 00000000002.14.1, 00000000002.32.1] 2024-04-12T03:29:29.857 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.34.6, 00000000002.49.1, 00000000004.52.6, 00000000002.29.1, 00000000004.42.6, 00000000002.20.1, 00000000002.18.1, 00000000004.38.6, 00000000002.20.1, 00000000002.14.1, 00000000002.32.1] 2024-04-12T03:29:34.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:34.858+0000 7fa15e9c9780 1 bucket sync caught up with source: 2024-04-12T03:29:34.859 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.34.6, 00000000002.49.1, 00000000004.52.6, 00000000002.29.1, 00000000004.42.6, 00000000002.20.1, 00000000002.18.1, 00000000004.38.6, 00000000002.20.1, 00000000002.14.1, 00000000002.32.1] 2024-04-12T03:29:34.859 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.34.6, 00000000002.49.1, 00000000004.52.6, 00000000002.29.1, 00000000004.42.6, 00000000002.20.1, 00000000002.18.1, 00000000004.38.6, 00000000002.20.1, 00000000002.14.1, 00000000002.32.1] 2024-04-12T03:29:34.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:34.858+0000 7fa15e9c9780 0 bucket checkpoint complete 2024-04-12T03:29:34.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-49 2024-04-12T03:29:34.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:34.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:34.877 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:34.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:34.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:34.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:34.973+0000 7f787d492780 1 bucket sync caught up with source: 2024-04-12T03:29:34.973 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.3213.6, , 00000000002.420.6, , 00000000002.191.6, , , 00000000002.768.6, , , ] 2024-04-12T03:29:34.974 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.3213.6, , 00000000002.420.6, , 00000000002.191.6, , , 00000000002.768.6, , , ] 2024-04-12T03:29:34.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:34.973+0000 7f787d492780 0 bucket checkpoint complete 2024-04-12T03:29:34.986 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-49 zones={a2, a1} 2024-04-12T03:29:34.986 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:34.986 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:34.987 DEBUG:boto:Method: HEAD 2024-04-12T03:29:34.987 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:29:34.987 DEBUG:boto:Data: 2024-04-12T03:29:34.987 DEBUG:boto:Headers: {} 2024-04-12T03:29:34.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:34.987 DEBUG:boto:Port: 8001 2024-04-12T03:29:34.987 DEBUG:boto:Params: {} 2024-04-12T03:29:34.987 DEBUG:boto:Token: None 2024-04-12T03:29:34.987 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:34 GMT /dgbpxg-49/ 2024-04-12T03:29:34.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWqIdwUOxCHYGJA/m4Wn8ZBi2k8= 2024-04-12T03:29:34.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:34 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWqIdwUOxCHYGJA/m4Wn8ZBi2k8=', 'Content-Length': '0'} 2024-04-12T03:29:34.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaa6fb0246ba5767-006618aa9e-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:34.990 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:34.990 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:34.990 DEBUG:boto:Method: HEAD 2024-04-12T03:29:34.990 DEBUG:boto:Path: /dgbpxg-49/ 2024-04-12T03:29:34.990 DEBUG:boto:Data: 2024-04-12T03:29:34.991 DEBUG:boto:Headers: {} 2024-04-12T03:29:34.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:34.991 DEBUG:boto:Port: 8000 2024-04-12T03:29:34.991 DEBUG:boto:Params: {} 2024-04-12T03:29:34.991 DEBUG:boto:Token: None 2024-04-12T03:29:34.991 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:34 GMT /dgbpxg-49/ 2024-04-12T03:29:34.991 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWqIdwUOxCHYGJA/m4Wn8ZBi2k8= 2024-04-12T03:29:34.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:34 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWqIdwUOxCHYGJA/m4Wn8ZBi2k8=', 'Content-Length': '0'} 2024-04-12T03:29:34.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000559682e94280b262-006618aa9e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:34.993 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:29:34.994 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:34.994 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:34.994 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.994 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.994 DEBUG:boto:Method: GET 2024-04-12T03:29:34.994 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.994 DEBUG:boto:Data: 2024-04-12T03:29:34.994 DEBUG:boto:Headers: {} 2024-04-12T03:29:34.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:34.994 DEBUG:boto:Port: 8001 2024-04-12T03:29:34.995 DEBUG:boto:Params: {} 2024-04-12T03:29:34.995 DEBUG:boto:Token: None 2024-04-12T03:29:34.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:34 GMT /dgbpxg-49/?versions 2024-04-12T03:29:34.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eThtFXKgJ0AD6dYiMNBB7MP+3io= 2024-04-12T03:29:34.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:34 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eThtFXKgJ0AD6dYiMNBB7MP+3io=', 'Content-Length': '0'} 2024-04-12T03:29:34.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da3703113a8d765-006618aa9e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:34.997 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:29:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:29:34.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:29:34.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:29:34.999 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:34.999 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:34.999 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.999 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.999 DEBUG:boto:Method: GET 2024-04-12T03:29:34.999 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:34.999 DEBUG:boto:Data: 2024-04-12T03:29:34.999 DEBUG:boto:Headers: {} 2024-04-12T03:29:34.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.000 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.000 DEBUG:boto:Params: {} 2024-04-12T03:29:35.000 DEBUG:boto:Token: None 2024-04-12T03:29:35.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/?versions 2024-04-12T03:29:35.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA= 2024-04-12T03:29:35.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA=', 'Content-Length': '0'} 2024-04-12T03:29:35.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbaf88779396aefd-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.002 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:29:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:29:35.003 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.004 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.004 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.004 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.004 DEBUG:boto:Method: GET 2024-04-12T03:29:35.004 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.004 DEBUG:boto:Data: 2024-04-12T03:29:35.004 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.004 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.004 DEBUG:boto:Params: {} 2024-04-12T03:29:35.004 DEBUG:boto:Token: None 2024-04-12T03:29:35.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/?versions 2024-04-12T03:29:35.005 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA= 2024-04-12T03:29:35.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA=', 'Content-Length': '0'} 2024-04-12T03:29:35.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a779a66829540b09-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.007 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.008 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.008 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.008 DEBUG:boto:path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.008 DEBUG:boto:auth_path=/dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.008 DEBUG:boto:Method: GET 2024-04-12T03:29:35.008 DEBUG:boto:Path: /dgbpxg-49/?versions&max-keys=999 2024-04-12T03:29:35.008 DEBUG:boto:Data: 2024-04-12T03:29:35.008 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.009 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.009 DEBUG:boto:Params: {} 2024-04-12T03:29:35.009 DEBUG:boto:Token: None 2024-04-12T03:29:35.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/?versions 2024-04-12T03:29:35.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA= 2024-04-12T03:29:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJIqd+FuLpjAqgeLkoLJxOUp7SA=', 'Content-Length': '0'} 2024-04-12T03:29:35.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b4b975e9255a20-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.011 DEBUG:boto:b'dgbpxg-49999falseobj1nulltrue2024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:35.012 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.012 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.012 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.012 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.013 DEBUG:boto:Method: GET 2024-04-12T03:29:35.013 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.013 DEBUG:boto:Data: 2024-04-12T03:29:35.013 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.013 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.013 DEBUG:boto:Params: {} 2024-04-12T03:29:35.013 DEBUG:boto:Token: None 2024-04-12T03:29:35.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM= 2024-04-12T03:29:35.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM=', 'Content-Length': '0'} 2024-04-12T03:29:35.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3a32de86dad0ac0-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.058 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.058 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.059 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.059 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.059 DEBUG:boto:Method: GET 2024-04-12T03:29:35.059 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.059 DEBUG:boto:Data: 2024-04-12T03:29:35.059 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.059 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.059 DEBUG:boto:Params: {} 2024-04-12T03:29:35.059 DEBUG:boto:Token: None 2024-04-12T03:29:35.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM= 2024-04-12T03:29:35.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM=', 'Content-Length': '0'} 2024-04-12T03:29:35.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f53d5393459d9f8-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.062 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.062 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.062 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.062 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.062 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.063 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.063 DEBUG:boto:Data: 2024-04-12T03:29:35.063 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.063 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.063 DEBUG:boto:Params: {} 2024-04-12T03:29:35.063 DEBUG:boto:Token: None 2024-04-12T03:29:35.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0/nKZbzjbAunsfA+Ytotkr4zMhI= 2024-04-12T03:29:35.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0/nKZbzjbAunsfA+Ytotkr4zMhI=', 'Content-Length': '0'} 2024-04-12T03:29:35.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f3a54d4c0469ac9-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.066 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.066 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.066 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.066 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.066 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.066 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.066 DEBUG:boto:Data: 2024-04-12T03:29:35.066 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.066 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.066 DEBUG:boto:Params: {} 2024-04-12T03:29:35.067 DEBUG:boto:Token: None 2024-04-12T03:29:35.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0/nKZbzjbAunsfA+Ytotkr4zMhI= 2024-04-12T03:29:35.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0/nKZbzjbAunsfA+Ytotkr4zMhI=', 'Content-Length': '0'} 2024-04-12T03:29:35.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f1eba6ad847578f-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:35.069 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.069 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.069 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.069 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.069 DEBUG:boto:Method: GET 2024-04-12T03:29:35.069 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.069 DEBUG:boto:Data: 2024-04-12T03:29:35.069 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.069 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.070 DEBUG:boto:Params: {} 2024-04-12T03:29:35.070 DEBUG:boto:Token: None 2024-04-12T03:29:35.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM= 2024-04-12T03:29:35.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM=', 'Content-Length': '0'} 2024-04-12T03:29:35.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000841ff3c10afa8bdb-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.072 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.072 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.072 DEBUG:boto:path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.072 DEBUG:boto:auth_path=/dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.072 DEBUG:boto:Method: GET 2024-04-12T03:29:35.072 DEBUG:boto:Path: /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.072 DEBUG:boto:Data: 2024-04-12T03:29:35.072 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.072 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.072 DEBUG:boto:Params: {} 2024-04-12T03:29:35.073 DEBUG:boto:Token: None 2024-04-12T03:29:35.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1?versionId=null 2024-04-12T03:29:35.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM= 2024-04-12T03:29:35.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gqcDHokp1dR75zQoMRDFyIJ6UTM=', 'Content-Length': '0'} 2024-04-12T03:29:35.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a61e270d72cfd07-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.075 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.075 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.075 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.075 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.075 DEBUG:boto:Method: GET 2024-04-12T03:29:35.076 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.076 DEBUG:boto:Data: 2024-04-12T03:29:35.076 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.076 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.076 DEBUG:boto:Params: {} 2024-04-12T03:29:35.076 DEBUG:boto:Token: None 2024-04-12T03:29:35.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ebf54480b7140b6-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.079 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.079 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.079 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.079 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.079 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.079 DEBUG:boto:Method: GET 2024-04-12T03:29:35.079 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.080 DEBUG:boto:Data: 2024-04-12T03:29:35.080 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.080 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.080 DEBUG:boto:Params: {} 2024-04-12T03:29:35.080 DEBUG:boto:Token: None 2024-04-12T03:29:35.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006437cc08058ac15a-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.126 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.127 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.127 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.127 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.127 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:29:35.127 DEBUG:boto:Data: 2024-04-12T03:29:35.127 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.127 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.128 DEBUG:boto:Params: {} 2024-04-12T03:29:35.128 DEBUG:boto:Token: None 2024-04-12T03:29:35.128 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1 2024-04-12T03:29:35.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWQh7mkHHdFp8OzCpz/6I83fRiI= 2024-04-12T03:29:35.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWQh7mkHHdFp8OzCpz/6I83fRiI=', 'Content-Length': '0'} 2024-04-12T03:29:35.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e026db16ad5ea3d-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.130 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.130 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.130 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.131 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:29:35.131 DEBUG:boto:Data: 2024-04-12T03:29:35.131 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.131 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.131 DEBUG:boto:Params: {} 2024-04-12T03:29:35.131 DEBUG:boto:Token: None 2024-04-12T03:29:35.131 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1 2024-04-12T03:29:35.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWQh7mkHHdFp8OzCpz/6I83fRiI= 2024-04-12T03:29:35.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWQh7mkHHdFp8OzCpz/6I83fRiI=', 'Content-Length': '0'} 2024-04-12T03:29:35.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4e9be98281c4ad1-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:35.133 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.133 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.133 DEBUG:boto:Method: GET 2024-04-12T03:29:35.133 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:29:35.133 DEBUG:boto:Data: 2024-04-12T03:29:35.134 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.134 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.134 DEBUG:boto:Params: {} 2024-04-12T03:29:35.134 DEBUG:boto:Token: None 2024-04-12T03:29:35.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1 2024-04-12T03:29:35.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sFalNLnmEuccL5X8QONRSHL7rMg= 2024-04-12T03:29:35.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sFalNLnmEuccL5X8QONRSHL7rMg=', 'Content-Length': '0'} 2024-04-12T03:29:35.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c07e6fbe57c79b7-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.136 DEBUG:boto:path=/dgbpxg-49/obj1 2024-04-12T03:29:35.136 DEBUG:boto:auth_path=/dgbpxg-49/obj1 2024-04-12T03:29:35.136 DEBUG:boto:Method: GET 2024-04-12T03:29:35.136 DEBUG:boto:Path: /dgbpxg-49/obj1 2024-04-12T03:29:35.136 DEBUG:boto:Data: 2024-04-12T03:29:35.137 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.137 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.137 DEBUG:boto:Params: {} 2024-04-12T03:29:35.137 DEBUG:boto:Token: None 2024-04-12T03:29:35.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj1 2024-04-12T03:29:35.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sFalNLnmEuccL5X8QONRSHL7rMg= 2024-04-12T03:29:35.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sFalNLnmEuccL5X8QONRSHL7rMg=', 'Content-Length': '0'} 2024-04-12T03:29:35.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014041dd9e4778532-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.139 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.139 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.139 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.139 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.139 DEBUG:boto:Method: GET 2024-04-12T03:29:35.139 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.140 DEBUG:boto:Data: 2024-04-12T03:29:35.140 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.140 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.140 DEBUG:boto:Params: {} 2024-04-12T03:29:35.140 DEBUG:boto:Token: None 2024-04-12T03:29:35.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf05ef55414dad6-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.143 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.143 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.143 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.143 DEBUG:boto:path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.143 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.143 DEBUG:boto:Method: GET 2024-04-12T03:29:35.143 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj1 2024-04-12T03:29:35.143 DEBUG:boto:Data: 2024-04-12T03:29:35.144 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.144 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.144 DEBUG:boto:Params: {} 2024-04-12T03:29:35.144 DEBUG:boto:Token: None 2024-04-12T03:29:35.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5877df6aea5578-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.146 DEBUG:boto:b'dgbpxg-49obj11000falseobj12024-04-12T03:28:26.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:35.147 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.147 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.147 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.147 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.147 DEBUG:boto:Method: GET 2024-04-12T03:29:35.147 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.147 DEBUG:boto:Data: 2024-04-12T03:29:35.147 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.148 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.148 DEBUG:boto:Params: {} 2024-04-12T03:29:35.148 DEBUG:boto:Token: None 2024-04-12T03:29:35.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0= 2024-04-12T03:29:35.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0=', 'Content-Length': '0'} 2024-04-12T03:29:35.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9ef21281d1a1b1e-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.150 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.150 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.150 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.150 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.150 DEBUG:boto:Method: GET 2024-04-12T03:29:35.150 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.151 DEBUG:boto:Data: 2024-04-12T03:29:35.151 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.151 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.151 DEBUG:boto:Params: {} 2024-04-12T03:29:35.151 DEBUG:boto:Token: None 2024-04-12T03:29:35.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0= 2024-04-12T03:29:35.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0=', 'Content-Length': '0'} 2024-04-12T03:29:35.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f496c9acdccda007-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.194 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.195 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.195 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.195 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.195 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.195 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.195 DEBUG:boto:Data: 2024-04-12T03:29:35.195 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.195 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.195 DEBUG:boto:Params: {} 2024-04-12T03:29:35.196 DEBUG:boto:Token: None 2024-04-12T03:29:35.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U2XzrPtOTDmquy6kFmrepFB8YpM= 2024-04-12T03:29:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U2XzrPtOTDmquy6kFmrepFB8YpM=', 'Content-Length': '0'} 2024-04-12T03:29:35.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f2721ea67ab1e7a-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.198 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.198 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.198 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.198 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.198 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.198 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.198 DEBUG:boto:Data: 2024-04-12T03:29:35.198 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.199 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.199 DEBUG:boto:Params: {} 2024-04-12T03:29:35.199 DEBUG:boto:Token: None 2024-04-12T03:29:35.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U2XzrPtOTDmquy6kFmrepFB8YpM= 2024-04-12T03:29:35.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U2XzrPtOTDmquy6kFmrepFB8YpM=', 'Content-Length': '0'} 2024-04-12T03:29:35.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b72b56779fc8ee1d-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:35.201 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.201 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.201 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.201 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.201 DEBUG:boto:Method: GET 2024-04-12T03:29:35.201 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.201 DEBUG:boto:Data: 2024-04-12T03:29:35.201 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.202 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.202 DEBUG:boto:Params: {} 2024-04-12T03:29:35.202 DEBUG:boto:Token: None 2024-04-12T03:29:35.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.202 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0= 2024-04-12T03:29:35.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0=', 'Content-Length': '0'} 2024-04-12T03:29:35.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068d14880649a6a67-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.204 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.204 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.204 DEBUG:boto:path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.204 DEBUG:boto:auth_path=/dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.204 DEBUG:boto:Method: GET 2024-04-12T03:29:35.204 DEBUG:boto:Path: /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.205 DEBUG:boto:Data: 2024-04-12T03:29:35.205 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.205 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.205 DEBUG:boto:Params: {} 2024-04-12T03:29:35.205 DEBUG:boto:Token: None 2024-04-12T03:29:35.205 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2?versionId=null 2024-04-12T03:29:35.205 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0= 2024-04-12T03:29:35.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EOLJJ0eef73RLZoRmbqwP1hGVX0=', 'Content-Length': '0'} 2024-04-12T03:29:35.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaf05e7769d53192-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.207 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.207 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.207 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.207 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.207 DEBUG:boto:Method: GET 2024-04-12T03:29:35.207 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.208 DEBUG:boto:Data: 2024-04-12T03:29:35.208 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.208 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.208 DEBUG:boto:Params: {} 2024-04-12T03:29:35.208 DEBUG:boto:Token: None 2024-04-12T03:29:35.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f6a71106cdcb51-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.210 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.211 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.211 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.211 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.211 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.211 DEBUG:boto:Method: GET 2024-04-12T03:29:35.211 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.211 DEBUG:boto:Data: 2024-04-12T03:29:35.211 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.211 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.212 DEBUG:boto:Params: {} 2024-04-12T03:29:35.212 DEBUG:boto:Token: None 2024-04-12T03:29:35.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df2d7184c4613ba-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.214 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.214 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.214 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.214 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.214 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:29:35.215 DEBUG:boto:Data: 2024-04-12T03:29:35.215 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.215 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.215 DEBUG:boto:Params: {} 2024-04-12T03:29:35.215 DEBUG:boto:Token: None 2024-04-12T03:29:35.215 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2 2024-04-12T03:29:35.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Vzs0Jiv7331KND4sXEcYyLgICU= 2024-04-12T03:29:35.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Vzs0Jiv7331KND4sXEcYyLgICU=', 'Content-Length': '0'} 2024-04-12T03:29:35.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d480fbab700107-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.217 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.217 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.217 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.217 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:29:35.217 DEBUG:boto:Data: 2024-04-12T03:29:35.217 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.217 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.217 DEBUG:boto:Params: {} 2024-04-12T03:29:35.218 DEBUG:boto:Token: None 2024-04-12T03:29:35.218 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2 2024-04-12T03:29:35.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Vzs0Jiv7331KND4sXEcYyLgICU= 2024-04-12T03:29:35.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Vzs0Jiv7331KND4sXEcYyLgICU=', 'Content-Length': '0'} 2024-04-12T03:29:35.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062278f959deaaea3-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:35.220 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.220 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.220 DEBUG:boto:Method: GET 2024-04-12T03:29:35.220 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:29:35.220 DEBUG:boto:Data: 2024-04-12T03:29:35.220 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.220 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.220 DEBUG:boto:Params: {} 2024-04-12T03:29:35.220 DEBUG:boto:Token: None 2024-04-12T03:29:35.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2 2024-04-12T03:29:35.221 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vnpTcyxP29chFm5uhn1OMIPg2A= 2024-04-12T03:29:35.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vnpTcyxP29chFm5uhn1OMIPg2A=', 'Content-Length': '0'} 2024-04-12T03:29:35.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a43a2ae63cc6ce20-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.222 DEBUG:boto:path=/dgbpxg-49/obj2 2024-04-12T03:29:35.222 DEBUG:boto:auth_path=/dgbpxg-49/obj2 2024-04-12T03:29:35.222 DEBUG:boto:Method: GET 2024-04-12T03:29:35.223 DEBUG:boto:Path: /dgbpxg-49/obj2 2024-04-12T03:29:35.223 DEBUG:boto:Data: 2024-04-12T03:29:35.223 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.223 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.223 DEBUG:boto:Params: {} 2024-04-12T03:29:35.223 DEBUG:boto:Token: None 2024-04-12T03:29:35.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj2 2024-04-12T03:29:35.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5vnpTcyxP29chFm5uhn1OMIPg2A= 2024-04-12T03:29:35.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5vnpTcyxP29chFm5uhn1OMIPg2A=', 'Content-Length': '0'} 2024-04-12T03:29:35.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a23b11121ed1d8cc-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.225 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.225 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.225 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.225 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.226 DEBUG:boto:Method: GET 2024-04-12T03:29:35.226 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.226 DEBUG:boto:Data: 2024-04-12T03:29:35.226 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.226 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.226 DEBUG:boto:Params: {} 2024-04-12T03:29:35.226 DEBUG:boto:Token: None 2024-04-12T03:29:35.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14e705aa42e30d7-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.229 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.229 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.229 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.229 DEBUG:boto:path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.229 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.229 DEBUG:boto:Method: GET 2024-04-12T03:29:35.230 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj2 2024-04-12T03:29:35.230 DEBUG:boto:Data: 2024-04-12T03:29:35.230 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.230 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.230 DEBUG:boto:Params: {} 2024-04-12T03:29:35.230 DEBUG:boto:Token: None 2024-04-12T03:29:35.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ddf1e972f2bcf2-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.232 DEBUG:boto:b'dgbpxg-49obj21000falseobj22024-04-12T03:28:26.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:35.233 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.233 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.233 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.233 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.233 DEBUG:boto:Method: GET 2024-04-12T03:29:35.233 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.233 DEBUG:boto:Data: 2024-04-12T03:29:35.233 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.234 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.234 DEBUG:boto:Params: {} 2024-04-12T03:29:35.234 DEBUG:boto:Token: None 2024-04-12T03:29:35.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.234 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ= 2024-04-12T03:29:35.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d65fe2bf08cbefe-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.236 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.236 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.236 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.236 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.236 DEBUG:boto:Method: GET 2024-04-12T03:29:35.236 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.236 DEBUG:boto:Data: 2024-04-12T03:29:35.236 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.237 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.237 DEBUG:boto:Params: {} 2024-04-12T03:29:35.237 DEBUG:boto:Token: None 2024-04-12T03:29:35.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.237 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ= 2024-04-12T03:29:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005ab832967b22c50-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.239 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.239 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.239 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.239 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.239 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.239 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.239 DEBUG:boto:Data: 2024-04-12T03:29:35.239 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.240 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.240 DEBUG:boto:Params: {} 2024-04-12T03:29:35.240 DEBUG:boto:Token: None 2024-04-12T03:29:35.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:krpgHBw6rzY9eF33HY3YTRJThs0= 2024-04-12T03:29:35.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:krpgHBw6rzY9eF33HY3YTRJThs0=', 'Content-Length': '0'} 2024-04-12T03:29:35.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d144f011c4ca0259-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.242 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.242 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.242 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.242 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.242 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.242 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.242 DEBUG:boto:Data: 2024-04-12T03:29:35.242 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.243 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.243 DEBUG:boto:Params: {} 2024-04-12T03:29:35.243 DEBUG:boto:Token: None 2024-04-12T03:29:35.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:krpgHBw6rzY9eF33HY3YTRJThs0= 2024-04-12T03:29:35.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:krpgHBw6rzY9eF33HY3YTRJThs0=', 'Content-Length': '0'} 2024-04-12T03:29:35.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb2f49aa4e615e92-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:35.245 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.245 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.245 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.245 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.245 DEBUG:boto:Method: GET 2024-04-12T03:29:35.245 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.246 DEBUG:boto:Data: 2024-04-12T03:29:35.246 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.246 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.246 DEBUG:boto:Params: {} 2024-04-12T03:29:35.246 DEBUG:boto:Token: None 2024-04-12T03:29:35.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ= 2024-04-12T03:29:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d8919f6d222378-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.248 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.248 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.248 DEBUG:boto:path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.248 DEBUG:boto:auth_path=/dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.248 DEBUG:boto:Method: GET 2024-04-12T03:29:35.249 DEBUG:boto:Path: /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.249 DEBUG:boto:Data: 2024-04-12T03:29:35.249 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.249 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.249 DEBUG:boto:Params: {} 2024-04-12T03:29:35.249 DEBUG:boto:Token: None 2024-04-12T03:29:35.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3?versionId=null 2024-04-12T03:29:35.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ= 2024-04-12T03:29:35.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KhlZpu0/Uqj2xca6tfL7LE43uWQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099843a430a0b8960-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.294 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.294 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.295 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.295 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.295 DEBUG:boto:Method: GET 2024-04-12T03:29:35.295 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.295 DEBUG:boto:Data: 2024-04-12T03:29:35.295 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.295 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.295 DEBUG:boto:Params: {} 2024-04-12T03:29:35.295 DEBUG:boto:Token: None 2024-04-12T03:29:35.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc16cb0e0881aa2a-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.298 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.298 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.298 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.298 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.298 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.298 DEBUG:boto:Method: GET 2024-04-12T03:29:35.298 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.298 DEBUG:boto:Data: 2024-04-12T03:29:35.298 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.299 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.299 DEBUG:boto:Params: {} 2024-04-12T03:29:35.299 DEBUG:boto:Token: None 2024-04-12T03:29:35.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0df24f3b06bfb16-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.301 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.301 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.301 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.302 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.302 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:29:35.302 DEBUG:boto:Data: 2024-04-12T03:29:35.302 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.302 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.302 DEBUG:boto:Params: {} 2024-04-12T03:29:35.302 DEBUG:boto:Token: None 2024-04-12T03:29:35.302 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3 2024-04-12T03:29:35.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7MpkrwE+hDCt8R8OWB+UYu1Q0Hk= 2024-04-12T03:29:35.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7MpkrwE+hDCt8R8OWB+UYu1Q0Hk=', 'Content-Length': '0'} 2024-04-12T03:29:35.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091eefc49fa312c9f-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.304 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.304 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.304 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.304 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:29:35.304 DEBUG:boto:Data: 2024-04-12T03:29:35.304 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.304 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.305 DEBUG:boto:Params: {} 2024-04-12T03:29:35.305 DEBUG:boto:Token: None 2024-04-12T03:29:35.305 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3 2024-04-12T03:29:35.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7MpkrwE+hDCt8R8OWB+UYu1Q0Hk= 2024-04-12T03:29:35.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7MpkrwE+hDCt8R8OWB+UYu1Q0Hk=', 'Content-Length': '0'} 2024-04-12T03:29:35.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bfe6ba428a15df3-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:35.307 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.307 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.307 DEBUG:boto:Method: GET 2024-04-12T03:29:35.307 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:29:35.307 DEBUG:boto:Data: 2024-04-12T03:29:35.307 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.308 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.308 DEBUG:boto:Params: {} 2024-04-12T03:29:35.308 DEBUG:boto:Token: None 2024-04-12T03:29:35.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3 2024-04-12T03:29:35.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M+CaiFCj9YSeVCDGPiHo7/sA20I= 2024-04-12T03:29:35.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M+CaiFCj9YSeVCDGPiHo7/sA20I=', 'Content-Length': '0'} 2024-04-12T03:29:35.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7d5f988a28e065c-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.310 DEBUG:boto:path=/dgbpxg-49/obj3 2024-04-12T03:29:35.310 DEBUG:boto:auth_path=/dgbpxg-49/obj3 2024-04-12T03:29:35.310 DEBUG:boto:Method: GET 2024-04-12T03:29:35.310 DEBUG:boto:Path: /dgbpxg-49/obj3 2024-04-12T03:29:35.310 DEBUG:boto:Data: 2024-04-12T03:29:35.310 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.310 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.310 DEBUG:boto:Params: {} 2024-04-12T03:29:35.311 DEBUG:boto:Token: None 2024-04-12T03:29:35.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj3 2024-04-12T03:29:35.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M+CaiFCj9YSeVCDGPiHo7/sA20I= 2024-04-12T03:29:35.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M+CaiFCj9YSeVCDGPiHo7/sA20I=', 'Content-Length': '0'} 2024-04-12T03:29:35.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cd6a2d58b5ea108-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.312 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.313 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.313 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.313 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.313 DEBUG:boto:Method: GET 2024-04-12T03:29:35.313 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.313 DEBUG:boto:Data: 2024-04-12T03:29:35.313 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.313 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.313 DEBUG:boto:Params: {} 2024-04-12T03:29:35.313 DEBUG:boto:Token: None 2024-04-12T03:29:35.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.314 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008705d7f2053397fe-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.316 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.316 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.316 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.316 DEBUG:boto:path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.316 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.316 DEBUG:boto:Method: GET 2024-04-12T03:29:35.316 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj3 2024-04-12T03:29:35.316 DEBUG:boto:Data: 2024-04-12T03:29:35.317 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.317 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.317 DEBUG:boto:Params: {} 2024-04-12T03:29:35.317 DEBUG:boto:Token: None 2024-04-12T03:29:35.317 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.317 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c21c1f14800bd105-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.320 DEBUG:boto:b'dgbpxg-49obj31000falseobj32024-04-12T03:28:26.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:35.320 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.320 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.320 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.321 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.321 DEBUG:boto:Method: GET 2024-04-12T03:29:35.321 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.321 DEBUG:boto:Data: 2024-04-12T03:29:35.321 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.321 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.321 DEBUG:boto:Params: {} 2024-04-12T03:29:35.321 DEBUG:boto:Token: None 2024-04-12T03:29:35.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.321 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE= 2024-04-12T03:29:35.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE=', 'Content-Length': '0'} 2024-04-12T03:29:35.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000496cfa147d7a1956-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.323 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.323 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.323 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.323 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.324 DEBUG:boto:Method: GET 2024-04-12T03:29:35.324 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.324 DEBUG:boto:Data: 2024-04-12T03:29:35.324 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.324 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.324 DEBUG:boto:Params: {} 2024-04-12T03:29:35.324 DEBUG:boto:Token: None 2024-04-12T03:29:35.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE= 2024-04-12T03:29:35.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE=', 'Content-Length': '0'} 2024-04-12T03:29:35.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0e0c7bdc7330b9b-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.366 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.366 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.366 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.367 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.367 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.367 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.367 DEBUG:boto:Data: 2024-04-12T03:29:35.367 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.367 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.367 DEBUG:boto:Params: {} 2024-04-12T03:29:35.367 DEBUG:boto:Token: None 2024-04-12T03:29:35.367 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hfSKQfGOICHJNHAa2RfekVpfy9w= 2024-04-12T03:29:35.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hfSKQfGOICHJNHAa2RfekVpfy9w=', 'Content-Length': '0'} 2024-04-12T03:29:35.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f164c1145604947-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.369 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.369 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.369 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.370 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.370 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.370 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.370 DEBUG:boto:Data: 2024-04-12T03:29:35.370 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.370 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.370 DEBUG:boto:Params: {} 2024-04-12T03:29:35.370 DEBUG:boto:Token: None 2024-04-12T03:29:35.370 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hfSKQfGOICHJNHAa2RfekVpfy9w= 2024-04-12T03:29:35.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hfSKQfGOICHJNHAa2RfekVpfy9w=', 'Content-Length': '0'} 2024-04-12T03:29:35.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b43cae4326c0194-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:35.372 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.373 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.373 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.373 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.373 DEBUG:boto:Method: GET 2024-04-12T03:29:35.373 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.373 DEBUG:boto:Data: 2024-04-12T03:29:35.373 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.373 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.373 DEBUG:boto:Params: {} 2024-04-12T03:29:35.373 DEBUG:boto:Token: None 2024-04-12T03:29:35.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.374 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE= 2024-04-12T03:29:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE=', 'Content-Length': '0'} 2024-04-12T03:29:35.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b090faa9ba2ff3cb-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.376 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.376 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.376 DEBUG:boto:path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.376 DEBUG:boto:auth_path=/dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.376 DEBUG:boto:Method: GET 2024-04-12T03:29:35.376 DEBUG:boto:Path: /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.376 DEBUG:boto:Data: 2024-04-12T03:29:35.376 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.376 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.376 DEBUG:boto:Params: {} 2024-04-12T03:29:35.376 DEBUG:boto:Token: None 2024-04-12T03:29:35.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4?versionId=null 2024-04-12T03:29:35.377 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE= 2024-04-12T03:29:35.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQZH4JlEtZE6kGv2e26U4Dx0bmE=', 'Content-Length': '0'} 2024-04-12T03:29:35.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000530556aebae7f757-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.379 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.379 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.379 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.379 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.379 DEBUG:boto:Method: GET 2024-04-12T03:29:35.379 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.379 DEBUG:boto:Data: 2024-04-12T03:29:35.379 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.379 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.379 DEBUG:boto:Params: {} 2024-04-12T03:29:35.380 DEBUG:boto:Token: None 2024-04-12T03:29:35.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161098c4def9503a-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.382 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.382 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.382 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.382 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.383 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.383 DEBUG:boto:Method: GET 2024-04-12T03:29:35.383 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.383 DEBUG:boto:Data: 2024-04-12T03:29:35.383 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.383 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.383 DEBUG:boto:Params: {} 2024-04-12T03:29:35.383 DEBUG:boto:Token: None 2024-04-12T03:29:35.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ecf914cff929f7-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.385 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.386 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.386 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.386 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.386 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:29:35.386 DEBUG:boto:Data: 2024-04-12T03:29:35.386 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.386 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.386 DEBUG:boto:Params: {} 2024-04-12T03:29:35.387 DEBUG:boto:Token: None 2024-04-12T03:29:35.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4 2024-04-12T03:29:35.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbhjDtZNo2+F4FkN3597FRYBXHY= 2024-04-12T03:29:35.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbhjDtZNo2+F4FkN3597FRYBXHY=', 'Content-Length': '0'} 2024-04-12T03:29:35.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092a7ba4c38d36196-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.388 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.388 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.388 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.389 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:29:35.389 DEBUG:boto:Data: 2024-04-12T03:29:35.389 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.389 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.389 DEBUG:boto:Params: {} 2024-04-12T03:29:35.389 DEBUG:boto:Token: None 2024-04-12T03:29:35.389 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4 2024-04-12T03:29:35.389 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DbhjDtZNo2+F4FkN3597FRYBXHY= 2024-04-12T03:29:35.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DbhjDtZNo2+F4FkN3597FRYBXHY=', 'Content-Length': '0'} 2024-04-12T03:29:35.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091dd6b7ff805f429-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:35.391 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.391 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.391 DEBUG:boto:Method: GET 2024-04-12T03:29:35.391 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:29:35.391 DEBUG:boto:Data: 2024-04-12T03:29:35.391 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.392 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.392 DEBUG:boto:Params: {} 2024-04-12T03:29:35.392 DEBUG:boto:Token: None 2024-04-12T03:29:35.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4 2024-04-12T03:29:35.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aBsqRt4Ai9rZBLnOZcJVtPSYuxk= 2024-04-12T03:29:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aBsqRt4Ai9rZBLnOZcJVtPSYuxk=', 'Content-Length': '0'} 2024-04-12T03:29:35.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8b4b26fdb8eb59e-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.394 DEBUG:boto:path=/dgbpxg-49/obj4 2024-04-12T03:29:35.394 DEBUG:boto:auth_path=/dgbpxg-49/obj4 2024-04-12T03:29:35.394 DEBUG:boto:Method: GET 2024-04-12T03:29:35.394 DEBUG:boto:Path: /dgbpxg-49/obj4 2024-04-12T03:29:35.394 DEBUG:boto:Data: 2024-04-12T03:29:35.394 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.394 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.395 DEBUG:boto:Params: {} 2024-04-12T03:29:35.395 DEBUG:boto:Token: None 2024-04-12T03:29:35.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj4 2024-04-12T03:29:35.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aBsqRt4Ai9rZBLnOZcJVtPSYuxk= 2024-04-12T03:29:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aBsqRt4Ai9rZBLnOZcJVtPSYuxk=', 'Content-Length': '0'} 2024-04-12T03:29:35.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce67f6ccdd5ba277-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.438 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.438 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.438 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.439 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.439 DEBUG:boto:Method: GET 2024-04-12T03:29:35.439 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.439 DEBUG:boto:Data: 2024-04-12T03:29:35.439 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.439 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.439 DEBUG:boto:Params: {} 2024-04-12T03:29:35.439 DEBUG:boto:Token: None 2024-04-12T03:29:35.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.439 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73928a19654227e-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.442 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.442 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.442 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.442 DEBUG:boto:path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.442 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.442 DEBUG:boto:Method: GET 2024-04-12T03:29:35.442 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj4 2024-04-12T03:29:35.443 DEBUG:boto:Data: 2024-04-12T03:29:35.443 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.443 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.443 DEBUG:boto:Params: {} 2024-04-12T03:29:35.443 DEBUG:boto:Token: None 2024-04-12T03:29:35.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e36fbed7ecf92c37-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.445 DEBUG:boto:b'dgbpxg-49obj41000falseobj42024-04-12T03:28:26.530Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:35.446 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.446 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.446 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.446 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.446 DEBUG:boto:Method: GET 2024-04-12T03:29:35.446 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.446 DEBUG:boto:Data: 2024-04-12T03:29:35.447 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.447 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.447 DEBUG:boto:Params: {} 2024-04-12T03:29:35.447 DEBUG:boto:Token: None 2024-04-12T03:29:35.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo= 2024-04-12T03:29:35.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo=', 'Content-Length': '0'} 2024-04-12T03:29:35.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000699abfcb36e2063e-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.449 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.449 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.449 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.450 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.450 DEBUG:boto:Method: GET 2024-04-12T03:29:35.450 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.450 DEBUG:boto:Data: 2024-04-12T03:29:35.450 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.450 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.450 DEBUG:boto:Params: {} 2024-04-12T03:29:35.450 DEBUG:boto:Token: None 2024-04-12T03:29:35.450 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo= 2024-04-12T03:29:35.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo=', 'Content-Length': '0'} 2024-04-12T03:29:35.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005db8b72a1d211884-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.453 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.453 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.453 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.453 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.453 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.453 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.453 DEBUG:boto:Data: 2024-04-12T03:29:35.453 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.454 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.454 DEBUG:boto:Params: {} 2024-04-12T03:29:35.454 DEBUG:boto:Token: None 2024-04-12T03:29:35.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O/SiqB2nuTJWgrWrzKtDQ0iXgqQ= 2024-04-12T03:29:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O/SiqB2nuTJWgrWrzKtDQ0iXgqQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000564bf4845015daea-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.456 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.456 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.456 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.456 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.456 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.456 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.457 DEBUG:boto:Data: 2024-04-12T03:29:35.457 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.457 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.457 DEBUG:boto:Params: {} 2024-04-12T03:29:35.457 DEBUG:boto:Token: None 2024-04-12T03:29:35.457 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.457 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O/SiqB2nuTJWgrWrzKtDQ0iXgqQ= 2024-04-12T03:29:35.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O/SiqB2nuTJWgrWrzKtDQ0iXgqQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d32adbc1f8f76a7-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:35.459 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.459 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.459 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.459 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.459 DEBUG:boto:Method: GET 2024-04-12T03:29:35.459 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.460 DEBUG:boto:Data: 2024-04-12T03:29:35.460 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.460 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.460 DEBUG:boto:Params: {} 2024-04-12T03:29:35.460 DEBUG:boto:Token: None 2024-04-12T03:29:35.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo= 2024-04-12T03:29:35.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo=', 'Content-Length': '0'} 2024-04-12T03:29:35.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d45b682028da407-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.462 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.462 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.462 DEBUG:boto:path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.462 DEBUG:boto:auth_path=/dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.462 DEBUG:boto:Method: GET 2024-04-12T03:29:35.463 DEBUG:boto:Path: /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.463 DEBUG:boto:Data: 2024-04-12T03:29:35.463 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.463 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.463 DEBUG:boto:Params: {} 2024-04-12T03:29:35.463 DEBUG:boto:Token: None 2024-04-12T03:29:35.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5?versionId=null 2024-04-12T03:29:35.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo= 2024-04-12T03:29:35.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YG3/ni21+75DqNgvabsF1U++4Jo=', 'Content-Length': '0'} 2024-04-12T03:29:35.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096872e26afb3b238-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.465 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.465 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.465 DEBUG:boto:path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.465 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.466 DEBUG:boto:Method: GET 2024-04-12T03:29:35.466 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.466 DEBUG:boto:Data: 2024-04-12T03:29:35.466 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.466 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.466 DEBUG:boto:Params: {} 2024-04-12T03:29:35.466 DEBUG:boto:Token: None 2024-04-12T03:29:35.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa56ffe2e22b1b6-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.468 DEBUG:boto:b'dgbpxg-49obj51000falseobj52024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.469 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.469 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.469 DEBUG:boto:path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.469 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.469 DEBUG:boto:Method: GET 2024-04-12T03:29:35.469 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.469 DEBUG:boto:Data: 2024-04-12T03:29:35.469 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.470 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.470 DEBUG:boto:Params: {} 2024-04-12T03:29:35.470 DEBUG:boto:Token: None 2024-04-12T03:29:35.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944bd5a7ac42dc92-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.472 DEBUG:boto:b'dgbpxg-49obj51000falseobj52024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.472 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.472 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.473 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.473 DEBUG:boto:Path: /dgbpxg-49/obj5 2024-04-12T03:29:35.473 DEBUG:boto:Data: 2024-04-12T03:29:35.473 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.473 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.473 DEBUG:boto:Params: {} 2024-04-12T03:29:35.473 DEBUG:boto:Token: None 2024-04-12T03:29:35.473 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5 2024-04-12T03:29:35.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gv3wnohBxEK6BK92IlJcBQfQGoo= 2024-04-12T03:29:35.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gv3wnohBxEK6BK92IlJcBQfQGoo=', 'Content-Length': '0'} 2024-04-12T03:29:35.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000782a8ee0836bf1e5-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.475 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.475 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.476 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.476 DEBUG:boto:Path: /dgbpxg-49/obj5 2024-04-12T03:29:35.476 DEBUG:boto:Data: 2024-04-12T03:29:35.476 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.476 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.476 DEBUG:boto:Params: {} 2024-04-12T03:29:35.476 DEBUG:boto:Token: None 2024-04-12T03:29:35.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5 2024-04-12T03:29:35.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gv3wnohBxEK6BK92IlJcBQfQGoo= 2024-04-12T03:29:35.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gv3wnohBxEK6BK92IlJcBQfQGoo=', 'Content-Length': '0'} 2024-04-12T03:29:35.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068a24701c93ffe80-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:29:35.478 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.478 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.479 DEBUG:boto:Method: GET 2024-04-12T03:29:35.479 DEBUG:boto:Path: /dgbpxg-49/obj5 2024-04-12T03:29:35.479 DEBUG:boto:Data: 2024-04-12T03:29:35.479 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.479 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.479 DEBUG:boto:Params: {} 2024-04-12T03:29:35.479 DEBUG:boto:Token: None 2024-04-12T03:29:35.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5 2024-04-12T03:29:35.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZA7qWPahMagjjfLSZprseBfiBwg= 2024-04-12T03:29:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZA7qWPahMagjjfLSZprseBfiBwg=', 'Content-Length': '0'} 2024-04-12T03:29:35.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed11cb78a2473fa3-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.522 DEBUG:boto:path=/dgbpxg-49/obj5 2024-04-12T03:29:35.522 DEBUG:boto:auth_path=/dgbpxg-49/obj5 2024-04-12T03:29:35.522 DEBUG:boto:Method: GET 2024-04-12T03:29:35.523 DEBUG:boto:Path: /dgbpxg-49/obj5 2024-04-12T03:29:35.523 DEBUG:boto:Data: 2024-04-12T03:29:35.523 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.523 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.523 DEBUG:boto:Params: {} 2024-04-12T03:29:35.523 DEBUG:boto:Token: None 2024-04-12T03:29:35.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj5 2024-04-12T03:29:35.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZA7qWPahMagjjfLSZprseBfiBwg= 2024-04-12T03:29:35.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZA7qWPahMagjjfLSZprseBfiBwg=', 'Content-Length': '0'} 2024-04-12T03:29:35.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d74f31abc35249c-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.525 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.525 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.525 DEBUG:boto:path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.525 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.526 DEBUG:boto:Method: GET 2024-04-12T03:29:35.526 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.526 DEBUG:boto:Data: 2024-04-12T03:29:35.526 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.526 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.526 DEBUG:boto:Params: {} 2024-04-12T03:29:35.526 DEBUG:boto:Token: None 2024-04-12T03:29:35.526 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ab21e7d7d187cd-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.529 DEBUG:boto:b'dgbpxg-49obj51000falseobj52024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.529 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.530 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.530 DEBUG:boto:path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.530 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.530 DEBUG:boto:Method: GET 2024-04-12T03:29:35.530 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj5 2024-04-12T03:29:35.530 DEBUG:boto:Data: 2024-04-12T03:29:35.530 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.530 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.530 DEBUG:boto:Params: {} 2024-04-12T03:29:35.530 DEBUG:boto:Token: None 2024-04-12T03:29:35.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc31abc87158552-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.534 DEBUG:boto:b'dgbpxg-49obj51000falseobj52024-04-12T03:28:58.643Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:35.534 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.534 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.534 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.534 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.535 DEBUG:boto:Method: GET 2024-04-12T03:29:35.535 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.535 DEBUG:boto:Data: 2024-04-12T03:29:35.535 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.535 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.535 DEBUG:boto:Params: {} 2024-04-12T03:29:35.535 DEBUG:boto:Token: None 2024-04-12T03:29:35.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ= 2024-04-12T03:29:35.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fabb669b5fb10ec5-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.537 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.538 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.538 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.538 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.538 DEBUG:boto:Method: GET 2024-04-12T03:29:35.538 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.538 DEBUG:boto:Data: 2024-04-12T03:29:35.538 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.538 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.538 DEBUG:boto:Params: {} 2024-04-12T03:29:35.538 DEBUG:boto:Token: None 2024-04-12T03:29:35.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ= 2024-04-12T03:29:35.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d116a5c70764277b-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.541 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.541 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.541 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.541 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.541 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.541 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.541 DEBUG:boto:Data: 2024-04-12T03:29:35.541 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.541 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.542 DEBUG:boto:Params: {} 2024-04-12T03:29:35.542 DEBUG:boto:Token: None 2024-04-12T03:29:35.542 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dlU2Le9wRAemG136wvpyWmgplV4= 2024-04-12T03:29:35.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dlU2Le9wRAemG136wvpyWmgplV4=', 'Content-Length': '0'} 2024-04-12T03:29:35.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b9f67a88e2094c8-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.544 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.544 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.544 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.544 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.544 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.544 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.544 DEBUG:boto:Data: 2024-04-12T03:29:35.544 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.545 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.545 DEBUG:boto:Params: {} 2024-04-12T03:29:35.545 DEBUG:boto:Token: None 2024-04-12T03:29:35.545 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dlU2Le9wRAemG136wvpyWmgplV4= 2024-04-12T03:29:35.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dlU2Le9wRAemG136wvpyWmgplV4=', 'Content-Length': '0'} 2024-04-12T03:29:35.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd9fb8380b804f3-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:35.547 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.547 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.547 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.547 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.547 DEBUG:boto:Method: GET 2024-04-12T03:29:35.548 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.548 DEBUG:boto:Data: 2024-04-12T03:29:35.548 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.548 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.548 DEBUG:boto:Params: {} 2024-04-12T03:29:35.548 DEBUG:boto:Token: None 2024-04-12T03:29:35.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ= 2024-04-12T03:29:35.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e846362492041470-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.550 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.550 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.551 DEBUG:boto:path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.551 DEBUG:boto:auth_path=/dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.551 DEBUG:boto:Method: GET 2024-04-12T03:29:35.551 DEBUG:boto:Path: /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.551 DEBUG:boto:Data: 2024-04-12T03:29:35.551 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.551 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.551 DEBUG:boto:Params: {} 2024-04-12T03:29:35.551 DEBUG:boto:Token: None 2024-04-12T03:29:35.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6?versionId=null 2024-04-12T03:29:35.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ= 2024-04-12T03:29:35.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BasBbiIj52opEcSbXzYV4OJ2TZQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e63343c14681170c-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.594 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.595 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.595 DEBUG:boto:path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.595 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.595 DEBUG:boto:Method: GET 2024-04-12T03:29:35.595 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.595 DEBUG:boto:Data: 2024-04-12T03:29:35.595 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.595 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.595 DEBUG:boto:Params: {} 2024-04-12T03:29:35.595 DEBUG:boto:Token: None 2024-04-12T03:29:35.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2c6ac1337ab0b1-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.598 DEBUG:boto:b'dgbpxg-49obj61000falseobj62024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.598 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.599 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.599 DEBUG:boto:path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.599 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.599 DEBUG:boto:Method: GET 2024-04-12T03:29:35.599 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.599 DEBUG:boto:Data: 2024-04-12T03:29:35.599 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.599 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.599 DEBUG:boto:Params: {} 2024-04-12T03:29:35.599 DEBUG:boto:Token: None 2024-04-12T03:29:35.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006afddcded569ab73-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.602 DEBUG:boto:b'dgbpxg-49obj61000falseobj62024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.602 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.602 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.602 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.603 DEBUG:boto:Path: /dgbpxg-49/obj6 2024-04-12T03:29:35.603 DEBUG:boto:Data: 2024-04-12T03:29:35.603 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.603 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.603 DEBUG:boto:Params: {} 2024-04-12T03:29:35.603 DEBUG:boto:Token: None 2024-04-12T03:29:35.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6 2024-04-12T03:29:35.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oCUqqvBFCAdEOvtP1nlrEztrxBQ= 2024-04-12T03:29:35.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oCUqqvBFCAdEOvtP1nlrEztrxBQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d6a78a6bc861e29-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.605 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.605 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.606 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.606 DEBUG:boto:Path: /dgbpxg-49/obj6 2024-04-12T03:29:35.606 DEBUG:boto:Data: 2024-04-12T03:29:35.606 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.606 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.606 DEBUG:boto:Params: {} 2024-04-12T03:29:35.606 DEBUG:boto:Token: None 2024-04-12T03:29:35.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6 2024-04-12T03:29:35.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oCUqqvBFCAdEOvtP1nlrEztrxBQ= 2024-04-12T03:29:35.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oCUqqvBFCAdEOvtP1nlrEztrxBQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a78a42622df20f5e-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:29:35.609 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.609 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.609 DEBUG:boto:Method: GET 2024-04-12T03:29:35.609 DEBUG:boto:Path: /dgbpxg-49/obj6 2024-04-12T03:29:35.609 DEBUG:boto:Data: 2024-04-12T03:29:35.609 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.609 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.609 DEBUG:boto:Params: {} 2024-04-12T03:29:35.610 DEBUG:boto:Token: None 2024-04-12T03:29:35.610 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6 2024-04-12T03:29:35.610 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1pGwUypXuqJ8mM60gqeVysCKFEs= 2024-04-12T03:29:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1pGwUypXuqJ8mM60gqeVysCKFEs=', 'Content-Length': '0'} 2024-04-12T03:29:35.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000541d8c696f65efdf-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.612 DEBUG:boto:path=/dgbpxg-49/obj6 2024-04-12T03:29:35.612 DEBUG:boto:auth_path=/dgbpxg-49/obj6 2024-04-12T03:29:35.612 DEBUG:boto:Method: GET 2024-04-12T03:29:35.612 DEBUG:boto:Path: /dgbpxg-49/obj6 2024-04-12T03:29:35.612 DEBUG:boto:Data: 2024-04-12T03:29:35.612 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.612 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.612 DEBUG:boto:Params: {} 2024-04-12T03:29:35.612 DEBUG:boto:Token: None 2024-04-12T03:29:35.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj6 2024-04-12T03:29:35.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1pGwUypXuqJ8mM60gqeVysCKFEs= 2024-04-12T03:29:35.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1pGwUypXuqJ8mM60gqeVysCKFEs=', 'Content-Length': '0'} 2024-04-12T03:29:35.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2b9773f71a4694f-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.615 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.615 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.615 DEBUG:boto:path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.615 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.615 DEBUG:boto:Method: GET 2024-04-12T03:29:35.615 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.615 DEBUG:boto:Data: 2024-04-12T03:29:35.615 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.616 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.616 DEBUG:boto:Params: {} 2024-04-12T03:29:35.616 DEBUG:boto:Token: None 2024-04-12T03:29:35.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba4fd0fab31171a-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.618 DEBUG:boto:b'dgbpxg-49obj61000falseobj62024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.619 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.619 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.619 DEBUG:boto:path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.619 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.619 DEBUG:boto:Method: GET 2024-04-12T03:29:35.619 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj6 2024-04-12T03:29:35.619 DEBUG:boto:Data: 2024-04-12T03:29:35.619 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.620 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.620 DEBUG:boto:Params: {} 2024-04-12T03:29:35.620 DEBUG:boto:Token: None 2024-04-12T03:29:35.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054419b95ed961861-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.622 DEBUG:boto:b'dgbpxg-49obj61000falseobj62024-04-12T03:28:58.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:35.623 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.623 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.623 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.623 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.623 DEBUG:boto:Method: GET 2024-04-12T03:29:35.623 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.623 DEBUG:boto:Data: 2024-04-12T03:29:35.623 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.623 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.623 DEBUG:boto:Params: {} 2024-04-12T03:29:35.624 DEBUG:boto:Token: None 2024-04-12T03:29:35.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ= 2024-04-12T03:29:35.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b07c7865b599800e-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.626 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.626 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.626 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.626 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.626 DEBUG:boto:Method: GET 2024-04-12T03:29:35.626 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.627 DEBUG:boto:Data: 2024-04-12T03:29:35.627 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.627 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.627 DEBUG:boto:Params: {} 2024-04-12T03:29:35.627 DEBUG:boto:Token: None 2024-04-12T03:29:35.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ= 2024-04-12T03:29:35.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081a1247d6e42d3a3-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.629 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.630 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.630 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.630 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.630 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.630 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.630 DEBUG:boto:Data: 2024-04-12T03:29:35.630 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.630 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.630 DEBUG:boto:Params: {} 2024-04-12T03:29:35.631 DEBUG:boto:Token: None 2024-04-12T03:29:35.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.631 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zS4PrYhI+AsuVqBULEA/9esu6e4= 2024-04-12T03:29:35.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zS4PrYhI+AsuVqBULEA/9esu6e4=', 'Content-Length': '0'} 2024-04-12T03:29:35.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae1674d250483c1b-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.633 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.633 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.633 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.633 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.633 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.633 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.633 DEBUG:boto:Data: 2024-04-12T03:29:35.633 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.633 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.634 DEBUG:boto:Params: {} 2024-04-12T03:29:35.634 DEBUG:boto:Token: None 2024-04-12T03:29:35.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.634 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zS4PrYhI+AsuVqBULEA/9esu6e4= 2024-04-12T03:29:35.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zS4PrYhI+AsuVqBULEA/9esu6e4=', 'Content-Length': '0'} 2024-04-12T03:29:35.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032fdac9e0ca97745-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:35.637 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.637 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.637 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.637 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.638 DEBUG:boto:Method: GET 2024-04-12T03:29:35.638 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.638 DEBUG:boto:Data: 2024-04-12T03:29:35.638 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.638 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.638 DEBUG:boto:Params: {} 2024-04-12T03:29:35.638 DEBUG:boto:Token: None 2024-04-12T03:29:35.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.638 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ= 2024-04-12T03:29:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000131f6da09c2f2728-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.640 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.640 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.640 DEBUG:boto:path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.640 DEBUG:boto:auth_path=/dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.640 DEBUG:boto:Method: GET 2024-04-12T03:29:35.641 DEBUG:boto:Path: /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.641 DEBUG:boto:Data: 2024-04-12T03:29:35.641 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.641 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.641 DEBUG:boto:Params: {} 2024-04-12T03:29:35.641 DEBUG:boto:Token: None 2024-04-12T03:29:35.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7?versionId=null 2024-04-12T03:29:35.641 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ= 2024-04-12T03:29:35.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hf/3cVnLWIz9c9QlfPmZm/afyEQ=', 'Content-Length': '0'} 2024-04-12T03:29:35.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d025671b4eac95d-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.643 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.643 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.643 DEBUG:boto:path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.643 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.643 DEBUG:boto:Method: GET 2024-04-12T03:29:35.644 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.644 DEBUG:boto:Data: 2024-04-12T03:29:35.644 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.644 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.644 DEBUG:boto:Params: {} 2024-04-12T03:29:35.644 DEBUG:boto:Token: None 2024-04-12T03:29:35.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107c0c703e397c94-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.646 DEBUG:boto:b'dgbpxg-49obj71000falseobj72024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.647 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.647 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.647 DEBUG:boto:path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.647 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.647 DEBUG:boto:Method: GET 2024-04-12T03:29:35.647 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.647 DEBUG:boto:Data: 2024-04-12T03:29:35.647 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.647 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.647 DEBUG:boto:Params: {} 2024-04-12T03:29:35.648 DEBUG:boto:Token: None 2024-04-12T03:29:35.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33cc9530a9b03be-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.690 DEBUG:boto:b'dgbpxg-49obj71000falseobj72024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.690 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.691 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.691 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.691 DEBUG:boto:Path: /dgbpxg-49/obj7 2024-04-12T03:29:35.691 DEBUG:boto:Data: 2024-04-12T03:29:35.691 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.691 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.691 DEBUG:boto:Params: {} 2024-04-12T03:29:35.691 DEBUG:boto:Token: None 2024-04-12T03:29:35.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7 2024-04-12T03:29:35.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mcwhjGDKHWIIf/s5Vqk5AJYjJWM= 2024-04-12T03:29:35.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mcwhjGDKHWIIf/s5Vqk5AJYjJWM=', 'Content-Length': '0'} 2024-04-12T03:29:35.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e69cf7c6fc31055-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.693 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.694 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.694 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.694 DEBUG:boto:Path: /dgbpxg-49/obj7 2024-04-12T03:29:35.694 DEBUG:boto:Data: 2024-04-12T03:29:35.694 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.694 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.694 DEBUG:boto:Params: {} 2024-04-12T03:29:35.694 DEBUG:boto:Token: None 2024-04-12T03:29:35.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7 2024-04-12T03:29:35.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mcwhjGDKHWIIf/s5Vqk5AJYjJWM= 2024-04-12T03:29:35.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mcwhjGDKHWIIf/s5Vqk5AJYjJWM=', 'Content-Length': '0'} 2024-04-12T03:29:35.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bc15f98f9d14b58-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:29:35.697 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.697 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.697 DEBUG:boto:Method: GET 2024-04-12T03:29:35.697 DEBUG:boto:Path: /dgbpxg-49/obj7 2024-04-12T03:29:35.697 DEBUG:boto:Data: 2024-04-12T03:29:35.697 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.697 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.697 DEBUG:boto:Params: {} 2024-04-12T03:29:35.697 DEBUG:boto:Token: None 2024-04-12T03:29:35.697 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7 2024-04-12T03:29:35.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8Y1sgMuv1HDXdHHIm3CWenaj9g= 2024-04-12T03:29:35.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8Y1sgMuv1HDXdHHIm3CWenaj9g=', 'Content-Length': '0'} 2024-04-12T03:29:35.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053081c6026deaf83-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.699 DEBUG:boto:path=/dgbpxg-49/obj7 2024-04-12T03:29:35.700 DEBUG:boto:auth_path=/dgbpxg-49/obj7 2024-04-12T03:29:35.700 DEBUG:boto:Method: GET 2024-04-12T03:29:35.700 DEBUG:boto:Path: /dgbpxg-49/obj7 2024-04-12T03:29:35.700 DEBUG:boto:Data: 2024-04-12T03:29:35.700 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.700 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.700 DEBUG:boto:Params: {} 2024-04-12T03:29:35.700 DEBUG:boto:Token: None 2024-04-12T03:29:35.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj7 2024-04-12T03:29:35.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8Y1sgMuv1HDXdHHIm3CWenaj9g= 2024-04-12T03:29:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8Y1sgMuv1HDXdHHIm3CWenaj9g=', 'Content-Length': '0'} 2024-04-12T03:29:35.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c567cfcf2f5f6d9-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.703 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.703 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.703 DEBUG:boto:path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.703 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.703 DEBUG:boto:Method: GET 2024-04-12T03:29:35.703 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.703 DEBUG:boto:Data: 2024-04-12T03:29:35.703 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.703 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.703 DEBUG:boto:Params: {} 2024-04-12T03:29:35.704 DEBUG:boto:Token: None 2024-04-12T03:29:35.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab6dd5e4c25c30e-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.706 DEBUG:boto:b'dgbpxg-49obj71000falseobj72024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.706 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.707 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.707 DEBUG:boto:path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.707 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.707 DEBUG:boto:Method: GET 2024-04-12T03:29:35.707 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj7 2024-04-12T03:29:35.707 DEBUG:boto:Data: 2024-04-12T03:29:35.707 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.707 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.707 DEBUG:boto:Params: {} 2024-04-12T03:29:35.708 DEBUG:boto:Token: None 2024-04-12T03:29:35.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084c0ca5907485c70-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.710 DEBUG:boto:b'dgbpxg-49obj71000falseobj72024-04-12T03:28:58.701Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:35.711 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.711 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.711 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.711 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.711 DEBUG:boto:Method: GET 2024-04-12T03:29:35.711 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.711 DEBUG:boto:Data: 2024-04-12T03:29:35.711 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.711 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.711 DEBUG:boto:Params: {} 2024-04-12T03:29:35.712 DEBUG:boto:Token: None 2024-04-12T03:29:35.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA= 2024-04-12T03:29:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA=', 'Content-Length': '0'} 2024-04-12T03:29:35.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000969972ab0e0b0c72-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.714 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.714 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.714 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.714 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.714 DEBUG:boto:Method: GET 2024-04-12T03:29:35.714 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.714 DEBUG:boto:Data: 2024-04-12T03:29:35.714 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.714 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.715 DEBUG:boto:Params: {} 2024-04-12T03:29:35.715 DEBUG:boto:Token: None 2024-04-12T03:29:35.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA= 2024-04-12T03:29:35.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA=', 'Content-Length': '0'} 2024-04-12T03:29:35.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cd6d7fac6a03ec9-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.717 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.717 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.717 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.717 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.717 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.717 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.718 DEBUG:boto:Data: 2024-04-12T03:29:35.718 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.718 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.718 DEBUG:boto:Params: {} 2024-04-12T03:29:35.718 DEBUG:boto:Token: None 2024-04-12T03:29:35.718 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TqpyNfBW7dQ+TlVi2BMju1j1Yx8= 2024-04-12T03:29:35.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TqpyNfBW7dQ+TlVi2BMju1j1Yx8=', 'Content-Length': '0'} 2024-04-12T03:29:35.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074be6466ac6b391e-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.720 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.720 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.720 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.720 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.720 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.721 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.721 DEBUG:boto:Data: 2024-04-12T03:29:35.721 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.721 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.721 DEBUG:boto:Params: {} 2024-04-12T03:29:35.721 DEBUG:boto:Token: None 2024-04-12T03:29:35.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TqpyNfBW7dQ+TlVi2BMju1j1Yx8= 2024-04-12T03:29:35.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TqpyNfBW7dQ+TlVi2BMju1j1Yx8=', 'Content-Length': '0'} 2024-04-12T03:29:35.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007922b817c83f436f-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:35.723 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.723 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.723 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.723 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.724 DEBUG:boto:Method: GET 2024-04-12T03:29:35.724 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.724 DEBUG:boto:Data: 2024-04-12T03:29:35.724 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.724 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.724 DEBUG:boto:Params: {} 2024-04-12T03:29:35.724 DEBUG:boto:Token: None 2024-04-12T03:29:35.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA= 2024-04-12T03:29:35.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA=', 'Content-Length': '0'} 2024-04-12T03:29:35.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e73dcc21776bf34-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.726 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.727 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.727 DEBUG:boto:path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.727 DEBUG:boto:auth_path=/dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.727 DEBUG:boto:Method: GET 2024-04-12T03:29:35.727 DEBUG:boto:Path: /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.727 DEBUG:boto:Data: 2024-04-12T03:29:35.727 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.727 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.727 DEBUG:boto:Params: {} 2024-04-12T03:29:35.727 DEBUG:boto:Token: None 2024-04-12T03:29:35.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8?versionId=null 2024-04-12T03:29:35.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA= 2024-04-12T03:29:35.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EpF4I9c/LBO69pmHbCYbm6C5qNA=', 'Content-Length': '0'} 2024-04-12T03:29:35.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf6da58e916d2bf2-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.770 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.770 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.771 DEBUG:boto:path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.771 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.771 DEBUG:boto:Method: GET 2024-04-12T03:29:35.771 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.771 DEBUG:boto:Data: 2024-04-12T03:29:35.771 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.771 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.771 DEBUG:boto:Params: {} 2024-04-12T03:29:35.771 DEBUG:boto:Token: None 2024-04-12T03:29:35.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a371890057ef4ed6-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.774 DEBUG:boto:b'dgbpxg-49obj81000falseobj82024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.774 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.774 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.774 DEBUG:boto:path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.775 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.775 DEBUG:boto:Method: GET 2024-04-12T03:29:35.775 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.775 DEBUG:boto:Data: 2024-04-12T03:29:35.775 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.775 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.775 DEBUG:boto:Params: {} 2024-04-12T03:29:35.775 DEBUG:boto:Token: None 2024-04-12T03:29:35.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc7042512a7da8a-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.778 DEBUG:boto:b'dgbpxg-49obj81000falseobj82024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.778 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.778 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.778 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.778 DEBUG:boto:Path: /dgbpxg-49/obj8 2024-04-12T03:29:35.778 DEBUG:boto:Data: 2024-04-12T03:29:35.779 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.779 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.779 DEBUG:boto:Params: {} 2024-04-12T03:29:35.779 DEBUG:boto:Token: None 2024-04-12T03:29:35.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8 2024-04-12T03:29:35.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2lnf4HLEevhSq+cPF1+5m5Tl9Pc= 2024-04-12T03:29:35.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2lnf4HLEevhSq+cPF1+5m5Tl9Pc=', 'Content-Length': '0'} 2024-04-12T03:29:35.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025f822bf7382b419-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.781 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.781 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.781 DEBUG:boto:Method: HEAD 2024-04-12T03:29:35.781 DEBUG:boto:Path: /dgbpxg-49/obj8 2024-04-12T03:29:35.781 DEBUG:boto:Data: 2024-04-12T03:29:35.781 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.782 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.782 DEBUG:boto:Params: {} 2024-04-12T03:29:35.782 DEBUG:boto:Token: None 2024-04-12T03:29:35.782 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8 2024-04-12T03:29:35.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2lnf4HLEevhSq+cPF1+5m5Tl9Pc= 2024-04-12T03:29:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2lnf4HLEevhSq+cPF1+5m5Tl9Pc=', 'Content-Length': '0'} 2024-04-12T03:29:35.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092675348b17e8e2b-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:29:35.784 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.784 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.784 DEBUG:boto:Method: GET 2024-04-12T03:29:35.784 DEBUG:boto:Path: /dgbpxg-49/obj8 2024-04-12T03:29:35.784 DEBUG:boto:Data: 2024-04-12T03:29:35.784 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.785 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.785 DEBUG:boto:Params: {} 2024-04-12T03:29:35.785 DEBUG:boto:Token: None 2024-04-12T03:29:35.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8 2024-04-12T03:29:35.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0NECMWLmaSfBrAO59BWeeWsb1oA= 2024-04-12T03:29:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0NECMWLmaSfBrAO59BWeeWsb1oA=', 'Content-Length': '0'} 2024-04-12T03:29:35.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f784c0cafb0b6bb4-006618aa9f-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.787 DEBUG:boto:path=/dgbpxg-49/obj8 2024-04-12T03:29:35.787 DEBUG:boto:auth_path=/dgbpxg-49/obj8 2024-04-12T03:29:35.787 DEBUG:boto:Method: GET 2024-04-12T03:29:35.787 DEBUG:boto:Path: /dgbpxg-49/obj8 2024-04-12T03:29:35.787 DEBUG:boto:Data: 2024-04-12T03:29:35.787 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.787 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.788 DEBUG:boto:Params: {} 2024-04-12T03:29:35.788 DEBUG:boto:Token: None 2024-04-12T03:29:35.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/obj8 2024-04-12T03:29:35.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0NECMWLmaSfBrAO59BWeeWsb1oA= 2024-04-12T03:29:35.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0NECMWLmaSfBrAO59BWeeWsb1oA=', 'Content-Length': '0'} 2024-04-12T03:29:35.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-49:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dd5aecf38397934-006618aa9f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.790 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.790 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.790 DEBUG:boto:path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.790 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.790 DEBUG:boto:Method: GET 2024-04-12T03:29:35.790 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.790 DEBUG:boto:Data: 2024-04-12T03:29:35.791 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:35.791 DEBUG:boto:Port: 8001 2024-04-12T03:29:35.791 DEBUG:boto:Params: {} 2024-04-12T03:29:35.791 DEBUG:boto:Token: None 2024-04-12T03:29:35.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000237983686f5edbbe-006618aa9f-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.793 DEBUG:boto:b'dgbpxg-49obj81000falseobj82024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.794 DEBUG:boto:path=/dgbpxg-49/ 2024-04-12T03:29:35.794 DEBUG:boto:auth_path=/dgbpxg-49/ 2024-04-12T03:29:35.794 DEBUG:boto:path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.794 DEBUG:boto:auth_path=/dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.794 DEBUG:boto:Method: GET 2024-04-12T03:29:35.794 DEBUG:boto:Path: /dgbpxg-49/?prefix=obj8 2024-04-12T03:29:35.794 DEBUG:boto:Data: 2024-04-12T03:29:35.794 DEBUG:boto:Headers: {} 2024-04-12T03:29:35.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:35.794 DEBUG:boto:Port: 8000 2024-04-12T03:29:35.795 DEBUG:boto:Params: {} 2024-04-12T03:29:35.795 DEBUG:boto:Token: None 2024-04-12T03:29:35.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:35 GMT /dgbpxg-49/ 2024-04-12T03:29:35.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI= 2024-04-12T03:29:35.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obNOmk/7bQ7V6aHjH8ZZSI18yGI=', 'Content-Length': '0'} 2024-04-12T03:29:35.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff1ea8426f217d1-006618aa9f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:35.797 DEBUG:boto:b'dgbpxg-49obj81000falseobj82024-04-12T03:28:58.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:35.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-49 zones={a2, a1} 2024-04-12T03:29:35.797 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:29:35.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.798 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:35.846 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:35.847 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005869", 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:04.605529Z" 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003753", 2024-04-12T03:29:35.893 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:15.847031Z" 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:14.814092Z" 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007552", 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:58.538502Z" 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.894 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:29:35.896 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005869', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009525', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:29:35.896 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:29:35.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.896 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:35.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:35.990 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:29:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005869", 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:04.605529Z" 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003753", 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:15.847031Z" 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:14.814092Z" 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007552", 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:58.538502Z" 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:29:35Z" 2024-04-12T03:29:35.996 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:29:35.998 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005869", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:04.605529Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003753", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:15.847031Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009525", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:14.814092Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007552", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:58.538502Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:29:35Z" } 2024-04-12T03:29:35.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005869', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:04.605529Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003753', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:15.847031Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009525', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:14.814092Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007552', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:58.538502Z'}}] 2024-04-12T03:29:35.998 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005869', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009525', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:29:35.998 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005869', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009525', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:29:35.998 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:29:35.998 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:29:35.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:35.999 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004032", 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:12.638464Z" 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002625", 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:12.619370Z" 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007692", 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:12.623177Z" 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:29:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.096 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:29:36.096 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004032', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000007692', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:29:36.097 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:29:36.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.097 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:36.195 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:29:36.195 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:29:36.196 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:29:36.196 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:29:36.196 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:29:36.196 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:29:36.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.197 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:29:36.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.198 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:29:36.198 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:36.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004032", 2024-04-12T03:29:36.199 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.199 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:12.638464Z" 2024-04-12T03:29:36.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:29:36.201 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.201 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:36.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002625", 2024-04-12T03:29:36.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:12.619370Z" 2024-04-12T03:29:36.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:29:36.203 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.203 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:36.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007692", 2024-04-12T03:29:36.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.205 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:12.623177Z" 2024-04-12T03:29:36.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:29:36.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:29:36.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005157", 2024-04-12T03:29:36.207 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:28:10.830455Z" 2024-04-12T03:29:36.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.208 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:29:36.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.209 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:29:36.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:29:36.210 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:29:36.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.210 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:29:36Z" 2024-04-12T03:29:36.211 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:29:36.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004032", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:12.638464Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:12.619370Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007692", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:12.623177Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005157", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:10.830455Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:29:36Z" } 2024-04-12T03:29:36.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004032', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:12.638464Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:12.619370Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007692', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:12.623177Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005157', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:10.830455Z'}}] 2024-04-12T03:29:36.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004032', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000007692', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:29:36.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004032', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000007692', 3: '00000000000000000000:00000000000000005157'} 2024-04-12T03:29:36.216 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:29:36.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-04-12T03:29:36.219 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-50 2024-04-12T03:29:36.220 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:36.220 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:36.220 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.220 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:36.221 DEBUG:boto:Data: 2024-04-12T03:29:36.221 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.221 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.222 DEBUG:boto:Params: {} 2024-04-12T03:29:36.222 DEBUG:boto:Token: None 2024-04-12T03:29:36.223 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/ 2024-04-12T03:29:36.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Do3VVcqzb977amX9z/SoO+LxMVQ= 2024-04-12T03:29:36.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Do3VVcqzb977amX9z/SoO+LxMVQ=', 'Content-Length': '0'} 2024-04-12T03:29:36.239 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099fdb29455ba05ec-006618aaa0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.240 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-51 2024-04-12T03:29:36.240 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:36.240 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:36.240 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.240 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:36.240 DEBUG:boto:Data: 2024-04-12T03:29:36.240 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.240 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.241 DEBUG:boto:Params: {} 2024-04-12T03:29:36.241 DEBUG:boto:Token: None 2024-04-12T03:29:36.241 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/ 2024-04-12T03:29:36.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sdpFmjSc4T2PA8Zn+IKTyxAdOQM= 2024-04-12T03:29:36.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sdpFmjSc4T2PA8Zn+IKTyxAdOQM=', 'Content-Length': '0'} 2024-04-12T03:29:36.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e5f589b91e7d1e6-006618aaa0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.265 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:36.265 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:36.266 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.266 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:36.266 DEBUG:boto:Data: 2024-04-12T03:29:36.266 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.266 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.266 DEBUG:boto:Params: {} 2024-04-12T03:29:36.266 DEBUG:boto:Token: None 2024-04-12T03:29:36.267 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/ 2024-04-12T03:29:36.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0= 2024-04-12T03:29:36.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0=', 'Content-Length': '0'} 2024-04-12T03:29:36.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000764e27d830fd0fad-006618aaa0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.270 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:36.270 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:36.270 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.270 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:29:36.270 DEBUG:boto:Data: 2024-04-12T03:29:36.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.271 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.271 DEBUG:boto:Params: {} 2024-04-12T03:29:36.271 DEBUG:boto:Token: None 2024-04-12T03:29:36.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/obj1 2024-04-12T03:29:36.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LafnEyUOFXkSiwY+ZXHeeg6WhCg= 2024-04-12T03:29:36.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LafnEyUOFXkSiwY+ZXHeeg6WhCg='} 2024-04-12T03:29:36.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8c6e35b2fd855ac-006618aaa0-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.281 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:36.281 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:36.281 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.281 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:36.282 DEBUG:boto:Data: 2024-04-12T03:29:36.282 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.282 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.282 DEBUG:boto:Params: {} 2024-04-12T03:29:36.282 DEBUG:boto:Token: None 2024-04-12T03:29:36.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/ 2024-04-12T03:29:36.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0= 2024-04-12T03:29:36.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0=', 'Content-Length': '0'} 2024-04-12T03:29:36.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad8009dcac164db-006618aaa0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.286 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:36.286 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:36.286 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.286 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:29:36.286 DEBUG:boto:Data: 2024-04-12T03:29:36.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.287 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.287 DEBUG:boto:Params: {} 2024-04-12T03:29:36.287 DEBUG:boto:Token: None 2024-04-12T03:29:36.288 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/obj2 2024-04-12T03:29:36.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mYkCQ6Pw6YJTbGTgodK/RHGNlbY= 2024-04-12T03:29:36.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mYkCQ6Pw6YJTbGTgodK/RHGNlbY='} 2024-04-12T03:29:36.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f867acdc5b709e2c-006618aaa0-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.340 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:36.340 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:36.340 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.341 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:36.341 DEBUG:boto:Data: 2024-04-12T03:29:36.341 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.341 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.341 DEBUG:boto:Params: {} 2024-04-12T03:29:36.342 DEBUG:boto:Token: None 2024-04-12T03:29:36.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/ 2024-04-12T03:29:36.342 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0= 2024-04-12T03:29:36.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0=', 'Content-Length': '0'} 2024-04-12T03:29:36.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000002aedf9042f76cc-006618aaa0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.359 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:36.359 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:36.359 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.359 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:29:36.360 DEBUG:boto:Data: 2024-04-12T03:29:36.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.360 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.360 DEBUG:boto:Params: {} 2024-04-12T03:29:36.360 DEBUG:boto:Token: None 2024-04-12T03:29:36.360 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/obj3 2024-04-12T03:29:36.361 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BR4IfIm36LfxSjts7pyNI8Z0Y1A= 2024-04-12T03:29:36.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BR4IfIm36LfxSjts7pyNI8Z0Y1A='} 2024-04-12T03:29:36.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fd53a2530661cc5-006618aaa0-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.371 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:36.371 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:36.372 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.372 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:36.372 DEBUG:boto:Data: 2024-04-12T03:29:36.372 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.372 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.373 DEBUG:boto:Params: {} 2024-04-12T03:29:36.373 DEBUG:boto:Token: None 2024-04-12T03:29:36.373 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/ 2024-04-12T03:29:36.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0= 2024-04-12T03:29:36.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eWMUiWWrCRRV/86rqUoyMhI5Xy0=', 'Content-Length': '0'} 2024-04-12T03:29:36.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016e2c4bda83a720c-006618aaa0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.394 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:36.394 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:36.394 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.394 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:29:36.394 DEBUG:boto:Data: 2024-04-12T03:29:36.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:36.394 DEBUG:boto:Port: 8000 2024-04-12T03:29:36.394 DEBUG:boto:Params: {} 2024-04-12T03:29:36.395 DEBUG:boto:Token: None 2024-04-12T03:29:36.395 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-50/obj4 2024-04-12T03:29:36.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0b2PgaxJYgG9xicb4xaMvtphj50= 2024-04-12T03:29:36.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0b2PgaxJYgG9xicb4xaMvtphj50='} 2024-04-12T03:29:36.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004812a4ccdfb9dac8-006618aaa0-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.439 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:36.439 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:36.440 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.440 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:36.440 DEBUG:boto:Data: 2024-04-12T03:29:36.440 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.440 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.440 DEBUG:boto:Params: {} 2024-04-12T03:29:36.440 DEBUG:boto:Token: None 2024-04-12T03:29:36.441 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/ 2024-04-12T03:29:36.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0= 2024-04-12T03:29:36.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0=', 'Content-Length': '0'} 2024-04-12T03:29:36.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f098d537c15482e-006618aaa0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.444 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:36.444 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:36.444 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.444 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:29:36.444 DEBUG:boto:Data: 2024-04-12T03:29:36.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.444 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.444 DEBUG:boto:Params: {} 2024-04-12T03:29:36.444 DEBUG:boto:Token: None 2024-04-12T03:29:36.445 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/obj1 2024-04-12T03:29:36.445 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wV8UY2V3lBDm+hD4Yz8S2AY1B0s= 2024-04-12T03:29:36.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wV8UY2V3lBDm+hD4Yz8S2AY1B0s='} 2024-04-12T03:29:36.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006d315c27d271a5f-006618aaa0-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.491 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:36.493 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:36.493 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.493 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:36.493 DEBUG:boto:Data: 2024-04-12T03:29:36.494 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.494 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.494 DEBUG:boto:Params: {} 2024-04-12T03:29:36.494 DEBUG:boto:Token: None 2024-04-12T03:29:36.494 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/ 2024-04-12T03:29:36.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0= 2024-04-12T03:29:36.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0=', 'Content-Length': '0'} 2024-04-12T03:29:36.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ca51a7b4447393b-006618aaa0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.497 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:36.498 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:36.498 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.498 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:29:36.498 DEBUG:boto:Data: 2024-04-12T03:29:36.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.498 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.498 DEBUG:boto:Params: {} 2024-04-12T03:29:36.499 DEBUG:boto:Token: None 2024-04-12T03:29:36.499 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/obj2 2024-04-12T03:29:36.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IRSZPut6bKK8lPoLyrlLYNdwF0w= 2024-04-12T03:29:36.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IRSZPut6bKK8lPoLyrlLYNdwF0w='} 2024-04-12T03:29:36.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d44d2bfadbd8b2c-006618aaa0-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.505 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:36.506 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:36.506 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.506 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:36.506 DEBUG:boto:Data: 2024-04-12T03:29:36.506 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.506 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.506 DEBUG:boto:Params: {} 2024-04-12T03:29:36.507 DEBUG:boto:Token: None 2024-04-12T03:29:36.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/ 2024-04-12T03:29:36.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0= 2024-04-12T03:29:36.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0=', 'Content-Length': '0'} 2024-04-12T03:29:36.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037087637bcf18fef-006618aaa0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.509 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:36.509 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:36.510 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.510 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:29:36.510 DEBUG:boto:Data: 2024-04-12T03:29:36.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.510 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.510 DEBUG:boto:Params: {} 2024-04-12T03:29:36.510 DEBUG:boto:Token: None 2024-04-12T03:29:36.511 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/obj3 2024-04-12T03:29:36.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vThPKPlxbBWY7BCErNrxBosNqXk= 2024-04-12T03:29:36.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vThPKPlxbBWY7BCErNrxBosNqXk='} 2024-04-12T03:29:36.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008709a38f777e0c09-006618aaa0-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.516 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:36.516 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:36.516 DEBUG:boto:Method: HEAD 2024-04-12T03:29:36.516 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:36.516 DEBUG:boto:Data: 2024-04-12T03:29:36.516 DEBUG:boto:Headers: {} 2024-04-12T03:29:36.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.517 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.517 DEBUG:boto:Params: {} 2024-04-12T03:29:36.517 DEBUG:boto:Token: None 2024-04-12T03:29:36.517 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/ 2024-04-12T03:29:36.517 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0= 2024-04-12T03:29:36.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c9P/5DxFb6oXNMgbcgTVWUhbgF0=', 'Content-Length': '0'} 2024-04-12T03:29:36.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093285cf5b13ee1f3-006618aaa0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.521 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:36.521 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:36.522 DEBUG:boto:Method: PUT 2024-04-12T03:29:36.522 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:29:36.522 DEBUG:boto:Data: 2024-04-12T03:29:36.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:29:36.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:36.522 DEBUG:boto:Port: 8001 2024-04-12T03:29:36.522 DEBUG:boto:Params: {} 2024-04-12T03:29:36.522 DEBUG:boto:Token: None 2024-04-12T03:29:36.523 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:29:36 GMT /dgbpxg-51/obj4 2024-04-12T03:29:36.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lDgkwQK4STP4IoQi9rFwfZ6j3oE= 2024-04-12T03:29:36.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:29:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lDgkwQK4STP4IoQi9rFwfZ6j3oE='} 2024-04-12T03:29:36.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060a51b536cb70a5d-006618aaa0-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:36.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.568 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:36.616 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:36.616 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892576.255008_916.1", 2024-04-12T03:29:36.667 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:36.255008Z" 2024-04-12T03:29:36.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.667 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:29:36.668 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.255008_916.1'} 2024-04-12T03:29:36.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:29:36.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:36.669 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:36.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:36.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:36.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:36.744+0000 7f6fc1d1e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:36.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:36.745+0000 7f6fc1d1e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:36.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.762 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892576.231560_909.1", 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:36.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:36.231560Z", 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:29:36Z" 2024-04-12T03:29:36.765 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:29:36.765 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892576.231560_909.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:36.231560Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:29:36Z" } 2024-04-12T03:29:36.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892576.231560_909.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:36.231560Z', 'realm_epoch': 2}}] 2024-04-12T03:29:36.766 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.255008_916.1'} 2024-04-12T03:29:36.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.231560_909.1'} 2024-04-12T03:29:36.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892576.255008_916.1 target=1_1712892576.231560_909.1 2024-04-12T03:29:41.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:41.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:41.768 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:41.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:41.854+0000 7fdf5f564780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:41.855+0000 7fdf5f564780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:41.870 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:29:41.870 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:41.871 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:41.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:29:41.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:41.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892576.255008_916.1", 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:36.255008Z", 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:29:41.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:29:41Z" 2024-04-12T03:29:41.876 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:29:41.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892576.255008_916.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:36.255008Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:29:41Z" } 2024-04-12T03:29:41.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892576.255008_916.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:36.255008Z', 'realm_epoch': 2}}] 2024-04-12T03:29:41.877 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.255008_916.1'} 2024-04-12T03:29:41.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.255008_916.1'} 2024-04-12T03:29:41.878 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:29:41.878 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-50 2024-04-12T03:29:41.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:41.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:41.878 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:41.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:41.966+0000 7face08f5780 1 waiting to reach incremental sync.. 2024-04-12T03:29:46.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:46.966+0000 7face08f5780 1 waiting to reach incremental sync.. 2024-04-12T03:29:51.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:51.968+0000 7face08f5780 1 waiting to reach incremental sync.. 2024-04-12T03:29:56.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:56.970+0000 7face08f5780 1 bucket sync caught up with source: 2024-04-12T03:29:56.974 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.29.6, , 00000000001.36.6, , 00000000001.42.6, , , 00000000001.31.6, , , ] 2024-04-12T03:29:56.974 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.29.6, , 00000000001.36.6, , 00000000001.42.6, , , 00000000001.31.6, , , ] 2024-04-12T03:29:56.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:56.970+0000 7face08f5780 0 bucket checkpoint complete 2024-04-12T03:29:56.988 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-50 2024-04-12T03:29:56.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:56.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:56.989 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.083+0000 7f7944671780 1 bucket sync caught up with source: 2024-04-12T03:29:57.084 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.89.6, , 00000000001.1197.6, , 00000000001.782.6, , , 00000000001.91.6, , , ] 2024-04-12T03:29:57.084 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.89.6, , 00000000001.1197.6, , 00000000001.782.6, , , 00000000001.91.6, , , ] 2024-04-12T03:29:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.083+0000 7f7944671780 0 bucket checkpoint complete 2024-04-12T03:29:57.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-50 zones={a2, a1} 2024-04-12T03:29:57.098 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.098 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.098 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.098 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:57.098 DEBUG:boto:Data: 2024-04-12T03:29:57.098 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.098 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.099 DEBUG:boto:Params: {} 2024-04-12T03:29:57.099 DEBUG:boto:Token: None 2024-04-12T03:29:57.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cqf6xdoxGlW2F3t27KtjUUQghRY= 2024-04-12T03:29:57.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cqf6xdoxGlW2F3t27KtjUUQghRY=', 'Content-Length': '0'} 2024-04-12T03:29:57.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000788d35b21b4d5f59-006618aab5-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.102 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.102 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.102 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.102 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:29:57.102 DEBUG:boto:Data: 2024-04-12T03:29:57.102 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.102 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.103 DEBUG:boto:Params: {} 2024-04-12T03:29:57.103 DEBUG:boto:Token: None 2024-04-12T03:29:57.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cqf6xdoxGlW2F3t27KtjUUQghRY= 2024-04-12T03:29:57.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cqf6xdoxGlW2F3t27KtjUUQghRY=', 'Content-Length': '0'} 2024-04-12T03:29:57.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c67fb830bc4744e-006618aab5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.105 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:29:57.105 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.105 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.106 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.106 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.106 DEBUG:boto:Method: GET 2024-04-12T03:29:57.106 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.106 DEBUG:boto:Data: 2024-04-12T03:29:57.106 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.106 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.106 DEBUG:boto:Params: {} 2024-04-12T03:29:57.107 DEBUG:boto:Token: None 2024-04-12T03:29:57.107 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/?versions 2024-04-12T03:29:57.107 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY= 2024-04-12T03:29:57.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY=', 'Content-Length': '0'} 2024-04-12T03:29:57.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d085ed1c4f0f861e-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.109 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:57.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:29:57.110 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.110 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.110 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.110 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.110 DEBUG:boto:Method: GET 2024-04-12T03:29:57.111 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.111 DEBUG:boto:Data: 2024-04-12T03:29:57.111 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.111 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.111 DEBUG:boto:Params: {} 2024-04-12T03:29:57.111 DEBUG:boto:Token: None 2024-04-12T03:29:57.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/?versions 2024-04-12T03:29:57.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY= 2024-04-12T03:29:57.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY=', 'Content-Length': '0'} 2024-04-12T03:29:57.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023708e3dd7533dec-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.114 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:57.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:57.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:57.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:57.115 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.115 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.115 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.115 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.115 DEBUG:boto:Method: GET 2024-04-12T03:29:57.115 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.115 DEBUG:boto:Data: 2024-04-12T03:29:57.115 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.116 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.116 DEBUG:boto:Params: {} 2024-04-12T03:29:57.116 DEBUG:boto:Token: None 2024-04-12T03:29:57.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/?versions 2024-04-12T03:29:57.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY= 2024-04-12T03:29:57.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY=', 'Content-Length': '0'} 2024-04-12T03:29:57.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb9464b06897210-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.120 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.121 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.121 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.121 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.121 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.121 DEBUG:boto:Method: GET 2024-04-12T03:29:57.121 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:29:57.121 DEBUG:boto:Data: 2024-04-12T03:29:57.121 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.121 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.121 DEBUG:boto:Params: {} 2024-04-12T03:29:57.122 DEBUG:boto:Token: None 2024-04-12T03:29:57.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/?versions 2024-04-12T03:29:57.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY= 2024-04-12T03:29:57.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bALsaDOiarZ/ykk+5EPYRx67zOY=', 'Content-Length': '0'} 2024-04-12T03:29:57.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016050ab071edc0c1-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.126 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.127 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.127 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.127 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.127 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.127 DEBUG:boto:Method: GET 2024-04-12T03:29:57.127 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.127 DEBUG:boto:Data: 2024-04-12T03:29:57.127 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.127 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.127 DEBUG:boto:Params: {} 2024-04-12T03:29:57.128 DEBUG:boto:Token: None 2024-04-12T03:29:57.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE= 2024-04-12T03:29:57.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE=', 'Content-Length': '0'} 2024-04-12T03:29:57.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a997f6f8495106d5-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.130 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.131 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.131 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.131 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.131 DEBUG:boto:Method: GET 2024-04-12T03:29:57.131 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.131 DEBUG:boto:Data: 2024-04-12T03:29:57.131 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.131 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.131 DEBUG:boto:Params: {} 2024-04-12T03:29:57.131 DEBUG:boto:Token: None 2024-04-12T03:29:57.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE= 2024-04-12T03:29:57.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE=', 'Content-Length': '0'} 2024-04-12T03:29:57.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000823c51d50351f7b3-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.134 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.134 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.134 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.134 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.134 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.134 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.134 DEBUG:boto:Data: 2024-04-12T03:29:57.134 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.135 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.135 DEBUG:boto:Params: {} 2024-04-12T03:29:57.135 DEBUG:boto:Token: None 2024-04-12T03:29:57.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cyl+4tdt0imgGfEQE0T8BCv0HM8= 2024-04-12T03:29:57.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cyl+4tdt0imgGfEQE0T8BCv0HM8=', 'Content-Length': '0'} 2024-04-12T03:29:57.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000189b84ceb1eaed6-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.137 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.137 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.137 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.137 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.137 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.137 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.137 DEBUG:boto:Data: 2024-04-12T03:29:57.137 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.137 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.138 DEBUG:boto:Params: {} 2024-04-12T03:29:57.138 DEBUG:boto:Token: None 2024-04-12T03:29:57.138 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cyl+4tdt0imgGfEQE0T8BCv0HM8= 2024-04-12T03:29:57.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cyl+4tdt0imgGfEQE0T8BCv0HM8=', 'Content-Length': '0'} 2024-04-12T03:29:57.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005a53fe0b1749fd5-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.140 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.140 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.140 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.140 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.140 DEBUG:boto:Method: GET 2024-04-12T03:29:57.141 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.141 DEBUG:boto:Data: 2024-04-12T03:29:57.141 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.141 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.141 DEBUG:boto:Params: {} 2024-04-12T03:29:57.141 DEBUG:boto:Token: None 2024-04-12T03:29:57.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE= 2024-04-12T03:29:57.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE=', 'Content-Length': '0'} 2024-04-12T03:29:57.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044639362b9354eb7-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.143 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.143 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.143 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.143 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.143 DEBUG:boto:Method: GET 2024-04-12T03:29:57.144 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.144 DEBUG:boto:Data: 2024-04-12T03:29:57.144 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.144 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.144 DEBUG:boto:Params: {} 2024-04-12T03:29:57.144 DEBUG:boto:Token: None 2024-04-12T03:29:57.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:29:57.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE= 2024-04-12T03:29:57.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QaHrd30h557k7nAxaoKkZrBC7hE=', 'Content-Length': '0'} 2024-04-12T03:29:57.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ea5a0c95d285a4d-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.146 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.146 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.146 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.146 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.146 DEBUG:boto:Method: GET 2024-04-12T03:29:57.147 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.147 DEBUG:boto:Data: 2024-04-12T03:29:57.147 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.147 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.147 DEBUG:boto:Params: {} 2024-04-12T03:29:57.147 DEBUG:boto:Token: None 2024-04-12T03:29:57.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e10e1f9d48ddc1-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.150 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.153 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.154 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.154 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.154 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.154 DEBUG:boto:Method: GET 2024-04-12T03:29:57.154 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.154 DEBUG:boto:Data: 2024-04-12T03:29:57.154 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.154 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.155 DEBUG:boto:Params: {} 2024-04-12T03:29:57.155 DEBUG:boto:Token: None 2024-04-12T03:29:57.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e9a29bdfbed0e6e-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.157 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.157 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.158 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.158 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.158 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:29:57.158 DEBUG:boto:Data: 2024-04-12T03:29:57.158 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.158 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.158 DEBUG:boto:Params: {} 2024-04-12T03:29:57.158 DEBUG:boto:Token: None 2024-04-12T03:29:57.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1 2024-04-12T03:29:57.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5zLJCqAfnhp+XWpa3US9zOxMmPg= 2024-04-12T03:29:57.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5zLJCqAfnhp+XWpa3US9zOxMmPg=', 'Content-Length': '0'} 2024-04-12T03:29:57.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0b4c08812c90859-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.160 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.160 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.160 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.160 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:29:57.161 DEBUG:boto:Data: 2024-04-12T03:29:57.161 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.161 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.161 DEBUG:boto:Params: {} 2024-04-12T03:29:57.161 DEBUG:boto:Token: None 2024-04-12T03:29:57.161 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1 2024-04-12T03:29:57.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5zLJCqAfnhp+XWpa3US9zOxMmPg= 2024-04-12T03:29:57.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5zLJCqAfnhp+XWpa3US9zOxMmPg=', 'Content-Length': '0'} 2024-04-12T03:29:57.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b63537bde6eb1f02-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.163 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.163 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.163 DEBUG:boto:Method: GET 2024-04-12T03:29:57.163 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:29:57.163 DEBUG:boto:Data: 2024-04-12T03:29:57.163 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.164 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.164 DEBUG:boto:Params: {} 2024-04-12T03:29:57.164 DEBUG:boto:Token: None 2024-04-12T03:29:57.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1 2024-04-12T03:29:57.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W5qVkSUDX2KbbKfSblWHnP5Kp04= 2024-04-12T03:29:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W5qVkSUDX2KbbKfSblWHnP5Kp04=', 'Content-Length': '0'} 2024-04-12T03:29:57.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bec25808973cde76-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.166 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:29:57.166 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:29:57.166 DEBUG:boto:Method: GET 2024-04-12T03:29:57.166 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:29:57.166 DEBUG:boto:Data: 2024-04-12T03:29:57.167 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.167 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.167 DEBUG:boto:Params: {} 2024-04-12T03:29:57.167 DEBUG:boto:Token: None 2024-04-12T03:29:57.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj1 2024-04-12T03:29:57.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W5qVkSUDX2KbbKfSblWHnP5Kp04= 2024-04-12T03:29:57.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W5qVkSUDX2KbbKfSblWHnP5Kp04=', 'Content-Length': '0'} 2024-04-12T03:29:57.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f70650e1a340397-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.169 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.169 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.169 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.169 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.170 DEBUG:boto:Method: GET 2024-04-12T03:29:57.170 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.170 DEBUG:boto:Data: 2024-04-12T03:29:57.170 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.170 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.170 DEBUG:boto:Params: {} 2024-04-12T03:29:57.170 DEBUG:boto:Token: None 2024-04-12T03:29:57.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.171 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a49ac494ca0db4-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.173 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.174 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.174 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.174 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.174 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.174 DEBUG:boto:Method: GET 2024-04-12T03:29:57.174 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:29:57.174 DEBUG:boto:Data: 2024-04-12T03:29:57.174 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.175 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.175 DEBUG:boto:Params: {} 2024-04-12T03:29:57.175 DEBUG:boto:Token: None 2024-04-12T03:29:57.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3674283931d20a-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.177 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:57.178 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.178 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.178 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.178 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.178 DEBUG:boto:Method: GET 2024-04-12T03:29:57.178 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.178 DEBUG:boto:Data: 2024-04-12T03:29:57.178 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.178 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.179 DEBUG:boto:Params: {} 2024-04-12T03:29:57.179 DEBUG:boto:Token: None 2024-04-12T03:29:57.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg= 2024-04-12T03:29:57.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg=', 'Content-Length': '0'} 2024-04-12T03:29:57.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062f7b3955c80d5d3-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.181 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.181 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.181 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.181 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.181 DEBUG:boto:Method: GET 2024-04-12T03:29:57.181 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.182 DEBUG:boto:Data: 2024-04-12T03:29:57.182 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.182 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.182 DEBUG:boto:Params: {} 2024-04-12T03:29:57.182 DEBUG:boto:Token: None 2024-04-12T03:29:57.182 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg= 2024-04-12T03:29:57.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg=', 'Content-Length': '0'} 2024-04-12T03:29:57.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000182b6e98b6269dab-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.184 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.185 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.185 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.185 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.185 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.185 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.185 DEBUG:boto:Data: 2024-04-12T03:29:57.185 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.185 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.185 DEBUG:boto:Params: {} 2024-04-12T03:29:57.185 DEBUG:boto:Token: None 2024-04-12T03:29:57.186 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pBiPOWYjyn4JDmNyMP/b2dwEZgw= 2024-04-12T03:29:57.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pBiPOWYjyn4JDmNyMP/b2dwEZgw=', 'Content-Length': '0'} 2024-04-12T03:29:57.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028222980951d6d1b-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.188 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.188 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.188 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.188 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.188 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.188 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.188 DEBUG:boto:Data: 2024-04-12T03:29:57.188 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.188 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.189 DEBUG:boto:Params: {} 2024-04-12T03:29:57.189 DEBUG:boto:Token: None 2024-04-12T03:29:57.189 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pBiPOWYjyn4JDmNyMP/b2dwEZgw= 2024-04-12T03:29:57.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pBiPOWYjyn4JDmNyMP/b2dwEZgw=', 'Content-Length': '0'} 2024-04-12T03:29:57.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004de3ab450475276f-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:57.191 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.192 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.192 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.192 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.192 DEBUG:boto:Method: GET 2024-04-12T03:29:57.192 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.192 DEBUG:boto:Data: 2024-04-12T03:29:57.192 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.192 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.192 DEBUG:boto:Params: {} 2024-04-12T03:29:57.192 DEBUG:boto:Token: None 2024-04-12T03:29:57.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg= 2024-04-12T03:29:57.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg=', 'Content-Length': '0'} 2024-04-12T03:29:57.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000900ae14eb472ab7b-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.195 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.195 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.195 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.195 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.195 DEBUG:boto:Method: GET 2024-04-12T03:29:57.195 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.195 DEBUG:boto:Data: 2024-04-12T03:29:57.195 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.195 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.196 DEBUG:boto:Params: {} 2024-04-12T03:29:57.196 DEBUG:boto:Token: None 2024-04-12T03:29:57.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:29:57.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg= 2024-04-12T03:29:57.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lX5MV5Xyd2O4xKlOPhIE/iqKleg=', 'Content-Length': '0'} 2024-04-12T03:29:57.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067d61cae949348b1-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.198 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.198 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.198 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.198 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.198 DEBUG:boto:Method: GET 2024-04-12T03:29:57.198 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.198 DEBUG:boto:Data: 2024-04-12T03:29:57.198 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.199 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.199 DEBUG:boto:Params: {} 2024-04-12T03:29:57.199 DEBUG:boto:Token: None 2024-04-12T03:29:57.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048471ae6c7530adc-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.358 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.359 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.359 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.359 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.359 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.359 DEBUG:boto:Method: GET 2024-04-12T03:29:57.359 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.359 DEBUG:boto:Data: 2024-04-12T03:29:57.359 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.360 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.360 DEBUG:boto:Params: {} 2024-04-12T03:29:57.360 DEBUG:boto:Token: None 2024-04-12T03:29:57.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c527252406ffb6-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.362 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.363 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.363 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.363 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.363 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:29:57.363 DEBUG:boto:Data: 2024-04-12T03:29:57.363 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.364 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.364 DEBUG:boto:Params: {} 2024-04-12T03:29:57.364 DEBUG:boto:Token: None 2024-04-12T03:29:57.364 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2 2024-04-12T03:29:57.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AQELoqWrstQArAUKdVH5larGu0o= 2024-04-12T03:29:57.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AQELoqWrstQArAUKdVH5larGu0o=', 'Content-Length': '0'} 2024-04-12T03:29:57.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3756a91757c07d3-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.366 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.366 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.366 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.366 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:29:57.366 DEBUG:boto:Data: 2024-04-12T03:29:57.366 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.367 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.367 DEBUG:boto:Params: {} 2024-04-12T03:29:57.367 DEBUG:boto:Token: None 2024-04-12T03:29:57.367 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2 2024-04-12T03:29:57.367 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AQELoqWrstQArAUKdVH5larGu0o= 2024-04-12T03:29:57.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AQELoqWrstQArAUKdVH5larGu0o=', 'Content-Length': '0'} 2024-04-12T03:29:57.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e758ed0a61046253-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:57.369 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.369 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.369 DEBUG:boto:Method: GET 2024-04-12T03:29:57.370 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:29:57.370 DEBUG:boto:Data: 2024-04-12T03:29:57.370 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.370 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.370 DEBUG:boto:Params: {} 2024-04-12T03:29:57.370 DEBUG:boto:Token: None 2024-04-12T03:29:57.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2 2024-04-12T03:29:57.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y6advBWqfyjaoej0BpI/na/y9Lk= 2024-04-12T03:29:57.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y6advBWqfyjaoej0BpI/na/y9Lk=', 'Content-Length': '0'} 2024-04-12T03:29:57.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb37747ffea900d1-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.372 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:29:57.372 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:29:57.372 DEBUG:boto:Method: GET 2024-04-12T03:29:57.372 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:29:57.373 DEBUG:boto:Data: 2024-04-12T03:29:57.373 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.373 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.373 DEBUG:boto:Params: {} 2024-04-12T03:29:57.373 DEBUG:boto:Token: None 2024-04-12T03:29:57.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj2 2024-04-12T03:29:57.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y6advBWqfyjaoej0BpI/na/y9Lk= 2024-04-12T03:29:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y6advBWqfyjaoej0BpI/na/y9Lk=', 'Content-Length': '0'} 2024-04-12T03:29:57.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000116f32cdf1246d14-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.418 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.419 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.419 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.419 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.419 DEBUG:boto:Method: GET 2024-04-12T03:29:57.419 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.419 DEBUG:boto:Data: 2024-04-12T03:29:57.419 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.419 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.419 DEBUG:boto:Params: {} 2024-04-12T03:29:57.419 DEBUG:boto:Token: None 2024-04-12T03:29:57.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.420 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008283d1b62e6d1206-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.422 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.423 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.423 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.423 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.423 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.423 DEBUG:boto:Method: GET 2024-04-12T03:29:57.423 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:29:57.423 DEBUG:boto:Data: 2024-04-12T03:29:57.424 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.424 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.424 DEBUG:boto:Params: {} 2024-04-12T03:29:57.424 DEBUG:boto:Token: None 2024-04-12T03:29:57.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077815c42bd9cb731-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.426 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:57.427 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.427 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.427 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.427 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.427 DEBUG:boto:Method: GET 2024-04-12T03:29:57.427 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.427 DEBUG:boto:Data: 2024-04-12T03:29:57.428 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.428 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.428 DEBUG:boto:Params: {} 2024-04-12T03:29:57.428 DEBUG:boto:Token: None 2024-04-12T03:29:57.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc= 2024-04-12T03:29:57.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc=', 'Content-Length': '0'} 2024-04-12T03:29:57.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d13adf419b467489-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.430 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.430 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.430 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.430 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.431 DEBUG:boto:Method: GET 2024-04-12T03:29:57.431 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.431 DEBUG:boto:Data: 2024-04-12T03:29:57.431 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.431 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.431 DEBUG:boto:Params: {} 2024-04-12T03:29:57.431 DEBUG:boto:Token: None 2024-04-12T03:29:57.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc= 2024-04-12T03:29:57.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc=', 'Content-Length': '0'} 2024-04-12T03:29:57.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088dbbbdb00a7b63a-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.433 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.433 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.434 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.434 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.434 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.434 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.434 DEBUG:boto:Data: 2024-04-12T03:29:57.434 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.434 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.434 DEBUG:boto:Params: {} 2024-04-12T03:29:57.435 DEBUG:boto:Token: None 2024-04-12T03:29:57.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCr+rP5GO+qk56ClXjtUjCvnJ58= 2024-04-12T03:29:57.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCr+rP5GO+qk56ClXjtUjCvnJ58=', 'Content-Length': '0'} 2024-04-12T03:29:57.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d491e8961c49687-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.437 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.437 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.437 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.437 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.437 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.437 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.437 DEBUG:boto:Data: 2024-04-12T03:29:57.437 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.438 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.438 DEBUG:boto:Params: {} 2024-04-12T03:29:57.438 DEBUG:boto:Token: None 2024-04-12T03:29:57.438 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.438 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCr+rP5GO+qk56ClXjtUjCvnJ58= 2024-04-12T03:29:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCr+rP5GO+qk56ClXjtUjCvnJ58=', 'Content-Length': '0'} 2024-04-12T03:29:57.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000090c247dd42f0fb4-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:57.440 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.440 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.440 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.440 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.440 DEBUG:boto:Method: GET 2024-04-12T03:29:57.441 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.441 DEBUG:boto:Data: 2024-04-12T03:29:57.441 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.441 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.441 DEBUG:boto:Params: {} 2024-04-12T03:29:57.441 DEBUG:boto:Token: None 2024-04-12T03:29:57.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc= 2024-04-12T03:29:57.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc=', 'Content-Length': '0'} 2024-04-12T03:29:57.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd6b4ea92a494f93-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.443 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.443 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.443 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.443 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.443 DEBUG:boto:Method: GET 2024-04-12T03:29:57.444 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.444 DEBUG:boto:Data: 2024-04-12T03:29:57.444 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.444 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.444 DEBUG:boto:Params: {} 2024-04-12T03:29:57.444 DEBUG:boto:Token: None 2024-04-12T03:29:57.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:29:57.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc= 2024-04-12T03:29:57.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPI1QQ0m6DtU8JcOOcAr9Ow0cvc=', 'Content-Length': '0'} 2024-04-12T03:29:57.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3bec32393025852-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.446 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.446 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.446 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.447 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.447 DEBUG:boto:Method: GET 2024-04-12T03:29:57.447 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.447 DEBUG:boto:Data: 2024-04-12T03:29:57.447 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.447 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.447 DEBUG:boto:Params: {} 2024-04-12T03:29:57.447 DEBUG:boto:Token: None 2024-04-12T03:29:57.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061969640c46c1c7c-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.450 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.450 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.450 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.451 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.451 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.451 DEBUG:boto:Method: GET 2024-04-12T03:29:57.451 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.451 DEBUG:boto:Data: 2024-04-12T03:29:57.451 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.451 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.451 DEBUG:boto:Params: {} 2024-04-12T03:29:57.451 DEBUG:boto:Token: None 2024-04-12T03:29:57.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2f014f85424a230-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.454 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.454 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.454 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.454 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.454 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:29:57.455 DEBUG:boto:Data: 2024-04-12T03:29:57.455 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.455 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.455 DEBUG:boto:Params: {} 2024-04-12T03:29:57.455 DEBUG:boto:Token: None 2024-04-12T03:29:57.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3 2024-04-12T03:29:57.455 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BXnkPJH9qbhyOoAUpMmKHPUTk8M= 2024-04-12T03:29:57.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BXnkPJH9qbhyOoAUpMmKHPUTk8M=', 'Content-Length': '0'} 2024-04-12T03:29:57.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffe395816d4e548b-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.457 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.457 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.457 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.457 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:29:57.457 DEBUG:boto:Data: 2024-04-12T03:29:57.457 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.458 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.458 DEBUG:boto:Params: {} 2024-04-12T03:29:57.458 DEBUG:boto:Token: None 2024-04-12T03:29:57.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3 2024-04-12T03:29:57.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BXnkPJH9qbhyOoAUpMmKHPUTk8M= 2024-04-12T03:29:57.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BXnkPJH9qbhyOoAUpMmKHPUTk8M=', 'Content-Length': '0'} 2024-04-12T03:29:57.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000515fccaf0b7011a6-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:57.460 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.460 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.460 DEBUG:boto:Method: GET 2024-04-12T03:29:57.460 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:29:57.460 DEBUG:boto:Data: 2024-04-12T03:29:57.460 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.461 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.461 DEBUG:boto:Params: {} 2024-04-12T03:29:57.461 DEBUG:boto:Token: None 2024-04-12T03:29:57.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3 2024-04-12T03:29:57.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H85Bf/VBvsuobd48GGEVgNBxIE0= 2024-04-12T03:29:57.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H85Bf/VBvsuobd48GGEVgNBxIE0=', 'Content-Length': '0'} 2024-04-12T03:29:57.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c5993e508934dec-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.463 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:29:57.463 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:29:57.463 DEBUG:boto:Method: GET 2024-04-12T03:29:57.463 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:29:57.463 DEBUG:boto:Data: 2024-04-12T03:29:57.463 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.463 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.464 DEBUG:boto:Params: {} 2024-04-12T03:29:57.464 DEBUG:boto:Token: None 2024-04-12T03:29:57.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj3 2024-04-12T03:29:57.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H85Bf/VBvsuobd48GGEVgNBxIE0= 2024-04-12T03:29:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H85Bf/VBvsuobd48GGEVgNBxIE0=', 'Content-Length': '0'} 2024-04-12T03:29:57.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c388cb5185af228-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.466 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.466 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.466 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.466 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.466 DEBUG:boto:Method: GET 2024-04-12T03:29:57.466 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.466 DEBUG:boto:Data: 2024-04-12T03:29:57.466 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.467 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.467 DEBUG:boto:Params: {} 2024-04-12T03:29:57.467 DEBUG:boto:Token: None 2024-04-12T03:29:57.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0b667590bd82b8-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.510 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.511 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.511 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.511 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.511 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.511 DEBUG:boto:Method: GET 2024-04-12T03:29:57.511 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:29:57.512 DEBUG:boto:Data: 2024-04-12T03:29:57.512 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.512 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.512 DEBUG:boto:Params: {} 2024-04-12T03:29:57.512 DEBUG:boto:Token: None 2024-04-12T03:29:57.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000999c9c978b6bc3e-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.515 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:57.515 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.515 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.515 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.515 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.516 DEBUG:boto:Method: GET 2024-04-12T03:29:57.516 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.516 DEBUG:boto:Data: 2024-04-12T03:29:57.516 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.516 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.516 DEBUG:boto:Params: {} 2024-04-12T03:29:57.516 DEBUG:boto:Token: None 2024-04-12T03:29:57.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4= 2024-04-12T03:29:57.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4=', 'Content-Length': '0'} 2024-04-12T03:29:57.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007eb452dae687ce83-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.518 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.518 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.518 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.519 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.519 DEBUG:boto:Method: GET 2024-04-12T03:29:57.519 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.519 DEBUG:boto:Data: 2024-04-12T03:29:57.519 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.519 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.519 DEBUG:boto:Params: {} 2024-04-12T03:29:57.519 DEBUG:boto:Token: None 2024-04-12T03:29:57.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4= 2024-04-12T03:29:57.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4=', 'Content-Length': '0'} 2024-04-12T03:29:57.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e3b23dd407379a8-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.562 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.562 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.563 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.563 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.563 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.563 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.563 DEBUG:boto:Data: 2024-04-12T03:29:57.563 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.563 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.563 DEBUG:boto:Params: {} 2024-04-12T03:29:57.564 DEBUG:boto:Token: None 2024-04-12T03:29:57.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ob1gA6hEtSBup57zUbt5gGQXDuQ= 2024-04-12T03:29:57.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ob1gA6hEtSBup57zUbt5gGQXDuQ=', 'Content-Length': '0'} 2024-04-12T03:29:57.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e9cbc5b8db2ee3d-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.566 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.566 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.566 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.566 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.566 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.566 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.566 DEBUG:boto:Data: 2024-04-12T03:29:57.567 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.567 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.567 DEBUG:boto:Params: {} 2024-04-12T03:29:57.567 DEBUG:boto:Token: None 2024-04-12T03:29:57.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ob1gA6hEtSBup57zUbt5gGQXDuQ= 2024-04-12T03:29:57.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ob1gA6hEtSBup57zUbt5gGQXDuQ=', 'Content-Length': '0'} 2024-04-12T03:29:57.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fac42684dc6d04cd-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:57.569 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.569 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.570 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.570 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.570 DEBUG:boto:Method: GET 2024-04-12T03:29:57.570 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.570 DEBUG:boto:Data: 2024-04-12T03:29:57.570 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.570 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.570 DEBUG:boto:Params: {} 2024-04-12T03:29:57.570 DEBUG:boto:Token: None 2024-04-12T03:29:57.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4= 2024-04-12T03:29:57.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4=', 'Content-Length': '0'} 2024-04-12T03:29:57.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eabfa007ea63f68-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.573 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.573 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.573 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.573 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.573 DEBUG:boto:Method: GET 2024-04-12T03:29:57.573 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.573 DEBUG:boto:Data: 2024-04-12T03:29:57.574 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.574 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.574 DEBUG:boto:Params: {} 2024-04-12T03:29:57.574 DEBUG:boto:Token: None 2024-04-12T03:29:57.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:29:57.574 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4= 2024-04-12T03:29:57.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qD5L2JY1dBRvQ5DvvuVBoYd14j4=', 'Content-Length': '0'} 2024-04-12T03:29:57.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f1c1d54e1256851-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.576 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.576 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.576 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.576 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.576 DEBUG:boto:Method: GET 2024-04-12T03:29:57.577 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.577 DEBUG:boto:Data: 2024-04-12T03:29:57.577 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.577 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.577 DEBUG:boto:Params: {} 2024-04-12T03:29:57.577 DEBUG:boto:Token: None 2024-04-12T03:29:57.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.577 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f2cac97ff6732e-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.580 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.580 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.580 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.581 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.581 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.581 DEBUG:boto:Method: GET 2024-04-12T03:29:57.581 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.581 DEBUG:boto:Data: 2024-04-12T03:29:57.581 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.581 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.581 DEBUG:boto:Params: {} 2024-04-12T03:29:57.581 DEBUG:boto:Token: None 2024-04-12T03:29:57.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e730b61cc148f903-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.584 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.584 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.584 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.584 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.584 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:29:57.585 DEBUG:boto:Data: 2024-04-12T03:29:57.585 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.585 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.585 DEBUG:boto:Params: {} 2024-04-12T03:29:57.585 DEBUG:boto:Token: None 2024-04-12T03:29:57.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4 2024-04-12T03:29:57.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTzco5FZERjdMO+sY5RkkBgxKio= 2024-04-12T03:29:57.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTzco5FZERjdMO+sY5RkkBgxKio=', 'Content-Length': '0'} 2024-04-12T03:29:57.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000799c999e49e2484d-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.587 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.587 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.588 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.588 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:29:57.588 DEBUG:boto:Data: 2024-04-12T03:29:57.588 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.588 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.588 DEBUG:boto:Params: {} 2024-04-12T03:29:57.588 DEBUG:boto:Token: None 2024-04-12T03:29:57.588 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4 2024-04-12T03:29:57.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTzco5FZERjdMO+sY5RkkBgxKio= 2024-04-12T03:29:57.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTzco5FZERjdMO+sY5RkkBgxKio=', 'Content-Length': '0'} 2024-04-12T03:29:57.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043193a114e02c755-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:57.591 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.591 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.591 DEBUG:boto:Method: GET 2024-04-12T03:29:57.591 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:29:57.591 DEBUG:boto:Data: 2024-04-12T03:29:57.591 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.591 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.591 DEBUG:boto:Params: {} 2024-04-12T03:29:57.592 DEBUG:boto:Token: None 2024-04-12T03:29:57.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4 2024-04-12T03:29:57.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YsqV6c/tzNCGHQYogAvdGOQt/+c= 2024-04-12T03:29:57.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YsqV6c/tzNCGHQYogAvdGOQt/+c=', 'Content-Length': '0'} 2024-04-12T03:29:57.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c2fb0170ce50a35-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.638 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:29:57.639 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:29:57.639 DEBUG:boto:Method: GET 2024-04-12T03:29:57.639 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:29:57.639 DEBUG:boto:Data: 2024-04-12T03:29:57.639 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.639 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.639 DEBUG:boto:Params: {} 2024-04-12T03:29:57.640 DEBUG:boto:Token: None 2024-04-12T03:29:57.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/obj4 2024-04-12T03:29:57.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YsqV6c/tzNCGHQYogAvdGOQt/+c= 2024-04-12T03:29:57.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YsqV6c/tzNCGHQYogAvdGOQt/+c=', 'Content-Length': '0'} 2024-04-12T03:29:57.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091ac0bfc25dfcbce-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.642 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.642 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.642 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.642 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.643 DEBUG:boto:Method: GET 2024-04-12T03:29:57.643 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.643 DEBUG:boto:Data: 2024-04-12T03:29:57.643 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.643 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.643 DEBUG:boto:Params: {} 2024-04-12T03:29:57.643 DEBUG:boto:Token: None 2024-04-12T03:29:57.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c828139c5daca21-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.646 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.646 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:29:57.646 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:29:57.646 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.646 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.646 DEBUG:boto:Method: GET 2024-04-12T03:29:57.647 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:29:57.647 DEBUG:boto:Data: 2024-04-12T03:29:57.647 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.647 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.647 DEBUG:boto:Params: {} 2024-04-12T03:29:57.647 DEBUG:boto:Token: None 2024-04-12T03:29:57.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-50/ 2024-04-12T03:29:57.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8= 2024-04-12T03:29:57.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JFlllzzry6GTBa3QwH+ae5iXrN8=', 'Content-Length': '0'} 2024-04-12T03:29:57.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29f5cfa0ea78243-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.650 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-50 zones={a2, a1} 2024-04-12T03:29:57.650 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-51 2024-04-12T03:29:57.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:57.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:57.651 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:57.707 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:57.707 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.745+0000 7f5b1c4fb780 1 bucket sync caught up with source: 2024-04-12T03:29:57.746 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.46.6, , 00000000001.34.6, , 00000000001.44.6, , , 00000000001.46.6, , , ] 2024-04-12T03:29:57.746 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.46.6, , 00000000001.34.6, , 00000000001.44.6, , , 00000000001.46.6, , , ] 2024-04-12T03:29:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.745+0000 7f5b1c4fb780 0 bucket checkpoint complete 2024-04-12T03:29:57.760 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-51 2024-04-12T03:29:57.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:57.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:57.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:57.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:57.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.848+0000 7f1888bbe780 1 bucket sync caught up with source: 2024-04-12T03:29:57.849 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.199.6, , 00000000001.423.6, , 00000000001.775.6, , , 00000000001.2179.6, , , ] 2024-04-12T03:29:57.849 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.199.6, , 00000000001.423.6, , 00000000001.775.6, , , 00000000001.2179.6, , , ] 2024-04-12T03:29:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:57.848+0000 7f1888bbe780 0 bucket checkpoint complete 2024-04-12T03:29:57.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-51 zones={a2, a1} 2024-04-12T03:29:57.861 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.862 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.862 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.862 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:57.862 DEBUG:boto:Data: 2024-04-12T03:29:57.862 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.862 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.862 DEBUG:boto:Params: {} 2024-04-12T03:29:57.862 DEBUG:boto:Token: None 2024-04-12T03:29:57.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/ 2024-04-12T03:29:57.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:093LPSYeep26ZAuQDVNHxN+h8+U= 2024-04-12T03:29:57.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:093LPSYeep26ZAuQDVNHxN+h8+U=', 'Content-Length': '0'} 2024-04-12T03:29:57.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc4e1db967ce590e-006618aab5-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.865 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.865 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.866 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.866 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:29:57.866 DEBUG:boto:Data: 2024-04-12T03:29:57.866 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.866 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.866 DEBUG:boto:Params: {} 2024-04-12T03:29:57.866 DEBUG:boto:Token: None 2024-04-12T03:29:57.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/ 2024-04-12T03:29:57.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:093LPSYeep26ZAuQDVNHxN+h8+U= 2024-04-12T03:29:57.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:093LPSYeep26ZAuQDVNHxN+h8+U=', 'Content-Length': '0'} 2024-04-12T03:29:57.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7f8a6012ad62ef4-006618aab5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:29:57.869 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.869 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.869 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.869 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.869 DEBUG:boto:Method: GET 2024-04-12T03:29:57.869 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.869 DEBUG:boto:Data: 2024-04-12T03:29:57.869 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.870 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.870 DEBUG:boto:Params: {} 2024-04-12T03:29:57.870 DEBUG:boto:Token: None 2024-04-12T03:29:57.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/?versions 2024-04-12T03:29:57.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw= 2024-04-12T03:29:57.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw=', 'Content-Length': '0'} 2024-04-12T03:29:57.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0ecf68ee6e143a-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.872 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:57.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:57.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:57.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:57.873 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:29:57.873 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.873 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.874 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.874 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.874 DEBUG:boto:Method: GET 2024-04-12T03:29:57.874 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.874 DEBUG:boto:Data: 2024-04-12T03:29:57.874 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.874 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.874 DEBUG:boto:Params: {} 2024-04-12T03:29:57.874 DEBUG:boto:Token: None 2024-04-12T03:29:57.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/?versions 2024-04-12T03:29:57.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw= 2024-04-12T03:29:57.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw=', 'Content-Length': '0'} 2024-04-12T03:29:57.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b6f3e2cfb21f2a-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.877 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:29:57.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:29:57.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:29:57.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:29:57.878 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.878 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.878 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.879 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.879 DEBUG:boto:Method: GET 2024-04-12T03:29:57.879 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.879 DEBUG:boto:Data: 2024-04-12T03:29:57.879 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.879 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.879 DEBUG:boto:Params: {} 2024-04-12T03:29:57.879 DEBUG:boto:Token: None 2024-04-12T03:29:57.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/?versions 2024-04-12T03:29:57.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw= 2024-04-12T03:29:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw=', 'Content-Length': '0'} 2024-04-12T03:29:57.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c95209e12cc85d07-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.882 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.882 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.882 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.883 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.883 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.883 DEBUG:boto:Method: GET 2024-04-12T03:29:57.883 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:29:57.883 DEBUG:boto:Data: 2024-04-12T03:29:57.883 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.883 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.883 DEBUG:boto:Params: {} 2024-04-12T03:29:57.883 DEBUG:boto:Token: None 2024-04-12T03:29:57.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/?versions 2024-04-12T03:29:57.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw= 2024-04-12T03:29:57.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmyINqk6MNrkrD80EGMgj5iQxvw=', 'Content-Length': '0'} 2024-04-12T03:29:57.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010cd0a1e4772db33-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.885 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.886 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.886 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.886 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.886 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.886 DEBUG:boto:Method: GET 2024-04-12T03:29:57.887 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.887 DEBUG:boto:Data: 2024-04-12T03:29:57.887 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.887 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.887 DEBUG:boto:Params: {} 2024-04-12T03:29:57.887 DEBUG:boto:Token: None 2024-04-12T03:29:57.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE= 2024-04-12T03:29:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE=', 'Content-Length': '0'} 2024-04-12T03:29:57.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ace512ff330bdac-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.889 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.889 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.889 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.889 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.890 DEBUG:boto:Method: GET 2024-04-12T03:29:57.890 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.890 DEBUG:boto:Data: 2024-04-12T03:29:57.890 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.890 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.890 DEBUG:boto:Params: {} 2024-04-12T03:29:57.890 DEBUG:boto:Token: None 2024-04-12T03:29:57.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE= 2024-04-12T03:29:57.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE=', 'Content-Length': '0'} 2024-04-12T03:29:57.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007bf0c670092945e-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.892 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.892 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.893 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.893 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.893 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.893 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.893 DEBUG:boto:Data: 2024-04-12T03:29:57.893 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.893 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.893 DEBUG:boto:Params: {} 2024-04-12T03:29:57.893 DEBUG:boto:Token: None 2024-04-12T03:29:57.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.894 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VfhTfnE6jtlveN+W8Cg8ikdmojM= 2024-04-12T03:29:57.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VfhTfnE6jtlveN+W8Cg8ikdmojM=', 'Content-Length': '0'} 2024-04-12T03:29:57.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a58bc5086050409-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.895 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.895 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.896 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.896 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.896 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.896 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.896 DEBUG:boto:Data: 2024-04-12T03:29:57.896 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.896 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.896 DEBUG:boto:Params: {} 2024-04-12T03:29:57.896 DEBUG:boto:Token: None 2024-04-12T03:29:57.897 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VfhTfnE6jtlveN+W8Cg8ikdmojM= 2024-04-12T03:29:57.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VfhTfnE6jtlveN+W8Cg8ikdmojM=', 'Content-Length': '0'} 2024-04-12T03:29:57.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000080386669ecb96ff-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.899 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.899 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.899 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.899 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.899 DEBUG:boto:Method: GET 2024-04-12T03:29:57.900 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.900 DEBUG:boto:Data: 2024-04-12T03:29:57.900 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.900 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.900 DEBUG:boto:Params: {} 2024-04-12T03:29:57.900 DEBUG:boto:Token: None 2024-04-12T03:29:57.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE= 2024-04-12T03:29:57.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE=', 'Content-Length': '0'} 2024-04-12T03:29:57.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000479749ad9c7064ae-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.902 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.903 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.903 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.903 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.903 DEBUG:boto:Method: GET 2024-04-12T03:29:57.903 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.903 DEBUG:boto:Data: 2024-04-12T03:29:57.903 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.903 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.904 DEBUG:boto:Params: {} 2024-04-12T03:29:57.904 DEBUG:boto:Token: None 2024-04-12T03:29:57.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:29:57.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE= 2024-04-12T03:29:57.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zQcqJPAfUex2GFSZuOvJiom8bCE=', 'Content-Length': '0'} 2024-04-12T03:29:57.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000252a29f1d1a75db9-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.906 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.906 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.906 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.906 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.906 DEBUG:boto:Method: GET 2024-04-12T03:29:57.906 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.906 DEBUG:boto:Data: 2024-04-12T03:29:57.906 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.907 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.907 DEBUG:boto:Params: {} 2024-04-12T03:29:57.907 DEBUG:boto:Token: None 2024-04-12T03:29:57.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/ 2024-04-12T03:29:57.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxm8/hM0la1hx+DcED/m3poDhVg= 2024-04-12T03:29:57.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxm8/hM0la1hx+DcED/m3poDhVg=', 'Content-Length': '0'} 2024-04-12T03:29:57.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004927df178cb0d4ed-006618aab5-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.950 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.951 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:57.951 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:57.951 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.951 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.951 DEBUG:boto:Method: GET 2024-04-12T03:29:57.952 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:29:57.952 DEBUG:boto:Data: 2024-04-12T03:29:57.952 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.952 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.952 DEBUG:boto:Params: {} 2024-04-12T03:29:57.952 DEBUG:boto:Token: None 2024-04-12T03:29:57.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/ 2024-04-12T03:29:57.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxm8/hM0la1hx+DcED/m3poDhVg= 2024-04-12T03:29:57.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxm8/hM0la1hx+DcED/m3poDhVg=', 'Content-Length': '0'} 2024-04-12T03:29:57.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95abcc602d6ae8c-006618aab5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.955 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:57.956 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.956 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.956 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.956 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:29:57.956 DEBUG:boto:Data: 2024-04-12T03:29:57.956 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.956 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.956 DEBUG:boto:Params: {} 2024-04-12T03:29:57.956 DEBUG:boto:Token: None 2024-04-12T03:29:57.957 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1 2024-04-12T03:29:57.957 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nfU1owCUIdty8vd2WqQR90+gZf8= 2024-04-12T03:29:57.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nfU1owCUIdty8vd2WqQR90+gZf8=', 'Content-Length': '0'} 2024-04-12T03:29:57.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738647d09b923282-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.959 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.959 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.959 DEBUG:boto:Method: HEAD 2024-04-12T03:29:57.959 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:29:57.959 DEBUG:boto:Data: 2024-04-12T03:29:57.959 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.959 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.959 DEBUG:boto:Params: {} 2024-04-12T03:29:57.959 DEBUG:boto:Token: None 2024-04-12T03:29:57.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1 2024-04-12T03:29:57.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nfU1owCUIdty8vd2WqQR90+gZf8= 2024-04-12T03:29:57.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nfU1owCUIdty8vd2WqQR90+gZf8=', 'Content-Length': '0'} 2024-04-12T03:29:57.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea66b4ca503762f5-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:29:57.962 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.962 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.962 DEBUG:boto:Method: GET 2024-04-12T03:29:57.962 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:29:57.963 DEBUG:boto:Data: 2024-04-12T03:29:57.963 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:57.963 DEBUG:boto:Port: 8001 2024-04-12T03:29:57.963 DEBUG:boto:Params: {} 2024-04-12T03:29:57.963 DEBUG:boto:Token: None 2024-04-12T03:29:57.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1 2024-04-12T03:29:57.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1eSJVoy8Q8G3CfhQ6B/PGmPbRUM= 2024-04-12T03:29:57.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1eSJVoy8Q8G3CfhQ6B/PGmPbRUM=', 'Content-Length': '0'} 2024-04-12T03:29:57.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f9a2aa706667a2c-006618aab5-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:57.965 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:29:57.965 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:29:57.965 DEBUG:boto:Method: GET 2024-04-12T03:29:57.965 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:29:57.965 DEBUG:boto:Data: 2024-04-12T03:29:57.966 DEBUG:boto:Headers: {} 2024-04-12T03:29:57.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:57.966 DEBUG:boto:Port: 8000 2024-04-12T03:29:57.966 DEBUG:boto:Params: {} 2024-04-12T03:29:57.966 DEBUG:boto:Token: None 2024-04-12T03:29:57.966 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:57 GMT /dgbpxg-51/obj1 2024-04-12T03:29:57.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1eSJVoy8Q8G3CfhQ6B/PGmPbRUM= 2024-04-12T03:29:57.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1eSJVoy8Q8G3CfhQ6B/PGmPbRUM=', 'Content-Length': '0'} 2024-04-12T03:29:57.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002602ca95ea79fba5-006618aab5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.010 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.011 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.011 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.011 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.011 DEBUG:boto:Method: GET 2024-04-12T03:29:58.011 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.011 DEBUG:boto:Data: 2024-04-12T03:29:58.011 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.011 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.012 DEBUG:boto:Params: {} 2024-04-12T03:29:58.012 DEBUG:boto:Token: None 2024-04-12T03:29:58.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3496f9e67641e1-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.015 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.015 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.015 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.015 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.015 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.016 DEBUG:boto:Method: GET 2024-04-12T03:29:58.016 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:29:58.016 DEBUG:boto:Data: 2024-04-12T03:29:58.016 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.016 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.016 DEBUG:boto:Params: {} 2024-04-12T03:29:58.016 DEBUG:boto:Token: None 2024-04-12T03:29:58.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd27b1b5bd6cf3f2-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.019 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:58.019 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.019 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.019 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.019 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.019 DEBUG:boto:Method: GET 2024-04-12T03:29:58.019 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.019 DEBUG:boto:Data: 2024-04-12T03:29:58.020 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.020 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.020 DEBUG:boto:Params: {} 2024-04-12T03:29:58.020 DEBUG:boto:Token: None 2024-04-12T03:29:58.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM= 2024-04-12T03:29:58.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM=', 'Content-Length': '0'} 2024-04-12T03:29:58.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c9308fcc032693-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.062 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.062 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.063 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.063 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.063 DEBUG:boto:Method: GET 2024-04-12T03:29:58.063 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.063 DEBUG:boto:Data: 2024-04-12T03:29:58.063 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.063 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.063 DEBUG:boto:Params: {} 2024-04-12T03:29:58.063 DEBUG:boto:Token: None 2024-04-12T03:29:58.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM= 2024-04-12T03:29:58.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM=', 'Content-Length': '0'} 2024-04-12T03:29:58.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7815a8c3cbde220-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.065 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.066 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.066 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.066 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.066 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.066 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.066 DEBUG:boto:Data: 2024-04-12T03:29:58.066 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.066 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.066 DEBUG:boto:Params: {} 2024-04-12T03:29:58.067 DEBUG:boto:Token: None 2024-04-12T03:29:58.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/YbTkq/WaC3Usrcb8bBUE24NFA= 2024-04-12T03:29:58.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/YbTkq/WaC3Usrcb8bBUE24NFA=', 'Content-Length': '0'} 2024-04-12T03:29:58.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011b077ad6a40b472-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.069 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.069 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.069 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.069 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.069 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.069 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.069 DEBUG:boto:Data: 2024-04-12T03:29:58.069 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.069 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.070 DEBUG:boto:Params: {} 2024-04-12T03:29:58.070 DEBUG:boto:Token: None 2024-04-12T03:29:58.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/YbTkq/WaC3Usrcb8bBUE24NFA= 2024-04-12T03:29:58.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/YbTkq/WaC3Usrcb8bBUE24NFA=', 'Content-Length': '0'} 2024-04-12T03:29:58.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ce7a6cfc552b55d-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:58.072 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.072 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.072 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.072 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.072 DEBUG:boto:Method: GET 2024-04-12T03:29:58.072 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.073 DEBUG:boto:Data: 2024-04-12T03:29:58.073 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.073 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.073 DEBUG:boto:Params: {} 2024-04-12T03:29:58.073 DEBUG:boto:Token: None 2024-04-12T03:29:58.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM= 2024-04-12T03:29:58.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM=', 'Content-Length': '0'} 2024-04-12T03:29:58.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eaee8263f5addb1-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.075 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.075 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.075 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.075 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.075 DEBUG:boto:Method: GET 2024-04-12T03:29:58.075 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.076 DEBUG:boto:Data: 2024-04-12T03:29:58.076 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.076 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.076 DEBUG:boto:Params: {} 2024-04-12T03:29:58.076 DEBUG:boto:Token: None 2024-04-12T03:29:58.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:29:58.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM= 2024-04-12T03:29:58.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1puT4iJTZp2nEm6iRd3nnWSzamM=', 'Content-Length': '0'} 2024-04-12T03:29:58.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034b4490c7d7f3c78-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.078 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.078 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.078 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.078 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.078 DEBUG:boto:Method: GET 2024-04-12T03:29:58.078 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.079 DEBUG:boto:Data: 2024-04-12T03:29:58.079 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.079 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.079 DEBUG:boto:Params: {} 2024-04-12T03:29:58.079 DEBUG:boto:Token: None 2024-04-12T03:29:58.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ad0cfb73aac801-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.081 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.082 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.082 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.082 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.082 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.082 DEBUG:boto:Method: GET 2024-04-12T03:29:58.082 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.082 DEBUG:boto:Data: 2024-04-12T03:29:58.082 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.083 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.083 DEBUG:boto:Params: {} 2024-04-12T03:29:58.083 DEBUG:boto:Token: None 2024-04-12T03:29:58.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4e9593b945cfc5-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.085 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.086 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.086 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.086 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.086 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:29:58.086 DEBUG:boto:Data: 2024-04-12T03:29:58.086 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.086 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.087 DEBUG:boto:Params: {} 2024-04-12T03:29:58.087 DEBUG:boto:Token: None 2024-04-12T03:29:58.087 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2 2024-04-12T03:29:58.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UpslBuy3wc8M3bUQvKkczozEVIU= 2024-04-12T03:29:58.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UpslBuy3wc8M3bUQvKkczozEVIU=', 'Content-Length': '0'} 2024-04-12T03:29:58.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f793ab90217f5ff-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.089 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.089 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.089 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.089 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:29:58.089 DEBUG:boto:Data: 2024-04-12T03:29:58.089 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.089 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.089 DEBUG:boto:Params: {} 2024-04-12T03:29:58.089 DEBUG:boto:Token: None 2024-04-12T03:29:58.090 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2 2024-04-12T03:29:58.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UpslBuy3wc8M3bUQvKkczozEVIU= 2024-04-12T03:29:58.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UpslBuy3wc8M3bUQvKkczozEVIU=', 'Content-Length': '0'} 2024-04-12T03:29:58.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021720fa908fca774-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:29:58.092 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.092 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.092 DEBUG:boto:Method: GET 2024-04-12T03:29:58.092 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:29:58.092 DEBUG:boto:Data: 2024-04-12T03:29:58.092 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.092 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.092 DEBUG:boto:Params: {} 2024-04-12T03:29:58.092 DEBUG:boto:Token: None 2024-04-12T03:29:58.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2 2024-04-12T03:29:58.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oAlHgTTgOj4C5xKVWl15iG9zs+k= 2024-04-12T03:29:58.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oAlHgTTgOj4C5xKVWl15iG9zs+k=', 'Content-Length': '0'} 2024-04-12T03:29:58.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d930b936b6e852e-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.095 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:29:58.095 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:29:58.095 DEBUG:boto:Method: GET 2024-04-12T03:29:58.095 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:29:58.095 DEBUG:boto:Data: 2024-04-12T03:29:58.095 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.095 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.095 DEBUG:boto:Params: {} 2024-04-12T03:29:58.095 DEBUG:boto:Token: None 2024-04-12T03:29:58.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj2 2024-04-12T03:29:58.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oAlHgTTgOj4C5xKVWl15iG9zs+k= 2024-04-12T03:29:58.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oAlHgTTgOj4C5xKVWl15iG9zs+k=', 'Content-Length': '0'} 2024-04-12T03:29:58.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d10e7716d524a5c-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.097 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.097 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.097 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.098 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.098 DEBUG:boto:Method: GET 2024-04-12T03:29:58.098 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.098 DEBUG:boto:Data: 2024-04-12T03:29:58.098 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.098 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.098 DEBUG:boto:Params: {} 2024-04-12T03:29:58.098 DEBUG:boto:Token: None 2024-04-12T03:29:58.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a49c236d7b6b2b3-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.142 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.143 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.143 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.143 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.143 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.143 DEBUG:boto:Method: GET 2024-04-12T03:29:58.143 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:29:58.143 DEBUG:boto:Data: 2024-04-12T03:29:58.143 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.143 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.144 DEBUG:boto:Params: {} 2024-04-12T03:29:58.144 DEBUG:boto:Token: None 2024-04-12T03:29:58.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f738bb2781708d-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.147 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:58.147 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.148 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.148 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.148 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.148 DEBUG:boto:Method: GET 2024-04-12T03:29:58.148 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.148 DEBUG:boto:Data: 2024-04-12T03:29:58.148 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.148 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.148 DEBUG:boto:Params: {} 2024-04-12T03:29:58.149 DEBUG:boto:Token: None 2024-04-12T03:29:58.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg= 2024-04-12T03:29:58.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg=', 'Content-Length': '0'} 2024-04-12T03:29:58.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc2773f0d3bf48f9-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.152 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.152 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.152 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.152 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.152 DEBUG:boto:Method: GET 2024-04-12T03:29:58.152 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.152 DEBUG:boto:Data: 2024-04-12T03:29:58.152 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.152 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.152 DEBUG:boto:Params: {} 2024-04-12T03:29:58.153 DEBUG:boto:Token: None 2024-04-12T03:29:58.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg= 2024-04-12T03:29:58.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg=', 'Content-Length': '0'} 2024-04-12T03:29:58.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000029f332639317f52-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.155 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.155 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.155 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.155 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.155 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.155 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.156 DEBUG:boto:Data: 2024-04-12T03:29:58.156 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.156 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.156 DEBUG:boto:Params: {} 2024-04-12T03:29:58.156 DEBUG:boto:Token: None 2024-04-12T03:29:58.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:764bU6FIKCJn//VV/lUbHEeJtD4= 2024-04-12T03:29:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:764bU6FIKCJn//VV/lUbHEeJtD4=', 'Content-Length': '0'} 2024-04-12T03:29:58.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ba025454f5d6284-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.158 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.159 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.159 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.159 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.159 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.159 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.159 DEBUG:boto:Data: 2024-04-12T03:29:58.159 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.159 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.159 DEBUG:boto:Params: {} 2024-04-12T03:29:58.160 DEBUG:boto:Token: None 2024-04-12T03:29:58.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:764bU6FIKCJn//VV/lUbHEeJtD4= 2024-04-12T03:29:58.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:764bU6FIKCJn//VV/lUbHEeJtD4=', 'Content-Length': '0'} 2024-04-12T03:29:58.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3ab0bad366e9242-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:58.162 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.162 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.162 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.162 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.162 DEBUG:boto:Method: GET 2024-04-12T03:29:58.162 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.162 DEBUG:boto:Data: 2024-04-12T03:29:58.163 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.163 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.163 DEBUG:boto:Params: {} 2024-04-12T03:29:58.163 DEBUG:boto:Token: None 2024-04-12T03:29:58.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg= 2024-04-12T03:29:58.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg=', 'Content-Length': '0'} 2024-04-12T03:29:58.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007515209727ca70fc-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.165 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.165 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.165 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.165 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.165 DEBUG:boto:Method: GET 2024-04-12T03:29:58.165 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.166 DEBUG:boto:Data: 2024-04-12T03:29:58.166 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.166 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.166 DEBUG:boto:Params: {} 2024-04-12T03:29:58.166 DEBUG:boto:Token: None 2024-04-12T03:29:58.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:29:58.166 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg= 2024-04-12T03:29:58.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PQwPLYqK09ErFzyiR0splKhnSOg=', 'Content-Length': '0'} 2024-04-12T03:29:58.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e33a0e4f641e4b0b-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.168 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.168 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.168 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.168 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.168 DEBUG:boto:Method: GET 2024-04-12T03:29:58.169 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.169 DEBUG:boto:Data: 2024-04-12T03:29:58.169 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.169 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.169 DEBUG:boto:Params: {} 2024-04-12T03:29:58.169 DEBUG:boto:Token: None 2024-04-12T03:29:58.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d7341d7be0352c4-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.172 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.172 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.172 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.172 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.172 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.173 DEBUG:boto:Method: GET 2024-04-12T03:29:58.173 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.173 DEBUG:boto:Data: 2024-04-12T03:29:58.173 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.173 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.173 DEBUG:boto:Params: {} 2024-04-12T03:29:58.173 DEBUG:boto:Token: None 2024-04-12T03:29:58.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.174 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333401d8e9fb4b8b-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.176 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.176 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.176 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.176 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.176 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:29:58.177 DEBUG:boto:Data: 2024-04-12T03:29:58.177 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.177 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.177 DEBUG:boto:Params: {} 2024-04-12T03:29:58.177 DEBUG:boto:Token: None 2024-04-12T03:29:58.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3 2024-04-12T03:29:58.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f4akJUeLcXnTi08yBnNERe9qSLM= 2024-04-12T03:29:58.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f4akJUeLcXnTi08yBnNERe9qSLM=', 'Content-Length': '0'} 2024-04-12T03:29:58.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e052ed1f47532031-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.180 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.180 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.180 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.180 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:29:58.180 DEBUG:boto:Data: 2024-04-12T03:29:58.180 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.180 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.180 DEBUG:boto:Params: {} 2024-04-12T03:29:58.180 DEBUG:boto:Token: None 2024-04-12T03:29:58.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3 2024-04-12T03:29:58.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f4akJUeLcXnTi08yBnNERe9qSLM= 2024-04-12T03:29:58.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f4akJUeLcXnTi08yBnNERe9qSLM=', 'Content-Length': '0'} 2024-04-12T03:29:58.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf2f57d3fda3b093-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:29:58.182 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.183 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.183 DEBUG:boto:Method: GET 2024-04-12T03:29:58.183 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:29:58.183 DEBUG:boto:Data: 2024-04-12T03:29:58.183 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.183 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.183 DEBUG:boto:Params: {} 2024-04-12T03:29:58.183 DEBUG:boto:Token: None 2024-04-12T03:29:58.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3 2024-04-12T03:29:58.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xisv7zxG1cGaL3VOwADCTi6Bb08= 2024-04-12T03:29:58.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xisv7zxG1cGaL3VOwADCTi6Bb08=', 'Content-Length': '0'} 2024-04-12T03:29:58.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ed15e6f8bda043b-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.226 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:29:58.226 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:29:58.227 DEBUG:boto:Method: GET 2024-04-12T03:29:58.227 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:29:58.227 DEBUG:boto:Data: 2024-04-12T03:29:58.227 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.227 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.227 DEBUG:boto:Params: {} 2024-04-12T03:29:58.227 DEBUG:boto:Token: None 2024-04-12T03:29:58.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj3 2024-04-12T03:29:58.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xisv7zxG1cGaL3VOwADCTi6Bb08= 2024-04-12T03:29:58.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xisv7zxG1cGaL3VOwADCTi6Bb08=', 'Content-Length': '0'} 2024-04-12T03:29:58.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e20a9b74c870c5-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.230 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.230 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.230 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.230 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.230 DEBUG:boto:Method: GET 2024-04-12T03:29:58.230 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.230 DEBUG:boto:Data: 2024-04-12T03:29:58.231 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.231 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.231 DEBUG:boto:Params: {} 2024-04-12T03:29:58.231 DEBUG:boto:Token: None 2024-04-12T03:29:58.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f1fa00e2e3dd11-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.234 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.234 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.234 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.234 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.234 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.234 DEBUG:boto:Method: GET 2024-04-12T03:29:58.235 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:29:58.235 DEBUG:boto:Data: 2024-04-12T03:29:58.235 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.235 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.235 DEBUG:boto:Params: {} 2024-04-12T03:29:58.235 DEBUG:boto:Token: None 2024-04-12T03:29:58.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7e03c44c5d126a-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.237 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:58.238 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.238 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.238 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.238 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.239 DEBUG:boto:Method: GET 2024-04-12T03:29:58.239 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.239 DEBUG:boto:Data: 2024-04-12T03:29:58.239 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.239 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.239 DEBUG:boto:Params: {} 2024-04-12T03:29:58.240 DEBUG:boto:Token: None 2024-04-12T03:29:58.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4= 2024-04-12T03:29:58.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4=', 'Content-Length': '0'} 2024-04-12T03:29:58.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f627e31a90c203c9-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.242 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.242 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.242 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.242 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.242 DEBUG:boto:Method: GET 2024-04-12T03:29:58.242 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.243 DEBUG:boto:Data: 2024-04-12T03:29:58.243 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.243 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.243 DEBUG:boto:Params: {} 2024-04-12T03:29:58.243 DEBUG:boto:Token: None 2024-04-12T03:29:58.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4= 2024-04-12T03:29:58.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4=', 'Content-Length': '0'} 2024-04-12T03:29:58.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000540e099cfafb7b32-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.245 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.246 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.246 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.246 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.246 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.246 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.246 DEBUG:boto:Data: 2024-04-12T03:29:58.246 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.246 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.247 DEBUG:boto:Params: {} 2024-04-12T03:29:58.247 DEBUG:boto:Token: None 2024-04-12T03:29:58.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cKPEzeH3U52USOJC1x5X2dmA7zs= 2024-04-12T03:29:58.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cKPEzeH3U52USOJC1x5X2dmA7zs=', 'Content-Length': '0'} 2024-04-12T03:29:58.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ac70c88b8f52eb9-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.249 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.249 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.249 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.249 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.249 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.249 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.249 DEBUG:boto:Data: 2024-04-12T03:29:58.249 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.249 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.250 DEBUG:boto:Params: {} 2024-04-12T03:29:58.250 DEBUG:boto:Token: None 2024-04-12T03:29:58.250 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.250 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cKPEzeH3U52USOJC1x5X2dmA7zs= 2024-04-12T03:29:58.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cKPEzeH3U52USOJC1x5X2dmA7zs=', 'Content-Length': '0'} 2024-04-12T03:29:58.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d47fa556be84ba9b-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:58.252 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.252 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.252 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.252 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.252 DEBUG:boto:Method: GET 2024-04-12T03:29:58.252 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.252 DEBUG:boto:Data: 2024-04-12T03:29:58.253 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.253 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.253 DEBUG:boto:Params: {} 2024-04-12T03:29:58.253 DEBUG:boto:Token: None 2024-04-12T03:29:58.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4= 2024-04-12T03:29:58.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4=', 'Content-Length': '0'} 2024-04-12T03:29:58.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd8c0b0d94267dba-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.255 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.255 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.255 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.255 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.255 DEBUG:boto:Method: GET 2024-04-12T03:29:58.255 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.255 DEBUG:boto:Data: 2024-04-12T03:29:58.256 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.256 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.256 DEBUG:boto:Params: {} 2024-04-12T03:29:58.256 DEBUG:boto:Token: None 2024-04-12T03:29:58.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:29:58.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4= 2024-04-12T03:29:58.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fWUvFv4uZayQT5qVZXAwGitbfW4=', 'Content-Length': '0'} 2024-04-12T03:29:58.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020d444859c6cc40f-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.258 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.258 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.258 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.258 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.258 DEBUG:boto:Method: GET 2024-04-12T03:29:58.258 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.259 DEBUG:boto:Data: 2024-04-12T03:29:58.259 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.259 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.259 DEBUG:boto:Params: {} 2024-04-12T03:29:58.259 DEBUG:boto:Token: None 2024-04-12T03:29:58.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af89407bf9aed916-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.261 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.262 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.262 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.262 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.262 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.262 DEBUG:boto:Method: GET 2024-04-12T03:29:58.262 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.262 DEBUG:boto:Data: 2024-04-12T03:29:58.262 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.263 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.263 DEBUG:boto:Params: {} 2024-04-12T03:29:58.263 DEBUG:boto:Token: None 2024-04-12T03:29:58.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b284f79e252831f9-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.265 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.266 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.266 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.266 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.266 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:29:58.266 DEBUG:boto:Data: 2024-04-12T03:29:58.266 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.266 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.266 DEBUG:boto:Params: {} 2024-04-12T03:29:58.267 DEBUG:boto:Token: None 2024-04-12T03:29:58.267 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4 2024-04-12T03:29:58.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dkkNHrhnFeY7plJXT5ZMbBWpfuY= 2024-04-12T03:29:58.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dkkNHrhnFeY7plJXT5ZMbBWpfuY=', 'Content-Length': '0'} 2024-04-12T03:29:58.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1e653abc967f763-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.269 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.269 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.269 DEBUG:boto:Method: HEAD 2024-04-12T03:29:58.269 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:29:58.269 DEBUG:boto:Data: 2024-04-12T03:29:58.269 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.269 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.269 DEBUG:boto:Params: {} 2024-04-12T03:29:58.270 DEBUG:boto:Token: None 2024-04-12T03:29:58.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4 2024-04-12T03:29:58.270 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dkkNHrhnFeY7plJXT5ZMbBWpfuY= 2024-04-12T03:29:58.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dkkNHrhnFeY7plJXT5ZMbBWpfuY=', 'Content-Length': '0'} 2024-04-12T03:29:58.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfb6e00b9f860b5f-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:29:58.272 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.272 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.272 DEBUG:boto:Method: GET 2024-04-12T03:29:58.272 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:29:58.272 DEBUG:boto:Data: 2024-04-12T03:29:58.272 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.272 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.273 DEBUG:boto:Params: {} 2024-04-12T03:29:58.273 DEBUG:boto:Token: None 2024-04-12T03:29:58.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4 2024-04-12T03:29:58.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PJ/hSH5coms0BEiOmq/knqqakH4= 2024-04-12T03:29:58.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PJ/hSH5coms0BEiOmq/knqqakH4=', 'Content-Length': '0'} 2024-04-12T03:29:58.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fefb5ad25441820-006618aab6-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.275 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:29:58.275 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:29:58.275 DEBUG:boto:Method: GET 2024-04-12T03:29:58.275 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:29:58.275 DEBUG:boto:Data: 2024-04-12T03:29:58.275 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.275 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.275 DEBUG:boto:Params: {} 2024-04-12T03:29:58.276 DEBUG:boto:Token: None 2024-04-12T03:29:58.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/obj4 2024-04-12T03:29:58.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PJ/hSH5coms0BEiOmq/knqqakH4= 2024-04-12T03:29:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PJ/hSH5coms0BEiOmq/knqqakH4=', 'Content-Length': '0'} 2024-04-12T03:29:58.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cef24251c9fa8677-006618aab6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.277 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.278 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.278 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.278 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.278 DEBUG:boto:Method: GET 2024-04-12T03:29:58.278 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.278 DEBUG:boto:Data: 2024-04-12T03:29:58.278 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:29:58.278 DEBUG:boto:Port: 8001 2024-04-12T03:29:58.278 DEBUG:boto:Params: {} 2024-04-12T03:29:58.279 DEBUG:boto:Token: None 2024-04-12T03:29:58.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052eac2cc998c5042-006618aab6-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.322 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.323 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:29:58.323 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:29:58.323 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.323 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.323 DEBUG:boto:Method: GET 2024-04-12T03:29:58.323 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:29:58.323 DEBUG:boto:Data: 2024-04-12T03:29:58.324 DEBUG:boto:Headers: {} 2024-04-12T03:29:58.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:29:58.324 DEBUG:boto:Port: 8000 2024-04-12T03:29:58.324 DEBUG:boto:Params: {} 2024-04-12T03:29:58.324 DEBUG:boto:Token: None 2024-04-12T03:29:58.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:29:58 GMT /dgbpxg-51/ 2024-04-12T03:29:58.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8= 2024-04-12T03:29:58.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:29:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0lQRtE1nudGlAV5rWw3qCAx2ul8=', 'Content-Length': '0'} 2024-04-12T03:29:58.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343ff8e8d6dde73e-006618aab6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:29:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:29:58.327 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:29:58.327 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-51 zones={a2, a1} 2024-04-12T03:29:58.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:29:58.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:29:58.328 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:29:58.377 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:58.378 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:58.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dgbpxg-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:29:58.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dgbpxg-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:29:58.444 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dgbpxg-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:29:58.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:58.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:58.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:58.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:58.559 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:58.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:58.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:58.634 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:29:58.646 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892598.545064_929.1", 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:29:58.545064Z" 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.648 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:29:58.649 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892598.545064_929.1'} 2024-04-12T03:29:58.649 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:29:58.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:58.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:29:58.650 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:29:58.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:58.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:58.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:58.725+0000 7fa90a642780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:58.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:29:58.726+0000 7fa90a642780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:29:58.740 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:58.741 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:58.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:58.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892576.255008_916.1", 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:29:58.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:36.255008Z", 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:29:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:29:58Z" 2024-04-12T03:29:58.746 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:29:58.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892576.255008_916.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:36.255008Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:29:58Z" } 2024-04-12T03:29:58.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892576.255008_916.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:36.255008Z', 'realm_epoch': 2}}] 2024-04-12T03:29:58.747 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892598.545064_929.1'} 2024-04-12T03:29:58.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892576.255008_916.1'} 2024-04-12T03:29:58.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892598.545064_929.1 target=1_1712892576.255008_916.1 2024-04-12T03:30:03.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:03.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:03.749 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:03.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:03.834+0000 7f9e93579780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:03.835+0000 7f9e93579780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:03.850 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:30:03.850 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:30:03.850 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:30:03.850 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:30:03.850 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:30:03.851 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:03.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:03.853 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:03.854 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892598.545064_929.1", 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:29:58.545064Z", 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:03.855 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:30:03Z" 2024-04-12T03:30:03.856 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:30:03.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892598.545064_929.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:29:58.545064Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:30:03Z" } 2024-04-12T03:30:03.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892598.545064_929.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:29:58.545064Z', 'realm_epoch': 2}}] 2024-04-12T03:30:03.857 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892598.545064_929.1'} 2024-04-12T03:30:03.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892598.545064_929.1'} 2024-04-12T03:30:03.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:30:03.858 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:03.858 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:03.858 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.858 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:03.858 DEBUG:boto:Data: 2024-04-12T03:30:03.858 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.858 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.859 DEBUG:boto:Params: {} 2024-04-12T03:30:03.859 DEBUG:boto:Token: None 2024-04-12T03:30:03.859 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/ 2024-04-12T03:30:03.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc= 2024-04-12T03:30:03.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc=', 'Content-Length': '0'} 2024-04-12T03:30:03.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004efe296c64abd67c-006618aabb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.862 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:03.862 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:03.862 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.862 DEBUG:boto:Path: /dgbpxg-50/obj5 2024-04-12T03:30:03.862 DEBUG:boto:Data: 2024-04-12T03:30:03.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.862 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.862 DEBUG:boto:Params: {} 2024-04-12T03:30:03.863 DEBUG:boto:Token: None 2024-04-12T03:30:03.863 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/obj5 2024-04-12T03:30:03.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CWnrXXk1mUv5Fw0zsDlNbJiXQUA= 2024-04-12T03:30:03.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CWnrXXk1mUv5Fw0zsDlNbJiXQUA='} 2024-04-12T03:30:03.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d006b79f920db70-006618aabb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.867 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:03.867 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:03.867 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.867 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:03.867 DEBUG:boto:Data: 2024-04-12T03:30:03.867 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.867 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.867 DEBUG:boto:Params: {} 2024-04-12T03:30:03.867 DEBUG:boto:Token: None 2024-04-12T03:30:03.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/ 2024-04-12T03:30:03.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc= 2024-04-12T03:30:03.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc=', 'Content-Length': '0'} 2024-04-12T03:30:03.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2bc37353aabbe9d-006618aabb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.870 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:03.870 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:03.870 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.870 DEBUG:boto:Path: /dgbpxg-50/obj6 2024-04-12T03:30:03.870 DEBUG:boto:Data: 2024-04-12T03:30:03.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.871 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.871 DEBUG:boto:Params: {} 2024-04-12T03:30:03.871 DEBUG:boto:Token: None 2024-04-12T03:30:03.871 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/obj6 2024-04-12T03:30:03.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7IIU68lwsNJ5JRCW1zSStmfY/ag= 2024-04-12T03:30:03.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7IIU68lwsNJ5JRCW1zSStmfY/ag='} 2024-04-12T03:30:03.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e7526ec40c8a6c1-006618aabb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.915 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:03.915 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:03.915 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.915 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:03.915 DEBUG:boto:Data: 2024-04-12T03:30:03.915 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.916 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.916 DEBUG:boto:Params: {} 2024-04-12T03:30:03.916 DEBUG:boto:Token: None 2024-04-12T03:30:03.916 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/ 2024-04-12T03:30:03.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc= 2024-04-12T03:30:03.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc=', 'Content-Length': '0'} 2024-04-12T03:30:03.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13adc6559ce68d2-006618aabb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.918 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:03.919 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:03.919 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.919 DEBUG:boto:Path: /dgbpxg-50/obj7 2024-04-12T03:30:03.919 DEBUG:boto:Data: 2024-04-12T03:30:03.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.919 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.919 DEBUG:boto:Params: {} 2024-04-12T03:30:03.919 DEBUG:boto:Token: None 2024-04-12T03:30:03.919 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/obj7 2024-04-12T03:30:03.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A+Ye2lX5iOvolqWKQvgQdlyV01c= 2024-04-12T03:30:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A+Ye2lX5iOvolqWKQvgQdlyV01c='} 2024-04-12T03:30:03.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2232d9fde86821a-006618aabb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.924 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:03.924 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:03.924 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.924 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:03.924 DEBUG:boto:Data: 2024-04-12T03:30:03.924 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.924 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.924 DEBUG:boto:Params: {} 2024-04-12T03:30:03.925 DEBUG:boto:Token: None 2024-04-12T03:30:03.925 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/ 2024-04-12T03:30:03.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc= 2024-04-12T03:30:03.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B8IBeFlJD+ZCY1PHTBk8hMAr6Pc=', 'Content-Length': '0'} 2024-04-12T03:30:03.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000444aab217328e8d4-006618aabb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.931 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:03.931 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:03.931 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.931 DEBUG:boto:Path: /dgbpxg-50/obj8 2024-04-12T03:30:03.931 DEBUG:boto:Data: 2024-04-12T03:30:03.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:03.932 DEBUG:boto:Port: 8000 2024-04-12T03:30:03.932 DEBUG:boto:Params: {} 2024-04-12T03:30:03.932 DEBUG:boto:Token: None 2024-04-12T03:30:03.932 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-50/obj8 2024-04-12T03:30:03.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hAi7iH33XEW6j+eiZFOwT0PHiyw= 2024-04-12T03:30:03.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hAi7iH33XEW6j+eiZFOwT0PHiyw='} 2024-04-12T03:30:03.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cbb0d8d06037e10-006618aabb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.975 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:03.975 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:03.975 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.975 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:03.975 DEBUG:boto:Data: 2024-04-12T03:30:03.975 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:03.975 DEBUG:boto:Port: 8001 2024-04-12T03:30:03.975 DEBUG:boto:Params: {} 2024-04-12T03:30:03.976 DEBUG:boto:Token: None 2024-04-12T03:30:03.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-51/ 2024-04-12T03:30:03.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jgcrbKIcJ637rvcXd0qpGLXCshc= 2024-04-12T03:30:03.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jgcrbKIcJ637rvcXd0qpGLXCshc=', 'Content-Length': '0'} 2024-04-12T03:30:03.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bb7bbd7283c0149-006618aabb-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.979 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:03.979 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:03.979 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.979 DEBUG:boto:Path: /dgbpxg-51/obj5 2024-04-12T03:30:03.979 DEBUG:boto:Data: 2024-04-12T03:30:03.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:03.979 DEBUG:boto:Port: 8001 2024-04-12T03:30:03.979 DEBUG:boto:Params: {} 2024-04-12T03:30:03.980 DEBUG:boto:Token: None 2024-04-12T03:30:03.980 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-51/obj5 2024-04-12T03:30:03.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GkzFm+JtB6uSQlbc+ElPDY8OoKI= 2024-04-12T03:30:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GkzFm+JtB6uSQlbc+ElPDY8OoKI='} 2024-04-12T03:30:03.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ebef72abbafeb9f-006618aabb-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.984 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:03.984 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:03.984 DEBUG:boto:Method: HEAD 2024-04-12T03:30:03.984 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:03.984 DEBUG:boto:Data: 2024-04-12T03:30:03.984 DEBUG:boto:Headers: {} 2024-04-12T03:30:03.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:03.984 DEBUG:boto:Port: 8001 2024-04-12T03:30:03.985 DEBUG:boto:Params: {} 2024-04-12T03:30:03.985 DEBUG:boto:Token: None 2024-04-12T03:30:03.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-51/ 2024-04-12T03:30:03.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jgcrbKIcJ637rvcXd0qpGLXCshc= 2024-04-12T03:30:03.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jgcrbKIcJ637rvcXd0qpGLXCshc=', 'Content-Length': '0'} 2024-04-12T03:30:03.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f406996d6a6f2e98-006618aabb-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:03.987 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:03.987 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:03.987 DEBUG:boto:Method: PUT 2024-04-12T03:30:03.987 DEBUG:boto:Path: /dgbpxg-51/obj6 2024-04-12T03:30:03.988 DEBUG:boto:Data: 2024-04-12T03:30:03.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:03.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:03.988 DEBUG:boto:Port: 8001 2024-04-12T03:30:03.988 DEBUG:boto:Params: {} 2024-04-12T03:30:03.988 DEBUG:boto:Token: None 2024-04-12T03:30:03.988 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:03 GMT /dgbpxg-51/obj6 2024-04-12T03:30:03.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ln1c2HpOraPZYeAyBEug9K2p33o= 2024-04-12T03:30:03.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ln1c2HpOraPZYeAyBEug9K2p33o='} 2024-04-12T03:30:04.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aadfdf68f40c0bd0-006618aabb-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:04.031 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:04.031 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:04.031 DEBUG:boto:Method: HEAD 2024-04-12T03:30:04.031 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:04.031 DEBUG:boto:Data: 2024-04-12T03:30:04.031 DEBUG:boto:Headers: {} 2024-04-12T03:30:04.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:04.031 DEBUG:boto:Port: 8001 2024-04-12T03:30:04.032 DEBUG:boto:Params: {} 2024-04-12T03:30:04.032 DEBUG:boto:Token: None 2024-04-12T03:30:04.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:04 GMT /dgbpxg-51/ 2024-04-12T03:30:04.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H3NfsR3j95WPayGAvU4Y7j79s5w= 2024-04-12T03:30:04.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H3NfsR3j95WPayGAvU4Y7j79s5w=', 'Content-Length': '0'} 2024-04-12T03:30:04.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc415b411cc2175-006618aabc-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:04.034 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:04.034 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:04.034 DEBUG:boto:Method: PUT 2024-04-12T03:30:04.034 DEBUG:boto:Path: /dgbpxg-51/obj7 2024-04-12T03:30:04.035 DEBUG:boto:Data: 2024-04-12T03:30:04.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:04.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:04.035 DEBUG:boto:Port: 8001 2024-04-12T03:30:04.035 DEBUG:boto:Params: {} 2024-04-12T03:30:04.035 DEBUG:boto:Token: None 2024-04-12T03:30:04.035 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:04 GMT /dgbpxg-51/obj7 2024-04-12T03:30:04.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nz67GAv9tVjpAkYzclPlLENzi94= 2024-04-12T03:30:04.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nz67GAv9tVjpAkYzclPlLENzi94='} 2024-04-12T03:30:04.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be4791091fea999d-006618aabc-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:04.039 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:04.039 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:04.039 DEBUG:boto:Method: HEAD 2024-04-12T03:30:04.039 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:04.039 DEBUG:boto:Data: 2024-04-12T03:30:04.039 DEBUG:boto:Headers: {} 2024-04-12T03:30:04.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:04.039 DEBUG:boto:Port: 8001 2024-04-12T03:30:04.040 DEBUG:boto:Params: {} 2024-04-12T03:30:04.040 DEBUG:boto:Token: None 2024-04-12T03:30:04.040 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:04 GMT /dgbpxg-51/ 2024-04-12T03:30:04.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H3NfsR3j95WPayGAvU4Y7j79s5w= 2024-04-12T03:30:04.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H3NfsR3j95WPayGAvU4Y7j79s5w=', 'Content-Length': '0'} 2024-04-12T03:30:04.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006fcc20a5c5b7db4-006618aabc-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:04.042 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:04.042 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:04.042 DEBUG:boto:Method: PUT 2024-04-12T03:30:04.043 DEBUG:boto:Path: /dgbpxg-51/obj8 2024-04-12T03:30:04.043 DEBUG:boto:Data: 2024-04-12T03:30:04.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:30:04.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:04.043 DEBUG:boto:Port: 8001 2024-04-12T03:30:04.043 DEBUG:boto:Params: {} 2024-04-12T03:30:04.043 DEBUG:boto:Token: None 2024-04-12T03:30:04.043 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:30:04 GMT /dgbpxg-51/obj8 2024-04-12T03:30:04.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QP1a70ffL8fBWE3kCqCw6xfJ0Gg= 2024-04-12T03:30:04.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QP1a70ffL8fBWE3kCqCw6xfJ0Gg='} 2024-04-12T03:30:04.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020e74a385dd6c13b-006618aabc-192d-a2'), ('Date', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:04.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dgbpxg-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:30:04.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dgbpxg-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:30:04.047 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dgbpxg-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:30:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:04.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dgbpxg-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:30:04.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dgbpxg-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:30:04.162 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dgbpxg-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:30:04.211 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:04.211 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:04.277 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-50 2024-04-12T03:30:04.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:04.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:04.278 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:04.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:04.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:04.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:04.367+0000 7f46cec86780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:04.368 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.29.6, , 00000000001.36.6, , 00000000001.42.6, , , 00000000001.31.6, , , ] 2024-04-12T03:30:04.368 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.38.1, 00000000002.20.1, 00000000004.40.1, 00000000002.21.1, 00000000004.49.1, 00000000002.17.1, 00000000002.48.1, 00000000004.39.1, 00000000002.18.1, 00000000002.35.1, 00000000002.21.1] 2024-04-12T03:30:09.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:09.368+0000 7f46cec86780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:09.369 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.29.6, , 00000000001.36.6, , 00000000001.42.6, , , 00000000001.31.6, , , ] 2024-04-12T03:30:09.369 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.38.1, 00000000002.20.1, 00000000004.40.1, 00000000002.21.1, 00000000004.49.1, 00000000002.17.1, 00000000002.48.1, 00000000004.39.1, 00000000002.18.1, 00000000002.35.1, 00000000002.21.1] 2024-04-12T03:30:14.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:14.369+0000 7f46cec86780 1 bucket sync caught up with source: 2024-04-12T03:30:14.370 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.38.1, 00000000002.20.1, 00000000004.40.1, 00000000002.21.1, 00000000004.49.1, 00000000002.17.1, 00000000002.48.1, 00000000004.39.1, 00000000002.18.1, 00000000002.35.1, 00000000002.21.1] 2024-04-12T03:30:14.371 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.38.1, 00000000002.20.1, 00000000004.40.1, 00000000002.21.1, 00000000004.49.1, 00000000002.17.1, 00000000002.48.1, 00000000004.39.1, 00000000002.18.1, 00000000002.35.1, 00000000002.21.1] 2024-04-12T03:30:14.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:14.369+0000 7f46cec86780 0 bucket checkpoint complete 2024-04-12T03:30:14.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-50 2024-04-12T03:30:14.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:14.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:14.389 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:14.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:14.450 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:14.486+0000 7ff94329d780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:14.487 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.94.1, 00000000002.1568.1, 00000000003.1199.1, 00000000002.400.1, 00000000003.790.1, 00000000002.512.1, 00000000002.1029.1, 00000000003.97.1, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:14.487 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:19.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:19.487+0000 7ff94329d780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:19.488 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.94.1, 00000000002.1568.1, 00000000003.1199.1, 00000000002.400.1, 00000000003.790.1, 00000000002.512.1, 00000000002.1029.1, 00000000003.97.1, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:19.488 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:24.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:24.488+0000 7ff94329d780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:24.490 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.94.1, 00000000002.1568.1, 00000000003.1199.1, 00000000002.400.1, 00000000003.790.1, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:24.490 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:29.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:29.489+0000 7ff94329d780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:29.491 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.94.1, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000003.790.1, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:29.491 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:34.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:34.491+0000 7ff94329d780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:34.492 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.94.1, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000003.790.1, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:34.493 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:39.492+0000 7ff94329d780 1 bucket sync caught up with source: 2024-04-12T03:30:39.493 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:39.493 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.102.6, 00000000002.1568.1, 00000000004.1203.6, 00000000002.400.1, 00000000004.794.6, 00000000002.512.1, 00000000002.1029.1, 00000000004.103.6, 00000000002.839.1, 00000000002.1406.1, 00000000002.1571.1] 2024-04-12T03:30:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:39.492+0000 7ff94329d780 0 bucket checkpoint complete 2024-04-12T03:30:39.514 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-50 zones={a2, a1} 2024-04-12T03:30:39.514 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.514 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.515 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.515 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:39.515 DEBUG:boto:Data: 2024-04-12T03:30:39.515 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.515 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.515 DEBUG:boto:Params: {} 2024-04-12T03:30:39.515 DEBUG:boto:Token: None 2024-04-12T03:30:39.516 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y4prMHT17kiP5oAJc34hjNgEItw= 2024-04-12T03:30:39.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4prMHT17kiP5oAJc34hjNgEItw=', 'Content-Length': '0'} 2024-04-12T03:30:39.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c222a043d60835a1-006618aadf-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.519 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.519 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.519 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.519 DEBUG:boto:Path: /dgbpxg-50/ 2024-04-12T03:30:39.520 DEBUG:boto:Data: 2024-04-12T03:30:39.520 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.520 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.520 DEBUG:boto:Params: {} 2024-04-12T03:30:39.521 DEBUG:boto:Token: None 2024-04-12T03:30:39.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.521 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y4prMHT17kiP5oAJc34hjNgEItw= 2024-04-12T03:30:39.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4prMHT17kiP5oAJc34hjNgEItw=', 'Content-Length': '0'} 2024-04-12T03:30:39.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da8611bb45d78356-006618aadf-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:30:39.525 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.525 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.525 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.525 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.525 DEBUG:boto:Method: GET 2024-04-12T03:30:39.525 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.525 DEBUG:boto:Data: 2024-04-12T03:30:39.525 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.526 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.526 DEBUG:boto:Params: {} 2024-04-12T03:30:39.526 DEBUG:boto:Token: None 2024-04-12T03:30:39.526 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/?versions 2024-04-12T03:30:39.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A= 2024-04-12T03:30:39.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A=', 'Content-Length': '0'} 2024-04-12T03:30:39.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ae57ac8344f621-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.529 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:30:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:30:39.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:30:39.531 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.531 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.531 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.531 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.531 DEBUG:boto:Method: GET 2024-04-12T03:30:39.531 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.531 DEBUG:boto:Data: 2024-04-12T03:30:39.531 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.531 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.531 DEBUG:boto:Params: {} 2024-04-12T03:30:39.532 DEBUG:boto:Token: None 2024-04-12T03:30:39.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/?versions 2024-04-12T03:30:39.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A= 2024-04-12T03:30:39.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A=', 'Content-Length': '0'} 2024-04-12T03:30:39.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef45ac21513cadc0-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.534 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:30:39.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:30:39.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:30:39.536 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.536 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.536 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.536 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.537 DEBUG:boto:Method: GET 2024-04-12T03:30:39.537 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.537 DEBUG:boto:Data: 2024-04-12T03:30:39.537 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.537 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.537 DEBUG:boto:Params: {} 2024-04-12T03:30:39.537 DEBUG:boto:Token: None 2024-04-12T03:30:39.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/?versions 2024-04-12T03:30:39.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A= 2024-04-12T03:30:39.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A=', 'Content-Length': '0'} 2024-04-12T03:30:39.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724b48ed90165aa1-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.540 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.540 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.541 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.541 DEBUG:boto:path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.541 DEBUG:boto:auth_path=/dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.541 DEBUG:boto:Method: GET 2024-04-12T03:30:39.541 DEBUG:boto:Path: /dgbpxg-50/?versions&max-keys=999 2024-04-12T03:30:39.541 DEBUG:boto:Data: 2024-04-12T03:30:39.541 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.541 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.541 DEBUG:boto:Params: {} 2024-04-12T03:30:39.541 DEBUG:boto:Token: None 2024-04-12T03:30:39.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/?versions 2024-04-12T03:30:39.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A= 2024-04-12T03:30:39.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eHwx3fkXKPkrxU68JhBbdrCat4A=', 'Content-Length': '0'} 2024-04-12T03:30:39.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c7017ba148baa3b-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.544 DEBUG:boto:b'dgbpxg-50999falseobj1nulltrue2024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:39.545 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.545 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.545 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.545 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.546 DEBUG:boto:Method: GET 2024-04-12T03:30:39.546 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.546 DEBUG:boto:Data: 2024-04-12T03:30:39.546 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.546 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.546 DEBUG:boto:Params: {} 2024-04-12T03:30:39.546 DEBUG:boto:Token: None 2024-04-12T03:30:39.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8= 2024-04-12T03:30:39.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8=', 'Content-Length': '0'} 2024-04-12T03:30:39.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a272e425bff7d728-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.590 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.590 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.591 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.591 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.591 DEBUG:boto:Method: GET 2024-04-12T03:30:39.591 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.591 DEBUG:boto:Data: 2024-04-12T03:30:39.591 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.591 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.591 DEBUG:boto:Params: {} 2024-04-12T03:30:39.591 DEBUG:boto:Token: None 2024-04-12T03:30:39.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8= 2024-04-12T03:30:39.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8=', 'Content-Length': '0'} 2024-04-12T03:30:39.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c2554b89d2534f0-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.594 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.594 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.594 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.594 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.594 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.594 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.595 DEBUG:boto:Data: 2024-04-12T03:30:39.595 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.595 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.595 DEBUG:boto:Params: {} 2024-04-12T03:30:39.595 DEBUG:boto:Token: None 2024-04-12T03:30:39.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YnlVNO4VkHUReccExjgzhdbtMBI= 2024-04-12T03:30:39.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YnlVNO4VkHUReccExjgzhdbtMBI=', 'Content-Length': '0'} 2024-04-12T03:30:39.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018e0a24da3fd2fb7-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.597 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.597 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.597 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.597 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.597 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.597 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.597 DEBUG:boto:Data: 2024-04-12T03:30:39.598 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.598 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.598 DEBUG:boto:Params: {} 2024-04-12T03:30:39.598 DEBUG:boto:Token: None 2024-04-12T03:30:39.598 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YnlVNO4VkHUReccExjgzhdbtMBI= 2024-04-12T03:30:39.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YnlVNO4VkHUReccExjgzhdbtMBI=', 'Content-Length': '0'} 2024-04-12T03:30:39.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005684c9e8f5b5f7b1-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:39.601 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.601 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.601 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.601 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.601 DEBUG:boto:Method: GET 2024-04-12T03:30:39.601 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.602 DEBUG:boto:Data: 2024-04-12T03:30:39.602 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.602 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.602 DEBUG:boto:Params: {} 2024-04-12T03:30:39.603 DEBUG:boto:Token: None 2024-04-12T03:30:39.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8= 2024-04-12T03:30:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8=', 'Content-Length': '0'} 2024-04-12T03:30:39.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001d501b20770d6fb-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.605 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.605 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.605 DEBUG:boto:path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.605 DEBUG:boto:auth_path=/dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.605 DEBUG:boto:Method: GET 2024-04-12T03:30:39.605 DEBUG:boto:Path: /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.606 DEBUG:boto:Data: 2024-04-12T03:30:39.606 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.606 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.606 DEBUG:boto:Params: {} 2024-04-12T03:30:39.606 DEBUG:boto:Token: None 2024-04-12T03:30:39.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1?versionId=null 2024-04-12T03:30:39.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8= 2024-04-12T03:30:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYblgXGEhZzhS/BSOC+2Iom3dJ8=', 'Content-Length': '0'} 2024-04-12T03:30:39.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9139c8868c8da2b-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.608 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.608 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.608 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.609 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.609 DEBUG:boto:Method: GET 2024-04-12T03:30:39.609 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.609 DEBUG:boto:Data: 2024-04-12T03:30:39.609 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.609 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.609 DEBUG:boto:Params: {} 2024-04-12T03:30:39.609 DEBUG:boto:Token: None 2024-04-12T03:30:39.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16dbd9551bebed7-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.613 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.613 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.613 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.613 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.613 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.614 DEBUG:boto:Method: GET 2024-04-12T03:30:39.614 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.614 DEBUG:boto:Data: 2024-04-12T03:30:39.614 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.614 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.614 DEBUG:boto:Params: {} 2024-04-12T03:30:39.614 DEBUG:boto:Token: None 2024-04-12T03:30:39.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a99c7340163e006-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.617 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.617 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.617 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.617 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.618 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:30:39.618 DEBUG:boto:Data: 2024-04-12T03:30:39.618 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.618 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.618 DEBUG:boto:Params: {} 2024-04-12T03:30:39.618 DEBUG:boto:Token: None 2024-04-12T03:30:39.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1 2024-04-12T03:30:39.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3ZFMAJwBzbDIzugY8+cd/5iuPbE= 2024-04-12T03:30:39.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3ZFMAJwBzbDIzugY8+cd/5iuPbE=', 'Content-Length': '0'} 2024-04-12T03:30:39.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000405a5bacd7090df7-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.621 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.621 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.621 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.621 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:30:39.622 DEBUG:boto:Data: 2024-04-12T03:30:39.622 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.622 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.622 DEBUG:boto:Params: {} 2024-04-12T03:30:39.622 DEBUG:boto:Token: None 2024-04-12T03:30:39.622 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1 2024-04-12T03:30:39.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3ZFMAJwBzbDIzugY8+cd/5iuPbE= 2024-04-12T03:30:39.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3ZFMAJwBzbDIzugY8+cd/5iuPbE=', 'Content-Length': '0'} 2024-04-12T03:30:39.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f725d5614d246254-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:39.625 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.625 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.625 DEBUG:boto:Method: GET 2024-04-12T03:30:39.625 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:30:39.625 DEBUG:boto:Data: 2024-04-12T03:30:39.625 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.626 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.626 DEBUG:boto:Params: {} 2024-04-12T03:30:39.626 DEBUG:boto:Token: None 2024-04-12T03:30:39.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1 2024-04-12T03:30:39.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HRm7Ja8mFJeHW/ajuLS33wy9sfE= 2024-04-12T03:30:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HRm7Ja8mFJeHW/ajuLS33wy9sfE=', 'Content-Length': '0'} 2024-04-12T03:30:39.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9afda6689d5cec-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.628 DEBUG:boto:path=/dgbpxg-50/obj1 2024-04-12T03:30:39.628 DEBUG:boto:auth_path=/dgbpxg-50/obj1 2024-04-12T03:30:39.628 DEBUG:boto:Method: GET 2024-04-12T03:30:39.628 DEBUG:boto:Path: /dgbpxg-50/obj1 2024-04-12T03:30:39.628 DEBUG:boto:Data: 2024-04-12T03:30:39.628 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.629 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.629 DEBUG:boto:Params: {} 2024-04-12T03:30:39.629 DEBUG:boto:Token: None 2024-04-12T03:30:39.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj1 2024-04-12T03:30:39.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HRm7Ja8mFJeHW/ajuLS33wy9sfE= 2024-04-12T03:30:39.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HRm7Ja8mFJeHW/ajuLS33wy9sfE=', 'Content-Length': '0'} 2024-04-12T03:30:39.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056c79cba3ae81d05-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.631 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.631 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.631 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.631 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.631 DEBUG:boto:Method: GET 2024-04-12T03:30:39.631 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.632 DEBUG:boto:Data: 2024-04-12T03:30:39.632 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.632 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.632 DEBUG:boto:Params: {} 2024-04-12T03:30:39.632 DEBUG:boto:Token: None 2024-04-12T03:30:39.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7de9a786f68104e-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.634 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.635 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.635 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.635 DEBUG:boto:path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.635 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.635 DEBUG:boto:Method: GET 2024-04-12T03:30:39.635 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj1 2024-04-12T03:30:39.635 DEBUG:boto:Data: 2024-04-12T03:30:39.635 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.635 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.635 DEBUG:boto:Params: {} 2024-04-12T03:30:39.636 DEBUG:boto:Token: None 2024-04-12T03:30:39.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5c8ddbd958f55c-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.678 DEBUG:boto:b'dgbpxg-50obj11000falseobj12024-04-12T03:29:36.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:39.679 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.679 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.679 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.679 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.679 DEBUG:boto:Method: GET 2024-04-12T03:30:39.679 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.679 DEBUG:boto:Data: 2024-04-12T03:30:39.679 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.680 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.680 DEBUG:boto:Params: {} 2024-04-12T03:30:39.680 DEBUG:boto:Token: None 2024-04-12T03:30:39.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw= 2024-04-12T03:30:39.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw=', 'Content-Length': '0'} 2024-04-12T03:30:39.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab0ffb6da04a72ce-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.682 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.682 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.683 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.683 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.683 DEBUG:boto:Method: GET 2024-04-12T03:30:39.683 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.683 DEBUG:boto:Data: 2024-04-12T03:30:39.683 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.683 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.684 DEBUG:boto:Params: {} 2024-04-12T03:30:39.684 DEBUG:boto:Token: None 2024-04-12T03:30:39.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw= 2024-04-12T03:30:39.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw=', 'Content-Length': '0'} 2024-04-12T03:30:39.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cb442704a1b8795-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.686 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.686 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.686 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.687 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.687 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.687 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.687 DEBUG:boto:Data: 2024-04-12T03:30:39.687 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.687 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.687 DEBUG:boto:Params: {} 2024-04-12T03:30:39.687 DEBUG:boto:Token: None 2024-04-12T03:30:39.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YM9pbFGcbA2NMoUNiLo/9c5zSk4= 2024-04-12T03:30:39.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YM9pbFGcbA2NMoUNiLo/9c5zSk4=', 'Content-Length': '0'} 2024-04-12T03:30:39.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007d52fbb5b0cc6bc-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.689 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.689 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.689 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.690 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.690 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.690 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.690 DEBUG:boto:Data: 2024-04-12T03:30:39.690 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.690 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.690 DEBUG:boto:Params: {} 2024-04-12T03:30:39.690 DEBUG:boto:Token: None 2024-04-12T03:30:39.690 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YM9pbFGcbA2NMoUNiLo/9c5zSk4= 2024-04-12T03:30:39.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YM9pbFGcbA2NMoUNiLo/9c5zSk4=', 'Content-Length': '0'} 2024-04-12T03:30:39.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b5c1a1849fbdb5c-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:39.693 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.693 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.693 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.693 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.693 DEBUG:boto:Method: GET 2024-04-12T03:30:39.693 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.693 DEBUG:boto:Data: 2024-04-12T03:30:39.693 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.693 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.693 DEBUG:boto:Params: {} 2024-04-12T03:30:39.694 DEBUG:boto:Token: None 2024-04-12T03:30:39.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.694 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw= 2024-04-12T03:30:39.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw=', 'Content-Length': '0'} 2024-04-12T03:30:39.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fa7959587fe787c-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.696 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.696 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.696 DEBUG:boto:path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.696 DEBUG:boto:auth_path=/dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.696 DEBUG:boto:Method: GET 2024-04-12T03:30:39.696 DEBUG:boto:Path: /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.696 DEBUG:boto:Data: 2024-04-12T03:30:39.697 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.697 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.697 DEBUG:boto:Params: {} 2024-04-12T03:30:39.697 DEBUG:boto:Token: None 2024-04-12T03:30:39.697 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2?versionId=null 2024-04-12T03:30:39.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw= 2024-04-12T03:30:39.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hMjYzKetOC2kEIU7bX04Bwu1nLw=', 'Content-Length': '0'} 2024-04-12T03:30:39.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009854d3fa7100ddb4-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.699 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.699 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.699 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.700 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.700 DEBUG:boto:Method: GET 2024-04-12T03:30:39.700 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.700 DEBUG:boto:Data: 2024-04-12T03:30:39.700 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.700 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.700 DEBUG:boto:Params: {} 2024-04-12T03:30:39.700 DEBUG:boto:Token: None 2024-04-12T03:30:39.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000947e7bce25ec9767-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.703 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.703 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.703 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.704 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.704 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.704 DEBUG:boto:Method: GET 2024-04-12T03:30:39.704 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.704 DEBUG:boto:Data: 2024-04-12T03:30:39.704 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.704 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.704 DEBUG:boto:Params: {} 2024-04-12T03:30:39.704 DEBUG:boto:Token: None 2024-04-12T03:30:39.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c744236977d495e1-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.707 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.707 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.708 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.708 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.708 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:30:39.708 DEBUG:boto:Data: 2024-04-12T03:30:39.708 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.708 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.708 DEBUG:boto:Params: {} 2024-04-12T03:30:39.708 DEBUG:boto:Token: None 2024-04-12T03:30:39.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2 2024-04-12T03:30:39.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bUKGDbDdBNFFXwObqVYlQKFBIbA= 2024-04-12T03:30:39.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bUKGDbDdBNFFXwObqVYlQKFBIbA=', 'Content-Length': '0'} 2024-04-12T03:30:39.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054f7e3cd693b5fa7-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.711 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.711 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.711 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.712 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:30:39.712 DEBUG:boto:Data: 2024-04-12T03:30:39.712 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.712 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.712 DEBUG:boto:Params: {} 2024-04-12T03:30:39.712 DEBUG:boto:Token: None 2024-04-12T03:30:39.712 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2 2024-04-12T03:30:39.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bUKGDbDdBNFFXwObqVYlQKFBIbA= 2024-04-12T03:30:39.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bUKGDbDdBNFFXwObqVYlQKFBIbA=', 'Content-Length': '0'} 2024-04-12T03:30:39.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b972b9f95063cac-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:39.714 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.714 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.715 DEBUG:boto:Method: GET 2024-04-12T03:30:39.715 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:30:39.715 DEBUG:boto:Data: 2024-04-12T03:30:39.715 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.715 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.715 DEBUG:boto:Params: {} 2024-04-12T03:30:39.715 DEBUG:boto:Token: None 2024-04-12T03:30:39.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2 2024-04-12T03:30:39.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:af7xU4ZsZ7qsp6KxP0Aw3WQBqXI= 2024-04-12T03:30:39.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:af7xU4ZsZ7qsp6KxP0Aw3WQBqXI=', 'Content-Length': '0'} 2024-04-12T03:30:39.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f489ea2042b01bdd-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.758 DEBUG:boto:path=/dgbpxg-50/obj2 2024-04-12T03:30:39.759 DEBUG:boto:auth_path=/dgbpxg-50/obj2 2024-04-12T03:30:39.759 DEBUG:boto:Method: GET 2024-04-12T03:30:39.759 DEBUG:boto:Path: /dgbpxg-50/obj2 2024-04-12T03:30:39.759 DEBUG:boto:Data: 2024-04-12T03:30:39.759 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.759 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.759 DEBUG:boto:Params: {} 2024-04-12T03:30:39.759 DEBUG:boto:Token: None 2024-04-12T03:30:39.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj2 2024-04-12T03:30:39.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:af7xU4ZsZ7qsp6KxP0Aw3WQBqXI= 2024-04-12T03:30:39.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:af7xU4ZsZ7qsp6KxP0Aw3WQBqXI=', 'Content-Length': '0'} 2024-04-12T03:30:39.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8c51898e492c5a7-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.762 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.762 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.762 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.762 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.762 DEBUG:boto:Method: GET 2024-04-12T03:30:39.762 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.762 DEBUG:boto:Data: 2024-04-12T03:30:39.762 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.763 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.763 DEBUG:boto:Params: {} 2024-04-12T03:30:39.763 DEBUG:boto:Token: None 2024-04-12T03:30:39.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478becb3c893965e-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.765 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.766 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.766 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.766 DEBUG:boto:path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.766 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.766 DEBUG:boto:Method: GET 2024-04-12T03:30:39.766 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj2 2024-04-12T03:30:39.766 DEBUG:boto:Data: 2024-04-12T03:30:39.766 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.767 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.767 DEBUG:boto:Params: {} 2024-04-12T03:30:39.767 DEBUG:boto:Token: None 2024-04-12T03:30:39.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003063fd1f276ebce4-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.769 DEBUG:boto:b'dgbpxg-50obj21000falseobj22024-04-12T03:29:36.293Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:39.770 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.770 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.770 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.770 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.770 DEBUG:boto:Method: GET 2024-04-12T03:30:39.770 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.770 DEBUG:boto:Data: 2024-04-12T03:30:39.770 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.771 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.771 DEBUG:boto:Params: {} 2024-04-12T03:30:39.771 DEBUG:boto:Token: None 2024-04-12T03:30:39.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw= 2024-04-12T03:30:39.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw=', 'Content-Length': '0'} 2024-04-12T03:30:39.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bfd1481f8a7c81e-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.773 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.773 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.773 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.773 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.773 DEBUG:boto:Method: GET 2024-04-12T03:30:39.773 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.773 DEBUG:boto:Data: 2024-04-12T03:30:39.774 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.774 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.774 DEBUG:boto:Params: {} 2024-04-12T03:30:39.774 DEBUG:boto:Token: None 2024-04-12T03:30:39.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.774 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw= 2024-04-12T03:30:39.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw=', 'Content-Length': '0'} 2024-04-12T03:30:39.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000165277fe273e439d-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.776 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.776 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.776 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.776 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.777 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.777 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.777 DEBUG:boto:Data: 2024-04-12T03:30:39.777 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.777 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.777 DEBUG:boto:Params: {} 2024-04-12T03:30:39.777 DEBUG:boto:Token: None 2024-04-12T03:30:39.777 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.778 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cy0dIl6VOEC9zUgB/yfs/HOevUQ= 2024-04-12T03:30:39.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cy0dIl6VOEC9zUgB/yfs/HOevUQ=', 'Content-Length': '0'} 2024-04-12T03:30:39.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008edd90f0ad39551a-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.779 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.780 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.780 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.780 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.780 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.780 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.780 DEBUG:boto:Data: 2024-04-12T03:30:39.780 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.780 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.780 DEBUG:boto:Params: {} 2024-04-12T03:30:39.781 DEBUG:boto:Token: None 2024-04-12T03:30:39.781 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.781 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cy0dIl6VOEC9zUgB/yfs/HOevUQ= 2024-04-12T03:30:39.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cy0dIl6VOEC9zUgB/yfs/HOevUQ=', 'Content-Length': '0'} 2024-04-12T03:30:39.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf417b3deb01bf00-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:39.783 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.783 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.783 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.783 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.783 DEBUG:boto:Method: GET 2024-04-12T03:30:39.783 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.784 DEBUG:boto:Data: 2024-04-12T03:30:39.784 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.784 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.784 DEBUG:boto:Params: {} 2024-04-12T03:30:39.784 DEBUG:boto:Token: None 2024-04-12T03:30:39.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw= 2024-04-12T03:30:39.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw=', 'Content-Length': '0'} 2024-04-12T03:30:39.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158a103ae03739cc-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.787 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.787 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.787 DEBUG:boto:path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.787 DEBUG:boto:auth_path=/dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.787 DEBUG:boto:Method: GET 2024-04-12T03:30:39.787 DEBUG:boto:Path: /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.787 DEBUG:boto:Data: 2024-04-12T03:30:39.787 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.787 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.787 DEBUG:boto:Params: {} 2024-04-12T03:30:39.788 DEBUG:boto:Token: None 2024-04-12T03:30:39.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3?versionId=null 2024-04-12T03:30:39.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw= 2024-04-12T03:30:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:by9WzCX6tjSS9VwBkoc9PJ1qqJw=', 'Content-Length': '0'} 2024-04-12T03:30:39.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d1d365e3d4526dd-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.790 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.790 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.790 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.790 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.790 DEBUG:boto:Method: GET 2024-04-12T03:30:39.790 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.790 DEBUG:boto:Data: 2024-04-12T03:30:39.791 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.791 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.791 DEBUG:boto:Params: {} 2024-04-12T03:30:39.791 DEBUG:boto:Token: None 2024-04-12T03:30:39.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbcd563efc808cbf-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.793 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.794 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.794 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.795 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.795 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.795 DEBUG:boto:Method: GET 2024-04-12T03:30:39.795 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.795 DEBUG:boto:Data: 2024-04-12T03:30:39.795 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.795 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.795 DEBUG:boto:Params: {} 2024-04-12T03:30:39.795 DEBUG:boto:Token: None 2024-04-12T03:30:39.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dc6cedfd76950d-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.798 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.798 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.798 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.798 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.799 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:30:39.799 DEBUG:boto:Data: 2024-04-12T03:30:39.799 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.799 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.799 DEBUG:boto:Params: {} 2024-04-12T03:30:39.799 DEBUG:boto:Token: None 2024-04-12T03:30:39.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3 2024-04-12T03:30:39.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zp75cb2RzByjVihmZtOXvQfZ8wI= 2024-04-12T03:30:39.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zp75cb2RzByjVihmZtOXvQfZ8wI=', 'Content-Length': '0'} 2024-04-12T03:30:39.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a498b3cdb828d7ec-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.801 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.801 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.801 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.801 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:30:39.801 DEBUG:boto:Data: 2024-04-12T03:30:39.801 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.802 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.802 DEBUG:boto:Params: {} 2024-04-12T03:30:39.802 DEBUG:boto:Token: None 2024-04-12T03:30:39.802 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3 2024-04-12T03:30:39.802 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zp75cb2RzByjVihmZtOXvQfZ8wI= 2024-04-12T03:30:39.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zp75cb2RzByjVihmZtOXvQfZ8wI=', 'Content-Length': '0'} 2024-04-12T03:30:39.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000339e88b70bc9e224-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:39.804 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.804 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.804 DEBUG:boto:Method: GET 2024-04-12T03:30:39.804 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:30:39.804 DEBUG:boto:Data: 2024-04-12T03:30:39.804 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.805 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.805 DEBUG:boto:Params: {} 2024-04-12T03:30:39.805 DEBUG:boto:Token: None 2024-04-12T03:30:39.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3 2024-04-12T03:30:39.805 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aIX0VmzC9SAW9mkMcVGinE89bk= 2024-04-12T03:30:39.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aIX0VmzC9SAW9mkMcVGinE89bk=', 'Content-Length': '0'} 2024-04-12T03:30:39.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b8ff400217a2e29-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.807 DEBUG:boto:path=/dgbpxg-50/obj3 2024-04-12T03:30:39.807 DEBUG:boto:auth_path=/dgbpxg-50/obj3 2024-04-12T03:30:39.807 DEBUG:boto:Method: GET 2024-04-12T03:30:39.807 DEBUG:boto:Path: /dgbpxg-50/obj3 2024-04-12T03:30:39.807 DEBUG:boto:Data: 2024-04-12T03:30:39.807 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.808 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.808 DEBUG:boto:Params: {} 2024-04-12T03:30:39.808 DEBUG:boto:Token: None 2024-04-12T03:30:39.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj3 2024-04-12T03:30:39.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aIX0VmzC9SAW9mkMcVGinE89bk= 2024-04-12T03:30:39.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aIX0VmzC9SAW9mkMcVGinE89bk=', 'Content-Length': '0'} 2024-04-12T03:30:39.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000383c4c1f9c62dc95-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.810 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.810 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.810 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.810 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.810 DEBUG:boto:Method: GET 2024-04-12T03:30:39.810 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.810 DEBUG:boto:Data: 2024-04-12T03:30:39.810 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.811 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.811 DEBUG:boto:Params: {} 2024-04-12T03:30:39.811 DEBUG:boto:Token: None 2024-04-12T03:30:39.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ceb63726fef07f8-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.858 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.859 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.859 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.859 DEBUG:boto:path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.859 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.859 DEBUG:boto:Method: GET 2024-04-12T03:30:39.860 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj3 2024-04-12T03:30:39.860 DEBUG:boto:Data: 2024-04-12T03:30:39.860 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.860 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.860 DEBUG:boto:Params: {} 2024-04-12T03:30:39.860 DEBUG:boto:Token: None 2024-04-12T03:30:39.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046990fcbbb550255-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.863 DEBUG:boto:b'dgbpxg-50obj31000falseobj32024-04-12T03:29:36.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:39.863 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.863 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.864 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.864 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.864 DEBUG:boto:Method: GET 2024-04-12T03:30:39.864 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.864 DEBUG:boto:Data: 2024-04-12T03:30:39.864 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.864 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.864 DEBUG:boto:Params: {} 2024-04-12T03:30:39.864 DEBUG:boto:Token: None 2024-04-12T03:30:39.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA= 2024-04-12T03:30:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA=', 'Content-Length': '0'} 2024-04-12T03:30:39.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a9ad302aabc4747-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.867 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.867 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.867 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.867 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.867 DEBUG:boto:Method: GET 2024-04-12T03:30:39.867 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.867 DEBUG:boto:Data: 2024-04-12T03:30:39.867 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.867 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.867 DEBUG:boto:Params: {} 2024-04-12T03:30:39.868 DEBUG:boto:Token: None 2024-04-12T03:30:39.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA= 2024-04-12T03:30:39.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA=', 'Content-Length': '0'} 2024-04-12T03:30:39.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea1738880ca7b190-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.870 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.870 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.870 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.870 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.870 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.870 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.870 DEBUG:boto:Data: 2024-04-12T03:30:39.871 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.871 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.871 DEBUG:boto:Params: {} 2024-04-12T03:30:39.871 DEBUG:boto:Token: None 2024-04-12T03:30:39.871 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eKT2Fp3sxHWrJVfvogFz8HrHy5Q= 2024-04-12T03:30:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eKT2Fp3sxHWrJVfvogFz8HrHy5Q=', 'Content-Length': '0'} 2024-04-12T03:30:39.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7a0f254d08a030e-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.873 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.873 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.873 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.873 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.873 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.874 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.874 DEBUG:boto:Data: 2024-04-12T03:30:39.874 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.874 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.874 DEBUG:boto:Params: {} 2024-04-12T03:30:39.874 DEBUG:boto:Token: None 2024-04-12T03:30:39.874 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.874 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eKT2Fp3sxHWrJVfvogFz8HrHy5Q= 2024-04-12T03:30:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eKT2Fp3sxHWrJVfvogFz8HrHy5Q=', 'Content-Length': '0'} 2024-04-12T03:30:39.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd14f456a11a9969-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:39.876 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.876 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.876 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.876 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.876 DEBUG:boto:Method: GET 2024-04-12T03:30:39.877 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.877 DEBUG:boto:Data: 2024-04-12T03:30:39.877 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.877 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.877 DEBUG:boto:Params: {} 2024-04-12T03:30:39.877 DEBUG:boto:Token: None 2024-04-12T03:30:39.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA= 2024-04-12T03:30:39.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA=', 'Content-Length': '0'} 2024-04-12T03:30:39.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9f3d5d446eafcfa-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.879 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.879 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.880 DEBUG:boto:path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.880 DEBUG:boto:auth_path=/dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.880 DEBUG:boto:Method: GET 2024-04-12T03:30:39.880 DEBUG:boto:Path: /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.880 DEBUG:boto:Data: 2024-04-12T03:30:39.880 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.880 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.880 DEBUG:boto:Params: {} 2024-04-12T03:30:39.880 DEBUG:boto:Token: None 2024-04-12T03:30:39.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4?versionId=null 2024-04-12T03:30:39.881 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA= 2024-04-12T03:30:39.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pzyBi00aBQmQW03oyb4RE9zrtiA=', 'Content-Length': '0'} 2024-04-12T03:30:39.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002952928032d686ae-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.883 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.883 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.883 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.883 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.883 DEBUG:boto:Method: GET 2024-04-12T03:30:39.883 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.883 DEBUG:boto:Data: 2024-04-12T03:30:39.883 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.884 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.884 DEBUG:boto:Params: {} 2024-04-12T03:30:39.884 DEBUG:boto:Token: None 2024-04-12T03:30:39.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f452cdff010a5e-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.926 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.927 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.927 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.927 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.927 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.927 DEBUG:boto:Method: GET 2024-04-12T03:30:39.927 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.928 DEBUG:boto:Data: 2024-04-12T03:30:39.928 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.928 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.928 DEBUG:boto:Params: {} 2024-04-12T03:30:39.928 DEBUG:boto:Token: None 2024-04-12T03:30:39.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670f3cc5168211e6-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.931 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.931 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.931 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.931 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.931 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:30:39.931 DEBUG:boto:Data: 2024-04-12T03:30:39.932 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.932 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.932 DEBUG:boto:Params: {} 2024-04-12T03:30:39.932 DEBUG:boto:Token: None 2024-04-12T03:30:39.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4 2024-04-12T03:30:39.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JCLqqgzocR/Cv2hJH92PW7S+i5A= 2024-04-12T03:30:39.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JCLqqgzocR/Cv2hJH92PW7S+i5A=', 'Content-Length': '0'} 2024-04-12T03:30:39.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e6ef1bbd1c5d7ae-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.934 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.934 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.935 DEBUG:boto:Method: HEAD 2024-04-12T03:30:39.935 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:30:39.935 DEBUG:boto:Data: 2024-04-12T03:30:39.935 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.935 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.935 DEBUG:boto:Params: {} 2024-04-12T03:30:39.935 DEBUG:boto:Token: None 2024-04-12T03:30:39.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4 2024-04-12T03:30:39.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JCLqqgzocR/Cv2hJH92PW7S+i5A= 2024-04-12T03:30:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JCLqqgzocR/Cv2hJH92PW7S+i5A=', 'Content-Length': '0'} 2024-04-12T03:30:39.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079809faf70e6bc1a-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:39.937 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.937 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.937 DEBUG:boto:Method: GET 2024-04-12T03:30:39.937 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:30:39.938 DEBUG:boto:Data: 2024-04-12T03:30:39.938 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.938 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.938 DEBUG:boto:Params: {} 2024-04-12T03:30:39.938 DEBUG:boto:Token: None 2024-04-12T03:30:39.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4 2024-04-12T03:30:39.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JESiuv5atc7m85wsveYm57wOM9w= 2024-04-12T03:30:39.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JESiuv5atc7m85wsveYm57wOM9w=', 'Content-Length': '0'} 2024-04-12T03:30:39.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d70087936dd592-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.940 DEBUG:boto:path=/dgbpxg-50/obj4 2024-04-12T03:30:39.940 DEBUG:boto:auth_path=/dgbpxg-50/obj4 2024-04-12T03:30:39.941 DEBUG:boto:Method: GET 2024-04-12T03:30:39.941 DEBUG:boto:Path: /dgbpxg-50/obj4 2024-04-12T03:30:39.941 DEBUG:boto:Data: 2024-04-12T03:30:39.941 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.941 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.941 DEBUG:boto:Params: {} 2024-04-12T03:30:39.941 DEBUG:boto:Token: None 2024-04-12T03:30:39.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj4 2024-04-12T03:30:39.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JESiuv5atc7m85wsveYm57wOM9w= 2024-04-12T03:30:39.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JESiuv5atc7m85wsveYm57wOM9w=', 'Content-Length': '0'} 2024-04-12T03:30:39.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b219662d7aa39586-006618aadf-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.986 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.987 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.987 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.987 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.987 DEBUG:boto:Method: GET 2024-04-12T03:30:39.987 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.987 DEBUG:boto:Data: 2024-04-12T03:30:39.987 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.988 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.988 DEBUG:boto:Params: {} 2024-04-12T03:30:39.988 DEBUG:boto:Token: None 2024-04-12T03:30:39.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000169c8bf6a4dff198-006618aadf-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.991 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.991 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:39.991 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:39.992 DEBUG:boto:path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.992 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.992 DEBUG:boto:Method: GET 2024-04-12T03:30:39.992 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj4 2024-04-12T03:30:39.992 DEBUG:boto:Data: 2024-04-12T03:30:39.992 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:39.992 DEBUG:boto:Port: 8000 2024-04-12T03:30:39.992 DEBUG:boto:Params: {} 2024-04-12T03:30:39.993 DEBUG:boto:Token: None 2024-04-12T03:30:39.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/ 2024-04-12T03:30:39.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY= 2024-04-12T03:30:39.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:47S/VdrWR4l/iIALqyHF80L2UYY=', 'Content-Length': '0'} 2024-04-12T03:30:39.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc2b6b991ccd1a7-006618aadf-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.995 DEBUG:boto:b'dgbpxg-50obj41000falseobj42024-04-12T03:29:36.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:39.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:39.996 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:39.996 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:39.996 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:39.996 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:39.996 DEBUG:boto:Method: GET 2024-04-12T03:30:39.996 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:39.996 DEBUG:boto:Data: 2024-04-12T03:30:39.997 DEBUG:boto:Headers: {} 2024-04-12T03:30:39.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:39.997 DEBUG:boto:Port: 8001 2024-04-12T03:30:39.997 DEBUG:boto:Params: {} 2024-04-12T03:30:39.997 DEBUG:boto:Token: None 2024-04-12T03:30:39.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:39 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:39.997 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m1ewWTHzizLZcjxnUJ7CKaTQ54o= 2024-04-12T03:30:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m1ewWTHzizLZcjxnUJ7CKaTQ54o=', 'Content-Length': '0'} 2024-04-12T03:30:39.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e8b8d193d8d440a-006618aadf-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:39.999 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.000 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.000 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.000 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.000 DEBUG:boto:Method: GET 2024-04-12T03:30:40.000 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.000 DEBUG:boto:Data: 2024-04-12T03:30:40.000 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.000 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.000 DEBUG:boto:Params: {} 2024-04-12T03:30:40.000 DEBUG:boto:Token: None 2024-04-12T03:30:40.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo= 2024-04-12T03:30:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo=', 'Content-Length': '0'} 2024-04-12T03:30:40.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006982694d3b968bb9-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.003 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.003 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.003 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.003 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.003 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.003 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.003 DEBUG:boto:Data: 2024-04-12T03:30:40.003 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.004 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.004 DEBUG:boto:Params: {} 2024-04-12T03:30:40.004 DEBUG:boto:Token: None 2024-04-12T03:30:40.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RAz587IG8DVDChKvUftx353XRZY= 2024-04-12T03:30:40.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RAz587IG8DVDChKvUftx353XRZY=', 'Content-Length': '0'} 2024-04-12T03:30:40.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dcfb98f4a026efc-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.006 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.006 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.006 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.006 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.006 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.006 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.007 DEBUG:boto:Data: 2024-04-12T03:30:40.007 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.007 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.007 DEBUG:boto:Params: {} 2024-04-12T03:30:40.007 DEBUG:boto:Token: None 2024-04-12T03:30:40.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RAz587IG8DVDChKvUftx353XRZY= 2024-04-12T03:30:40.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RAz587IG8DVDChKvUftx353XRZY=', 'Content-Length': '0'} 2024-04-12T03:30:40.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b797fe87d931f4d2-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:40.009 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.010 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.010 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.010 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.010 DEBUG:boto:Method: GET 2024-04-12T03:30:40.010 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.010 DEBUG:boto:Data: 2024-04-12T03:30:40.010 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.011 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.011 DEBUG:boto:Params: {} 2024-04-12T03:30:40.011 DEBUG:boto:Token: None 2024-04-12T03:30:40.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo= 2024-04-12T03:30:40.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo=', 'Content-Length': '0'} 2024-04-12T03:30:40.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b72eb4dd5a10699d-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.054 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.054 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.055 DEBUG:boto:path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.055 DEBUG:boto:auth_path=/dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.055 DEBUG:boto:Method: GET 2024-04-12T03:30:40.055 DEBUG:boto:Path: /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.055 DEBUG:boto:Data: 2024-04-12T03:30:40.055 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.055 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.055 DEBUG:boto:Params: {} 2024-04-12T03:30:40.056 DEBUG:boto:Token: None 2024-04-12T03:30:40.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5?versionId=null 2024-04-12T03:30:40.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo= 2024-04-12T03:30:40.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZRoT4ifp7T6d87mEI//e1IR26xo=', 'Content-Length': '0'} 2024-04-12T03:30:40.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035dbe4d417a95bd2-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.058 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.058 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.058 DEBUG:boto:path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.058 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.059 DEBUG:boto:Method: GET 2024-04-12T03:30:40.059 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.059 DEBUG:boto:Data: 2024-04-12T03:30:40.059 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.059 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.059 DEBUG:boto:Params: {} 2024-04-12T03:30:40.059 DEBUG:boto:Token: None 2024-04-12T03:30:40.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c8dd87ad7029cf-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.062 DEBUG:boto:b'dgbpxg-50obj51000falseobj52024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.062 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.063 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.063 DEBUG:boto:path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.063 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.063 DEBUG:boto:Method: GET 2024-04-12T03:30:40.063 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.063 DEBUG:boto:Data: 2024-04-12T03:30:40.063 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.063 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.063 DEBUG:boto:Params: {} 2024-04-12T03:30:40.064 DEBUG:boto:Token: None 2024-04-12T03:30:40.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000901456fb8e663687-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.066 DEBUG:boto:b'dgbpxg-50obj51000falseobj52024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.067 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.067 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.067 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.067 DEBUG:boto:Path: /dgbpxg-50/obj5 2024-04-12T03:30:40.067 DEBUG:boto:Data: 2024-04-12T03:30:40.067 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.067 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.067 DEBUG:boto:Params: {} 2024-04-12T03:30:40.068 DEBUG:boto:Token: None 2024-04-12T03:30:40.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5 2024-04-12T03:30:40.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T7cvaOetS552mjzb7I7xnJ4tqvM= 2024-04-12T03:30:40.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T7cvaOetS552mjzb7I7xnJ4tqvM=', 'Content-Length': '0'} 2024-04-12T03:30:40.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a41b04bf433fd1ba-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.070 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.070 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.070 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.070 DEBUG:boto:Path: /dgbpxg-50/obj5 2024-04-12T03:30:40.070 DEBUG:boto:Data: 2024-04-12T03:30:40.070 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.070 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.070 DEBUG:boto:Params: {} 2024-04-12T03:30:40.071 DEBUG:boto:Token: None 2024-04-12T03:30:40.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5 2024-04-12T03:30:40.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T7cvaOetS552mjzb7I7xnJ4tqvM= 2024-04-12T03:30:40.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T7cvaOetS552mjzb7I7xnJ4tqvM=', 'Content-Length': '0'} 2024-04-12T03:30:40.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e70c736ba8ad1eb4-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:40.073 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.073 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.073 DEBUG:boto:Method: GET 2024-04-12T03:30:40.073 DEBUG:boto:Path: /dgbpxg-50/obj5 2024-04-12T03:30:40.073 DEBUG:boto:Data: 2024-04-12T03:30:40.073 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.074 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.074 DEBUG:boto:Params: {} 2024-04-12T03:30:40.074 DEBUG:boto:Token: None 2024-04-12T03:30:40.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5 2024-04-12T03:30:40.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:INOLjqx8lPkA2SIu+Qc5JugtLlo= 2024-04-12T03:30:40.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:INOLjqx8lPkA2SIu+Qc5JugtLlo=', 'Content-Length': '0'} 2024-04-12T03:30:40.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ef39fac5d2337d3-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.076 DEBUG:boto:path=/dgbpxg-50/obj5 2024-04-12T03:30:40.076 DEBUG:boto:auth_path=/dgbpxg-50/obj5 2024-04-12T03:30:40.076 DEBUG:boto:Method: GET 2024-04-12T03:30:40.076 DEBUG:boto:Path: /dgbpxg-50/obj5 2024-04-12T03:30:40.076 DEBUG:boto:Data: 2024-04-12T03:30:40.077 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.077 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.077 DEBUG:boto:Params: {} 2024-04-12T03:30:40.077 DEBUG:boto:Token: None 2024-04-12T03:30:40.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj5 2024-04-12T03:30:40.077 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:INOLjqx8lPkA2SIu+Qc5JugtLlo= 2024-04-12T03:30:40.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:INOLjqx8lPkA2SIu+Qc5JugtLlo=', 'Content-Length': '0'} 2024-04-12T03:30:40.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083ddacbc0d7a7525-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.079 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.079 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.080 DEBUG:boto:path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.080 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.080 DEBUG:boto:Method: GET 2024-04-12T03:30:40.080 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.080 DEBUG:boto:Data: 2024-04-12T03:30:40.080 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.080 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.080 DEBUG:boto:Params: {} 2024-04-12T03:30:40.080 DEBUG:boto:Token: None 2024-04-12T03:30:40.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801245335eb368a6-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.083 DEBUG:boto:b'dgbpxg-50obj51000falseobj52024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.083 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.083 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.084 DEBUG:boto:path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.084 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.084 DEBUG:boto:Method: GET 2024-04-12T03:30:40.084 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj5 2024-04-12T03:30:40.084 DEBUG:boto:Data: 2024-04-12T03:30:40.084 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.084 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.084 DEBUG:boto:Params: {} 2024-04-12T03:30:40.084 DEBUG:boto:Token: None 2024-04-12T03:30:40.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072b87e5691fc866b-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.087 DEBUG:boto:b'dgbpxg-50obj51000falseobj52024-04-12T03:30:03.864Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:40.088 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.088 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.088 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.088 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.088 DEBUG:boto:Method: GET 2024-04-12T03:30:40.088 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.088 DEBUG:boto:Data: 2024-04-12T03:30:40.088 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.088 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.088 DEBUG:boto:Params: {} 2024-04-12T03:30:40.089 DEBUG:boto:Token: None 2024-04-12T03:30:40.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s= 2024-04-12T03:30:40.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s=', 'Content-Length': '0'} 2024-04-12T03:30:40.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094f2493489816aff-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.091 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.091 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.091 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.091 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.092 DEBUG:boto:Method: GET 2024-04-12T03:30:40.092 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.092 DEBUG:boto:Data: 2024-04-12T03:30:40.092 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.092 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.092 DEBUG:boto:Params: {} 2024-04-12T03:30:40.092 DEBUG:boto:Token: None 2024-04-12T03:30:40.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s= 2024-04-12T03:30:40.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s=', 'Content-Length': '0'} 2024-04-12T03:30:40.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c229a88cf14df5c2-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.094 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.094 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.095 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.095 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.095 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.095 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.095 DEBUG:boto:Data: 2024-04-12T03:30:40.095 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.095 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.095 DEBUG:boto:Params: {} 2024-04-12T03:30:40.095 DEBUG:boto:Token: None 2024-04-12T03:30:40.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68591caWOKzNpcx2r5L0js+i3a0= 2024-04-12T03:30:40.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68591caWOKzNpcx2r5L0js+i3a0=', 'Content-Length': '0'} 2024-04-12T03:30:40.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003513c868774976e-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.097 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.098 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.098 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.098 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.098 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.098 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.098 DEBUG:boto:Data: 2024-04-12T03:30:40.098 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.098 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.099 DEBUG:boto:Params: {} 2024-04-12T03:30:40.099 DEBUG:boto:Token: None 2024-04-12T03:30:40.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68591caWOKzNpcx2r5L0js+i3a0= 2024-04-12T03:30:40.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68591caWOKzNpcx2r5L0js+i3a0=', 'Content-Length': '0'} 2024-04-12T03:30:40.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da5e071abcd44f48-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:40.101 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.101 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.101 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.101 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.102 DEBUG:boto:Method: GET 2024-04-12T03:30:40.102 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.102 DEBUG:boto:Data: 2024-04-12T03:30:40.102 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.102 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.102 DEBUG:boto:Params: {} 2024-04-12T03:30:40.102 DEBUG:boto:Token: None 2024-04-12T03:30:40.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s= 2024-04-12T03:30:40.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s=', 'Content-Length': '0'} 2024-04-12T03:30:40.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e6b2aee81f4d211-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.146 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.147 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.147 DEBUG:boto:path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.147 DEBUG:boto:auth_path=/dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.147 DEBUG:boto:Method: GET 2024-04-12T03:30:40.147 DEBUG:boto:Path: /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.147 DEBUG:boto:Data: 2024-04-12T03:30:40.147 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.147 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.148 DEBUG:boto:Params: {} 2024-04-12T03:30:40.148 DEBUG:boto:Token: None 2024-04-12T03:30:40.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6?versionId=null 2024-04-12T03:30:40.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s= 2024-04-12T03:30:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AgWK33fJblOQE1vtJoZHSk80O1s=', 'Content-Length': '0'} 2024-04-12T03:30:40.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014b61443a900bce7-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.150 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.150 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.150 DEBUG:boto:path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.151 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.151 DEBUG:boto:Method: GET 2024-04-12T03:30:40.151 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.151 DEBUG:boto:Data: 2024-04-12T03:30:40.151 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.151 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.151 DEBUG:boto:Params: {} 2024-04-12T03:30:40.151 DEBUG:boto:Token: None 2024-04-12T03:30:40.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871b4ca90620f9fa-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.154 DEBUG:boto:b'dgbpxg-50obj61000falseobj62024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.154 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.154 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.154 DEBUG:boto:path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.155 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.155 DEBUG:boto:Method: GET 2024-04-12T03:30:40.155 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.155 DEBUG:boto:Data: 2024-04-12T03:30:40.155 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.155 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.155 DEBUG:boto:Params: {} 2024-04-12T03:30:40.155 DEBUG:boto:Token: None 2024-04-12T03:30:40.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ae5b07dafbc347-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.158 DEBUG:boto:b'dgbpxg-50obj61000falseobj62024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.158 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.158 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.158 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.158 DEBUG:boto:Path: /dgbpxg-50/obj6 2024-04-12T03:30:40.158 DEBUG:boto:Data: 2024-04-12T03:30:40.159 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.159 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.159 DEBUG:boto:Params: {} 2024-04-12T03:30:40.159 DEBUG:boto:Token: None 2024-04-12T03:30:40.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6 2024-04-12T03:30:40.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:koCsxpk8YRaVROSOuEZyd4EnJpo= 2024-04-12T03:30:40.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:koCsxpk8YRaVROSOuEZyd4EnJpo=', 'Content-Length': '0'} 2024-04-12T03:30:40.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000524241962eb81254-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.161 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.161 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.161 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.161 DEBUG:boto:Path: /dgbpxg-50/obj6 2024-04-12T03:30:40.161 DEBUG:boto:Data: 2024-04-12T03:30:40.161 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.161 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.162 DEBUG:boto:Params: {} 2024-04-12T03:30:40.162 DEBUG:boto:Token: None 2024-04-12T03:30:40.162 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6 2024-04-12T03:30:40.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:koCsxpk8YRaVROSOuEZyd4EnJpo= 2024-04-12T03:30:40.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:koCsxpk8YRaVROSOuEZyd4EnJpo=', 'Content-Length': '0'} 2024-04-12T03:30:40.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ef84441feb1e27e-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:40.164 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.164 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.164 DEBUG:boto:Method: GET 2024-04-12T03:30:40.164 DEBUG:boto:Path: /dgbpxg-50/obj6 2024-04-12T03:30:40.164 DEBUG:boto:Data: 2024-04-12T03:30:40.164 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.164 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.165 DEBUG:boto:Params: {} 2024-04-12T03:30:40.165 DEBUG:boto:Token: None 2024-04-12T03:30:40.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6 2024-04-12T03:30:40.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gY+JXY1CLLSHV96i7wsMIyu2jHg= 2024-04-12T03:30:40.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY+JXY1CLLSHV96i7wsMIyu2jHg=', 'Content-Length': '0'} 2024-04-12T03:30:40.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055d97a381a695ad7-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.167 DEBUG:boto:path=/dgbpxg-50/obj6 2024-04-12T03:30:40.167 DEBUG:boto:auth_path=/dgbpxg-50/obj6 2024-04-12T03:30:40.167 DEBUG:boto:Method: GET 2024-04-12T03:30:40.167 DEBUG:boto:Path: /dgbpxg-50/obj6 2024-04-12T03:30:40.168 DEBUG:boto:Data: 2024-04-12T03:30:40.168 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.168 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.168 DEBUG:boto:Params: {} 2024-04-12T03:30:40.168 DEBUG:boto:Token: None 2024-04-12T03:30:40.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj6 2024-04-12T03:30:40.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gY+JXY1CLLSHV96i7wsMIyu2jHg= 2024-04-12T03:30:40.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY+JXY1CLLSHV96i7wsMIyu2jHg=', 'Content-Length': '0'} 2024-04-12T03:30:40.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c035a52903090e7f-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.170 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.170 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.170 DEBUG:boto:path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.170 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.170 DEBUG:boto:Method: GET 2024-04-12T03:30:40.171 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.171 DEBUG:boto:Data: 2024-04-12T03:30:40.171 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.171 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.171 DEBUG:boto:Params: {} 2024-04-12T03:30:40.171 DEBUG:boto:Token: None 2024-04-12T03:30:40.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.171 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ecb88cdcdd7b941-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.173 DEBUG:boto:b'dgbpxg-50obj61000falseobj62024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.174 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.174 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.174 DEBUG:boto:path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.174 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.174 DEBUG:boto:Method: GET 2024-04-12T03:30:40.174 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj6 2024-04-12T03:30:40.174 DEBUG:boto:Data: 2024-04-12T03:30:40.174 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.175 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.175 DEBUG:boto:Params: {} 2024-04-12T03:30:40.175 DEBUG:boto:Token: None 2024-04-12T03:30:40.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbd8306bed63c31-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.177 DEBUG:boto:b'dgbpxg-50obj61000falseobj62024-04-12T03:30:03.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:40.177 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.177 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.178 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.178 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.178 DEBUG:boto:Method: GET 2024-04-12T03:30:40.178 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.178 DEBUG:boto:Data: 2024-04-12T03:30:40.178 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.178 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.178 DEBUG:boto:Params: {} 2024-04-12T03:30:40.178 DEBUG:boto:Token: None 2024-04-12T03:30:40.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk= 2024-04-12T03:30:40.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk=', 'Content-Length': '0'} 2024-04-12T03:30:40.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000417004ac3cc6a76f-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.181 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.181 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.181 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.181 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.181 DEBUG:boto:Method: GET 2024-04-12T03:30:40.181 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.181 DEBUG:boto:Data: 2024-04-12T03:30:40.182 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.182 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.182 DEBUG:boto:Params: {} 2024-04-12T03:30:40.182 DEBUG:boto:Token: None 2024-04-12T03:30:40.182 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk= 2024-04-12T03:30:40.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk=', 'Content-Length': '0'} 2024-04-12T03:30:40.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e1b534dcf87b123-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.184 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.184 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.184 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.184 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.184 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.185 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.185 DEBUG:boto:Data: 2024-04-12T03:30:40.185 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.185 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.185 DEBUG:boto:Params: {} 2024-04-12T03:30:40.185 DEBUG:boto:Token: None 2024-04-12T03:30:40.185 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.185 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NDhpn+/k0WReGGUd5WQBWASLMsk= 2024-04-12T03:30:40.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NDhpn+/k0WReGGUd5WQBWASLMsk=', 'Content-Length': '0'} 2024-04-12T03:30:40.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e81cdfb35ed502f6-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.187 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.187 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.187 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.188 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.188 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.188 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.188 DEBUG:boto:Data: 2024-04-12T03:30:40.188 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.188 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.188 DEBUG:boto:Params: {} 2024-04-12T03:30:40.188 DEBUG:boto:Token: None 2024-04-12T03:30:40.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NDhpn+/k0WReGGUd5WQBWASLMsk= 2024-04-12T03:30:40.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NDhpn+/k0WReGGUd5WQBWASLMsk=', 'Content-Length': '0'} 2024-04-12T03:30:40.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1d31e2c2f7aa46e-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:40.190 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.191 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.191 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.191 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.191 DEBUG:boto:Method: GET 2024-04-12T03:30:40.191 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.191 DEBUG:boto:Data: 2024-04-12T03:30:40.191 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.191 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.191 DEBUG:boto:Params: {} 2024-04-12T03:30:40.191 DEBUG:boto:Token: None 2024-04-12T03:30:40.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk= 2024-04-12T03:30:40.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk=', 'Content-Length': '0'} 2024-04-12T03:30:40.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074bee91af079f2cc-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.238 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.239 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.239 DEBUG:boto:path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.239 DEBUG:boto:auth_path=/dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.239 DEBUG:boto:Method: GET 2024-04-12T03:30:40.239 DEBUG:boto:Path: /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.239 DEBUG:boto:Data: 2024-04-12T03:30:40.239 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.240 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.240 DEBUG:boto:Params: {} 2024-04-12T03:30:40.240 DEBUG:boto:Token: None 2024-04-12T03:30:40.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7?versionId=null 2024-04-12T03:30:40.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk= 2024-04-12T03:30:40.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7wmnNDJOVCWTUfZo2fiNmB6Wbk=', 'Content-Length': '0'} 2024-04-12T03:30:40.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005544eeb3ec147a17-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.242 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.242 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.242 DEBUG:boto:path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.243 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.243 DEBUG:boto:Method: GET 2024-04-12T03:30:40.243 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.243 DEBUG:boto:Data: 2024-04-12T03:30:40.243 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.243 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.243 DEBUG:boto:Params: {} 2024-04-12T03:30:40.243 DEBUG:boto:Token: None 2024-04-12T03:30:40.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4c9286451d4333e-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.246 DEBUG:boto:b'dgbpxg-50obj71000falseobj72024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.246 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.246 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.246 DEBUG:boto:path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.246 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.247 DEBUG:boto:Method: GET 2024-04-12T03:30:40.247 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.247 DEBUG:boto:Data: 2024-04-12T03:30:40.247 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.247 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.247 DEBUG:boto:Params: {} 2024-04-12T03:30:40.247 DEBUG:boto:Token: None 2024-04-12T03:30:40.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09e11a8d0bd0625-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.250 DEBUG:boto:b'dgbpxg-50obj71000falseobj72024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.250 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.250 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.251 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.251 DEBUG:boto:Path: /dgbpxg-50/obj7 2024-04-12T03:30:40.251 DEBUG:boto:Data: 2024-04-12T03:30:40.251 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.251 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.251 DEBUG:boto:Params: {} 2024-04-12T03:30:40.251 DEBUG:boto:Token: None 2024-04-12T03:30:40.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7 2024-04-12T03:30:40.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xftPt8RV5SpYPJ3X+DTIXgi4ihU= 2024-04-12T03:30:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xftPt8RV5SpYPJ3X+DTIXgi4ihU=', 'Content-Length': '0'} 2024-04-12T03:30:40.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9e52563be626ae6-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.253 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.253 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.253 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.253 DEBUG:boto:Path: /dgbpxg-50/obj7 2024-04-12T03:30:40.254 DEBUG:boto:Data: 2024-04-12T03:30:40.254 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.254 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.254 DEBUG:boto:Params: {} 2024-04-12T03:30:40.254 DEBUG:boto:Token: None 2024-04-12T03:30:40.254 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7 2024-04-12T03:30:40.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xftPt8RV5SpYPJ3X+DTIXgi4ihU= 2024-04-12T03:30:40.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xftPt8RV5SpYPJ3X+DTIXgi4ihU=', 'Content-Length': '0'} 2024-04-12T03:30:40.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe7ad42bea17afe2-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:40.256 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.256 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.256 DEBUG:boto:Method: GET 2024-04-12T03:30:40.256 DEBUG:boto:Path: /dgbpxg-50/obj7 2024-04-12T03:30:40.257 DEBUG:boto:Data: 2024-04-12T03:30:40.257 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.257 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.257 DEBUG:boto:Params: {} 2024-04-12T03:30:40.257 DEBUG:boto:Token: None 2024-04-12T03:30:40.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7 2024-04-12T03:30:40.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pVqTuJq1VBSQdv43cfZYvnq6vfk= 2024-04-12T03:30:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pVqTuJq1VBSQdv43cfZYvnq6vfk=', 'Content-Length': '0'} 2024-04-12T03:30:40.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a687d71426561fc9-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.259 DEBUG:boto:path=/dgbpxg-50/obj7 2024-04-12T03:30:40.259 DEBUG:boto:auth_path=/dgbpxg-50/obj7 2024-04-12T03:30:40.259 DEBUG:boto:Method: GET 2024-04-12T03:30:40.259 DEBUG:boto:Path: /dgbpxg-50/obj7 2024-04-12T03:30:40.259 DEBUG:boto:Data: 2024-04-12T03:30:40.260 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.260 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.260 DEBUG:boto:Params: {} 2024-04-12T03:30:40.260 DEBUG:boto:Token: None 2024-04-12T03:30:40.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj7 2024-04-12T03:30:40.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pVqTuJq1VBSQdv43cfZYvnq6vfk= 2024-04-12T03:30:40.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pVqTuJq1VBSQdv43cfZYvnq6vfk=', 'Content-Length': '0'} 2024-04-12T03:30:40.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce7161b4b1c8ffeb-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.262 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.262 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.262 DEBUG:boto:path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.262 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.262 DEBUG:boto:Method: GET 2024-04-12T03:30:40.262 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.263 DEBUG:boto:Data: 2024-04-12T03:30:40.263 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.263 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.263 DEBUG:boto:Params: {} 2024-04-12T03:30:40.263 DEBUG:boto:Token: None 2024-04-12T03:30:40.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687995c3eaca4a10-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.266 DEBUG:boto:b'dgbpxg-50obj71000falseobj72024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.266 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.266 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.266 DEBUG:boto:path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.266 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.267 DEBUG:boto:Method: GET 2024-04-12T03:30:40.267 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj7 2024-04-12T03:30:40.267 DEBUG:boto:Data: 2024-04-12T03:30:40.267 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.267 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.267 DEBUG:boto:Params: {} 2024-04-12T03:30:40.267 DEBUG:boto:Token: None 2024-04-12T03:30:40.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c98dc16dcfb8c1-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.270 DEBUG:boto:b'dgbpxg-50obj71000falseobj72024-04-12T03:30:03.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:40.270 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.270 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.270 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.270 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.270 DEBUG:boto:Method: GET 2024-04-12T03:30:40.271 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.271 DEBUG:boto:Data: 2024-04-12T03:30:40.271 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.271 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.271 DEBUG:boto:Params: {} 2024-04-12T03:30:40.271 DEBUG:boto:Token: None 2024-04-12T03:30:40.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc= 2024-04-12T03:30:40.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc=', 'Content-Length': '0'} 2024-04-12T03:30:40.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f00c4b66ae27077d-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.273 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.274 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.274 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.274 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.274 DEBUG:boto:Method: GET 2024-04-12T03:30:40.274 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.274 DEBUG:boto:Data: 2024-04-12T03:30:40.274 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.274 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.274 DEBUG:boto:Params: {} 2024-04-12T03:30:40.275 DEBUG:boto:Token: None 2024-04-12T03:30:40.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc= 2024-04-12T03:30:40.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc=', 'Content-Length': '0'} 2024-04-12T03:30:40.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c029047e892eb31-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.277 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.277 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.277 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.277 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.277 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.277 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.277 DEBUG:boto:Data: 2024-04-12T03:30:40.277 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.278 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.278 DEBUG:boto:Params: {} 2024-04-12T03:30:40.278 DEBUG:boto:Token: None 2024-04-12T03:30:40.278 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.278 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:epFF+049aBraPwQqU5n4sfIMDtc= 2024-04-12T03:30:40.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:epFF+049aBraPwQqU5n4sfIMDtc=', 'Content-Length': '0'} 2024-04-12T03:30:40.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080235b6ba202be8d-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.280 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.280 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.280 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.280 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.280 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.281 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.281 DEBUG:boto:Data: 2024-04-12T03:30:40.281 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.281 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.281 DEBUG:boto:Params: {} 2024-04-12T03:30:40.281 DEBUG:boto:Token: None 2024-04-12T03:30:40.281 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:epFF+049aBraPwQqU5n4sfIMDtc= 2024-04-12T03:30:40.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:epFF+049aBraPwQqU5n4sfIMDtc=', 'Content-Length': '0'} 2024-04-12T03:30:40.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb1ed1b7fc3d2806-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:40.283 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.284 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.284 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.284 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.284 DEBUG:boto:Method: GET 2024-04-12T03:30:40.284 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.284 DEBUG:boto:Data: 2024-04-12T03:30:40.284 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.284 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.284 DEBUG:boto:Params: {} 2024-04-12T03:30:40.284 DEBUG:boto:Token: None 2024-04-12T03:30:40.285 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.285 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc= 2024-04-12T03:30:40.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc=', 'Content-Length': '0'} 2024-04-12T03:30:40.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd110ddc81093a12-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.286 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.287 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.287 DEBUG:boto:path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.287 DEBUG:boto:auth_path=/dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.287 DEBUG:boto:Method: GET 2024-04-12T03:30:40.287 DEBUG:boto:Path: /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.287 DEBUG:boto:Data: 2024-04-12T03:30:40.287 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.287 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.287 DEBUG:boto:Params: {} 2024-04-12T03:30:40.288 DEBUG:boto:Token: None 2024-04-12T03:30:40.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8?versionId=null 2024-04-12T03:30:40.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc= 2024-04-12T03:30:40.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dp7pjAQDo4cHcpNmiE1ijvANkbc=', 'Content-Length': '0'} 2024-04-12T03:30:40.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003eac6fab5afdcda8-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.330 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.331 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.331 DEBUG:boto:path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.331 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.331 DEBUG:boto:Method: GET 2024-04-12T03:30:40.331 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.331 DEBUG:boto:Data: 2024-04-12T03:30:40.331 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.331 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.331 DEBUG:boto:Params: {} 2024-04-12T03:30:40.331 DEBUG:boto:Token: None 2024-04-12T03:30:40.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053373949d048c958-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.334 DEBUG:boto:b'dgbpxg-50obj81000falseobj82024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.334 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.334 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.335 DEBUG:boto:path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.335 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.335 DEBUG:boto:Method: GET 2024-04-12T03:30:40.335 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.335 DEBUG:boto:Data: 2024-04-12T03:30:40.335 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.335 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.335 DEBUG:boto:Params: {} 2024-04-12T03:30:40.335 DEBUG:boto:Token: None 2024-04-12T03:30:40.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac707de0e8f1441b-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.338 DEBUG:boto:b'dgbpxg-50obj81000falseobj82024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.338 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.338 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.338 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.338 DEBUG:boto:Path: /dgbpxg-50/obj8 2024-04-12T03:30:40.339 DEBUG:boto:Data: 2024-04-12T03:30:40.339 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.339 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.339 DEBUG:boto:Params: {} 2024-04-12T03:30:40.339 DEBUG:boto:Token: None 2024-04-12T03:30:40.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8 2024-04-12T03:30:40.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8+jP/uinlljC2vKbnWX6+S0JuI= 2024-04-12T03:30:40.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8+jP/uinlljC2vKbnWX6+S0JuI=', 'Content-Length': '0'} 2024-04-12T03:30:40.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe2b54e724fcd844-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.341 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.341 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.341 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.341 DEBUG:boto:Path: /dgbpxg-50/obj8 2024-04-12T03:30:40.341 DEBUG:boto:Data: 2024-04-12T03:30:40.341 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.341 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.342 DEBUG:boto:Params: {} 2024-04-12T03:30:40.342 DEBUG:boto:Token: None 2024-04-12T03:30:40.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8 2024-04-12T03:30:40.342 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8+jP/uinlljC2vKbnWX6+S0JuI= 2024-04-12T03:30:40.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8+jP/uinlljC2vKbnWX6+S0JuI=', 'Content-Length': '0'} 2024-04-12T03:30:40.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e72f8a9227d0b91-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:40.344 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.344 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.344 DEBUG:boto:Method: GET 2024-04-12T03:30:40.344 DEBUG:boto:Path: /dgbpxg-50/obj8 2024-04-12T03:30:40.344 DEBUG:boto:Data: 2024-04-12T03:30:40.345 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.345 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.345 DEBUG:boto:Params: {} 2024-04-12T03:30:40.345 DEBUG:boto:Token: None 2024-04-12T03:30:40.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8 2024-04-12T03:30:40.345 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ig2Hehol38UMhHmF3g3r4D3ilgE= 2024-04-12T03:30:40.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ig2Hehol38UMhHmF3g3r4D3ilgE=', 'Content-Length': '0'} 2024-04-12T03:30:40.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-50:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000420d7eefffdfdf8-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.347 DEBUG:boto:path=/dgbpxg-50/obj8 2024-04-12T03:30:40.347 DEBUG:boto:auth_path=/dgbpxg-50/obj8 2024-04-12T03:30:40.347 DEBUG:boto:Method: GET 2024-04-12T03:30:40.347 DEBUG:boto:Path: /dgbpxg-50/obj8 2024-04-12T03:30:40.348 DEBUG:boto:Data: 2024-04-12T03:30:40.348 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.348 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.348 DEBUG:boto:Params: {} 2024-04-12T03:30:40.348 DEBUG:boto:Token: None 2024-04-12T03:30:40.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/obj8 2024-04-12T03:30:40.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ig2Hehol38UMhHmF3g3r4D3ilgE= 2024-04-12T03:30:40.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ig2Hehol38UMhHmF3g3r4D3ilgE=', 'Content-Length': '0'} 2024-04-12T03:30:40.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4d5f19b207cde12-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.351 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.351 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.351 DEBUG:boto:path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.351 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.351 DEBUG:boto:Method: GET 2024-04-12T03:30:40.351 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.351 DEBUG:boto:Data: 2024-04-12T03:30:40.351 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.351 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.351 DEBUG:boto:Params: {} 2024-04-12T03:30:40.352 DEBUG:boto:Token: None 2024-04-12T03:30:40.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063300f9643dff157-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.398 DEBUG:boto:b'dgbpxg-50obj81000falseobj82024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.399 DEBUG:boto:path=/dgbpxg-50/ 2024-04-12T03:30:40.399 DEBUG:boto:auth_path=/dgbpxg-50/ 2024-04-12T03:30:40.399 DEBUG:boto:path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.399 DEBUG:boto:auth_path=/dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.399 DEBUG:boto:Method: GET 2024-04-12T03:30:40.399 DEBUG:boto:Path: /dgbpxg-50/?prefix=obj8 2024-04-12T03:30:40.399 DEBUG:boto:Data: 2024-04-12T03:30:40.399 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.400 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.400 DEBUG:boto:Params: {} 2024-04-12T03:30:40.400 DEBUG:boto:Token: None 2024-04-12T03:30:40.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-50/ 2024-04-12T03:30:40.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM= 2024-04-12T03:30:40.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JWm5mpg3N7TPyDGMAqNyF+A0tsM=', 'Content-Length': '0'} 2024-04-12T03:30:40.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cacfe314014e404d-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.402 DEBUG:boto:b'dgbpxg-50obj81000falseobj82024-04-12T03:30:03.934Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-50 zones={a2, a1} 2024-04-12T03:30:40.403 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-51 2024-04-12T03:30:40.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:40.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:40.404 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:40.458 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:40.459 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:40.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:40.495+0000 7f49d2c1c780 1 bucket sync caught up with source: 2024-04-12T03:30:40.496 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.52.6, 00000000002.50.1, 00000000004.38.6, 00000000002.23.1, 00000000004.52.6, 00000000002.19.1, 00000000002.32.1, 00000000004.50.6, 00000000002.28.1, 00000000002.20.1, 00000000002.32.1] 2024-04-12T03:30:40.496 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.52.6, 00000000002.50.1, 00000000004.38.6, 00000000002.23.1, 00000000004.52.6, 00000000002.19.1, 00000000002.32.1, 00000000004.50.6, 00000000002.28.1, 00000000002.20.1, 00000000002.32.1] 2024-04-12T03:30:40.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:40.495+0000 7f49d2c1c780 0 bucket checkpoint complete 2024-04-12T03:30:40.509 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-51 2024-04-12T03:30:40.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:40.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:40.510 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:40.566 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:40.566 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:40.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:40.600+0000 7fa2bde80780 1 bucket sync caught up with source: 2024-04-12T03:30:40.601 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.210.1, 00000000002.211.1, 00000000004.430.1, 00000000002.1157.1, 00000000004.789.1, 00000000002.836.1, 00000000002.634.1, 00000000004.2186.1, 00000000002.3494.1, 00000000002.838.1, 00000000002.2471.1] 2024-04-12T03:30:40.601 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.210.1, 00000000002.211.1, 00000000004.430.1, 00000000002.1157.1, 00000000004.789.1, 00000000002.836.1, 00000000002.634.1, 00000000004.2186.1, 00000000002.3494.1, 00000000002.838.1, 00000000002.2471.1] 2024-04-12T03:30:40.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:40.600+0000 7fa2bde80780 0 bucket checkpoint complete 2024-04-12T03:30:40.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-51 zones={a2, a1} 2024-04-12T03:30:40.615 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.615 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.615 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.615 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:40.615 DEBUG:boto:Data: 2024-04-12T03:30:40.615 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.615 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.615 DEBUG:boto:Params: {} 2024-04-12T03:30:40.615 DEBUG:boto:Token: None 2024-04-12T03:30:40.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKkfaFj5XE48mj85Omro0H9vor4= 2024-04-12T03:30:40.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKkfaFj5XE48mj85Omro0H9vor4=', 'Content-Length': '0'} 2024-04-12T03:30:40.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000501554409048b4e2-006618aae0-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.618 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.619 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.619 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.619 DEBUG:boto:Path: /dgbpxg-51/ 2024-04-12T03:30:40.619 DEBUG:boto:Data: 2024-04-12T03:30:40.619 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.619 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.619 DEBUG:boto:Params: {} 2024-04-12T03:30:40.619 DEBUG:boto:Token: None 2024-04-12T03:30:40.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKkfaFj5XE48mj85Omro0H9vor4= 2024-04-12T03:30:40.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKkfaFj5XE48mj85Omro0H9vor4=', 'Content-Length': '0'} 2024-04-12T03:30:40.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059831869f564af0d-006618aae0-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.622 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:30:40.622 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.622 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.622 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.622 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.622 DEBUG:boto:Method: GET 2024-04-12T03:30:40.622 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.622 DEBUG:boto:Data: 2024-04-12T03:30:40.623 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.623 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.623 DEBUG:boto:Params: {} 2024-04-12T03:30:40.623 DEBUG:boto:Token: None 2024-04-12T03:30:40.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/?versions 2024-04-12T03:30:40.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c= 2024-04-12T03:30:40.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c=', 'Content-Length': '0'} 2024-04-12T03:30:40.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1651d41a536504b-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.626 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:30:40.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:30:40.627 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.627 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.627 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.627 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.628 DEBUG:boto:Method: GET 2024-04-12T03:30:40.628 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.628 DEBUG:boto:Data: 2024-04-12T03:30:40.628 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.628 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.628 DEBUG:boto:Params: {} 2024-04-12T03:30:40.628 DEBUG:boto:Token: None 2024-04-12T03:30:40.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/?versions 2024-04-12T03:30:40.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c= 2024-04-12T03:30:40.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c=', 'Content-Length': '0'} 2024-04-12T03:30:40.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18c1484e1017549-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.631 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:30:40.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:30:40.632 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.632 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.632 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.632 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.633 DEBUG:boto:Method: GET 2024-04-12T03:30:40.633 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.633 DEBUG:boto:Data: 2024-04-12T03:30:40.633 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.633 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.633 DEBUG:boto:Params: {} 2024-04-12T03:30:40.633 DEBUG:boto:Token: None 2024-04-12T03:30:40.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/?versions 2024-04-12T03:30:40.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c= 2024-04-12T03:30:40.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c=', 'Content-Length': '0'} 2024-04-12T03:30:40.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005715edb4e4fbb53e-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.636 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.636 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.637 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.637 DEBUG:boto:path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.637 DEBUG:boto:auth_path=/dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.637 DEBUG:boto:Method: GET 2024-04-12T03:30:40.637 DEBUG:boto:Path: /dgbpxg-51/?versions&max-keys=999 2024-04-12T03:30:40.637 DEBUG:boto:Data: 2024-04-12T03:30:40.637 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.637 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.637 DEBUG:boto:Params: {} 2024-04-12T03:30:40.637 DEBUG:boto:Token: None 2024-04-12T03:30:40.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/?versions 2024-04-12T03:30:40.638 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c= 2024-04-12T03:30:40.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMDx0uKbfpgVWdBQ2ZuHiwc6F5c=', 'Content-Length': '0'} 2024-04-12T03:30:40.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02ef7d9d66aa2dd-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.640 DEBUG:boto:b'dgbpxg-51999falseobj1nulltrue2024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:40.641 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.641 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.641 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.641 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.641 DEBUG:boto:Method: GET 2024-04-12T03:30:40.641 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.641 DEBUG:boto:Data: 2024-04-12T03:30:40.641 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.641 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.642 DEBUG:boto:Params: {} 2024-04-12T03:30:40.642 DEBUG:boto:Token: None 2024-04-12T03:30:40.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE= 2024-04-12T03:30:40.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE=', 'Content-Length': '0'} 2024-04-12T03:30:40.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f01b444cdbd8edae-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.644 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.644 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.644 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.644 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.645 DEBUG:boto:Method: GET 2024-04-12T03:30:40.645 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.645 DEBUG:boto:Data: 2024-04-12T03:30:40.645 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.645 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.645 DEBUG:boto:Params: {} 2024-04-12T03:30:40.645 DEBUG:boto:Token: None 2024-04-12T03:30:40.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE= 2024-04-12T03:30:40.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE=', 'Content-Length': '0'} 2024-04-12T03:30:40.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ef9a74a14c58a66-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.647 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.647 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.647 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.648 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.648 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.648 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.648 DEBUG:boto:Data: 2024-04-12T03:30:40.648 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.648 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.648 DEBUG:boto:Params: {} 2024-04-12T03:30:40.648 DEBUG:boto:Token: None 2024-04-12T03:30:40.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bHC7Vuth10zwInlMZD5y4ZjjYmo= 2024-04-12T03:30:40.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bHC7Vuth10zwInlMZD5y4ZjjYmo=', 'Content-Length': '0'} 2024-04-12T03:30:40.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065d723c58780184b-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.650 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.650 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.650 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.650 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.651 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.651 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.651 DEBUG:boto:Data: 2024-04-12T03:30:40.651 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.651 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.651 DEBUG:boto:Params: {} 2024-04-12T03:30:40.651 DEBUG:boto:Token: None 2024-04-12T03:30:40.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bHC7Vuth10zwInlMZD5y4ZjjYmo= 2024-04-12T03:30:40.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bHC7Vuth10zwInlMZD5y4ZjjYmo=', 'Content-Length': '0'} 2024-04-12T03:30:40.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c29b55b7bdeaaac-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:40.653 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.653 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.654 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.654 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.654 DEBUG:boto:Method: GET 2024-04-12T03:30:40.654 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.654 DEBUG:boto:Data: 2024-04-12T03:30:40.654 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.654 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.654 DEBUG:boto:Params: {} 2024-04-12T03:30:40.654 DEBUG:boto:Token: None 2024-04-12T03:30:40.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE= 2024-04-12T03:30:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE=', 'Content-Length': '0'} 2024-04-12T03:30:40.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc494c680f4cb018-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.657 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.657 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.657 DEBUG:boto:path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.657 DEBUG:boto:auth_path=/dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.657 DEBUG:boto:Method: GET 2024-04-12T03:30:40.657 DEBUG:boto:Path: /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.657 DEBUG:boto:Data: 2024-04-12T03:30:40.657 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.658 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.658 DEBUG:boto:Params: {} 2024-04-12T03:30:40.658 DEBUG:boto:Token: None 2024-04-12T03:30:40.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1?versionId=null 2024-04-12T03:30:40.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE= 2024-04-12T03:30:40.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QfbLGlfe425eglK+f4q3FHLbE=', 'Content-Length': '0'} 2024-04-12T03:30:40.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f59040844cba41d2-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.702 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.702 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.703 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.703 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.703 DEBUG:boto:Method: GET 2024-04-12T03:30:40.703 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.703 DEBUG:boto:Data: 2024-04-12T03:30:40.703 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.703 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.703 DEBUG:boto:Params: {} 2024-04-12T03:30:40.703 DEBUG:boto:Token: None 2024-04-12T03:30:40.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037984f2f828cee8d-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.706 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.706 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.706 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.707 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.707 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.707 DEBUG:boto:Method: GET 2024-04-12T03:30:40.707 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.707 DEBUG:boto:Data: 2024-04-12T03:30:40.707 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.707 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.707 DEBUG:boto:Params: {} 2024-04-12T03:30:40.707 DEBUG:boto:Token: None 2024-04-12T03:30:40.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee092c528058cda5-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.710 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.710 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.710 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.710 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.711 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:30:40.711 DEBUG:boto:Data: 2024-04-12T03:30:40.711 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.711 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.711 DEBUG:boto:Params: {} 2024-04-12T03:30:40.711 DEBUG:boto:Token: None 2024-04-12T03:30:40.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1 2024-04-12T03:30:40.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eGj7JV9/X1R1RjQh2JEvWsAh2QU= 2024-04-12T03:30:40.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eGj7JV9/X1R1RjQh2JEvWsAh2QU=', 'Content-Length': '0'} 2024-04-12T03:30:40.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004495b6d4efd72584-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.713 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.713 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.713 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.713 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:30:40.713 DEBUG:boto:Data: 2024-04-12T03:30:40.713 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.714 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.714 DEBUG:boto:Params: {} 2024-04-12T03:30:40.714 DEBUG:boto:Token: None 2024-04-12T03:30:40.714 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1 2024-04-12T03:30:40.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eGj7JV9/X1R1RjQh2JEvWsAh2QU= 2024-04-12T03:30:40.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eGj7JV9/X1R1RjQh2JEvWsAh2QU=', 'Content-Length': '0'} 2024-04-12T03:30:40.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002718c2e678e4f587-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:30:40.716 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.716 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.716 DEBUG:boto:Method: GET 2024-04-12T03:30:40.716 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:30:40.717 DEBUG:boto:Data: 2024-04-12T03:30:40.717 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.717 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.717 DEBUG:boto:Params: {} 2024-04-12T03:30:40.717 DEBUG:boto:Token: None 2024-04-12T03:30:40.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1 2024-04-12T03:30:40.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sDN5aA5bx+eoBjzv1FKgoJKEhZ8= 2024-04-12T03:30:40.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sDN5aA5bx+eoBjzv1FKgoJKEhZ8=', 'Content-Length': '0'} 2024-04-12T03:30:40.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003df3529c6b9cb994-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.719 DEBUG:boto:path=/dgbpxg-51/obj1 2024-04-12T03:30:40.719 DEBUG:boto:auth_path=/dgbpxg-51/obj1 2024-04-12T03:30:40.719 DEBUG:boto:Method: GET 2024-04-12T03:30:40.719 DEBUG:boto:Path: /dgbpxg-51/obj1 2024-04-12T03:30:40.719 DEBUG:boto:Data: 2024-04-12T03:30:40.720 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.720 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.720 DEBUG:boto:Params: {} 2024-04-12T03:30:40.720 DEBUG:boto:Token: None 2024-04-12T03:30:40.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj1 2024-04-12T03:30:40.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sDN5aA5bx+eoBjzv1FKgoJKEhZ8= 2024-04-12T03:30:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sDN5aA5bx+eoBjzv1FKgoJKEhZ8=', 'Content-Length': '0'} 2024-04-12T03:30:40.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000824e14fd1f51d955-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.722 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.722 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.722 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.722 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.722 DEBUG:boto:Method: GET 2024-04-12T03:30:40.723 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.723 DEBUG:boto:Data: 2024-04-12T03:30:40.723 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.723 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.723 DEBUG:boto:Params: {} 2024-04-12T03:30:40.723 DEBUG:boto:Token: None 2024-04-12T03:30:40.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.723 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099464a5bc660fb5b-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.770 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.771 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.771 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.771 DEBUG:boto:path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.771 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.771 DEBUG:boto:Method: GET 2024-04-12T03:30:40.771 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj1 2024-04-12T03:30:40.771 DEBUG:boto:Data: 2024-04-12T03:30:40.771 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.771 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.771 DEBUG:boto:Params: {} 2024-04-12T03:30:40.772 DEBUG:boto:Token: None 2024-04-12T03:30:40.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795ce7909f5be7a5-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.774 DEBUG:boto:b'dgbpxg-51obj11000falseobj12024-04-12T03:29:36.447Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:40.775 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.775 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.775 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.775 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.775 DEBUG:boto:Method: GET 2024-04-12T03:30:40.775 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.775 DEBUG:boto:Data: 2024-04-12T03:30:40.775 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.776 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.776 DEBUG:boto:Params: {} 2024-04-12T03:30:40.776 DEBUG:boto:Token: None 2024-04-12T03:30:40.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0= 2024-04-12T03:30:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0=', 'Content-Length': '0'} 2024-04-12T03:30:40.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de39cc40503d8094-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.778 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.778 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.778 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.778 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.778 DEBUG:boto:Method: GET 2024-04-12T03:30:40.778 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.778 DEBUG:boto:Data: 2024-04-12T03:30:40.778 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.779 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.779 DEBUG:boto:Params: {} 2024-04-12T03:30:40.779 DEBUG:boto:Token: None 2024-04-12T03:30:40.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0= 2024-04-12T03:30:40.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0=', 'Content-Length': '0'} 2024-04-12T03:30:40.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023ed84556d6067b7-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.781 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.781 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.781 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.781 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.781 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.781 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.781 DEBUG:boto:Data: 2024-04-12T03:30:40.781 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.782 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.782 DEBUG:boto:Params: {} 2024-04-12T03:30:40.782 DEBUG:boto:Token: None 2024-04-12T03:30:40.782 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R4Mvy8lNMaj8fzZIwiYtTGub2Bk= 2024-04-12T03:30:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R4Mvy8lNMaj8fzZIwiYtTGub2Bk=', 'Content-Length': '0'} 2024-04-12T03:30:40.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a36cd439ee17e32-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.784 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.784 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.784 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.784 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.784 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.784 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.785 DEBUG:boto:Data: 2024-04-12T03:30:40.785 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.785 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.785 DEBUG:boto:Params: {} 2024-04-12T03:30:40.785 DEBUG:boto:Token: None 2024-04-12T03:30:40.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R4Mvy8lNMaj8fzZIwiYtTGub2Bk= 2024-04-12T03:30:40.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R4Mvy8lNMaj8fzZIwiYtTGub2Bk=', 'Content-Length': '0'} 2024-04-12T03:30:40.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d68142802670dd8-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:40.787 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.787 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.787 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.787 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.787 DEBUG:boto:Method: GET 2024-04-12T03:30:40.788 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.788 DEBUG:boto:Data: 2024-04-12T03:30:40.788 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.788 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.788 DEBUG:boto:Params: {} 2024-04-12T03:30:40.788 DEBUG:boto:Token: None 2024-04-12T03:30:40.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0= 2024-04-12T03:30:40.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0=', 'Content-Length': '0'} 2024-04-12T03:30:40.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d5f00d4bae4b35a-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.790 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.790 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.791 DEBUG:boto:path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.791 DEBUG:boto:auth_path=/dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.791 DEBUG:boto:Method: GET 2024-04-12T03:30:40.791 DEBUG:boto:Path: /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.791 DEBUG:boto:Data: 2024-04-12T03:30:40.791 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.791 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.791 DEBUG:boto:Params: {} 2024-04-12T03:30:40.791 DEBUG:boto:Token: None 2024-04-12T03:30:40.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2?versionId=null 2024-04-12T03:30:40.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0= 2024-04-12T03:30:40.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mMyDbpbbo1gRmCA+OstRRdFd5/0=', 'Content-Length': '0'} 2024-04-12T03:30:40.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000267c8fbcddb1c8-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.838 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.839 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.839 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.839 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.839 DEBUG:boto:Method: GET 2024-04-12T03:30:40.839 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.839 DEBUG:boto:Data: 2024-04-12T03:30:40.839 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.839 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.839 DEBUG:boto:Params: {} 2024-04-12T03:30:40.839 DEBUG:boto:Token: None 2024-04-12T03:30:40.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003784ce8a82154fc7-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.842 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.843 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.843 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.843 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.843 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.843 DEBUG:boto:Method: GET 2024-04-12T03:30:40.843 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.843 DEBUG:boto:Data: 2024-04-12T03:30:40.843 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.843 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.843 DEBUG:boto:Params: {} 2024-04-12T03:30:40.843 DEBUG:boto:Token: None 2024-04-12T03:30:40.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9067daaee32331e-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.846 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.846 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.847 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.847 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.847 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:30:40.847 DEBUG:boto:Data: 2024-04-12T03:30:40.847 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.847 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.847 DEBUG:boto:Params: {} 2024-04-12T03:30:40.847 DEBUG:boto:Token: None 2024-04-12T03:30:40.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2 2024-04-12T03:30:40.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aY4kThxDhikjZQDuMzSfzxah0GM= 2024-04-12T03:30:40.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aY4kThxDhikjZQDuMzSfzxah0GM=', 'Content-Length': '0'} 2024-04-12T03:30:40.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066fa1173bb38d7c1-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.849 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.849 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.850 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.850 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:30:40.850 DEBUG:boto:Data: 2024-04-12T03:30:40.850 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.850 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.850 DEBUG:boto:Params: {} 2024-04-12T03:30:40.850 DEBUG:boto:Token: None 2024-04-12T03:30:40.850 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2 2024-04-12T03:30:40.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aY4kThxDhikjZQDuMzSfzxah0GM= 2024-04-12T03:30:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aY4kThxDhikjZQDuMzSfzxah0GM=', 'Content-Length': '0'} 2024-04-12T03:30:40.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d5a4987afbb3faa-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:30:40.852 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.853 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.853 DEBUG:boto:Method: GET 2024-04-12T03:30:40.853 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:30:40.853 DEBUG:boto:Data: 2024-04-12T03:30:40.853 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.853 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.853 DEBUG:boto:Params: {} 2024-04-12T03:30:40.853 DEBUG:boto:Token: None 2024-04-12T03:30:40.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2 2024-04-12T03:30:40.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxpdiipwzYcUAA1hIXVCA+NdiEs= 2024-04-12T03:30:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxpdiipwzYcUAA1hIXVCA+NdiEs=', 'Content-Length': '0'} 2024-04-12T03:30:40.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065d1bd6c4290b5e6-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.855 DEBUG:boto:path=/dgbpxg-51/obj2 2024-04-12T03:30:40.855 DEBUG:boto:auth_path=/dgbpxg-51/obj2 2024-04-12T03:30:40.855 DEBUG:boto:Method: GET 2024-04-12T03:30:40.855 DEBUG:boto:Path: /dgbpxg-51/obj2 2024-04-12T03:30:40.856 DEBUG:boto:Data: 2024-04-12T03:30:40.856 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.856 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.856 DEBUG:boto:Params: {} 2024-04-12T03:30:40.856 DEBUG:boto:Token: None 2024-04-12T03:30:40.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj2 2024-04-12T03:30:40.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxpdiipwzYcUAA1hIXVCA+NdiEs= 2024-04-12T03:30:40.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxpdiipwzYcUAA1hIXVCA+NdiEs=', 'Content-Length': '0'} 2024-04-12T03:30:40.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054e5e2c0ad7dac78-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.858 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.858 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.858 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.858 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.858 DEBUG:boto:Method: GET 2024-04-12T03:30:40.859 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.859 DEBUG:boto:Data: 2024-04-12T03:30:40.859 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.859 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.859 DEBUG:boto:Params: {} 2024-04-12T03:30:40.859 DEBUG:boto:Token: None 2024-04-12T03:30:40.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006398586a3f497b0a-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.861 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.862 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.862 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.862 DEBUG:boto:path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.862 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.862 DEBUG:boto:Method: GET 2024-04-12T03:30:40.862 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj2 2024-04-12T03:30:40.862 DEBUG:boto:Data: 2024-04-12T03:30:40.862 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.862 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.863 DEBUG:boto:Params: {} 2024-04-12T03:30:40.863 DEBUG:boto:Token: None 2024-04-12T03:30:40.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee79c945416a525-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.865 DEBUG:boto:b'dgbpxg-51obj21000falseobj22024-04-12T03:29:36.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:40.865 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.866 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.866 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.866 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.866 DEBUG:boto:Method: GET 2024-04-12T03:30:40.866 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.866 DEBUG:boto:Data: 2024-04-12T03:30:40.866 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.866 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.866 DEBUG:boto:Params: {} 2024-04-12T03:30:40.866 DEBUG:boto:Token: None 2024-04-12T03:30:40.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk= 2024-04-12T03:30:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk=', 'Content-Length': '0'} 2024-04-12T03:30:40.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b98a93098b5591f-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.869 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.869 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.869 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.869 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.869 DEBUG:boto:Method: GET 2024-04-12T03:30:40.869 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.869 DEBUG:boto:Data: 2024-04-12T03:30:40.869 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.869 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.870 DEBUG:boto:Params: {} 2024-04-12T03:30:40.870 DEBUG:boto:Token: None 2024-04-12T03:30:40.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk= 2024-04-12T03:30:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk=', 'Content-Length': '0'} 2024-04-12T03:30:40.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b13d6b49a9c23c0-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.872 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.872 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.872 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.872 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.872 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.873 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.873 DEBUG:boto:Data: 2024-04-12T03:30:40.873 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.873 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.873 DEBUG:boto:Params: {} 2024-04-12T03:30:40.873 DEBUG:boto:Token: None 2024-04-12T03:30:40.873 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwZtzm9Oi5JmgtRNwntZOW3RkAA= 2024-04-12T03:30:40.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwZtzm9Oi5JmgtRNwntZOW3RkAA=', 'Content-Length': '0'} 2024-04-12T03:30:40.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feca32989e5ac2b6-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.875 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.875 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.875 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.875 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.875 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.875 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.875 DEBUG:boto:Data: 2024-04-12T03:30:40.875 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.875 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.876 DEBUG:boto:Params: {} 2024-04-12T03:30:40.876 DEBUG:boto:Token: None 2024-04-12T03:30:40.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwZtzm9Oi5JmgtRNwntZOW3RkAA= 2024-04-12T03:30:40.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwZtzm9Oi5JmgtRNwntZOW3RkAA=', 'Content-Length': '0'} 2024-04-12T03:30:40.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044c7b0dd113b8407-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:40.878 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.878 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.878 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.878 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.878 DEBUG:boto:Method: GET 2024-04-12T03:30:40.878 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.878 DEBUG:boto:Data: 2024-04-12T03:30:40.878 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.878 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.878 DEBUG:boto:Params: {} 2024-04-12T03:30:40.879 DEBUG:boto:Token: None 2024-04-12T03:30:40.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk= 2024-04-12T03:30:40.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk=', 'Content-Length': '0'} 2024-04-12T03:30:40.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dabcd657dc6bd1c-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.922 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.922 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.923 DEBUG:boto:path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.923 DEBUG:boto:auth_path=/dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.923 DEBUG:boto:Method: GET 2024-04-12T03:30:40.923 DEBUG:boto:Path: /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.923 DEBUG:boto:Data: 2024-04-12T03:30:40.923 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.923 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.923 DEBUG:boto:Params: {} 2024-04-12T03:30:40.923 DEBUG:boto:Token: None 2024-04-12T03:30:40.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3?versionId=null 2024-04-12T03:30:40.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk= 2024-04-12T03:30:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tXEi4/qQ3LBp2YCrGC72HTQGzmk=', 'Content-Length': '0'} 2024-04-12T03:30:40.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060fd9d70d0e1bfbf-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.926 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.926 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.926 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.926 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.926 DEBUG:boto:Method: GET 2024-04-12T03:30:40.926 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.926 DEBUG:boto:Data: 2024-04-12T03:30:40.926 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.927 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.927 DEBUG:boto:Params: {} 2024-04-12T03:30:40.927 DEBUG:boto:Token: None 2024-04-12T03:30:40.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ff7ca436a225aa2-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.929 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.930 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.930 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.930 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.930 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.930 DEBUG:boto:Method: GET 2024-04-12T03:30:40.930 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.930 DEBUG:boto:Data: 2024-04-12T03:30:40.930 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.930 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.930 DEBUG:boto:Params: {} 2024-04-12T03:30:40.931 DEBUG:boto:Token: None 2024-04-12T03:30:40.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d88d52b4eaad1a-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.933 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.933 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.933 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.934 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.934 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:30:40.934 DEBUG:boto:Data: 2024-04-12T03:30:40.934 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.934 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.934 DEBUG:boto:Params: {} 2024-04-12T03:30:40.934 DEBUG:boto:Token: None 2024-04-12T03:30:40.934 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3 2024-04-12T03:30:40.934 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8DwwFf8t35VJiP61rHtoO25+p2Y= 2024-04-12T03:30:40.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8DwwFf8t35VJiP61rHtoO25+p2Y=', 'Content-Length': '0'} 2024-04-12T03:30:40.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000922522e4bf99a6a5-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.936 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.936 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.936 DEBUG:boto:Method: HEAD 2024-04-12T03:30:40.936 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:30:40.936 DEBUG:boto:Data: 2024-04-12T03:30:40.936 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.936 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.937 DEBUG:boto:Params: {} 2024-04-12T03:30:40.937 DEBUG:boto:Token: None 2024-04-12T03:30:40.937 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3 2024-04-12T03:30:40.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8DwwFf8t35VJiP61rHtoO25+p2Y= 2024-04-12T03:30:40.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8DwwFf8t35VJiP61rHtoO25+p2Y=', 'Content-Length': '0'} 2024-04-12T03:30:40.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c21949428bcf8c4e-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:30:40.939 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.939 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.939 DEBUG:boto:Method: GET 2024-04-12T03:30:40.939 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:30:40.939 DEBUG:boto:Data: 2024-04-12T03:30:40.939 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.939 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.939 DEBUG:boto:Params: {} 2024-04-12T03:30:40.939 DEBUG:boto:Token: None 2024-04-12T03:30:40.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3 2024-04-12T03:30:40.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BBUmkTAanZn543Y0PIQvwuIf1oI= 2024-04-12T03:30:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BBUmkTAanZn543Y0PIQvwuIf1oI=', 'Content-Length': '0'} 2024-04-12T03:30:40.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000512f64a9b8372f5d-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.941 DEBUG:boto:path=/dgbpxg-51/obj3 2024-04-12T03:30:40.942 DEBUG:boto:auth_path=/dgbpxg-51/obj3 2024-04-12T03:30:40.942 DEBUG:boto:Method: GET 2024-04-12T03:30:40.942 DEBUG:boto:Path: /dgbpxg-51/obj3 2024-04-12T03:30:40.942 DEBUG:boto:Data: 2024-04-12T03:30:40.942 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.942 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.942 DEBUG:boto:Params: {} 2024-04-12T03:30:40.942 DEBUG:boto:Token: None 2024-04-12T03:30:40.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj3 2024-04-12T03:30:40.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BBUmkTAanZn543Y0PIQvwuIf1oI= 2024-04-12T03:30:40.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BBUmkTAanZn543Y0PIQvwuIf1oI=', 'Content-Length': '0'} 2024-04-12T03:30:40.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001244171e1f835d0e-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.944 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.944 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.944 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.945 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.945 DEBUG:boto:Method: GET 2024-04-12T03:30:40.945 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.945 DEBUG:boto:Data: 2024-04-12T03:30:40.945 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.945 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.945 DEBUG:boto:Params: {} 2024-04-12T03:30:40.945 DEBUG:boto:Token: None 2024-04-12T03:30:40.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa7a72538fe999b2-006618aae0-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.948 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.948 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:40.948 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:40.948 DEBUG:boto:path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.948 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.948 DEBUG:boto:Method: GET 2024-04-12T03:30:40.948 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj3 2024-04-12T03:30:40.948 DEBUG:boto:Data: 2024-04-12T03:30:40.949 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.949 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.949 DEBUG:boto:Params: {} 2024-04-12T03:30:40.949 DEBUG:boto:Token: None 2024-04-12T03:30:40.949 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/ 2024-04-12T03:30:40.949 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU= 2024-04-12T03:30:40.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KypgMsNGnY0Vp2Sdi9NNnDpuxfU=', 'Content-Length': '0'} 2024-04-12T03:30:40.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1cacbad6af67a1-006618aae0-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.951 DEBUG:boto:b'dgbpxg-51obj31000falseobj32024-04-12T03:29:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:40.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:40.951 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:40.952 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:40.952 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.952 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.952 DEBUG:boto:Method: GET 2024-04-12T03:30:40.952 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.952 DEBUG:boto:Data: 2024-04-12T03:30:40.952 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:40.952 DEBUG:boto:Port: 8001 2024-04-12T03:30:40.952 DEBUG:boto:Params: {} 2024-04-12T03:30:40.952 DEBUG:boto:Token: None 2024-04-12T03:30:40.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yxxEi9zpqdz/HXj0s1l1rN2xf1Q= 2024-04-12T03:30:40.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yxxEi9zpqdz/HXj0s1l1rN2xf1Q=', 'Content-Length': '0'} 2024-04-12T03:30:40.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dd4e6bf91a05a58-006618aae0-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:40.955 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:40.955 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:40.955 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.955 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.955 DEBUG:boto:Method: GET 2024-04-12T03:30:40.955 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.955 DEBUG:boto:Data: 2024-04-12T03:30:40.955 DEBUG:boto:Headers: {} 2024-04-12T03:30:40.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:40.955 DEBUG:boto:Port: 8000 2024-04-12T03:30:40.955 DEBUG:boto:Params: {} 2024-04-12T03:30:40.956 DEBUG:boto:Token: None 2024-04-12T03:30:40.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:40 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:40.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yxxEi9zpqdz/HXj0s1l1rN2xf1Q= 2024-04-12T03:30:40.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yxxEi9zpqdz/HXj0s1l1rN2xf1Q=', 'Content-Length': '0'} 2024-04-12T03:30:40.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fef7c4247d26251-006618aae0-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.002 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.002 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.003 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.003 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.003 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.003 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.003 DEBUG:boto:Data: 2024-04-12T03:30:41.003 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.003 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.003 DEBUG:boto:Params: {} 2024-04-12T03:30:41.004 DEBUG:boto:Token: None 2024-04-12T03:30:41.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mHFxzMinwxFLi5K3RI/6CisGbA0= 2024-04-12T03:30:41.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mHFxzMinwxFLi5K3RI/6CisGbA0=', 'Content-Length': '0'} 2024-04-12T03:30:41.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f827f1f6ac612714-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.006 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.006 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.007 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.007 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.007 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.007 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.007 DEBUG:boto:Data: 2024-04-12T03:30:41.007 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.007 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.007 DEBUG:boto:Params: {} 2024-04-12T03:30:41.008 DEBUG:boto:Token: None 2024-04-12T03:30:41.008 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mHFxzMinwxFLi5K3RI/6CisGbA0= 2024-04-12T03:30:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mHFxzMinwxFLi5K3RI/6CisGbA0=', 'Content-Length': '0'} 2024-04-12T03:30:41.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050285f9854980466-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:41.010 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.010 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.010 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.010 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.011 DEBUG:boto:Method: GET 2024-04-12T03:30:41.011 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.011 DEBUG:boto:Data: 2024-04-12T03:30:41.011 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.011 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.011 DEBUG:boto:Params: {} 2024-04-12T03:30:41.011 DEBUG:boto:Token: None 2024-04-12T03:30:41.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y9YjCpCaTR8EpMnoA2D0fpMDxT0= 2024-04-12T03:30:41.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y9YjCpCaTR8EpMnoA2D0fpMDxT0=', 'Content-Length': '0'} 2024-04-12T03:30:41.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbfad21ef00a0c17-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.013 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.013 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.014 DEBUG:boto:path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.014 DEBUG:boto:auth_path=/dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.014 DEBUG:boto:Method: GET 2024-04-12T03:30:41.014 DEBUG:boto:Path: /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.014 DEBUG:boto:Data: 2024-04-12T03:30:41.014 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.014 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.014 DEBUG:boto:Params: {} 2024-04-12T03:30:41.015 DEBUG:boto:Token: None 2024-04-12T03:30:41.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4?versionId=null 2024-04-12T03:30:41.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y9YjCpCaTR8EpMnoA2D0fpMDxT0= 2024-04-12T03:30:41.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y9YjCpCaTR8EpMnoA2D0fpMDxT0=', 'Content-Length': '0'} 2024-04-12T03:30:41.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043ba57abeebb6623-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.017 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.017 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.017 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.017 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.017 DEBUG:boto:Method: GET 2024-04-12T03:30:41.017 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.017 DEBUG:boto:Data: 2024-04-12T03:30:41.017 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.018 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.018 DEBUG:boto:Params: {} 2024-04-12T03:30:41.018 DEBUG:boto:Token: None 2024-04-12T03:30:41.018 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.018 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d034009acb5ae3db-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.020 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.021 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.021 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.021 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.021 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.021 DEBUG:boto:Method: GET 2024-04-12T03:30:41.021 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.021 DEBUG:boto:Data: 2024-04-12T03:30:41.022 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.022 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.022 DEBUG:boto:Params: {} 2024-04-12T03:30:41.022 DEBUG:boto:Token: None 2024-04-12T03:30:41.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f72c408597d2c6c2-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.025 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.025 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.025 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.025 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.025 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:30:41.025 DEBUG:boto:Data: 2024-04-12T03:30:41.026 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.026 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.026 DEBUG:boto:Params: {} 2024-04-12T03:30:41.026 DEBUG:boto:Token: None 2024-04-12T03:30:41.026 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4 2024-04-12T03:30:41.026 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u20/5XnmjWuZqrITtSqIGNg2x0w= 2024-04-12T03:30:41.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u20/5XnmjWuZqrITtSqIGNg2x0w=', 'Content-Length': '0'} 2024-04-12T03:30:41.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d80132710d4266c-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.028 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.028 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.028 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.028 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:30:41.028 DEBUG:boto:Data: 2024-04-12T03:30:41.028 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.029 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.029 DEBUG:boto:Params: {} 2024-04-12T03:30:41.029 DEBUG:boto:Token: None 2024-04-12T03:30:41.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4 2024-04-12T03:30:41.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u20/5XnmjWuZqrITtSqIGNg2x0w= 2024-04-12T03:30:41.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u20/5XnmjWuZqrITtSqIGNg2x0w=', 'Content-Length': '0'} 2024-04-12T03:30:41.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3157691bfe4c7ae-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:30:41.031 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.031 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.031 DEBUG:boto:Method: GET 2024-04-12T03:30:41.031 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:30:41.031 DEBUG:boto:Data: 2024-04-12T03:30:41.032 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.032 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.032 DEBUG:boto:Params: {} 2024-04-12T03:30:41.032 DEBUG:boto:Token: None 2024-04-12T03:30:41.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4 2024-04-12T03:30:41.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xNGfoQlIsh4I+Fotb/n6Ue8OEus= 2024-04-12T03:30:41.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xNGfoQlIsh4I+Fotb/n6Ue8OEus=', 'Content-Length': '0'} 2024-04-12T03:30:41.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0ff46126ab9ba7-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.034 DEBUG:boto:path=/dgbpxg-51/obj4 2024-04-12T03:30:41.034 DEBUG:boto:auth_path=/dgbpxg-51/obj4 2024-04-12T03:30:41.034 DEBUG:boto:Method: GET 2024-04-12T03:30:41.034 DEBUG:boto:Path: /dgbpxg-51/obj4 2024-04-12T03:30:41.035 DEBUG:boto:Data: 2024-04-12T03:30:41.035 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.035 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.035 DEBUG:boto:Params: {} 2024-04-12T03:30:41.035 DEBUG:boto:Token: None 2024-04-12T03:30:41.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj4 2024-04-12T03:30:41.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xNGfoQlIsh4I+Fotb/n6Ue8OEus= 2024-04-12T03:30:41.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xNGfoQlIsh4I+Fotb/n6Ue8OEus=', 'Content-Length': '0'} 2024-04-12T03:30:41.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:29:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5d500912edfb853-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.037 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.037 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.037 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.038 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.038 DEBUG:boto:Method: GET 2024-04-12T03:30:41.038 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.038 DEBUG:boto:Data: 2024-04-12T03:30:41.038 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.038 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.038 DEBUG:boto:Params: {} 2024-04-12T03:30:41.038 DEBUG:boto:Token: None 2024-04-12T03:30:41.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf71048f2c03918a-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.041 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.041 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.042 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.042 DEBUG:boto:path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.042 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.042 DEBUG:boto:Method: GET 2024-04-12T03:30:41.042 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj4 2024-04-12T03:30:41.042 DEBUG:boto:Data: 2024-04-12T03:30:41.042 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.042 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.043 DEBUG:boto:Params: {} 2024-04-12T03:30:41.043 DEBUG:boto:Token: None 2024-04-12T03:30:41.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004edde6afd932165e-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.045 DEBUG:boto:b'dgbpxg-51obj41000falseobj42024-04-12T03:29:36.524Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:41.046 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.046 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.046 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.046 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.046 DEBUG:boto:Method: GET 2024-04-12T03:30:41.046 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.046 DEBUG:boto:Data: 2024-04-12T03:30:41.046 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.047 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.047 DEBUG:boto:Params: {} 2024-04-12T03:30:41.047 DEBUG:boto:Token: None 2024-04-12T03:30:41.047 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.047 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY= 2024-04-12T03:30:41.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY=', 'Content-Length': '0'} 2024-04-12T03:30:41.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afa6d5ad25940c5e-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.049 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.049 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.049 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.049 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.049 DEBUG:boto:Method: GET 2024-04-12T03:30:41.050 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.050 DEBUG:boto:Data: 2024-04-12T03:30:41.050 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.050 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.050 DEBUG:boto:Params: {} 2024-04-12T03:30:41.050 DEBUG:boto:Token: None 2024-04-12T03:30:41.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY= 2024-04-12T03:30:41.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY=', 'Content-Length': '0'} 2024-04-12T03:30:41.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b5440e8c392be81-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.053 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.053 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.053 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.053 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.053 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.053 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.053 DEBUG:boto:Data: 2024-04-12T03:30:41.054 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.054 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.054 DEBUG:boto:Params: {} 2024-04-12T03:30:41.054 DEBUG:boto:Token: None 2024-04-12T03:30:41.054 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.054 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:owU7l+CCjP47qUu0elEw2L7BKFQ= 2024-04-12T03:30:41.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:owU7l+CCjP47qUu0elEw2L7BKFQ=', 'Content-Length': '0'} 2024-04-12T03:30:41.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f24bd830d99d694-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.056 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.056 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.057 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.057 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.057 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.057 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.057 DEBUG:boto:Data: 2024-04-12T03:30:41.057 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.057 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.057 DEBUG:boto:Params: {} 2024-04-12T03:30:41.058 DEBUG:boto:Token: None 2024-04-12T03:30:41.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:owU7l+CCjP47qUu0elEw2L7BKFQ= 2024-04-12T03:30:41.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:owU7l+CCjP47qUu0elEw2L7BKFQ=', 'Content-Length': '0'} 2024-04-12T03:30:41.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa03dcd13f20dba5-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:41.060 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.060 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.060 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.060 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.060 DEBUG:boto:Method: GET 2024-04-12T03:30:41.060 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.060 DEBUG:boto:Data: 2024-04-12T03:30:41.060 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.061 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.061 DEBUG:boto:Params: {} 2024-04-12T03:30:41.061 DEBUG:boto:Token: None 2024-04-12T03:30:41.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY= 2024-04-12T03:30:41.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY=', 'Content-Length': '0'} 2024-04-12T03:30:41.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020e1da4b13fe1fb4-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.063 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.063 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.063 DEBUG:boto:path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.063 DEBUG:boto:auth_path=/dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.063 DEBUG:boto:Method: GET 2024-04-12T03:30:41.063 DEBUG:boto:Path: /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.064 DEBUG:boto:Data: 2024-04-12T03:30:41.064 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.064 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.064 DEBUG:boto:Params: {} 2024-04-12T03:30:41.064 DEBUG:boto:Token: None 2024-04-12T03:30:41.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5?versionId=null 2024-04-12T03:30:41.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY= 2024-04-12T03:30:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x5ntIgkyrM6InMwI2Ec4VxjKoAY=', 'Content-Length': '0'} 2024-04-12T03:30:41.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004da578b49e9a97ea-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.066 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.066 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.066 DEBUG:boto:path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.066 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.067 DEBUG:boto:Method: GET 2024-04-12T03:30:41.067 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.067 DEBUG:boto:Data: 2024-04-12T03:30:41.067 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.067 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.067 DEBUG:boto:Params: {} 2024-04-12T03:30:41.067 DEBUG:boto:Token: None 2024-04-12T03:30:41.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beade02e97fdab5d-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.110 DEBUG:boto:b'dgbpxg-51obj51000falseobj52024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.111 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.111 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.111 DEBUG:boto:path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.111 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.111 DEBUG:boto:Method: GET 2024-04-12T03:30:41.111 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.111 DEBUG:boto:Data: 2024-04-12T03:30:41.111 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.111 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.111 DEBUG:boto:Params: {} 2024-04-12T03:30:41.112 DEBUG:boto:Token: None 2024-04-12T03:30:41.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cf3112ce00eb601-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.114 DEBUG:boto:b'dgbpxg-51obj51000falseobj52024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.115 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.115 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.115 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.115 DEBUG:boto:Path: /dgbpxg-51/obj5 2024-04-12T03:30:41.115 DEBUG:boto:Data: 2024-04-12T03:30:41.115 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.115 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.115 DEBUG:boto:Params: {} 2024-04-12T03:30:41.116 DEBUG:boto:Token: None 2024-04-12T03:30:41.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5 2024-04-12T03:30:41.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OohC2Bozxo0ZZO7T/1xQ5BL97D4= 2024-04-12T03:30:41.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OohC2Bozxo0ZZO7T/1xQ5BL97D4=', 'Content-Length': '0'} 2024-04-12T03:30:41.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f68761fa8a4950eb-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.118 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.118 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.118 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.118 DEBUG:boto:Path: /dgbpxg-51/obj5 2024-04-12T03:30:41.118 DEBUG:boto:Data: 2024-04-12T03:30:41.118 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.118 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.118 DEBUG:boto:Params: {} 2024-04-12T03:30:41.119 DEBUG:boto:Token: None 2024-04-12T03:30:41.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5 2024-04-12T03:30:41.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OohC2Bozxo0ZZO7T/1xQ5BL97D4= 2024-04-12T03:30:41.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OohC2Bozxo0ZZO7T/1xQ5BL97D4=', 'Content-Length': '0'} 2024-04-12T03:30:41.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc5e1a9d1ad8af8-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:30:41.121 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.121 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.121 DEBUG:boto:Method: GET 2024-04-12T03:30:41.121 DEBUG:boto:Path: /dgbpxg-51/obj5 2024-04-12T03:30:41.121 DEBUG:boto:Data: 2024-04-12T03:30:41.122 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.122 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.122 DEBUG:boto:Params: {} 2024-04-12T03:30:41.122 DEBUG:boto:Token: None 2024-04-12T03:30:41.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5 2024-04-12T03:30:41.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j1HIyoLc3oKubN44wkv7DXUbPZ0= 2024-04-12T03:30:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j1HIyoLc3oKubN44wkv7DXUbPZ0=', 'Content-Length': '0'} 2024-04-12T03:30:41.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4863618bc7de91f-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.124 DEBUG:boto:path=/dgbpxg-51/obj5 2024-04-12T03:30:41.124 DEBUG:boto:auth_path=/dgbpxg-51/obj5 2024-04-12T03:30:41.124 DEBUG:boto:Method: GET 2024-04-12T03:30:41.124 DEBUG:boto:Path: /dgbpxg-51/obj5 2024-04-12T03:30:41.125 DEBUG:boto:Data: 2024-04-12T03:30:41.125 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.125 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.125 DEBUG:boto:Params: {} 2024-04-12T03:30:41.125 DEBUG:boto:Token: None 2024-04-12T03:30:41.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj5 2024-04-12T03:30:41.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j1HIyoLc3oKubN44wkv7DXUbPZ0= 2024-04-12T03:30:41.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j1HIyoLc3oKubN44wkv7DXUbPZ0=', 'Content-Length': '0'} 2024-04-12T03:30:41.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb840bef466e395-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.170 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.170 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.171 DEBUG:boto:path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.171 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.171 DEBUG:boto:Method: GET 2024-04-12T03:30:41.171 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.171 DEBUG:boto:Data: 2024-04-12T03:30:41.171 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.171 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.171 DEBUG:boto:Params: {} 2024-04-12T03:30:41.171 DEBUG:boto:Token: None 2024-04-12T03:30:41.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000933e74ad05c4a395-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.174 DEBUG:boto:b'dgbpxg-51obj51000falseobj52024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.174 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.175 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.175 DEBUG:boto:path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.175 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.175 DEBUG:boto:Method: GET 2024-04-12T03:30:41.175 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj5 2024-04-12T03:30:41.175 DEBUG:boto:Data: 2024-04-12T03:30:41.175 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.175 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.175 DEBUG:boto:Params: {} 2024-04-12T03:30:41.176 DEBUG:boto:Token: None 2024-04-12T03:30:41.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35c3e628c9a3bbb-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.178 DEBUG:boto:b'dgbpxg-51obj51000falseobj52024-04-12T03:30:03.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:41.179 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.179 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.179 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.179 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.179 DEBUG:boto:Method: GET 2024-04-12T03:30:41.179 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.179 DEBUG:boto:Data: 2024-04-12T03:30:41.179 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.179 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.179 DEBUG:boto:Params: {} 2024-04-12T03:30:41.179 DEBUG:boto:Token: None 2024-04-12T03:30:41.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8= 2024-04-12T03:30:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8=', 'Content-Length': '0'} 2024-04-12T03:30:41.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072013e791755b728-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.182 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.182 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.182 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.182 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.182 DEBUG:boto:Method: GET 2024-04-12T03:30:41.182 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.182 DEBUG:boto:Data: 2024-04-12T03:30:41.183 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.183 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.183 DEBUG:boto:Params: {} 2024-04-12T03:30:41.183 DEBUG:boto:Token: None 2024-04-12T03:30:41.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8= 2024-04-12T03:30:41.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8=', 'Content-Length': '0'} 2024-04-12T03:30:41.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070575bedad4cd6a0-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.185 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.186 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.186 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.186 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.186 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.186 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.186 DEBUG:boto:Data: 2024-04-12T03:30:41.186 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.186 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.186 DEBUG:boto:Params: {} 2024-04-12T03:30:41.187 DEBUG:boto:Token: None 2024-04-12T03:30:41.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TNpeoiYDdsI/m+b5rlkhnhSUBuk= 2024-04-12T03:30:41.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TNpeoiYDdsI/m+b5rlkhnhSUBuk=', 'Content-Length': '0'} 2024-04-12T03:30:41.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d9a84ada08fc450-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.189 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.189 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.189 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.189 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.189 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.189 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.189 DEBUG:boto:Data: 2024-04-12T03:30:41.189 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.189 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.189 DEBUG:boto:Params: {} 2024-04-12T03:30:41.190 DEBUG:boto:Token: None 2024-04-12T03:30:41.190 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TNpeoiYDdsI/m+b5rlkhnhSUBuk= 2024-04-12T03:30:41.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TNpeoiYDdsI/m+b5rlkhnhSUBuk=', 'Content-Length': '0'} 2024-04-12T03:30:41.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001620f9a84549bc8e-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:41.192 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.192 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.192 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.192 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.192 DEBUG:boto:Method: GET 2024-04-12T03:30:41.192 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.193 DEBUG:boto:Data: 2024-04-12T03:30:41.193 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.193 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.193 DEBUG:boto:Params: {} 2024-04-12T03:30:41.193 DEBUG:boto:Token: None 2024-04-12T03:30:41.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8= 2024-04-12T03:30:41.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8=', 'Content-Length': '0'} 2024-04-12T03:30:41.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d1cc7fc84c612c-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.195 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.196 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.196 DEBUG:boto:path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.196 DEBUG:boto:auth_path=/dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.196 DEBUG:boto:Method: GET 2024-04-12T03:30:41.196 DEBUG:boto:Path: /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.196 DEBUG:boto:Data: 2024-04-12T03:30:41.196 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.196 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.196 DEBUG:boto:Params: {} 2024-04-12T03:30:41.197 DEBUG:boto:Token: None 2024-04-12T03:30:41.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6?versionId=null 2024-04-12T03:30:41.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8= 2024-04-12T03:30:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xEUZkBRsSCEkq6u/josaY5O4EX8=', 'Content-Length': '0'} 2024-04-12T03:30:41.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1920ac3cfbcd01c-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.199 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.199 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.199 DEBUG:boto:path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.199 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.199 DEBUG:boto:Method: GET 2024-04-12T03:30:41.199 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.200 DEBUG:boto:Data: 2024-04-12T03:30:41.200 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.200 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.200 DEBUG:boto:Params: {} 2024-04-12T03:30:41.200 DEBUG:boto:Token: None 2024-04-12T03:30:41.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ae0cc696dbf8ef-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.203 DEBUG:boto:b'dgbpxg-51obj61000falseobj62024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.203 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.203 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.203 DEBUG:boto:path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.203 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.203 DEBUG:boto:Method: GET 2024-04-12T03:30:41.203 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.203 DEBUG:boto:Data: 2024-04-12T03:30:41.204 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.204 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.204 DEBUG:boto:Params: {} 2024-04-12T03:30:41.204 DEBUG:boto:Token: None 2024-04-12T03:30:41.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e02249d255b3c66-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.206 DEBUG:boto:b'dgbpxg-51obj61000falseobj62024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.207 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.207 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.207 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.207 DEBUG:boto:Path: /dgbpxg-51/obj6 2024-04-12T03:30:41.207 DEBUG:boto:Data: 2024-04-12T03:30:41.207 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.207 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.207 DEBUG:boto:Params: {} 2024-04-12T03:30:41.207 DEBUG:boto:Token: None 2024-04-12T03:30:41.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6 2024-04-12T03:30:41.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nZHtFnm/RSiBmRYpLG2EFa3SXK4= 2024-04-12T03:30:41.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nZHtFnm/RSiBmRYpLG2EFa3SXK4=', 'Content-Length': '0'} 2024-04-12T03:30:41.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e9de2c3a4b046e6-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.209 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.210 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.210 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.210 DEBUG:boto:Path: /dgbpxg-51/obj6 2024-04-12T03:30:41.210 DEBUG:boto:Data: 2024-04-12T03:30:41.210 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.210 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.210 DEBUG:boto:Params: {} 2024-04-12T03:30:41.210 DEBUG:boto:Token: None 2024-04-12T03:30:41.210 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6 2024-04-12T03:30:41.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nZHtFnm/RSiBmRYpLG2EFa3SXK4= 2024-04-12T03:30:41.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nZHtFnm/RSiBmRYpLG2EFa3SXK4=', 'Content-Length': '0'} 2024-04-12T03:30:41.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6d353e07d05efb5-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:30:41.212 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.213 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.213 DEBUG:boto:Method: GET 2024-04-12T03:30:41.213 DEBUG:boto:Path: /dgbpxg-51/obj6 2024-04-12T03:30:41.213 DEBUG:boto:Data: 2024-04-12T03:30:41.213 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.213 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.213 DEBUG:boto:Params: {} 2024-04-12T03:30:41.213 DEBUG:boto:Token: None 2024-04-12T03:30:41.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6 2024-04-12T03:30:41.214 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:g2ovW15h93HX2osT5qB49gjqcvI= 2024-04-12T03:30:41.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:g2ovW15h93HX2osT5qB49gjqcvI=', 'Content-Length': '0'} 2024-04-12T03:30:41.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028d12be055eafbe9-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.258 DEBUG:boto:path=/dgbpxg-51/obj6 2024-04-12T03:30:41.259 DEBUG:boto:auth_path=/dgbpxg-51/obj6 2024-04-12T03:30:41.259 DEBUG:boto:Method: GET 2024-04-12T03:30:41.259 DEBUG:boto:Path: /dgbpxg-51/obj6 2024-04-12T03:30:41.259 DEBUG:boto:Data: 2024-04-12T03:30:41.259 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.259 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.259 DEBUG:boto:Params: {} 2024-04-12T03:30:41.259 DEBUG:boto:Token: None 2024-04-12T03:30:41.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj6 2024-04-12T03:30:41.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:g2ovW15h93HX2osT5qB49gjqcvI= 2024-04-12T03:30:41.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:g2ovW15h93HX2osT5qB49gjqcvI=', 'Content-Length': '0'} 2024-04-12T03:30:41.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ad921c89f42f0b-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.262 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.262 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.262 DEBUG:boto:path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.262 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.262 DEBUG:boto:Method: GET 2024-04-12T03:30:41.262 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.262 DEBUG:boto:Data: 2024-04-12T03:30:41.263 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.263 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.263 DEBUG:boto:Params: {} 2024-04-12T03:30:41.263 DEBUG:boto:Token: None 2024-04-12T03:30:41.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a309a5f71169fd-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.266 DEBUG:boto:b'dgbpxg-51obj61000falseobj62024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.266 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.266 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.267 DEBUG:boto:path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.267 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.267 DEBUG:boto:Method: GET 2024-04-12T03:30:41.267 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj6 2024-04-12T03:30:41.267 DEBUG:boto:Data: 2024-04-12T03:30:41.267 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.267 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.267 DEBUG:boto:Params: {} 2024-04-12T03:30:41.268 DEBUG:boto:Token: None 2024-04-12T03:30:41.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c747928f3ab2e73b-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.270 DEBUG:boto:b'dgbpxg-51obj61000falseobj62024-04-12T03:30:03.990Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:41.271 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.271 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.271 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.271 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.271 DEBUG:boto:Method: GET 2024-04-12T03:30:41.271 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.271 DEBUG:boto:Data: 2024-04-12T03:30:41.271 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.272 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.272 DEBUG:boto:Params: {} 2024-04-12T03:30:41.272 DEBUG:boto:Token: None 2024-04-12T03:30:41.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws= 2024-04-12T03:30:41.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws=', 'Content-Length': '0'} 2024-04-12T03:30:41.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000616d7c0ab020e6b6-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.274 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.274 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.274 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.274 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.274 DEBUG:boto:Method: GET 2024-04-12T03:30:41.275 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.275 DEBUG:boto:Data: 2024-04-12T03:30:41.275 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.275 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.275 DEBUG:boto:Params: {} 2024-04-12T03:30:41.275 DEBUG:boto:Token: None 2024-04-12T03:30:41.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws= 2024-04-12T03:30:41.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws=', 'Content-Length': '0'} 2024-04-12T03:30:41.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d04690b7d0776350-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.277 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.278 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.278 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.278 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.278 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.278 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.278 DEBUG:boto:Data: 2024-04-12T03:30:41.278 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.278 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.279 DEBUG:boto:Params: {} 2024-04-12T03:30:41.279 DEBUG:boto:Token: None 2024-04-12T03:30:41.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ksYYogjzuOCeUG+iAbyOoMCl9sA= 2024-04-12T03:30:41.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ksYYogjzuOCeUG+iAbyOoMCl9sA=', 'Content-Length': '0'} 2024-04-12T03:30:41.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008ac73df4124e601-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.281 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.281 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.281 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.281 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.281 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.281 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.281 DEBUG:boto:Data: 2024-04-12T03:30:41.281 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.282 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.282 DEBUG:boto:Params: {} 2024-04-12T03:30:41.282 DEBUG:boto:Token: None 2024-04-12T03:30:41.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.282 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ksYYogjzuOCeUG+iAbyOoMCl9sA= 2024-04-12T03:30:41.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ksYYogjzuOCeUG+iAbyOoMCl9sA=', 'Content-Length': '0'} 2024-04-12T03:30:41.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084d5493c0a233323-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:41.284 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.284 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.284 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.285 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.285 DEBUG:boto:Method: GET 2024-04-12T03:30:41.285 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.285 DEBUG:boto:Data: 2024-04-12T03:30:41.285 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.285 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.285 DEBUG:boto:Params: {} 2024-04-12T03:30:41.286 DEBUG:boto:Token: None 2024-04-12T03:30:41.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws= 2024-04-12T03:30:41.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws=', 'Content-Length': '0'} 2024-04-12T03:30:41.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000956ad6605c97be33-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.288 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.288 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.289 DEBUG:boto:path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.289 DEBUG:boto:auth_path=/dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.289 DEBUG:boto:Method: GET 2024-04-12T03:30:41.289 DEBUG:boto:Path: /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.289 DEBUG:boto:Data: 2024-04-12T03:30:41.289 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.289 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.289 DEBUG:boto:Params: {} 2024-04-12T03:30:41.289 DEBUG:boto:Token: None 2024-04-12T03:30:41.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7?versionId=null 2024-04-12T03:30:41.290 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws= 2024-04-12T03:30:41.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fvb7vbZiggewag1RoIrQoLwG/Ws=', 'Content-Length': '0'} 2024-04-12T03:30:41.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000346a9ee7b6236020-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.292 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.292 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.292 DEBUG:boto:path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.292 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.292 DEBUG:boto:Method: GET 2024-04-12T03:30:41.292 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.292 DEBUG:boto:Data: 2024-04-12T03:30:41.293 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.293 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.293 DEBUG:boto:Params: {} 2024-04-12T03:30:41.293 DEBUG:boto:Token: None 2024-04-12T03:30:41.293 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.293 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907232d9b10a5f13-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.295 DEBUG:boto:b'dgbpxg-51obj71000falseobj72024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.296 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.296 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.296 DEBUG:boto:path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.296 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.296 DEBUG:boto:Method: GET 2024-04-12T03:30:41.296 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.297 DEBUG:boto:Data: 2024-04-12T03:30:41.297 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.297 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.297 DEBUG:boto:Params: {} 2024-04-12T03:30:41.297 DEBUG:boto:Token: None 2024-04-12T03:30:41.297 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.297 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d772714bd72d402-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.300 DEBUG:boto:b'dgbpxg-51obj71000falseobj72024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.300 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.300 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.301 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.301 DEBUG:boto:Path: /dgbpxg-51/obj7 2024-04-12T03:30:41.301 DEBUG:boto:Data: 2024-04-12T03:30:41.301 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.301 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.301 DEBUG:boto:Params: {} 2024-04-12T03:30:41.301 DEBUG:boto:Token: None 2024-04-12T03:30:41.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7 2024-04-12T03:30:41.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IfZETpTHwk7RLPoB/Mp4aVcjNPU= 2024-04-12T03:30:41.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IfZETpTHwk7RLPoB/Mp4aVcjNPU=', 'Content-Length': '0'} 2024-04-12T03:30:41.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000050f38bac06e1c3f-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.303 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.304 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.304 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.304 DEBUG:boto:Path: /dgbpxg-51/obj7 2024-04-12T03:30:41.304 DEBUG:boto:Data: 2024-04-12T03:30:41.304 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.304 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.304 DEBUG:boto:Params: {} 2024-04-12T03:30:41.304 DEBUG:boto:Token: None 2024-04-12T03:30:41.305 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7 2024-04-12T03:30:41.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IfZETpTHwk7RLPoB/Mp4aVcjNPU= 2024-04-12T03:30:41.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IfZETpTHwk7RLPoB/Mp4aVcjNPU=', 'Content-Length': '0'} 2024-04-12T03:30:41.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f87fec92cb0d6c86-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:30:41.307 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.307 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.307 DEBUG:boto:Method: GET 2024-04-12T03:30:41.307 DEBUG:boto:Path: /dgbpxg-51/obj7 2024-04-12T03:30:41.307 DEBUG:boto:Data: 2024-04-12T03:30:41.308 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.308 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.308 DEBUG:boto:Params: {} 2024-04-12T03:30:41.308 DEBUG:boto:Token: None 2024-04-12T03:30:41.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7 2024-04-12T03:30:41.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OjQ2R1CzqtrrMy/zNWqJ6DFPajg= 2024-04-12T03:30:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OjQ2R1CzqtrrMy/zNWqJ6DFPajg=', 'Content-Length': '0'} 2024-04-12T03:30:41.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6a5338dd526602d-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.310 DEBUG:boto:path=/dgbpxg-51/obj7 2024-04-12T03:30:41.310 DEBUG:boto:auth_path=/dgbpxg-51/obj7 2024-04-12T03:30:41.310 DEBUG:boto:Method: GET 2024-04-12T03:30:41.311 DEBUG:boto:Path: /dgbpxg-51/obj7 2024-04-12T03:30:41.311 DEBUG:boto:Data: 2024-04-12T03:30:41.311 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.311 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.311 DEBUG:boto:Params: {} 2024-04-12T03:30:41.311 DEBUG:boto:Token: None 2024-04-12T03:30:41.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj7 2024-04-12T03:30:41.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OjQ2R1CzqtrrMy/zNWqJ6DFPajg= 2024-04-12T03:30:41.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OjQ2R1CzqtrrMy/zNWqJ6DFPajg=', 'Content-Length': '0'} 2024-04-12T03:30:41.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006563289b8c72ffb0-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.354 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.355 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.355 DEBUG:boto:path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.355 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.355 DEBUG:boto:Method: GET 2024-04-12T03:30:41.355 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.355 DEBUG:boto:Data: 2024-04-12T03:30:41.355 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.355 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.355 DEBUG:boto:Params: {} 2024-04-12T03:30:41.356 DEBUG:boto:Token: None 2024-04-12T03:30:41.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3bd4cfa7340ddd-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.358 DEBUG:boto:b'dgbpxg-51obj71000falseobj72024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.359 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.359 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.359 DEBUG:boto:path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.359 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.359 DEBUG:boto:Method: GET 2024-04-12T03:30:41.359 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj7 2024-04-12T03:30:41.359 DEBUG:boto:Data: 2024-04-12T03:30:41.359 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.360 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.360 DEBUG:boto:Params: {} 2024-04-12T03:30:41.360 DEBUG:boto:Token: None 2024-04-12T03:30:41.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005204a2a31491c6d7-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.362 DEBUG:boto:b'dgbpxg-51obj71000falseobj72024-04-12T03:30:04.037Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:41.363 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.363 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.363 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.363 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.363 DEBUG:boto:Method: GET 2024-04-12T03:30:41.363 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.363 DEBUG:boto:Data: 2024-04-12T03:30:41.364 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.364 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.364 DEBUG:boto:Params: {} 2024-04-12T03:30:41.364 DEBUG:boto:Token: None 2024-04-12T03:30:41.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I= 2024-04-12T03:30:41.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I=', 'Content-Length': '0'} 2024-04-12T03:30:41.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c28bcfb5c7dae82-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.406 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.406 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.407 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.407 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.407 DEBUG:boto:Method: GET 2024-04-12T03:30:41.407 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.407 DEBUG:boto:Data: 2024-04-12T03:30:41.407 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.407 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.407 DEBUG:boto:Params: {} 2024-04-12T03:30:41.407 DEBUG:boto:Token: None 2024-04-12T03:30:41.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I= 2024-04-12T03:30:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I=', 'Content-Length': '0'} 2024-04-12T03:30:41.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068899c6ca2d47055-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.410 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.410 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.410 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.410 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.410 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.410 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.410 DEBUG:boto:Data: 2024-04-12T03:30:41.410 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.410 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.411 DEBUG:boto:Params: {} 2024-04-12T03:30:41.411 DEBUG:boto:Token: None 2024-04-12T03:30:41.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GASFaTBd2z6CB0FqrzgBpUNZaSU= 2024-04-12T03:30:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GASFaTBd2z6CB0FqrzgBpUNZaSU=', 'Content-Length': '0'} 2024-04-12T03:30:41.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000949bf269c222b49e-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.413 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.413 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.413 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.413 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.413 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.413 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.413 DEBUG:boto:Data: 2024-04-12T03:30:41.414 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.414 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.414 DEBUG:boto:Params: {} 2024-04-12T03:30:41.414 DEBUG:boto:Token: None 2024-04-12T03:30:41.414 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GASFaTBd2z6CB0FqrzgBpUNZaSU= 2024-04-12T03:30:41.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GASFaTBd2z6CB0FqrzgBpUNZaSU=', 'Content-Length': '0'} 2024-04-12T03:30:41.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d3dd8a3d9be541d-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:41.416 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.416 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.416 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.417 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.417 DEBUG:boto:Method: GET 2024-04-12T03:30:41.417 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.417 DEBUG:boto:Data: 2024-04-12T03:30:41.417 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.417 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.417 DEBUG:boto:Params: {} 2024-04-12T03:30:41.417 DEBUG:boto:Token: None 2024-04-12T03:30:41.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I= 2024-04-12T03:30:41.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I=', 'Content-Length': '0'} 2024-04-12T03:30:41.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced3a2d495833390-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.419 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.420 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.420 DEBUG:boto:path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.420 DEBUG:boto:auth_path=/dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.420 DEBUG:boto:Method: GET 2024-04-12T03:30:41.420 DEBUG:boto:Path: /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.420 DEBUG:boto:Data: 2024-04-12T03:30:41.420 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.420 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.420 DEBUG:boto:Params: {} 2024-04-12T03:30:41.421 DEBUG:boto:Token: None 2024-04-12T03:30:41.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8?versionId=null 2024-04-12T03:30:41.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I= 2024-04-12T03:30:41.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MvaQy6/WtB6ytt6LpKdqrD3w98I=', 'Content-Length': '0'} 2024-04-12T03:30:41.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b22fc99df60820f-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.423 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.423 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.423 DEBUG:boto:path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.423 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.423 DEBUG:boto:Method: GET 2024-04-12T03:30:41.423 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.423 DEBUG:boto:Data: 2024-04-12T03:30:41.423 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.424 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.424 DEBUG:boto:Params: {} 2024-04-12T03:30:41.424 DEBUG:boto:Token: None 2024-04-12T03:30:41.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac4e2bc7cf163b0-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.426 DEBUG:boto:b'dgbpxg-51obj81000falseobj82024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.426 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.427 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.427 DEBUG:boto:path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.427 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.427 DEBUG:boto:Method: GET 2024-04-12T03:30:41.427 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.427 DEBUG:boto:Data: 2024-04-12T03:30:41.427 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.427 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.427 DEBUG:boto:Params: {} 2024-04-12T03:30:41.428 DEBUG:boto:Token: None 2024-04-12T03:30:41.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ffd3fec96eb8671-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.430 DEBUG:boto:b'dgbpxg-51obj81000falseobj82024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.430 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.431 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.431 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.431 DEBUG:boto:Path: /dgbpxg-51/obj8 2024-04-12T03:30:41.431 DEBUG:boto:Data: 2024-04-12T03:30:41.431 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.431 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.431 DEBUG:boto:Params: {} 2024-04-12T03:30:41.431 DEBUG:boto:Token: None 2024-04-12T03:30:41.432 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8 2024-04-12T03:30:41.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R/cuuGOKm1qGBuvU4kTLplUtN+o= 2024-04-12T03:30:41.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R/cuuGOKm1qGBuvU4kTLplUtN+o=', 'Content-Length': '0'} 2024-04-12T03:30:41.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e91356b5c4f9b22c-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.434 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.434 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.434 DEBUG:boto:Method: HEAD 2024-04-12T03:30:41.434 DEBUG:boto:Path: /dgbpxg-51/obj8 2024-04-12T03:30:41.434 DEBUG:boto:Data: 2024-04-12T03:30:41.434 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.434 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.434 DEBUG:boto:Params: {} 2024-04-12T03:30:41.434 DEBUG:boto:Token: None 2024-04-12T03:30:41.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8 2024-04-12T03:30:41.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R/cuuGOKm1qGBuvU4kTLplUtN+o= 2024-04-12T03:30:41.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R/cuuGOKm1qGBuvU4kTLplUtN+o=', 'Content-Length': '0'} 2024-04-12T03:30:41.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eadc3ed697dde54-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:30:41.437 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.437 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.437 DEBUG:boto:Method: GET 2024-04-12T03:30:41.437 DEBUG:boto:Path: /dgbpxg-51/obj8 2024-04-12T03:30:41.437 DEBUG:boto:Data: 2024-04-12T03:30:41.437 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.437 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.437 DEBUG:boto:Params: {} 2024-04-12T03:30:41.437 DEBUG:boto:Token: None 2024-04-12T03:30:41.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8 2024-04-12T03:30:41.438 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cZU7wzEcWFk8xL7xQ5bJNq5Ft1Q= 2024-04-12T03:30:41.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cZU7wzEcWFk8xL7xQ5bJNq5Ft1Q=', 'Content-Length': '0'} 2024-04-12T03:30:41.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfc21b63e2237c80-006618aae1-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.440 DEBUG:boto:path=/dgbpxg-51/obj8 2024-04-12T03:30:41.440 DEBUG:boto:auth_path=/dgbpxg-51/obj8 2024-04-12T03:30:41.440 DEBUG:boto:Method: GET 2024-04-12T03:30:41.440 DEBUG:boto:Path: /dgbpxg-51/obj8 2024-04-12T03:30:41.440 DEBUG:boto:Data: 2024-04-12T03:30:41.440 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.440 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.441 DEBUG:boto:Params: {} 2024-04-12T03:30:41.441 DEBUG:boto:Token: None 2024-04-12T03:30:41.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/obj8 2024-04-12T03:30:41.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cZU7wzEcWFk8xL7xQ5bJNq5Ft1Q= 2024-04-12T03:30:41.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cZU7wzEcWFk8xL7xQ5bJNq5Ft1Q=', 'Content-Length': '0'} 2024-04-12T03:30:41.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-51:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243b97ad77d8491d-006618aae1-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.443 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.443 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.443 DEBUG:boto:path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.443 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.443 DEBUG:boto:Method: GET 2024-04-12T03:30:41.443 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.443 DEBUG:boto:Data: 2024-04-12T03:30:41.444 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.444 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.444 DEBUG:boto:Params: {} 2024-04-12T03:30:41.444 DEBUG:boto:Token: None 2024-04-12T03:30:41.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc41b531aaba16cf-006618aae1-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.446 DEBUG:boto:b'dgbpxg-51obj81000falseobj82024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.447 DEBUG:boto:path=/dgbpxg-51/ 2024-04-12T03:30:41.447 DEBUG:boto:auth_path=/dgbpxg-51/ 2024-04-12T03:30:41.447 DEBUG:boto:path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.447 DEBUG:boto:auth_path=/dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.447 DEBUG:boto:Method: GET 2024-04-12T03:30:41.447 DEBUG:boto:Path: /dgbpxg-51/?prefix=obj8 2024-04-12T03:30:41.447 DEBUG:boto:Data: 2024-04-12T03:30:41.447 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.447 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.448 DEBUG:boto:Params: {} 2024-04-12T03:30:41.448 DEBUG:boto:Token: None 2024-04-12T03:30:41.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-51/ 2024-04-12T03:30:41.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY= 2024-04-12T03:30:41.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNCv3EcOFm7G0D400rOCJ4nenyY=', 'Content-Length': '0'} 2024-04-12T03:30:41.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f65243d54c85421-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.490 DEBUG:boto:b'dgbpxg-51obj81000falseobj82024-04-12T03:30:04.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:41.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-51 zones={a2, a1} 2024-04-12T03:30:41.491 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:30:41.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.492 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:41.587 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000008129", 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.103135Z" 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005445", 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.104453Z" 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012345", 2024-04-12T03:30:41.588 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.105426Z" 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009244", 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:04.259523Z" 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.589 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:30:41.590 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000008129', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012345', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:30:41.590 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:30:41.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.591 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:30:41.693 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 13630994088189014181 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000008129", 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.694 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.103135Z" 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005445", 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.104453Z" 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012345", 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.696 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.105426Z" 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009244", 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:04.259523Z" 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:30:41Z" 2024-04-12T03:30:41.698 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:30:41.699 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13630994088189014181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008129", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.103135Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.104453Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.105426Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009244", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:04.259523Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:30:41Z" } 2024-04-12T03:30:41.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.103135Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.104453Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.105426Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009244', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:04.259523Z'}}] 2024-04-12T03:30:41.700 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008129', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012345', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:30:41.700 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008129', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012345', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:30:41.700 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:30:41.700 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:30:41.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:41.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:41.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:41.797 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:30:41.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005868", 2024-04-12T03:30:41.797 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.338370Z" 2024-04-12T03:30:41.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003894", 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.339713Z" 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009666", 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.341021Z" 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:30:41.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:19.342171Z" 2024-04-12T03:30:41.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.799 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:30:41.800 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005868', 1: '00000000000000000000:00000000000000003894', 2: '00000000000000000000:00000000000000009666', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:30:41.800 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:30:41.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:41.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:41.850 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:41.850 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:41.892 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005868", 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.338370Z" 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:30:41.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003894", 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.339713Z" 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009666", 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.341021Z" 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:19.342171Z" 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:30:41.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:30:41Z" 2024-04-12T03:30:41.898 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:30:41.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005868", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.338370Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.339713Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009666", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.341021Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006849", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:19.342171Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:30:41Z" } 2024-04-12T03:30:41.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005868', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.338370Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.339713Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009666', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.341021Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006849', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:19.342171Z'}}] 2024-04-12T03:30:41.899 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005868', 1: '00000000000000000000:00000000000000003894', 2: '00000000000000000000:00000000000000009666', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:30:41.899 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005868', 1: '00000000000000000000:00000000000000003894', 2: '00000000000000000000:00000000000000009666', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:30:41.899 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:30:41.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-04-12T03:30:41.901 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-52 2024-04-12T03:30:41.901 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:41.902 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:41.902 DEBUG:boto:Method: PUT 2024-04-12T03:30:41.902 DEBUG:boto:Path: /dgbpxg-52/ 2024-04-12T03:30:41.902 DEBUG:boto:Data: 2024-04-12T03:30:41.902 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.902 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.902 DEBUG:boto:Params: {} 2024-04-12T03:30:41.902 DEBUG:boto:Token: None 2024-04-12T03:30:41.903 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-52/ 2024-04-12T03:30:41.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BAgFF/mDIFsPC0wNs5xXHEbALpM= 2024-04-12T03:30:41.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BAgFF/mDIFsPC0wNs5xXHEbALpM=', 'Content-Length': '0'} 2024-04-12T03:30:41.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000693c80de6170d001-006618aae1-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.918 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-53 2024-04-12T03:30:41.918 DEBUG:boto:path=/dgbpxg-53/ 2024-04-12T03:30:41.918 DEBUG:boto:auth_path=/dgbpxg-53/ 2024-04-12T03:30:41.918 DEBUG:boto:Method: PUT 2024-04-12T03:30:41.918 DEBUG:boto:Path: /dgbpxg-53/ 2024-04-12T03:30:41.918 DEBUG:boto:Data: 2024-04-12T03:30:41.918 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:41.918 DEBUG:boto:Port: 8001 2024-04-12T03:30:41.919 DEBUG:boto:Params: {} 2024-04-12T03:30:41.919 DEBUG:boto:Token: None 2024-04-12T03:30:41.919 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-53/ 2024-04-12T03:30:41.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fb1p+qLxOT5KZyF9v6uCO5Nz/rI= 2024-04-12T03:30:41.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fb1p+qLxOT5KZyF9v6uCO5Nz/rI=', 'Content-Length': '0'} 2024-04-12T03:30:41.944 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000129fb2969a8cf7c1-006618aae1-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:41.944 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:41.944 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:41.944 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploads 2024-04-12T03:30:41.945 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploads 2024-04-12T03:30:41.945 DEBUG:boto:Method: POST 2024-04-12T03:30:41.945 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploads 2024-04-12T03:30:41.945 DEBUG:boto:Data: 2024-04-12T03:30:41.945 DEBUG:boto:Headers: {} 2024-04-12T03:30:41.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:41.945 DEBUG:boto:Port: 8000 2024-04-12T03:30:41.945 DEBUG:boto:Params: {} 2024-04-12T03:30:41.945 DEBUG:boto:Token: None 2024-04-12T03:30:41.945 DEBUG:boto:StringToSign: POST Fri, 12 Apr 2024 03:30:41 GMT /dgbpxg-52/MULTIPART?uploads 2024-04-12T03:30:41.946 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pHfIng9nZ35aPyCqnK5WGiVri7Y= 2024-04-12T03:30:41.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pHfIng9nZ35aPyCqnK5WGiVri7Y=', 'Content-Length': '0'} 2024-04-12T03:30:43.873 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d9bfc14e1180a47-006618aae1-1921-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 12 Apr 2024 03:30:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:43.873 DEBUG:boto:b'dgbpxg-52MULTIPART2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa' 2024-04-12T03:30:43.913 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:43.913 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:43.913 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=1 2024-04-12T03:30:43.914 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=1 2024-04-12T03:30:43.914 DEBUG:boto:Method: PUT 2024-04-12T03:30:43.914 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=1 2024-04-12T03:30:43.914 DEBUG:boto:Data: 2024-04-12T03:30:43.914 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:30:43.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:43.914 DEBUG:boto:Port: 8000 2024-04-12T03:30:43.914 DEBUG:boto:Params: {} 2024-04-12T03:30:43.914 DEBUG:boto:Token: None 2024-04-12T03:30:43.915 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 12 Apr 2024 03:30:43 GMT /dgbpxg-52/MULTIPART?partNumber=1&uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:43.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0DPUQJBZT0veAZfcqS+OXrGa8Ww= 2024-04-12T03:30:43.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0DPUQJBZT0veAZfcqS+OXrGa8Ww='} 2024-04-12T03:30:43.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f16f97b764e88b54-006618aae3-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.011 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.011 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.011 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=2 2024-04-12T03:30:44.012 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=2 2024-04-12T03:30:44.012 DEBUG:boto:Method: PUT 2024-04-12T03:30:44.012 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=2 2024-04-12T03:30:44.012 DEBUG:boto:Data: 2024-04-12T03:30:44.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:30:44.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:44.012 DEBUG:boto:Port: 8000 2024-04-12T03:30:44.012 DEBUG:boto:Params: {} 2024-04-12T03:30:44.012 DEBUG:boto:Token: None 2024-04-12T03:30:44.013 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 12 Apr 2024 03:30:44 GMT /dgbpxg-52/MULTIPART?partNumber=2&uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m0QAW0iTZR1mqDLKN37BhrvnvTU= 2024-04-12T03:30:44.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m0QAW0iTZR1mqDLKN37BhrvnvTU='} 2024-04-12T03:30:44.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc4d5fe3467294b9-006618aae4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.084 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.084 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.084 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=3 2024-04-12T03:30:44.084 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=3 2024-04-12T03:30:44.084 DEBUG:boto:Method: PUT 2024-04-12T03:30:44.084 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=3 2024-04-12T03:30:44.084 DEBUG:boto:Data: 2024-04-12T03:30:44.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:30:44.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:44.085 DEBUG:boto:Port: 8000 2024-04-12T03:30:44.085 DEBUG:boto:Params: {} 2024-04-12T03:30:44.085 DEBUG:boto:Token: None 2024-04-12T03:30:44.085 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 12 Apr 2024 03:30:44 GMT /dgbpxg-52/MULTIPART?partNumber=3&uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qdX1Wm/ooSmWQJ1NLpOjmOQ7/nk= 2024-04-12T03:30:44.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qdX1Wm/ooSmWQJ1NLpOjmOQ7/nk='} 2024-04-12T03:30:44.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fc4a94fb17cf14d-006618aae4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.157 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.157 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.157 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=4 2024-04-12T03:30:44.157 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=4 2024-04-12T03:30:44.158 DEBUG:boto:Method: PUT 2024-04-12T03:30:44.158 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa&partNumber=4 2024-04-12T03:30:44.158 DEBUG:boto:Data: 2024-04-12T03:30:44.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:30:44.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:44.158 DEBUG:boto:Port: 8000 2024-04-12T03:30:44.158 DEBUG:boto:Params: {} 2024-04-12T03:30:44.158 DEBUG:boto:Token: None 2024-04-12T03:30:44.158 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 12 Apr 2024 03:30:44 GMT /dgbpxg-52/MULTIPART?partNumber=4&uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C9JFclS7HW6N5rV0ooMt++mB+Eg= 2024-04-12T03:30:44.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C9JFclS7HW6N5rV0ooMt++mB+Eg='} 2024-04-12T03:30:44.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014a62d5f0141114a-006618aae4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.210 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.210 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.210 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.210 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.210 DEBUG:boto:Method: GET 2024-04-12T03:30:44.211 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.211 DEBUG:boto:Data: 2024-04-12T03:30:44.211 DEBUG:boto:Headers: {} 2024-04-12T03:30:44.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:44.211 DEBUG:boto:Port: 8000 2024-04-12T03:30:44.211 DEBUG:boto:Params: {} 2024-04-12T03:30:44.211 DEBUG:boto:Token: None 2024-04-12T03:30:44.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:44 GMT /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VX4LZiwQeUAfyLmPeRwdsEyAny0= 2024-04-12T03:30:44.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VX4LZiwQeUAfyLmPeRwdsEyAny0=', 'Content-Length': '0'} 2024-04-12T03:30:44.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005416654fc45295f9-006618aae4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.255 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.255 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:44.255 DEBUG:boto:path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.255 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.255 DEBUG:boto:Method: POST 2024-04-12T03:30:44.255 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.255 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-12T03:30:44.255 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-12T03:30:44.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:44.255 DEBUG:boto:Port: 8000 2024-04-12T03:30:44.256 DEBUG:boto:Params: {} 2024-04-12T03:30:44.256 DEBUG:boto:Token: None 2024-04-12T03:30:44.256 DEBUG:boto:StringToSign: POST text/xml Fri, 12 Apr 2024 03:30:44 GMT /dgbpxg-52/MULTIPART?uploadId=2~ZvRfmuP5gjFfeTBSb68G2ujL5bIRFXa 2024-04-12T03:30:44.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ybU4kXVk/CzPxLCF71mBY+Be/F0= 2024-04-12T03:30:44.256 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ybU4kXVk/CzPxLCF71mBY+Be/F0=', 'Content-Length': '460'} 2024-04-12T03:30:44.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef5fe9b3c65471b4-006618aae4-1921-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:44.265 DEBUG:boto:http://smithi099.front.sepia.ceph.com:8000/dgbpxg-52/MULTIPARTdgbpxg-52MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-12T03:30:44.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:44.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:44.266 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:44.313 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:44.313 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:44.343 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:30:44.355 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:30:44.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:30:44.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.357 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892641.934631_967.1", 2024-04-12T03:30:44.357 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:30:41.934631Z" 2024-04-12T03:30:44.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.357 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:30:44.358 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892641.934631_967.1'} 2024-04-12T03:30:44.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:30:44.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:44.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:44.359 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:44.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:44.407 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:44.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:44.436+0000 7f589eaa5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:44.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:44.437+0000 7f589eaa5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:44.450 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:30:44.450 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:30:44.450 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:30:44.450 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:30:44.450 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:30:44.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:30:44.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:44.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:30:44.454 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892641.934631_967.1", 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:30:41.934631Z", 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.455 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:30:44.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:30:44.456 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:30:44.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:30:44.456 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:30:44Z" 2024-04-12T03:30:44.456 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:30:44.457 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892641.934631_967.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:41.934631Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:30:44Z" } 2024-04-12T03:30:44.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892641.934631_967.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:41.934631Z', 'realm_epoch': 2}}] 2024-04-12T03:30:44.457 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892641.934631_967.1'} 2024-04-12T03:30:44.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892641.934631_967.1'} 2024-04-12T03:30:44.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:30:44.457 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-52 2024-04-12T03:30:44.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:44.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:44.458 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:44.513 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:44.513 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:44.547+0000 7fb9bce74780 1 waiting to reach incremental sync.. 2024-04-12T03:30:49.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:49.548+0000 7fb9bce74780 1 waiting to reach incremental sync.. 2024-04-12T03:30:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:54.548+0000 7fb9bce74780 1 waiting to reach incremental sync.. 2024-04-12T03:30:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:59.550+0000 7fb9bce74780 1 bucket sync caught up with source: 2024-04-12T03:30:59.551 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000011.69.6, , , , , ] 2024-04-12T03:30:59.551 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000011.69.6, , , , , ] 2024-04-12T03:30:59.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:59.550+0000 7fb9bce74780 0 bucket checkpoint complete 2024-04-12T03:30:59.569 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-52 2024-04-12T03:30:59.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:59.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:59.570 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-52 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:59.636 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:59.636 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:59.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:59.672+0000 7f859902d780 1 bucket sync caught up with source: 2024-04-12T03:30:59.673 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.803.6, , , , , ] 2024-04-12T03:30:59.673 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.803.6, , , , , ] 2024-04-12T03:30:59.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:30:59.672+0000 7f859902d780 0 bucket checkpoint complete 2024-04-12T03:30:59.687 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-52 zones={a2, a1} 2024-04-12T03:30:59.687 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.687 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.687 DEBUG:boto:Method: HEAD 2024-04-12T03:30:59.687 DEBUG:boto:Path: /dgbpxg-52/ 2024-04-12T03:30:59.687 DEBUG:boto:Data: 2024-04-12T03:30:59.687 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.688 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.688 DEBUG:boto:Params: {} 2024-04-12T03:30:59.688 DEBUG:boto:Token: None 2024-04-12T03:30:59.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/ 2024-04-12T03:30:59.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f3SXpBPGXhRx7YWd0Ja3hJ/1reg= 2024-04-12T03:30:59.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f3SXpBPGXhRx7YWd0Ja3hJ/1reg=', 'Content-Length': '0'} 2024-04-12T03:30:59.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ec2c4928ee5734a-006618aaf3-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.691 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.691 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.691 DEBUG:boto:Method: HEAD 2024-04-12T03:30:59.691 DEBUG:boto:Path: /dgbpxg-52/ 2024-04-12T03:30:59.691 DEBUG:boto:Data: 2024-04-12T03:30:59.691 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:59.691 DEBUG:boto:Port: 8000 2024-04-12T03:30:59.692 DEBUG:boto:Params: {} 2024-04-12T03:30:59.692 DEBUG:boto:Token: None 2024-04-12T03:30:59.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/ 2024-04-12T03:30:59.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f3SXpBPGXhRx7YWd0Ja3hJ/1reg= 2024-04-12T03:30:59.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f3SXpBPGXhRx7YWd0Ja3hJ/1reg=', 'Content-Length': '0'} 2024-04-12T03:30:59.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044f084a781695c3a-006618aaf3-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:30:59.694 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.694 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.694 DEBUG:boto:path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.695 DEBUG:boto:auth_path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.695 DEBUG:boto:Method: GET 2024-04-12T03:30:59.695 DEBUG:boto:Path: /dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.695 DEBUG:boto:Data: 2024-04-12T03:30:59.695 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.695 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.695 DEBUG:boto:Params: {} 2024-04-12T03:30:59.695 DEBUG:boto:Token: None 2024-04-12T03:30:59.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/?versions 2024-04-12T03:30:59.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc= 2024-04-12T03:30:59.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc=', 'Content-Length': '0'} 2024-04-12T03:30:59.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d5e28c92c1953b5-006618aaf3-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.698 DEBUG:boto:b'dgbpxg-52999falseMULTIPARTnulltrue2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:59.698 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-12T03:30:59.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:30:59.699 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.699 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.699 DEBUG:boto:path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.699 DEBUG:boto:auth_path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.699 DEBUG:boto:Method: GET 2024-04-12T03:30:59.699 DEBUG:boto:Path: /dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.699 DEBUG:boto:Data: 2024-04-12T03:30:59.699 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:59.699 DEBUG:boto:Port: 8000 2024-04-12T03:30:59.700 DEBUG:boto:Params: {} 2024-04-12T03:30:59.700 DEBUG:boto:Token: None 2024-04-12T03:30:59.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/?versions 2024-04-12T03:30:59.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc= 2024-04-12T03:30:59.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc=', 'Content-Length': '0'} 2024-04-12T03:30:59.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1689824ce1ab374-006618aaf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.702 DEBUG:boto:b'dgbpxg-52999falseMULTIPARTnulltrue2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:59.702 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-12T03:30:59.703 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.703 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.703 DEBUG:boto:path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.703 DEBUG:boto:auth_path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.703 DEBUG:boto:Method: GET 2024-04-12T03:30:59.703 DEBUG:boto:Path: /dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.703 DEBUG:boto:Data: 2024-04-12T03:30:59.703 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.703 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.703 DEBUG:boto:Params: {} 2024-04-12T03:30:59.704 DEBUG:boto:Token: None 2024-04-12T03:30:59.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/?versions 2024-04-12T03:30:59.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc= 2024-04-12T03:30:59.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc=', 'Content-Length': '0'} 2024-04-12T03:30:59.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc327e34fd310cac-006618aaf3-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.707 DEBUG:boto:b'dgbpxg-52999falseMULTIPARTnulltrue2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:59.707 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:30:59.707 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:30:59.707 DEBUG:boto:path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.708 DEBUG:boto:auth_path=/dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.708 DEBUG:boto:Method: GET 2024-04-12T03:30:59.708 DEBUG:boto:Path: /dgbpxg-52/?versions&max-keys=999 2024-04-12T03:30:59.708 DEBUG:boto:Data: 2024-04-12T03:30:59.708 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:59.708 DEBUG:boto:Port: 8000 2024-04-12T03:30:59.708 DEBUG:boto:Params: {} 2024-04-12T03:30:59.708 DEBUG:boto:Token: None 2024-04-12T03:30:59.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/?versions 2024-04-12T03:30:59.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc= 2024-04-12T03:30:59.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SZrvyZDroZQVeByS1D2ucl0xcAc=', 'Content-Length': '0'} 2024-04-12T03:30:59.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132043c90a03c85e-006618aaf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.711 DEBUG:boto:b'dgbpxg-52999falseMULTIPARTnulltrue2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:59.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:30:59.711 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.711 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.712 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.712 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.712 DEBUG:boto:Method: GET 2024-04-12T03:30:59.712 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.712 DEBUG:boto:Data: 2024-04-12T03:30:59.712 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.712 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.712 DEBUG:boto:Params: {} 2024-04-12T03:30:59.712 DEBUG:boto:Token: None 2024-04-12T03:30:59.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.713 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4= 2024-04-12T03:30:59.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4=', 'Content-Length': '0'} 2024-04-12T03:30:59.717 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-52:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001a8383c4f67242ca-006618aaf3-192d-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.799 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.799 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.799 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.799 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.801 DEBUG:boto:Method: GET 2024-04-12T03:30:59.801 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.801 DEBUG:boto:Data: 2024-04-12T03:30:59.801 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:59.802 DEBUG:boto:Port: 8000 2024-04-12T03:30:59.802 DEBUG:boto:Params: {} 2024-04-12T03:30:59.802 DEBUG:boto:Token: None 2024-04-12T03:30:59.802 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.802 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4= 2024-04-12T03:30:59.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4=', 'Content-Length': '0'} 2024-04-12T03:30:59.811 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006ec452b5fb14c478-006618aaf3-1921-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.922 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.923 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.923 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.923 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.923 DEBUG:boto:Method: HEAD 2024-04-12T03:30:59.923 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.923 DEBUG:boto:Data: 2024-04-12T03:30:59.923 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.923 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.923 DEBUG:boto:Params: {} 2024-04-12T03:30:59.923 DEBUG:boto:Token: None 2024-04-12T03:30:59.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9iqNl17ou7fITD3QmDGvzZNSW/A= 2024-04-12T03:30:59.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9iqNl17ou7fITD3QmDGvzZNSW/A=', 'Content-Length': '0'} 2024-04-12T03:30:59.926 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-52:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e15700a0287bc786-006618aaf3-192d-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.926 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.926 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.927 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.927 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.927 DEBUG:boto:Method: HEAD 2024-04-12T03:30:59.927 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.927 DEBUG:boto:Data: 2024-04-12T03:30:59.927 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:30:59.927 DEBUG:boto:Port: 8000 2024-04-12T03:30:59.927 DEBUG:boto:Params: {} 2024-04-12T03:30:59.928 DEBUG:boto:Token: None 2024-04-12T03:30:59.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9iqNl17ou7fITD3QmDGvzZNSW/A= 2024-04-12T03:30:59.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9iqNl17ou7fITD3QmDGvzZNSW/A=', 'Content-Length': '0'} 2024-04-12T03:30:59.930 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000081c5a430a628982c-006618aaf3-1921-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:59.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:30:59.930 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.930 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:30:59.930 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.930 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.931 DEBUG:boto:Method: GET 2024-04-12T03:30:59.931 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.931 DEBUG:boto:Data: 2024-04-12T03:30:59.931 DEBUG:boto:Headers: {} 2024-04-12T03:30:59.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:30:59.931 DEBUG:boto:Port: 8001 2024-04-12T03:30:59.931 DEBUG:boto:Params: {} 2024-04-12T03:30:59.931 DEBUG:boto:Token: None 2024-04-12T03:30:59.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:59 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:30:59.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4= 2024-04-12T03:30:59.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CtMlGdl9PEMc40226gzkgLahUy4=', 'Content-Length': '0'} 2024-04-12T03:30:59.935 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-52:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000dfba1069e232729d-006618aaf3-192d-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.031 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.031 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.031 DEBUG:boto:path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:31:00.031 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:31:00.031 DEBUG:boto:Method: GET 2024-04-12T03:31:00.031 DEBUG:boto:Path: /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:31:00.031 DEBUG:boto:Data: 2024-04-12T03:31:00.031 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.032 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.032 DEBUG:boto:Params: {} 2024-04-12T03:31:00.032 DEBUG:boto:Token: None 2024-04-12T03:31:00.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/MULTIPART?versionId=null 2024-04-12T03:31:00.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GhB6UeyeDiiK1Yo1XUokTAYUmB4= 2024-04-12T03:31:00.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GhB6UeyeDiiK1Yo1XUokTAYUmB4=', 'Content-Length': '0'} 2024-04-12T03:31:00.041 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000017592b78610593ee-006618aaf4-1921-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.130 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:31:00.130 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:31:00.130 DEBUG:boto:path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.130 DEBUG:boto:auth_path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.130 DEBUG:boto:Method: GET 2024-04-12T03:31:00.130 DEBUG:boto:Path: /dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.131 DEBUG:boto:Data: 2024-04-12T03:31:00.131 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:00.131 DEBUG:boto:Port: 8001 2024-04-12T03:31:00.131 DEBUG:boto:Params: {} 2024-04-12T03:31:00.131 DEBUG:boto:Token: None 2024-04-12T03:31:00.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/ 2024-04-12T03:31:00.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc= 2024-04-12T03:31:00.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc=', 'Content-Length': '0'} 2024-04-12T03:31:00.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024432e28dc0e0f2-006618aaf4-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.134 DEBUG:boto:b'dgbpxg-52MULTIPART1000falseMULTIPART2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:00.134 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:31:00.134 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:31:00.134 DEBUG:boto:path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.135 DEBUG:boto:auth_path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.135 DEBUG:boto:Method: GET 2024-04-12T03:31:00.135 DEBUG:boto:Path: /dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.135 DEBUG:boto:Data: 2024-04-12T03:31:00.135 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.135 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.135 DEBUG:boto:Params: {} 2024-04-12T03:31:00.135 DEBUG:boto:Token: None 2024-04-12T03:31:00.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/ 2024-04-12T03:31:00.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc= 2024-04-12T03:31:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc=', 'Content-Length': '0'} 2024-04-12T03:31:00.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798750c1df7262fa-006618aaf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.138 DEBUG:boto:b'dgbpxg-52MULTIPART1000falseMULTIPART2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:00.138 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.138 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.138 DEBUG:boto:Method: HEAD 2024-04-12T03:31:00.138 DEBUG:boto:Path: /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.139 DEBUG:boto:Data: 2024-04-12T03:31:00.139 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:00.139 DEBUG:boto:Port: 8001 2024-04-12T03:31:00.139 DEBUG:boto:Params: {} 2024-04-12T03:31:00.139 DEBUG:boto:Token: None 2024-04-12T03:31:00.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsMY2u1PpF4bLjmgYdyYjEetSjk= 2024-04-12T03:31:00.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsMY2u1PpF4bLjmgYdyYjEetSjk=', 'Content-Length': '0'} 2024-04-12T03:31:00.141 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-52:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008e0239bfb0340d0f-006618aaf4-192d-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.141 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.141 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.141 DEBUG:boto:Method: HEAD 2024-04-12T03:31:00.141 DEBUG:boto:Path: /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.141 DEBUG:boto:Data: 2024-04-12T03:31:00.141 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.142 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.142 DEBUG:boto:Params: {} 2024-04-12T03:31:00.142 DEBUG:boto:Token: None 2024-04-12T03:31:00.142 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.142 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsMY2u1PpF4bLjmgYdyYjEetSjk= 2024-04-12T03:31:00.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsMY2u1PpF4bLjmgYdyYjEetSjk=', 'Content-Length': '0'} 2024-04-12T03:31:00.143 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004d2d007cd64370e4-006618aaf4-1921-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:31:00.144 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.144 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.144 DEBUG:boto:Method: GET 2024-04-12T03:31:00.144 DEBUG:boto:Path: /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.144 DEBUG:boto:Data: 2024-04-12T03:31:00.144 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:00.144 DEBUG:boto:Port: 8001 2024-04-12T03:31:00.145 DEBUG:boto:Params: {} 2024-04-12T03:31:00.145 DEBUG:boto:Token: None 2024-04-12T03:31:00.145 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.145 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqA/kUzxrm5Y14mrKcM2sWObE1Y= 2024-04-12T03:31:00.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqA/kUzxrm5Y14mrKcM2sWObE1Y=', 'Content-Length': '0'} 2024-04-12T03:31:00.148 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-52:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b265e18e33e1a139-006618aaf4-192d-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.230 DEBUG:boto:path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.230 DEBUG:boto:auth_path=/dgbpxg-52/MULTIPART 2024-04-12T03:31:00.230 DEBUG:boto:Method: GET 2024-04-12T03:31:00.230 DEBUG:boto:Path: /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.230 DEBUG:boto:Data: 2024-04-12T03:31:00.230 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.231 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.231 DEBUG:boto:Params: {} 2024-04-12T03:31:00.231 DEBUG:boto:Token: None 2024-04-12T03:31:00.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/MULTIPART 2024-04-12T03:31:00.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqA/kUzxrm5Y14mrKcM2sWObE1Y= 2024-04-12T03:31:00.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqA/kUzxrm5Y14mrKcM2sWObE1Y=', 'Content-Length': '0'} 2024-04-12T03:31:00.240 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e95f3b1330c58e01-006618aaf4-1921-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.329 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:31:00.329 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:31:00.329 DEBUG:boto:path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.330 DEBUG:boto:auth_path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.330 DEBUG:boto:Method: GET 2024-04-12T03:31:00.330 DEBUG:boto:Path: /dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.330 DEBUG:boto:Data: 2024-04-12T03:31:00.330 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:00.330 DEBUG:boto:Port: 8001 2024-04-12T03:31:00.330 DEBUG:boto:Params: {} 2024-04-12T03:31:00.330 DEBUG:boto:Token: None 2024-04-12T03:31:00.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/ 2024-04-12T03:31:00.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc= 2024-04-12T03:31:00.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc=', 'Content-Length': '0'} 2024-04-12T03:31:00.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90c3ed8a1b5aeab-006618aaf4-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.333 DEBUG:boto:b'dgbpxg-52MULTIPART1000falseMULTIPART2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:00.333 DEBUG:boto:path=/dgbpxg-52/ 2024-04-12T03:31:00.334 DEBUG:boto:auth_path=/dgbpxg-52/ 2024-04-12T03:31:00.334 DEBUG:boto:path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.334 DEBUG:boto:auth_path=/dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.334 DEBUG:boto:Method: GET 2024-04-12T03:31:00.334 DEBUG:boto:Path: /dgbpxg-52/?prefix=MULTIPART 2024-04-12T03:31:00.334 DEBUG:boto:Data: 2024-04-12T03:31:00.334 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.334 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.334 DEBUG:boto:Params: {} 2024-04-12T03:31:00.335 DEBUG:boto:Token: None 2024-04-12T03:31:00.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-52/ 2024-04-12T03:31:00.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc= 2024-04-12T03:31:00.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZP+xLGDye2jEBbYD6NhQekNF+Wc=', 'Content-Length': '0'} 2024-04-12T03:31:00.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051aa31603277c01c-006618aaf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.337 DEBUG:boto:b'dgbpxg-52MULTIPART1000falseMULTIPART2024-04-12T03:30:44.260Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:00.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-52 zones={a2, a1} 2024-04-12T03:31:00.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2024-04-12T03:31:00.338 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-54 2024-04-12T03:31:00.338 DEBUG:boto:path=/dgbpxg-54/ 2024-04-12T03:31:00.338 DEBUG:boto:auth_path=/dgbpxg-54/ 2024-04-12T03:31:00.339 DEBUG:boto:Method: PUT 2024-04-12T03:31:00.339 DEBUG:boto:Path: /dgbpxg-54/ 2024-04-12T03:31:00.339 DEBUG:boto:Data: 2024-04-12T03:31:00.339 DEBUG:boto:Headers: {} 2024-04-12T03:31:00.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.339 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.339 DEBUG:boto:Params: {} 2024-04-12T03:31:00.339 DEBUG:boto:Token: None 2024-04-12T03:31:00.339 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:00 GMT /dgbpxg-54/ 2024-04-12T03:31:00.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1qQniipi3Dxhkfq0MotGefGTe6Q= 2024-04-12T03:31:00.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1qQniipi3Dxhkfq0MotGefGTe6Q=', 'Content-Length': '0'} 2024-04-12T03:31:00.354 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ff4336b92e8382e-006618aaf4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.355 DEBUG:boto:path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:00.355 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:00.355 DEBUG:boto:Method: PUT 2024-04-12T03:31:00.355 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:00.355 DEBUG:boto:Data: 2024-04-12T03:31:00.355 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:31:00.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.355 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.356 DEBUG:boto:Params: {} 2024-04-12T03:31:00.356 DEBUG:boto:Token: None 2024-04-12T03:31:00.356 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:31:00 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:00.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/itOKoZvYExf30yIkjE3NCtNsjw= 2024-04-12T03:31:00.356 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/itOKoZvYExf30yIkjE3NCtNsjw='} 2024-04-12T03:31:00.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000000966a0053b8d1bbc-006618aaf4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.362 DEBUG:boto:path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:00.362 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:00.362 DEBUG:boto:Method: PUT 2024-04-12T03:31:00.363 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:00.363 DEBUG:boto:Data: 2024-04-12T03:31:00.363 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:31:00.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:00.363 DEBUG:boto:Port: 8000 2024-04-12T03:31:00.363 DEBUG:boto:Params: {} 2024-04-12T03:31:00.363 DEBUG:boto:Token: None 2024-04-12T03:31:00.363 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:31:00 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:00.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TtvqI0NrbqnBR6qJ7l7tzPhtyJM= 2024-04-12T03:31:00.364 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TtvqI0NrbqnBR6qJ7l7tzPhtyJM='} 2024-04-12T03:31:00.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZGdicHhnLTU0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000060038a8a90ec9643-006618aaf4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:00.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:00.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:00.407 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:00.456 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:00.456 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892660.353429_973.1", 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:31:00.353429Z" 2024-04-12T03:31:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.499 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:31:00.500 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892660.353429_973.1'} 2024-04-12T03:31:00.500 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:31:00.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:00.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:00.500 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:00.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:00.579+0000 7f321bff0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:00.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:00.583+0000 7f321bff0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:00.597 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:31:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:31:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892660.346625_970.1", 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:00.346625Z", 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:31:00Z" 2024-04-12T03:31:00.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:31:00.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892660.346625_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:00.346625Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:00Z" } 2024-04-12T03:31:00.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892660.346625_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:00.346625Z', 'realm_epoch': 2}}] 2024-04-12T03:31:00.605 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892660.353429_973.1'} 2024-04-12T03:31:00.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892660.346625_970.1'} 2024-04-12T03:31:00.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892660.353429_973.1 target=1_1712892660.346625_970.1 2024-04-12T03:31:05.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:05.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:05.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:05.664 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:05.664 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:05.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:05.696+0000 7f53e4bdc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:05.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:05.698+0000 7f53e4bdc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:05.713 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:31:05.713 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:31:05.713 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:31:05.713 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:05.714 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:05.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:05.716 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:05.717 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:05.718 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892660.353429_973.1", 2024-04-12T03:31:05.718 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:05.718 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:05.718 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:05.719 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:00.353429Z", 2024-04-12T03:31:05.719 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:05.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:05.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:31:05Z" 2024-04-12T03:31:05.720 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:31:05.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892660.353429_973.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:00.353429Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:05Z" } 2024-04-12T03:31:05.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892660.353429_973.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:00.353429Z', 'realm_epoch': 2}}] 2024-04-12T03:31:05.721 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892660.353429_973.1'} 2024-04-12T03:31:05.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892660.353429_973.1'} 2024-04-12T03:31:05.722 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:31:05.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:05.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:05.722 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:05.778 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:05.778 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:05.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:05.810+0000 7fd730c8b780 1 waiting to reach incremental sync.. 2024-04-12T03:31:10.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:10.811+0000 7fd730c8b780 1 waiting to reach incremental sync.. 2024-04-12T03:31:15.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:15.813+0000 7fd730c8b780 1 bucket sync caught up with source: 2024-04-12T03:31:15.815 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.6, , , , 00000000001.52.6, , , , , ] 2024-04-12T03:31:15.815 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.40.6, , , , 00000000001.52.6, , , , , ] 2024-04-12T03:31:15.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:15.813+0000 7fd730c8b780 0 bucket checkpoint complete 2024-04-12T03:31:15.830 DEBUG:boto:path=/dgbpxg-54/ 2024-04-12T03:31:15.830 DEBUG:boto:auth_path=/dgbpxg-54/ 2024-04-12T03:31:15.830 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.830 DEBUG:boto:Path: /dgbpxg-54/ 2024-04-12T03:31:15.831 DEBUG:boto:Data: 2024-04-12T03:31:15.831 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:15.831 DEBUG:boto:Port: 8001 2024-04-12T03:31:15.831 DEBUG:boto:Params: {} 2024-04-12T03:31:15.831 DEBUG:boto:Token: None 2024-04-12T03:31:15.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-54/ 2024-04-12T03:31:15.831 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:efRj1k/xPlQ2MaftdTC+C20YDhI= 2024-04-12T03:31:15.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:efRj1k/xPlQ2MaftdTC+C20YDhI=', 'Content-Length': '0'} 2024-04-12T03:31:15.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6d3a6e8a79a4313-006618ab03-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.834 DEBUG:boto:path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.834 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.834 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.834 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.834 DEBUG:boto:Data: 2024-04-12T03:31:15.835 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-12T03:31:15.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:15.835 DEBUG:boto:Port: 8001 2024-04-12T03:31:15.835 DEBUG:boto:Params: {} 2024-04-12T03:31:15.835 DEBUG:boto:Token: None 2024-04-12T03:31:15.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S63/UJxa9d0dW6hb2vs/JNh/x6M= 2024-04-12T03:31:15.835 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S63/UJxa9d0dW6hb2vs/JNh/x6M=', 'Content-Length': '0'} 2024-04-12T03:31:15.837 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-54:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000130a947e6b0bfd3f-006618ab03-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.837 DEBUG:boto:path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.837 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.837 DEBUG:boto:Method: GET 2024-04-12T03:31:15.837 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.838 DEBUG:boto:Data: 2024-04-12T03:31:15.838 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-12T03:31:15.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:15.838 DEBUG:boto:Port: 8001 2024-04-12T03:31:15.838 DEBUG:boto:Params: {} 2024-04-12T03:31:15.838 DEBUG:boto:Token: None 2024-04-12T03:31:15.838 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:15 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dgbpxg-54/testobj-sse-c 2024-04-12T03:31:15.838 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7QZbwn57PAgEMeeT6+zanyC1CTE= 2024-04-12T03:31:15.838 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7QZbwn57PAgEMeeT6+zanyC1CTE=', 'Content-Length': '0'} 2024-04-12T03:31:15.841 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-54:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000dfb0bbf59948ad0c-006618ab03-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.841 DEBUG:boto:path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.841 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.841 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.841 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.841 DEBUG:boto:Data: 2024-04-12T03:31:15.841 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:15.841 DEBUG:boto:Port: 8001 2024-04-12T03:31:15.842 DEBUG:boto:Params: {} 2024-04-12T03:31:15.842 DEBUG:boto:Token: None 2024-04-12T03:31:15.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.842 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A73l9YkF+8vpkFL9cgzf0HNdNdc= 2024-04-12T03:31:15.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A73l9YkF+8vpkFL9cgzf0HNdNdc=', 'Content-Length': '0'} 2024-04-12T03:31:15.844 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-54:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000558e4f3d0db07cfc-006618ab03-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.844 DEBUG:boto:path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.844 DEBUG:boto:auth_path=/dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.844 DEBUG:boto:Method: GET 2024-04-12T03:31:15.844 DEBUG:boto:Path: /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.844 DEBUG:boto:Data: 2024-04-12T03:31:15.845 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:15.845 DEBUG:boto:Port: 8001 2024-04-12T03:31:15.845 DEBUG:boto:Params: {} 2024-04-12T03:31:15.845 DEBUG:boto:Token: None 2024-04-12T03:31:15.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-54/testobj-sse-kms 2024-04-12T03:31:15.845 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BuMFVCKKn4V2zE5eBOZy4sivlH8= 2024-04-12T03:31:15.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BuMFVCKKn4V2zE5eBOZy4sivlH8=', 'Content-Length': '0'} 2024-04-12T03:31:15.847 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-54:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000009d84e92a8bbed48c-006618ab03-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2024-04-12T03:31:15.848 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-55 2024-04-12T03:31:15.848 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:15.848 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:15.848 DEBUG:boto:Method: PUT 2024-04-12T03:31:15.849 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:15.849 DEBUG:boto:Data: 2024-04-12T03:31:15.849 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.849 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.849 DEBUG:boto:Params: {} 2024-04-12T03:31:15.849 DEBUG:boto:Token: None 2024-04-12T03:31:15.849 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/ 2024-04-12T03:31:15.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2KYJ7YrIjaQbXpuHFQ9P2emUHb4= 2024-04-12T03:31:15.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2KYJ7YrIjaQbXpuHFQ9P2emUHb4=', 'Content-Length': '0'} 2024-04-12T03:31:15.865 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001844710d09a2af4c-006618ab03-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.866 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:15.867 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:15.867 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.867 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:15.867 DEBUG:boto:Data: 2024-04-12T03:31:15.867 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.867 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.868 DEBUG:boto:Params: {} 2024-04-12T03:31:15.868 DEBUG:boto:Token: None 2024-04-12T03:31:15.868 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/ 2024-04-12T03:31:15.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0= 2024-04-12T03:31:15.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0=', 'Content-Length': '0'} 2024-04-12T03:31:15.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008177ca90b94acf9c-006618ab03-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.871 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:15.871 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:15.871 DEBUG:boto:Method: PUT 2024-04-12T03:31:15.871 DEBUG:boto:Path: /dgbpxg-55/a 2024-04-12T03:31:15.871 DEBUG:boto:Data: 2024-04-12T03:31:15.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:15.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.871 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.871 DEBUG:boto:Params: {} 2024-04-12T03:31:15.871 DEBUG:boto:Token: None 2024-04-12T03:31:15.872 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/a 2024-04-12T03:31:15.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:86DTTnifkwnj+g6ugWe+MFVCtEQ= 2024-04-12T03:31:15.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:86DTTnifkwnj+g6ugWe+MFVCtEQ='} 2024-04-12T03:31:15.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001546b386544e0fda-006618ab03-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.878 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:15.878 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:15.878 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.878 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:15.878 DEBUG:boto:Data: 2024-04-12T03:31:15.878 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.878 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.878 DEBUG:boto:Params: {} 2024-04-12T03:31:15.878 DEBUG:boto:Token: None 2024-04-12T03:31:15.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/ 2024-04-12T03:31:15.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0= 2024-04-12T03:31:15.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0=', 'Content-Length': '0'} 2024-04-12T03:31:15.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004725e55cf5543375-006618ab03-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.881 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:15.881 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:15.881 DEBUG:boto:Method: PUT 2024-04-12T03:31:15.881 DEBUG:boto:Path: /dgbpxg-55/b 2024-04-12T03:31:15.881 DEBUG:boto:Data: 2024-04-12T03:31:15.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:15.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.882 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.882 DEBUG:boto:Params: {} 2024-04-12T03:31:15.882 DEBUG:boto:Token: None 2024-04-12T03:31:15.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/b 2024-04-12T03:31:15.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HGEMUK2LkTXRnFhhHzfZxD4vkg4= 2024-04-12T03:31:15.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HGEMUK2LkTXRnFhhHzfZxD4vkg4='} 2024-04-12T03:31:15.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000544e8cfe90dc5a23-006618ab03-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.927 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:15.927 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:15.927 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.927 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:15.927 DEBUG:boto:Data: 2024-04-12T03:31:15.927 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.927 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.927 DEBUG:boto:Params: {} 2024-04-12T03:31:15.928 DEBUG:boto:Token: None 2024-04-12T03:31:15.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/ 2024-04-12T03:31:15.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0= 2024-04-12T03:31:15.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0=', 'Content-Length': '0'} 2024-04-12T03:31:15.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000215b6b8bec761e55-006618ab03-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.930 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:15.930 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:15.930 DEBUG:boto:Method: PUT 2024-04-12T03:31:15.931 DEBUG:boto:Path: /dgbpxg-55/c 2024-04-12T03:31:15.931 DEBUG:boto:Data: 2024-04-12T03:31:15.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:15.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.931 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.931 DEBUG:boto:Params: {} 2024-04-12T03:31:15.931 DEBUG:boto:Token: None 2024-04-12T03:31:15.931 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/c 2024-04-12T03:31:15.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DqqHnB7mciuGG97DdYRMRNhdGmM= 2024-04-12T03:31:15.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DqqHnB7mciuGG97DdYRMRNhdGmM='} 2024-04-12T03:31:15.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8ce2a792b57e27f-006618ab03-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.937 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:15.937 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:15.937 DEBUG:boto:Method: HEAD 2024-04-12T03:31:15.937 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:15.937 DEBUG:boto:Data: 2024-04-12T03:31:15.937 DEBUG:boto:Headers: {} 2024-04-12T03:31:15.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.937 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.937 DEBUG:boto:Params: {} 2024-04-12T03:31:15.938 DEBUG:boto:Token: None 2024-04-12T03:31:15.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/ 2024-04-12T03:31:15.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0= 2024-04-12T03:31:15.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QZMRIZtdHDfXliL/LZdXHMKAVr0=', 'Content-Length': '0'} 2024-04-12T03:31:15.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8e63348fecc9d12-006618ab03-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.941 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:15.941 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:15.941 DEBUG:boto:Method: PUT 2024-04-12T03:31:15.941 DEBUG:boto:Path: /dgbpxg-55/d 2024-04-12T03:31:15.941 DEBUG:boto:Data: 2024-04-12T03:31:15.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:15.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:15.941 DEBUG:boto:Port: 8000 2024-04-12T03:31:15.942 DEBUG:boto:Params: {} 2024-04-12T03:31:15.942 DEBUG:boto:Token: None 2024-04-12T03:31:15.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:15 GMT /dgbpxg-55/d 2024-04-12T03:31:15.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PsL2OerkbocVz6StkB7PZOPVlOs= 2024-04-12T03:31:15.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PsL2OerkbocVz6StkB7PZOPVlOs='} 2024-04-12T03:31:15.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3cfe63bdcec06f9-006618ab03-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:15.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:15.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:15.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:15.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:15.999 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:31:16.039 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:16.040 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892675.863803_977.1", 2024-04-12T03:31:16.042 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:31:15.863803Z" 2024-04-12T03:31:16.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.042 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:31:16.043 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892675.863803_977.1'} 2024-04-12T03:31:16.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:31:16.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:16.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:16.044 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:16.120+0000 7fda0afcc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:16.121+0000 7fda0afcc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:16.134 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:31:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892675.863803_977.1", 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:15.863803Z", 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:31:16Z" 2024-04-12T03:31:16.140 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:31:16.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892675.863803_977.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:15.863803Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:16Z" } 2024-04-12T03:31:16.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892675.863803_977.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:15.863803Z', 'realm_epoch': 2}}] 2024-04-12T03:31:16.142 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892675.863803_977.1'} 2024-04-12T03:31:16.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892675.863803_977.1'} 2024-04-12T03:31:16.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:31:16.142 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-55 2024-04-12T03:31:16.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:16.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:16.142 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:16.197 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:16.197 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:16.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:16.231+0000 7fd546c83780 1 waiting to reach incremental sync.. 2024-04-12T03:31:17.009 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:31:17.008+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:31:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:21.232+0000 7fd546c83780 1 waiting to reach incremental sync.. 2024-04-12T03:31:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:26.232+0000 7fd546c83780 1 waiting to reach incremental sync.. 2024-04-12T03:31:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:31.232+0000 7fd546c83780 1 waiting to reach incremental sync.. 2024-04-12T03:31:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:36.234+0000 7fd546c83780 1 bucket sync caught up with source: 2024-04-12T03:31:36.235 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.20.6, , , , 00000000001.55.6, , , 00000000001.42.6, , 00000000001.24.6] 2024-04-12T03:31:36.235 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.20.6, , , , 00000000001.55.6, , , 00000000001.42.6, , 00000000001.24.6] 2024-04-12T03:31:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:36.234+0000 7fd546c83780 0 bucket checkpoint complete 2024-04-12T03:31:36.253 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-55 2024-04-12T03:31:36.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:36.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:36.254 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:36.314 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:36.314 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:36.349+0000 7f160b25d780 1 waiting to reach incremental sync.. 2024-04-12T03:31:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:41.351+0000 7f160b25d780 1 bucket sync caught up with source: 2024-04-12T03:31:41.352 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3036.6, , , , 00000000001.216.6, , , 00000000001.106.6, , 00000000001.1576.6] 2024-04-12T03:31:41.352 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3036.6, , , , 00000000001.216.6, , , 00000000001.106.6, , 00000000001.1576.6] 2024-04-12T03:31:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:41.351+0000 7f160b25d780 0 bucket checkpoint complete 2024-04-12T03:31:41.368 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-55 zones={a2, a1} 2024-04-12T03:31:41.369 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.369 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.369 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.369 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:41.369 DEBUG:boto:Data: 2024-04-12T03:31:41.369 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.369 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.369 DEBUG:boto:Params: {} 2024-04-12T03:31:41.370 DEBUG:boto:Token: None 2024-04-12T03:31:41.370 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GOEhdVUaNYspBAx/7GTlLfeuevQ= 2024-04-12T03:31:41.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GOEhdVUaNYspBAx/7GTlLfeuevQ=', 'Content-Length': '0'} 2024-04-12T03:31:41.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c45365d6ed3b8d5e-006618ab1d-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.373 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.373 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.373 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.373 DEBUG:boto:Path: /dgbpxg-55/ 2024-04-12T03:31:41.373 DEBUG:boto:Data: 2024-04-12T03:31:41.374 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.374 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.374 DEBUG:boto:Params: {} 2024-04-12T03:31:41.374 DEBUG:boto:Token: None 2024-04-12T03:31:41.374 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.374 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GOEhdVUaNYspBAx/7GTlLfeuevQ= 2024-04-12T03:31:41.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GOEhdVUaNYspBAx/7GTlLfeuevQ=', 'Content-Length': '0'} 2024-04-12T03:31:41.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb570810b04cdbdc-006618ab1d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:31:41.377 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.377 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.377 DEBUG:boto:path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.377 DEBUG:boto:auth_path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.377 DEBUG:boto:Method: GET 2024-04-12T03:31:41.377 DEBUG:boto:Path: /dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.378 DEBUG:boto:Data: 2024-04-12T03:31:41.378 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.378 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.378 DEBUG:boto:Params: {} 2024-04-12T03:31:41.378 DEBUG:boto:Token: None 2024-04-12T03:31:41.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/?versions 2024-04-12T03:31:41.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs= 2024-04-12T03:31:41.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs=', 'Content-Length': '0'} 2024-04-12T03:31:41.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075cd1c1302a99791-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.381 DEBUG:boto:b'dgbpxg-55999falseanulltrue2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:31:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:31:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:31:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:31:41.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:31:41.382 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.382 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.383 DEBUG:boto:path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.383 DEBUG:boto:auth_path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.383 DEBUG:boto:Method: GET 2024-04-12T03:31:41.383 DEBUG:boto:Path: /dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.383 DEBUG:boto:Data: 2024-04-12T03:31:41.383 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.383 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.383 DEBUG:boto:Params: {} 2024-04-12T03:31:41.383 DEBUG:boto:Token: None 2024-04-12T03:31:41.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/?versions 2024-04-12T03:31:41.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs= 2024-04-12T03:31:41.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs=', 'Content-Length': '0'} 2024-04-12T03:31:41.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ef0d53e8ed27b0-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.386 DEBUG:boto:b'dgbpxg-55999falseanulltrue2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.387 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:31:41.387 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:31:41.387 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:31:41.387 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:31:41.387 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.387 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.387 DEBUG:boto:path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.387 DEBUG:boto:auth_path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.388 DEBUG:boto:Method: GET 2024-04-12T03:31:41.388 DEBUG:boto:Path: /dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.388 DEBUG:boto:Data: 2024-04-12T03:31:41.388 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.388 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.388 DEBUG:boto:Params: {} 2024-04-12T03:31:41.388 DEBUG:boto:Token: None 2024-04-12T03:31:41.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/?versions 2024-04-12T03:31:41.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs= 2024-04-12T03:31:41.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs=', 'Content-Length': '0'} 2024-04-12T03:31:41.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000999dc7e324e79b60-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.391 DEBUG:boto:b'dgbpxg-55999falseanulltrue2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.391 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.392 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.392 DEBUG:boto:path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.392 DEBUG:boto:auth_path=/dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.392 DEBUG:boto:Method: GET 2024-04-12T03:31:41.392 DEBUG:boto:Path: /dgbpxg-55/?versions&max-keys=999 2024-04-12T03:31:41.392 DEBUG:boto:Data: 2024-04-12T03:31:41.392 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.392 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.392 DEBUG:boto:Params: {} 2024-04-12T03:31:41.393 DEBUG:boto:Token: None 2024-04-12T03:31:41.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/?versions 2024-04-12T03:31:41.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs= 2024-04-12T03:31:41.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Akl5l80RDECNMCgG7sl20V0nqs=', 'Content-Length': '0'} 2024-04-12T03:31:41.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548a03cf8fd9a5ca-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.395 DEBUG:boto:b'dgbpxg-55999falseanulltrue2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:31:41.396 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.396 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.396 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.396 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.397 DEBUG:boto:Method: GET 2024-04-12T03:31:41.397 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.397 DEBUG:boto:Data: 2024-04-12T03:31:41.397 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.397 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.397 DEBUG:boto:Params: {} 2024-04-12T03:31:41.397 DEBUG:boto:Token: None 2024-04-12T03:31:41.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg= 2024-04-12T03:31:41.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg=', 'Content-Length': '0'} 2024-04-12T03:31:41.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd0f6d438b9eb5c-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.400 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.400 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.400 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.400 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.400 DEBUG:boto:Method: GET 2024-04-12T03:31:41.400 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.400 DEBUG:boto:Data: 2024-04-12T03:31:41.400 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.401 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.401 DEBUG:boto:Params: {} 2024-04-12T03:31:41.401 DEBUG:boto:Token: None 2024-04-12T03:31:41.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg= 2024-04-12T03:31:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg=', 'Content-Length': '0'} 2024-04-12T03:31:41.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83aeb0967519e00-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.446 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.446 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.447 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.447 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.447 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.447 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.447 DEBUG:boto:Data: 2024-04-12T03:31:41.447 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.447 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.447 DEBUG:boto:Params: {} 2024-04-12T03:31:41.447 DEBUG:boto:Token: None 2024-04-12T03:31:41.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nPujYYToMkjvWp+XciYHOXYn6ms= 2024-04-12T03:31:41.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nPujYYToMkjvWp+XciYHOXYn6ms=', 'Content-Length': '0'} 2024-04-12T03:31:41.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008755bf4f7fb151d7-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.450 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.450 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.450 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.450 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.450 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.450 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.451 DEBUG:boto:Data: 2024-04-12T03:31:41.451 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.451 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.451 DEBUG:boto:Params: {} 2024-04-12T03:31:41.451 DEBUG:boto:Token: None 2024-04-12T03:31:41.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nPujYYToMkjvWp+XciYHOXYn6ms= 2024-04-12T03:31:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nPujYYToMkjvWp+XciYHOXYn6ms=', 'Content-Length': '0'} 2024-04-12T03:31:41.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103114cd4bd57e29-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:31:41.453 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.453 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.453 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.453 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.454 DEBUG:boto:Method: GET 2024-04-12T03:31:41.454 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.454 DEBUG:boto:Data: 2024-04-12T03:31:41.454 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.454 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.454 DEBUG:boto:Params: {} 2024-04-12T03:31:41.454 DEBUG:boto:Token: None 2024-04-12T03:31:41.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg= 2024-04-12T03:31:41.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg=', 'Content-Length': '0'} 2024-04-12T03:31:41.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dabdc9985c607d-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.457 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.457 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.457 DEBUG:boto:path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.457 DEBUG:boto:auth_path=/dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.457 DEBUG:boto:Method: GET 2024-04-12T03:31:41.457 DEBUG:boto:Path: /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.457 DEBUG:boto:Data: 2024-04-12T03:31:41.457 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.457 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.458 DEBUG:boto:Params: {} 2024-04-12T03:31:41.458 DEBUG:boto:Token: None 2024-04-12T03:31:41.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a?versionId=null 2024-04-12T03:31:41.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg= 2024-04-12T03:31:41.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BwajDzVicpE+Jzx/VmbXOaok4Fg=', 'Content-Length': '0'} 2024-04-12T03:31:41.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2854189eb11a642-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.460 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.460 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.460 DEBUG:boto:path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.460 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.461 DEBUG:boto:Method: GET 2024-04-12T03:31:41.461 DEBUG:boto:Path: /dgbpxg-55/?prefix=a 2024-04-12T03:31:41.461 DEBUG:boto:Data: 2024-04-12T03:31:41.461 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.461 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.461 DEBUG:boto:Params: {} 2024-04-12T03:31:41.461 DEBUG:boto:Token: None 2024-04-12T03:31:41.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001041d22be193970b-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.464 DEBUG:boto:b'dgbpxg-55a1000falsea2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.464 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.464 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.465 DEBUG:boto:path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.465 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.465 DEBUG:boto:Method: GET 2024-04-12T03:31:41.465 DEBUG:boto:Path: /dgbpxg-55/?prefix=a 2024-04-12T03:31:41.465 DEBUG:boto:Data: 2024-04-12T03:31:41.465 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.465 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.465 DEBUG:boto:Params: {} 2024-04-12T03:31:41.465 DEBUG:boto:Token: None 2024-04-12T03:31:41.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4595f27dd89bfbe-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.468 DEBUG:boto:b'dgbpxg-55a1000falsea2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.468 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.468 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.469 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.469 DEBUG:boto:Path: /dgbpxg-55/a 2024-04-12T03:31:41.469 DEBUG:boto:Data: 2024-04-12T03:31:41.469 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.469 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.469 DEBUG:boto:Params: {} 2024-04-12T03:31:41.469 DEBUG:boto:Token: None 2024-04-12T03:31:41.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a 2024-04-12T03:31:41.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gXAWFFsN9/bwq41vKKmj1C+Zw8E= 2024-04-12T03:31:41.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gXAWFFsN9/bwq41vKKmj1C+Zw8E=', 'Content-Length': '0'} 2024-04-12T03:31:41.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d8113b7b4674b83-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.471 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.471 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.471 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.472 DEBUG:boto:Path: /dgbpxg-55/a 2024-04-12T03:31:41.472 DEBUG:boto:Data: 2024-04-12T03:31:41.472 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.472 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.472 DEBUG:boto:Params: {} 2024-04-12T03:31:41.472 DEBUG:boto:Token: None 2024-04-12T03:31:41.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a 2024-04-12T03:31:41.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gXAWFFsN9/bwq41vKKmj1C+Zw8E= 2024-04-12T03:31:41.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gXAWFFsN9/bwq41vKKmj1C+Zw8E=', 'Content-Length': '0'} 2024-04-12T03:31:41.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939f3c55a7e9942d-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:31:41.474 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.475 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.475 DEBUG:boto:Method: GET 2024-04-12T03:31:41.475 DEBUG:boto:Path: /dgbpxg-55/a 2024-04-12T03:31:41.475 DEBUG:boto:Data: 2024-04-12T03:31:41.475 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.475 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.475 DEBUG:boto:Params: {} 2024-04-12T03:31:41.475 DEBUG:boto:Token: None 2024-04-12T03:31:41.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a 2024-04-12T03:31:41.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYfPNrMYOk8O0dZk0bUqAloaDzM= 2024-04-12T03:31:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYfPNrMYOk8O0dZk0bUqAloaDzM=', 'Content-Length': '0'} 2024-04-12T03:31:41.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f515038e992338-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.522 DEBUG:boto:path=/dgbpxg-55/a 2024-04-12T03:31:41.523 DEBUG:boto:auth_path=/dgbpxg-55/a 2024-04-12T03:31:41.523 DEBUG:boto:Method: GET 2024-04-12T03:31:41.523 DEBUG:boto:Path: /dgbpxg-55/a 2024-04-12T03:31:41.523 DEBUG:boto:Data: 2024-04-12T03:31:41.523 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.523 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.523 DEBUG:boto:Params: {} 2024-04-12T03:31:41.523 DEBUG:boto:Token: None 2024-04-12T03:31:41.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/a 2024-04-12T03:31:41.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYfPNrMYOk8O0dZk0bUqAloaDzM= 2024-04-12T03:31:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYfPNrMYOk8O0dZk0bUqAloaDzM=', 'Content-Length': '0'} 2024-04-12T03:31:41.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c575281e0a10e4-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.525 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.526 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.526 DEBUG:boto:path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.526 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.526 DEBUG:boto:Method: GET 2024-04-12T03:31:41.526 DEBUG:boto:Path: /dgbpxg-55/?prefix=a 2024-04-12T03:31:41.526 DEBUG:boto:Data: 2024-04-12T03:31:41.526 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.526 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.526 DEBUG:boto:Params: {} 2024-04-12T03:31:41.526 DEBUG:boto:Token: None 2024-04-12T03:31:41.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94d27d4cbb2f1f4-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.529 DEBUG:boto:b'dgbpxg-55a1000falsea2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.530 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.530 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.530 DEBUG:boto:path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.530 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=a 2024-04-12T03:31:41.530 DEBUG:boto:Method: GET 2024-04-12T03:31:41.530 DEBUG:boto:Path: /dgbpxg-55/?prefix=a 2024-04-12T03:31:41.530 DEBUG:boto:Data: 2024-04-12T03:31:41.530 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.531 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.531 DEBUG:boto:Params: {} 2024-04-12T03:31:41.531 DEBUG:boto:Token: None 2024-04-12T03:31:41.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3488ece7434a5a-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.534 DEBUG:boto:b'dgbpxg-55a1000falsea2024-04-12T03:31:15.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:31:41.534 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.535 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.535 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.535 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.535 DEBUG:boto:Method: GET 2024-04-12T03:31:41.535 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.535 DEBUG:boto:Data: 2024-04-12T03:31:41.535 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.535 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.535 DEBUG:boto:Params: {} 2024-04-12T03:31:41.535 DEBUG:boto:Token: None 2024-04-12T03:31:41.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4= 2024-04-12T03:31:41.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4=', 'Content-Length': '0'} 2024-04-12T03:31:41.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad8f9b8aa81e59b-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.537 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.538 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.538 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.538 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.538 DEBUG:boto:Method: GET 2024-04-12T03:31:41.538 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.538 DEBUG:boto:Data: 2024-04-12T03:31:41.538 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.538 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.538 DEBUG:boto:Params: {} 2024-04-12T03:31:41.539 DEBUG:boto:Token: None 2024-04-12T03:31:41.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4= 2024-04-12T03:31:41.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4=', 'Content-Length': '0'} 2024-04-12T03:31:41.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591c82d8d9343fba-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.541 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.541 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.541 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.541 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.541 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.541 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.541 DEBUG:boto:Data: 2024-04-12T03:31:41.541 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.542 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.542 DEBUG:boto:Params: {} 2024-04-12T03:31:41.542 DEBUG:boto:Token: None 2024-04-12T03:31:41.542 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G1PaGS7bGNg2mW9RKWV36umkYjg= 2024-04-12T03:31:41.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G1PaGS7bGNg2mW9RKWV36umkYjg=', 'Content-Length': '0'} 2024-04-12T03:31:41.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092d2b4cc436f4ac-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.544 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.544 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.544 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.544 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.544 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.544 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.544 DEBUG:boto:Data: 2024-04-12T03:31:41.544 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.544 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.545 DEBUG:boto:Params: {} 2024-04-12T03:31:41.545 DEBUG:boto:Token: None 2024-04-12T03:31:41.545 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G1PaGS7bGNg2mW9RKWV36umkYjg= 2024-04-12T03:31:41.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G1PaGS7bGNg2mW9RKWV36umkYjg=', 'Content-Length': '0'} 2024-04-12T03:31:41.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857a30651c1800a6-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:31:41.547 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.547 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.547 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.547 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.547 DEBUG:boto:Method: GET 2024-04-12T03:31:41.547 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.547 DEBUG:boto:Data: 2024-04-12T03:31:41.547 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.547 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.548 DEBUG:boto:Params: {} 2024-04-12T03:31:41.548 DEBUG:boto:Token: None 2024-04-12T03:31:41.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4= 2024-04-12T03:31:41.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4=', 'Content-Length': '0'} 2024-04-12T03:31:41.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fbfc4ea0f72c57-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.550 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.550 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.550 DEBUG:boto:path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.550 DEBUG:boto:auth_path=/dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.550 DEBUG:boto:Method: GET 2024-04-12T03:31:41.550 DEBUG:boto:Path: /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.550 DEBUG:boto:Data: 2024-04-12T03:31:41.551 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.551 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.551 DEBUG:boto:Params: {} 2024-04-12T03:31:41.551 DEBUG:boto:Token: None 2024-04-12T03:31:41.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b?versionId=null 2024-04-12T03:31:41.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4= 2024-04-12T03:31:41.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+dIZolcMyaTIUv8bQHTTgFngv4=', 'Content-Length': '0'} 2024-04-12T03:31:41.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed33d7ab985cdf84-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.553 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.553 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.553 DEBUG:boto:path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.553 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.554 DEBUG:boto:Method: GET 2024-04-12T03:31:41.554 DEBUG:boto:Path: /dgbpxg-55/?prefix=b 2024-04-12T03:31:41.554 DEBUG:boto:Data: 2024-04-12T03:31:41.554 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.554 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.554 DEBUG:boto:Params: {} 2024-04-12T03:31:41.554 DEBUG:boto:Token: None 2024-04-12T03:31:41.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebd842edd6eb0f4-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.557 DEBUG:boto:b'dgbpxg-55b1000falseb2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.557 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.557 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.558 DEBUG:boto:path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.558 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.558 DEBUG:boto:Method: GET 2024-04-12T03:31:41.558 DEBUG:boto:Path: /dgbpxg-55/?prefix=b 2024-04-12T03:31:41.558 DEBUG:boto:Data: 2024-04-12T03:31:41.558 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.558 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.558 DEBUG:boto:Params: {} 2024-04-12T03:31:41.558 DEBUG:boto:Token: None 2024-04-12T03:31:41.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.559 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019721f1ac0e82a8e-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.561 DEBUG:boto:b'dgbpxg-55b1000falseb2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.562 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.562 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.562 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.562 DEBUG:boto:Path: /dgbpxg-55/b 2024-04-12T03:31:41.562 DEBUG:boto:Data: 2024-04-12T03:31:41.562 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.562 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.562 DEBUG:boto:Params: {} 2024-04-12T03:31:41.563 DEBUG:boto:Token: None 2024-04-12T03:31:41.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b 2024-04-12T03:31:41.563 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9HxOGIYg8KPcKkRvWuKEVhBYJp4= 2024-04-12T03:31:41.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9HxOGIYg8KPcKkRvWuKEVhBYJp4=', 'Content-Length': '0'} 2024-04-12T03:31:41.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6551cb8024fe186-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.565 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.565 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.565 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.565 DEBUG:boto:Path: /dgbpxg-55/b 2024-04-12T03:31:41.565 DEBUG:boto:Data: 2024-04-12T03:31:41.565 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.565 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.565 DEBUG:boto:Params: {} 2024-04-12T03:31:41.565 DEBUG:boto:Token: None 2024-04-12T03:31:41.566 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b 2024-04-12T03:31:41.566 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9HxOGIYg8KPcKkRvWuKEVhBYJp4= 2024-04-12T03:31:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9HxOGIYg8KPcKkRvWuKEVhBYJp4=', 'Content-Length': '0'} 2024-04-12T03:31:41.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2a1240d581df89-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:31:41.568 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.568 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.568 DEBUG:boto:Method: GET 2024-04-12T03:31:41.568 DEBUG:boto:Path: /dgbpxg-55/b 2024-04-12T03:31:41.568 DEBUG:boto:Data: 2024-04-12T03:31:41.568 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.568 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.568 DEBUG:boto:Params: {} 2024-04-12T03:31:41.569 DEBUG:boto:Token: None 2024-04-12T03:31:41.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b 2024-04-12T03:31:41.569 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GYKEg8M+iGOLOJLSO9W1kp91qSw= 2024-04-12T03:31:41.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYKEg8M+iGOLOJLSO9W1kp91qSw=', 'Content-Length': '0'} 2024-04-12T03:31:41.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d785b4fbbb09f4-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.571 DEBUG:boto:path=/dgbpxg-55/b 2024-04-12T03:31:41.571 DEBUG:boto:auth_path=/dgbpxg-55/b 2024-04-12T03:31:41.571 DEBUG:boto:Method: GET 2024-04-12T03:31:41.571 DEBUG:boto:Path: /dgbpxg-55/b 2024-04-12T03:31:41.571 DEBUG:boto:Data: 2024-04-12T03:31:41.571 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.572 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.572 DEBUG:boto:Params: {} 2024-04-12T03:31:41.572 DEBUG:boto:Token: None 2024-04-12T03:31:41.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/b 2024-04-12T03:31:41.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GYKEg8M+iGOLOJLSO9W1kp91qSw= 2024-04-12T03:31:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYKEg8M+iGOLOJLSO9W1kp91qSw=', 'Content-Length': '0'} 2024-04-12T03:31:41.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba57607a93ca182-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.574 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.574 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.574 DEBUG:boto:path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.575 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.575 DEBUG:boto:Method: GET 2024-04-12T03:31:41.575 DEBUG:boto:Path: /dgbpxg-55/?prefix=b 2024-04-12T03:31:41.575 DEBUG:boto:Data: 2024-04-12T03:31:41.575 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.575 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.575 DEBUG:boto:Params: {} 2024-04-12T03:31:41.575 DEBUG:boto:Token: None 2024-04-12T03:31:41.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af83b51521e448f-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.578 DEBUG:boto:b'dgbpxg-55b1000falseb2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.579 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.579 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.579 DEBUG:boto:path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.579 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=b 2024-04-12T03:31:41.579 DEBUG:boto:Method: GET 2024-04-12T03:31:41.579 DEBUG:boto:Path: /dgbpxg-55/?prefix=b 2024-04-12T03:31:41.579 DEBUG:boto:Data: 2024-04-12T03:31:41.579 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.579 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.579 DEBUG:boto:Params: {} 2024-04-12T03:31:41.580 DEBUG:boto:Token: None 2024-04-12T03:31:41.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf89cd37e1dbcbd8-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.582 DEBUG:boto:b'dgbpxg-55b1000falseb2024-04-12T03:31:15.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:31:41.583 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.583 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.583 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.583 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.584 DEBUG:boto:Method: GET 2024-04-12T03:31:41.584 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.584 DEBUG:boto:Data: 2024-04-12T03:31:41.584 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.584 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.584 DEBUG:boto:Params: {} 2024-04-12T03:31:41.585 DEBUG:boto:Token: None 2024-04-12T03:31:41.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs= 2024-04-12T03:31:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs=', 'Content-Length': '0'} 2024-04-12T03:31:41.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f9611de4c97444-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.630 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.631 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.631 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.631 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.631 DEBUG:boto:Method: GET 2024-04-12T03:31:41.631 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.631 DEBUG:boto:Data: 2024-04-12T03:31:41.631 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.631 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.631 DEBUG:boto:Params: {} 2024-04-12T03:31:41.632 DEBUG:boto:Token: None 2024-04-12T03:31:41.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs= 2024-04-12T03:31:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs=', 'Content-Length': '0'} 2024-04-12T03:31:41.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1fdc38fae88893-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.634 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.635 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.635 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.635 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.635 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.635 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.635 DEBUG:boto:Data: 2024-04-12T03:31:41.635 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.635 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.635 DEBUG:boto:Params: {} 2024-04-12T03:31:41.635 DEBUG:boto:Token: None 2024-04-12T03:31:41.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Uv1bXKQApzvU0oSUbijKcI+UO8= 2024-04-12T03:31:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Uv1bXKQApzvU0oSUbijKcI+UO8=', 'Content-Length': '0'} 2024-04-12T03:31:41.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000635aab9c1e081d3f-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.638 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.638 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.638 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.638 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.638 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.638 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.638 DEBUG:boto:Data: 2024-04-12T03:31:41.638 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.639 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.639 DEBUG:boto:Params: {} 2024-04-12T03:31:41.639 DEBUG:boto:Token: None 2024-04-12T03:31:41.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Uv1bXKQApzvU0oSUbijKcI+UO8= 2024-04-12T03:31:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Uv1bXKQApzvU0oSUbijKcI+UO8=', 'Content-Length': '0'} 2024-04-12T03:31:41.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301626ac3c31e656-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:31:41.641 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.641 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.641 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.641 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.641 DEBUG:boto:Method: GET 2024-04-12T03:31:41.641 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.642 DEBUG:boto:Data: 2024-04-12T03:31:41.642 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.642 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.642 DEBUG:boto:Params: {} 2024-04-12T03:31:41.642 DEBUG:boto:Token: None 2024-04-12T03:31:41.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs= 2024-04-12T03:31:41.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs=', 'Content-Length': '0'} 2024-04-12T03:31:41.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d188c83490002417-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.644 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.644 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.644 DEBUG:boto:path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.644 DEBUG:boto:auth_path=/dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.644 DEBUG:boto:Method: GET 2024-04-12T03:31:41.645 DEBUG:boto:Path: /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.645 DEBUG:boto:Data: 2024-04-12T03:31:41.645 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.645 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.645 DEBUG:boto:Params: {} 2024-04-12T03:31:41.645 DEBUG:boto:Token: None 2024-04-12T03:31:41.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c?versionId=null 2024-04-12T03:31:41.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs= 2024-04-12T03:31:41.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D3D9cYhTxWSi46Zh6mZ0iPAjALs=', 'Content-Length': '0'} 2024-04-12T03:31:41.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ef838ddb8ebfc8-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.647 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.647 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.647 DEBUG:boto:path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.647 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.648 DEBUG:boto:Method: GET 2024-04-12T03:31:41.648 DEBUG:boto:Path: /dgbpxg-55/?prefix=c 2024-04-12T03:31:41.648 DEBUG:boto:Data: 2024-04-12T03:31:41.648 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.648 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.648 DEBUG:boto:Params: {} 2024-04-12T03:31:41.648 DEBUG:boto:Token: None 2024-04-12T03:31:41.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d9d1eef367c859-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.651 DEBUG:boto:b'dgbpxg-55c1000falsec2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.651 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.651 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.651 DEBUG:boto:path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.651 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.652 DEBUG:boto:Method: GET 2024-04-12T03:31:41.652 DEBUG:boto:Path: /dgbpxg-55/?prefix=c 2024-04-12T03:31:41.652 DEBUG:boto:Data: 2024-04-12T03:31:41.652 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.652 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.652 DEBUG:boto:Params: {} 2024-04-12T03:31:41.652 DEBUG:boto:Token: None 2024-04-12T03:31:41.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f372588e6262508a-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.702 DEBUG:boto:b'dgbpxg-55c1000falsec2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.703 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.703 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.703 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.703 DEBUG:boto:Path: /dgbpxg-55/c 2024-04-12T03:31:41.703 DEBUG:boto:Data: 2024-04-12T03:31:41.703 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.703 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.704 DEBUG:boto:Params: {} 2024-04-12T03:31:41.704 DEBUG:boto:Token: None 2024-04-12T03:31:41.704 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c 2024-04-12T03:31:41.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6hFEpSspem32QLhNkrb9bY1h44= 2024-04-12T03:31:41.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6hFEpSspem32QLhNkrb9bY1h44=', 'Content-Length': '0'} 2024-04-12T03:31:41.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9430a0cdbadcaa-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.706 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.706 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.706 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.706 DEBUG:boto:Path: /dgbpxg-55/c 2024-04-12T03:31:41.706 DEBUG:boto:Data: 2024-04-12T03:31:41.706 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.707 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.707 DEBUG:boto:Params: {} 2024-04-12T03:31:41.707 DEBUG:boto:Token: None 2024-04-12T03:31:41.707 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c 2024-04-12T03:31:41.707 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6hFEpSspem32QLhNkrb9bY1h44= 2024-04-12T03:31:41.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6hFEpSspem32QLhNkrb9bY1h44=', 'Content-Length': '0'} 2024-04-12T03:31:41.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7fa7090d62a4ce-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:31:41.709 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.709 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.709 DEBUG:boto:Method: GET 2024-04-12T03:31:41.709 DEBUG:boto:Path: /dgbpxg-55/c 2024-04-12T03:31:41.709 DEBUG:boto:Data: 2024-04-12T03:31:41.709 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.710 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.710 DEBUG:boto:Params: {} 2024-04-12T03:31:41.710 DEBUG:boto:Token: None 2024-04-12T03:31:41.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c 2024-04-12T03:31:41.710 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XyHBIKEDl/1DZiFL38Pyq9WAu1c= 2024-04-12T03:31:41.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XyHBIKEDl/1DZiFL38Pyq9WAu1c=', 'Content-Length': '0'} 2024-04-12T03:31:41.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488d1ac82d7f66d3-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.712 DEBUG:boto:path=/dgbpxg-55/c 2024-04-12T03:31:41.712 DEBUG:boto:auth_path=/dgbpxg-55/c 2024-04-12T03:31:41.712 DEBUG:boto:Method: GET 2024-04-12T03:31:41.712 DEBUG:boto:Path: /dgbpxg-55/c 2024-04-12T03:31:41.712 DEBUG:boto:Data: 2024-04-12T03:31:41.712 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.713 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.713 DEBUG:boto:Params: {} 2024-04-12T03:31:41.713 DEBUG:boto:Token: None 2024-04-12T03:31:41.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/c 2024-04-12T03:31:41.713 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XyHBIKEDl/1DZiFL38Pyq9WAu1c= 2024-04-12T03:31:41.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XyHBIKEDl/1DZiFL38Pyq9WAu1c=', 'Content-Length': '0'} 2024-04-12T03:31:41.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592b6d88ff9c26a9-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.715 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.715 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.715 DEBUG:boto:path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.715 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.716 DEBUG:boto:Method: GET 2024-04-12T03:31:41.716 DEBUG:boto:Path: /dgbpxg-55/?prefix=c 2024-04-12T03:31:41.717 DEBUG:boto:Data: 2024-04-12T03:31:41.717 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.717 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.717 DEBUG:boto:Params: {} 2024-04-12T03:31:41.717 DEBUG:boto:Token: None 2024-04-12T03:31:41.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61ca9c3af2bdb2e-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.719 DEBUG:boto:b'dgbpxg-55c1000falsec2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.720 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.720 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.720 DEBUG:boto:path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.720 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=c 2024-04-12T03:31:41.720 DEBUG:boto:Method: GET 2024-04-12T03:31:41.720 DEBUG:boto:Path: /dgbpxg-55/?prefix=c 2024-04-12T03:31:41.720 DEBUG:boto:Data: 2024-04-12T03:31:41.721 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.721 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.721 DEBUG:boto:Params: {} 2024-04-12T03:31:41.721 DEBUG:boto:Token: None 2024-04-12T03:31:41.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf74c21cb717521-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.723 DEBUG:boto:b'dgbpxg-55c1000falsec2024-04-12T03:31:15.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:31:41.724 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.724 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.724 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.724 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.724 DEBUG:boto:Method: GET 2024-04-12T03:31:41.724 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.724 DEBUG:boto:Data: 2024-04-12T03:31:41.724 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.725 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.725 DEBUG:boto:Params: {} 2024-04-12T03:31:41.725 DEBUG:boto:Token: None 2024-04-12T03:31:41.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8= 2024-04-12T03:31:41.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8=', 'Content-Length': '0'} 2024-04-12T03:31:41.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d288e4d068ac3e36-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.727 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.727 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.727 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.727 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.727 DEBUG:boto:Method: GET 2024-04-12T03:31:41.728 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.728 DEBUG:boto:Data: 2024-04-12T03:31:41.728 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.728 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.728 DEBUG:boto:Params: {} 2024-04-12T03:31:41.728 DEBUG:boto:Token: None 2024-04-12T03:31:41.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8= 2024-04-12T03:31:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8=', 'Content-Length': '0'} 2024-04-12T03:31:41.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269559ad14101de1-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.770 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.770 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.771 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.771 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.771 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.771 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.771 DEBUG:boto:Data: 2024-04-12T03:31:41.771 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.771 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.771 DEBUG:boto:Params: {} 2024-04-12T03:31:41.771 DEBUG:boto:Token: None 2024-04-12T03:31:41.772 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZXuTFaz5v0Tn0E3V4M96IvAp+2I= 2024-04-12T03:31:41.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZXuTFaz5v0Tn0E3V4M96IvAp+2I=', 'Content-Length': '0'} 2024-04-12T03:31:41.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fbb3be95bc81fc8-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.774 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.774 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.774 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.774 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.774 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.774 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.775 DEBUG:boto:Data: 2024-04-12T03:31:41.775 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.775 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.775 DEBUG:boto:Params: {} 2024-04-12T03:31:41.775 DEBUG:boto:Token: None 2024-04-12T03:31:41.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZXuTFaz5v0Tn0E3V4M96IvAp+2I= 2024-04-12T03:31:41.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZXuTFaz5v0Tn0E3V4M96IvAp+2I=', 'Content-Length': '0'} 2024-04-12T03:31:41.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079789da7558c0dc2-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:31:41.777 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.777 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.777 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.778 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.778 DEBUG:boto:Method: GET 2024-04-12T03:31:41.778 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.778 DEBUG:boto:Data: 2024-04-12T03:31:41.778 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.778 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.778 DEBUG:boto:Params: {} 2024-04-12T03:31:41.778 DEBUG:boto:Token: None 2024-04-12T03:31:41.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8= 2024-04-12T03:31:41.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8=', 'Content-Length': '0'} 2024-04-12T03:31:41.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0a83aa086a4cc78-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.781 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.781 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.781 DEBUG:boto:path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.781 DEBUG:boto:auth_path=/dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.781 DEBUG:boto:Method: GET 2024-04-12T03:31:41.781 DEBUG:boto:Path: /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.781 DEBUG:boto:Data: 2024-04-12T03:31:41.782 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.782 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.782 DEBUG:boto:Params: {} 2024-04-12T03:31:41.782 DEBUG:boto:Token: None 2024-04-12T03:31:41.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d?versionId=null 2024-04-12T03:31:41.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8= 2024-04-12T03:31:41.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rgj22mBD1M6MuguRuM+midSiDi8=', 'Content-Length': '0'} 2024-04-12T03:31:41.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a26f9dd614e425-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.784 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.784 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.784 DEBUG:boto:path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.784 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.785 DEBUG:boto:Method: GET 2024-04-12T03:31:41.785 DEBUG:boto:Path: /dgbpxg-55/?prefix=d 2024-04-12T03:31:41.785 DEBUG:boto:Data: 2024-04-12T03:31:41.785 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.785 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.785 DEBUG:boto:Params: {} 2024-04-12T03:31:41.785 DEBUG:boto:Token: None 2024-04-12T03:31:41.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b74abcdc3dbe5d-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.788 DEBUG:boto:b'dgbpxg-55d1000falsed2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.788 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.788 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.788 DEBUG:boto:path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.789 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.789 DEBUG:boto:Method: GET 2024-04-12T03:31:41.789 DEBUG:boto:Path: /dgbpxg-55/?prefix=d 2024-04-12T03:31:41.789 DEBUG:boto:Data: 2024-04-12T03:31:41.789 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.789 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.790 DEBUG:boto:Params: {} 2024-04-12T03:31:41.790 DEBUG:boto:Token: None 2024-04-12T03:31:41.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0937cc663b3ada-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.792 DEBUG:boto:b'dgbpxg-55d1000falsed2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.792 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.792 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.793 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.793 DEBUG:boto:Path: /dgbpxg-55/d 2024-04-12T03:31:41.793 DEBUG:boto:Data: 2024-04-12T03:31:41.793 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.793 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.793 DEBUG:boto:Params: {} 2024-04-12T03:31:41.793 DEBUG:boto:Token: None 2024-04-12T03:31:41.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d 2024-04-12T03:31:41.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Abot1UGNcZWjrRBI2PzfNopUrXg= 2024-04-12T03:31:41.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Abot1UGNcZWjrRBI2PzfNopUrXg=', 'Content-Length': '0'} 2024-04-12T03:31:41.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005474656b798cb0cc-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.795 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.795 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.795 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.795 DEBUG:boto:Path: /dgbpxg-55/d 2024-04-12T03:31:41.795 DEBUG:boto:Data: 2024-04-12T03:31:41.796 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.796 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.796 DEBUG:boto:Params: {} 2024-04-12T03:31:41.796 DEBUG:boto:Token: None 2024-04-12T03:31:41.796 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d 2024-04-12T03:31:41.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Abot1UGNcZWjrRBI2PzfNopUrXg= 2024-04-12T03:31:41.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Abot1UGNcZWjrRBI2PzfNopUrXg=', 'Content-Length': '0'} 2024-04-12T03:31:41.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005025c0fbfcac15a6-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:31:41.798 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.798 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.798 DEBUG:boto:Method: GET 2024-04-12T03:31:41.798 DEBUG:boto:Path: /dgbpxg-55/d 2024-04-12T03:31:41.798 DEBUG:boto:Data: 2024-04-12T03:31:41.798 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.799 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.799 DEBUG:boto:Params: {} 2024-04-12T03:31:41.799 DEBUG:boto:Token: None 2024-04-12T03:31:41.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d 2024-04-12T03:31:41.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsFAU/uQJX/qLAKXh3uasu+ttNw= 2024-04-12T03:31:41.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsFAU/uQJX/qLAKXh3uasu+ttNw=', 'Content-Length': '0'} 2024-04-12T03:31:41.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3a38aea9d960f2-006618ab1d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.842 DEBUG:boto:path=/dgbpxg-55/d 2024-04-12T03:31:41.843 DEBUG:boto:auth_path=/dgbpxg-55/d 2024-04-12T03:31:41.843 DEBUG:boto:Method: GET 2024-04-12T03:31:41.843 DEBUG:boto:Path: /dgbpxg-55/d 2024-04-12T03:31:41.843 DEBUG:boto:Data: 2024-04-12T03:31:41.843 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.843 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.844 DEBUG:boto:Params: {} 2024-04-12T03:31:41.844 DEBUG:boto:Token: None 2024-04-12T03:31:41.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/d 2024-04-12T03:31:41.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsFAU/uQJX/qLAKXh3uasu+ttNw= 2024-04-12T03:31:41.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsFAU/uQJX/qLAKXh3uasu+ttNw=', 'Content-Length': '0'} 2024-04-12T03:31:41.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094903eb5c1cba68a-006618ab1d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.846 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.847 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.847 DEBUG:boto:path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.847 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.847 DEBUG:boto:Method: GET 2024-04-12T03:31:41.847 DEBUG:boto:Path: /dgbpxg-55/?prefix=d 2024-04-12T03:31:41.847 DEBUG:boto:Data: 2024-04-12T03:31:41.847 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:31:41.847 DEBUG:boto:Port: 8001 2024-04-12T03:31:41.848 DEBUG:boto:Params: {} 2024-04-12T03:31:41.848 DEBUG:boto:Token: None 2024-04-12T03:31:41.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e219f2a122ac3f98-006618ab1d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.850 DEBUG:boto:b'dgbpxg-55d1000falsed2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.851 DEBUG:boto:path=/dgbpxg-55/ 2024-04-12T03:31:41.851 DEBUG:boto:auth_path=/dgbpxg-55/ 2024-04-12T03:31:41.851 DEBUG:boto:path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.851 DEBUG:boto:auth_path=/dgbpxg-55/?prefix=d 2024-04-12T03:31:41.851 DEBUG:boto:Method: GET 2024-04-12T03:31:41.851 DEBUG:boto:Path: /dgbpxg-55/?prefix=d 2024-04-12T03:31:41.851 DEBUG:boto:Data: 2024-04-12T03:31:41.851 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.852 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.852 DEBUG:boto:Params: {} 2024-04-12T03:31:41.852 DEBUG:boto:Token: None 2024-04-12T03:31:41.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-55/ 2024-04-12T03:31:41.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM= 2024-04-12T03:31:41.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tctqnEDYQ5Yr/lLAAWVHdWPVQvM=', 'Content-Length': '0'} 2024-04-12T03:31:41.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0862efc90d77c55-006618ab1d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.856 DEBUG:boto:b'dgbpxg-55d1000falsed2024-04-12T03:31:15.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:41.856 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-55 zones={a2, a1} 2024-04-12T03:31:41.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:31:41.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:31:41.857 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:31:41.901 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:41.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:41.945 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-56 2024-04-12T03:31:41.945 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:31:41.946 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:31:41.946 DEBUG:boto:Method: PUT 2024-04-12T03:31:41.946 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:31:41.946 DEBUG:boto:Data: 2024-04-12T03:31:41.946 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.946 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.946 DEBUG:boto:Params: {} 2024-04-12T03:31:41.946 DEBUG:boto:Token: None 2024-04-12T03:31:41.947 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-56/ 2024-04-12T03:31:41.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yQ9gxJoS42l6HHU7yjclWDbPWf8= 2024-04-12T03:31:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yQ9gxJoS42l6HHU7yjclWDbPWf8=', 'Content-Length': '0'} 2024-04-12T03:31:41.961 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f819d6b4f06dbb9-006618ab1d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.961 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:31:41.961 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:31:41.962 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.962 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:31:41.962 DEBUG:boto:Data: 2024-04-12T03:31:41.962 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.962 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.962 DEBUG:boto:Params: {} 2024-04-12T03:31:41.962 DEBUG:boto:Token: None 2024-04-12T03:31:41.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-56/ 2024-04-12T03:31:41.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJEVdEoyOpiK0/DfEaULLEP70dU= 2024-04-12T03:31:41.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJEVdEoyOpiK0/DfEaULLEP70dU=', 'Content-Length': '0'} 2024-04-12T03:31:41.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040d7481f6823ff10-006618ab1d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.965 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:31:41.965 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:31:41.965 DEBUG:boto:Method: PUT 2024-04-12T03:31:41.965 DEBUG:boto:Path: /dgbpxg-56/a 2024-04-12T03:31:41.965 DEBUG:boto:Data: 2024-04-12T03:31:41.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:41.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.966 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.966 DEBUG:boto:Params: {} 2024-04-12T03:31:41.966 DEBUG:boto:Token: None 2024-04-12T03:31:41.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-56/a 2024-04-12T03:31:41.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XfTLtXcgqA8Qcx72hOTureLWrPY= 2024-04-12T03:31:41.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XfTLtXcgqA8Qcx72hOTureLWrPY='} 2024-04-12T03:31:41.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d576572d42ee4a4-006618ab1d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.971 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:31:41.972 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:31:41.972 DEBUG:boto:Method: HEAD 2024-04-12T03:31:41.972 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:31:41.972 DEBUG:boto:Data: 2024-04-12T03:31:41.972 DEBUG:boto:Headers: {} 2024-04-12T03:31:41.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.972 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.972 DEBUG:boto:Params: {} 2024-04-12T03:31:41.972 DEBUG:boto:Token: None 2024-04-12T03:31:41.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-56/ 2024-04-12T03:31:41.973 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJEVdEoyOpiK0/DfEaULLEP70dU= 2024-04-12T03:31:41.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJEVdEoyOpiK0/DfEaULLEP70dU=', 'Content-Length': '0'} 2024-04-12T03:31:41.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdd7def2d780a1c1-006618ab1d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:41.975 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:31:41.975 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:31:41.975 DEBUG:boto:Method: PUT 2024-04-12T03:31:41.975 DEBUG:boto:Path: /dgbpxg-56/b 2024-04-12T03:31:41.975 DEBUG:boto:Data: 2024-04-12T03:31:41.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:41.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:41.976 DEBUG:boto:Port: 8000 2024-04-12T03:31:41.976 DEBUG:boto:Params: {} 2024-04-12T03:31:41.976 DEBUG:boto:Token: None 2024-04-12T03:31:41.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:41 GMT /dgbpxg-56/b 2024-04-12T03:31:41.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ptb4kv0Re0U46wsrEZC1ljSSZvs= 2024-04-12T03:31:41.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ptb4kv0Re0U46wsrEZC1ljSSZvs='} 2024-04-12T03:31:42.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a11d296c7ea9173-006618ab1d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:42.019 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:31:42.019 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:31:42.019 DEBUG:boto:Method: HEAD 2024-04-12T03:31:42.019 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:31:42.019 DEBUG:boto:Data: 2024-04-12T03:31:42.019 DEBUG:boto:Headers: {} 2024-04-12T03:31:42.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:42.020 DEBUG:boto:Port: 8000 2024-04-12T03:31:42.020 DEBUG:boto:Params: {} 2024-04-12T03:31:42.020 DEBUG:boto:Token: None 2024-04-12T03:31:42.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:42 GMT /dgbpxg-56/ 2024-04-12T03:31:42.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vxiYb1OTLdy/IZCJ6rHb9UOmYGk= 2024-04-12T03:31:42.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vxiYb1OTLdy/IZCJ6rHb9UOmYGk=', 'Content-Length': '0'} 2024-04-12T03:31:42.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c17401144818e2b3-006618ab1e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:42.023 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:31:42.023 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:31:42.023 DEBUG:boto:Method: PUT 2024-04-12T03:31:42.023 DEBUG:boto:Path: /dgbpxg-56/c 2024-04-12T03:31:42.023 DEBUG:boto:Data: 2024-04-12T03:31:42.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:42.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:42.023 DEBUG:boto:Port: 8000 2024-04-12T03:31:42.023 DEBUG:boto:Params: {} 2024-04-12T03:31:42.023 DEBUG:boto:Token: None 2024-04-12T03:31:42.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:42 GMT /dgbpxg-56/c 2024-04-12T03:31:42.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:24RcU8U72WlsipEOypaeBBQj/Z4= 2024-04-12T03:31:42.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:24RcU8U72WlsipEOypaeBBQj/Z4='} 2024-04-12T03:31:42.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbdcb4277dffbbed-006618ab1e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:42.029 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:31:42.029 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:31:42.029 DEBUG:boto:Method: HEAD 2024-04-12T03:31:42.029 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:31:42.029 DEBUG:boto:Data: 2024-04-12T03:31:42.029 DEBUG:boto:Headers: {} 2024-04-12T03:31:42.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:42.029 DEBUG:boto:Port: 8000 2024-04-12T03:31:42.030 DEBUG:boto:Params: {} 2024-04-12T03:31:42.030 DEBUG:boto:Token: None 2024-04-12T03:31:42.030 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:42 GMT /dgbpxg-56/ 2024-04-12T03:31:42.030 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vxiYb1OTLdy/IZCJ6rHb9UOmYGk= 2024-04-12T03:31:42.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vxiYb1OTLdy/IZCJ6rHb9UOmYGk=', 'Content-Length': '0'} 2024-04-12T03:31:42.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a9c05ca08116ded-006618ab1e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:42.032 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:31:42.032 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:31:42.033 DEBUG:boto:Method: PUT 2024-04-12T03:31:42.033 DEBUG:boto:Path: /dgbpxg-56/d 2024-04-12T03:31:42.033 DEBUG:boto:Data: 2024-04-12T03:31:42.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:31:42.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:31:42.033 DEBUG:boto:Port: 8000 2024-04-12T03:31:42.033 DEBUG:boto:Params: {} 2024-04-12T03:31:42.033 DEBUG:boto:Token: None 2024-04-12T03:31:42.033 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:31:42 GMT /dgbpxg-56/d 2024-04-12T03:31:42.033 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0VEwe9KQ1Dnh8JZupN4xyXFSiA= 2024-04-12T03:31:42.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0VEwe9KQ1Dnh8JZupN4xyXFSiA='} 2024-04-12T03:31:42.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aaeb4474cc0c23ae-006618ab1e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:42.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:42.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:42.079 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:42.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:42.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:42.155 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:31:42.168 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:31:42.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:42.168 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:31:42.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892701.959688_993.1", 2024-04-12T03:31:42.169 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:31:41.959688Z" 2024-04-12T03:31:42.170 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.170 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:31:42.171 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892701.959688_993.1'} 2024-04-12T03:31:42.171 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:31:42.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:42.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:42.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:42.221 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:42.221 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:42.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:42.249+0000 7f7843040780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:42.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:42.250+0000 7f7843040780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:42.263 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.264 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.265 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:42.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892701.959688_993.1", 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:41.959688Z", 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:31:42.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:31:42Z" 2024-04-12T03:31:42.269 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:31:42.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892701.959688_993.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:41.959688Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:42Z" } 2024-04-12T03:31:42.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892701.959688_993.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:41.959688Z', 'realm_epoch': 2}}] 2024-04-12T03:31:42.270 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892701.959688_993.1'} 2024-04-12T03:31:42.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892701.959688_993.1'} 2024-04-12T03:31:42.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:31:42.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-56 2024-04-12T03:31:42.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:42.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:42.271 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:42.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:42.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:42.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:42.361+0000 7ff770e15780 1 waiting to reach incremental sync.. 2024-04-12T03:31:47.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:47.360+0000 7ff770e15780 1 waiting to reach incremental sync.. 2024-04-12T03:31:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:52.361+0000 7ff770e15780 1 waiting to reach incremental sync.. 2024-04-12T03:31:57.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:57.363+0000 7ff770e15780 1 bucket sync caught up with source: 2024-04-12T03:31:57.364 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.31.6, , , , 00000000001.39.6, , , 00000000001.23.6, , 00000000001.43.6] 2024-04-12T03:31:57.364 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.31.6, , , , 00000000001.39.6, , , 00000000001.23.6, , 00000000001.43.6] 2024-04-12T03:31:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:57.363+0000 7ff770e15780 0 bucket checkpoint complete 2024-04-12T03:31:57.382 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-56 2024-04-12T03:31:57.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:57.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:57.382 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:57.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:31:57.478+0000 7f15c0f42780 1 waiting to reach incremental sync.. 2024-04-12T03:32:02.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:02.480+0000 7f15c0f42780 1 bucket sync caught up with source: 2024-04-12T03:32:02.481 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3499.6, , , , 00000000001.1630.6, , , 00000000001.3039.6, , 00000000001.1206.6] 2024-04-12T03:32:02.481 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3499.6, , , , 00000000001.1630.6, , , 00000000001.3039.6, , 00000000001.1206.6] 2024-04-12T03:32:02.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:02.480+0000 7f15c0f42780 0 bucket checkpoint complete 2024-04-12T03:32:02.497 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-56 zones={a2, a1} 2024-04-12T03:32:02.497 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.497 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.497 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.497 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:32:02.498 DEBUG:boto:Data: 2024-04-12T03:32:02.498 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.498 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.498 DEBUG:boto:Params: {} 2024-04-12T03:32:02.498 DEBUG:boto:Token: None 2024-04-12T03:32:02.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.498 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CeLcCgUAkmK75f0ZI9Ywm9SuKZk= 2024-04-12T03:32:02.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CeLcCgUAkmK75f0ZI9Ywm9SuKZk=', 'Content-Length': '0'} 2024-04-12T03:32:02.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000642618f571ca866f-006618ab32-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.501 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.501 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.502 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.502 DEBUG:boto:Path: /dgbpxg-56/ 2024-04-12T03:32:02.502 DEBUG:boto:Data: 2024-04-12T03:32:02.502 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.502 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.502 DEBUG:boto:Params: {} 2024-04-12T03:32:02.502 DEBUG:boto:Token: None 2024-04-12T03:32:02.502 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.502 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CeLcCgUAkmK75f0ZI9Ywm9SuKZk= 2024-04-12T03:32:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CeLcCgUAkmK75f0ZI9Ywm9SuKZk=', 'Content-Length': '0'} 2024-04-12T03:32:02.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d38011e92c32d1c8-006618ab32-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:32:02.505 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.505 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.505 DEBUG:boto:path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.505 DEBUG:boto:auth_path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.505 DEBUG:boto:Method: GET 2024-04-12T03:32:02.505 DEBUG:boto:Path: /dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.505 DEBUG:boto:Data: 2024-04-12T03:32:02.505 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.506 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.506 DEBUG:boto:Params: {} 2024-04-12T03:32:02.506 DEBUG:boto:Token: None 2024-04-12T03:32:02.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/?versions 2024-04-12T03:32:02.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k= 2024-04-12T03:32:02.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k=', 'Content-Length': '0'} 2024-04-12T03:32:02.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903757ee8200d5e9-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.508 DEBUG:boto:b'dgbpxg-56999falseanulltrue2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.509 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:32:02.509 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:32:02.509 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:32:02.509 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:32:02.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:32:02.510 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.510 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.510 DEBUG:boto:path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.510 DEBUG:boto:auth_path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.510 DEBUG:boto:Method: GET 2024-04-12T03:32:02.510 DEBUG:boto:Path: /dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.510 DEBUG:boto:Data: 2024-04-12T03:32:02.510 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.511 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.511 DEBUG:boto:Params: {} 2024-04-12T03:32:02.511 DEBUG:boto:Token: None 2024-04-12T03:32:02.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/?versions 2024-04-12T03:32:02.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k= 2024-04-12T03:32:02.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k=', 'Content-Length': '0'} 2024-04-12T03:32:02.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec784670e6d3f45b-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.513 DEBUG:boto:b'dgbpxg-56999falseanulltrue2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:32:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:32:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:32:02.517 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:32:02.517 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.517 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.517 DEBUG:boto:path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.517 DEBUG:boto:auth_path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.518 DEBUG:boto:Method: GET 2024-04-12T03:32:02.518 DEBUG:boto:Path: /dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.518 DEBUG:boto:Data: 2024-04-12T03:32:02.518 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.518 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.518 DEBUG:boto:Params: {} 2024-04-12T03:32:02.518 DEBUG:boto:Token: None 2024-04-12T03:32:02.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/?versions 2024-04-12T03:32:02.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k= 2024-04-12T03:32:02.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k=', 'Content-Length': '0'} 2024-04-12T03:32:02.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ad6c2a5a7b5791-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.521 DEBUG:boto:b'dgbpxg-56999falseanulltrue2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.521 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.522 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.522 DEBUG:boto:path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.522 DEBUG:boto:auth_path=/dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.522 DEBUG:boto:Method: GET 2024-04-12T03:32:02.522 DEBUG:boto:Path: /dgbpxg-56/?versions&max-keys=999 2024-04-12T03:32:02.522 DEBUG:boto:Data: 2024-04-12T03:32:02.522 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.522 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.522 DEBUG:boto:Params: {} 2024-04-12T03:32:02.523 DEBUG:boto:Token: None 2024-04-12T03:32:02.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/?versions 2024-04-12T03:32:02.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k= 2024-04-12T03:32:02.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrHnUzbs6ImVupipp/HYlJ6Z21k=', 'Content-Length': '0'} 2024-04-12T03:32:02.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a68378b568c87cf4-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.525 DEBUG:boto:b'dgbpxg-56999falseanulltrue2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:02.526 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.526 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.526 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.526 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.526 DEBUG:boto:Method: GET 2024-04-12T03:32:02.526 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.527 DEBUG:boto:Data: 2024-04-12T03:32:02.527 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.527 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.527 DEBUG:boto:Params: {} 2024-04-12T03:32:02.527 DEBUG:boto:Token: None 2024-04-12T03:32:02.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100= 2024-04-12T03:32:02.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100=', 'Content-Length': '0'} 2024-04-12T03:32:02.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cffe67c74f7d95-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.529 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.529 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.530 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.530 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.530 DEBUG:boto:Method: GET 2024-04-12T03:32:02.530 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.530 DEBUG:boto:Data: 2024-04-12T03:32:02.530 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.530 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.530 DEBUG:boto:Params: {} 2024-04-12T03:32:02.530 DEBUG:boto:Token: None 2024-04-12T03:32:02.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100= 2024-04-12T03:32:02.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100=', 'Content-Length': '0'} 2024-04-12T03:32:02.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004453fb0239b9e142-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.533 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.533 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.533 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.533 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.533 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.533 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.533 DEBUG:boto:Data: 2024-04-12T03:32:02.533 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.534 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.534 DEBUG:boto:Params: {} 2024-04-12T03:32:02.534 DEBUG:boto:Token: None 2024-04-12T03:32:02.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VSuxmB5XzctnCv4pQ1FVP+PkRt0= 2024-04-12T03:32:02.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VSuxmB5XzctnCv4pQ1FVP+PkRt0=', 'Content-Length': '0'} 2024-04-12T03:32:02.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd1af75b768336c-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.536 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.536 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.536 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.536 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.537 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.537 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.537 DEBUG:boto:Data: 2024-04-12T03:32:02.537 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.537 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.537 DEBUG:boto:Params: {} 2024-04-12T03:32:02.537 DEBUG:boto:Token: None 2024-04-12T03:32:02.537 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VSuxmB5XzctnCv4pQ1FVP+PkRt0= 2024-04-12T03:32:02.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VSuxmB5XzctnCv4pQ1FVP+PkRt0=', 'Content-Length': '0'} 2024-04-12T03:32:02.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b861e8f36684bb34-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:02.539 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.540 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.540 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.540 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.540 DEBUG:boto:Method: GET 2024-04-12T03:32:02.540 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.540 DEBUG:boto:Data: 2024-04-12T03:32:02.540 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.540 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.540 DEBUG:boto:Params: {} 2024-04-12T03:32:02.540 DEBUG:boto:Token: None 2024-04-12T03:32:02.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100= 2024-04-12T03:32:02.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100=', 'Content-Length': '0'} 2024-04-12T03:32:02.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a305a0d7010a59-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.543 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.543 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.543 DEBUG:boto:path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.543 DEBUG:boto:auth_path=/dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.543 DEBUG:boto:Method: GET 2024-04-12T03:32:02.543 DEBUG:boto:Path: /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.543 DEBUG:boto:Data: 2024-04-12T03:32:02.543 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.544 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.544 DEBUG:boto:Params: {} 2024-04-12T03:32:02.544 DEBUG:boto:Token: None 2024-04-12T03:32:02.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a?versionId=null 2024-04-12T03:32:02.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100= 2024-04-12T03:32:02.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wV5EfHRqlnlHmOrNtcHMVJk9100=', 'Content-Length': '0'} 2024-04-12T03:32:02.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051dd0c1ec71ab887-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.547 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.547 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.548 DEBUG:boto:path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.548 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.548 DEBUG:boto:Method: GET 2024-04-12T03:32:02.548 DEBUG:boto:Path: /dgbpxg-56/?prefix=a 2024-04-12T03:32:02.548 DEBUG:boto:Data: 2024-04-12T03:32:02.548 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.548 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.548 DEBUG:boto:Params: {} 2024-04-12T03:32:02.548 DEBUG:boto:Token: None 2024-04-12T03:32:02.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043f004b8f5cd54c2-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.551 DEBUG:boto:b'dgbpxg-56a1000falsea2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.551 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.551 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.551 DEBUG:boto:path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.552 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.552 DEBUG:boto:Method: GET 2024-04-12T03:32:02.552 DEBUG:boto:Path: /dgbpxg-56/?prefix=a 2024-04-12T03:32:02.552 DEBUG:boto:Data: 2024-04-12T03:32:02.552 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.552 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.552 DEBUG:boto:Params: {} 2024-04-12T03:32:02.552 DEBUG:boto:Token: None 2024-04-12T03:32:02.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f13895fbaf299bf-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.555 DEBUG:boto:b'dgbpxg-56a1000falsea2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.556 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.556 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.556 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.556 DEBUG:boto:Path: /dgbpxg-56/a 2024-04-12T03:32:02.556 DEBUG:boto:Data: 2024-04-12T03:32:02.556 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.556 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.556 DEBUG:boto:Params: {} 2024-04-12T03:32:02.557 DEBUG:boto:Token: None 2024-04-12T03:32:02.557 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a 2024-04-12T03:32:02.557 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pr+sYxczySOVSJ9RJLVkqGikZlg= 2024-04-12T03:32:02.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pr+sYxczySOVSJ9RJLVkqGikZlg=', 'Content-Length': '0'} 2024-04-12T03:32:02.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd631bf6b5f8edf9-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.559 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.559 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.559 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.559 DEBUG:boto:Path: /dgbpxg-56/a 2024-04-12T03:32:02.559 DEBUG:boto:Data: 2024-04-12T03:32:02.559 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.559 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.559 DEBUG:boto:Params: {} 2024-04-12T03:32:02.560 DEBUG:boto:Token: None 2024-04-12T03:32:02.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a 2024-04-12T03:32:02.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pr+sYxczySOVSJ9RJLVkqGikZlg= 2024-04-12T03:32:02.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pr+sYxczySOVSJ9RJLVkqGikZlg=', 'Content-Length': '0'} 2024-04-12T03:32:02.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eff16575e2d9a03-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:02.562 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.562 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.562 DEBUG:boto:Method: GET 2024-04-12T03:32:02.562 DEBUG:boto:Path: /dgbpxg-56/a 2024-04-12T03:32:02.562 DEBUG:boto:Data: 2024-04-12T03:32:02.562 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.563 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.563 DEBUG:boto:Params: {} 2024-04-12T03:32:02.563 DEBUG:boto:Token: None 2024-04-12T03:32:02.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a 2024-04-12T03:32:02.563 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dctEuwqLzl7xXpPqqqwg9P4wuXc= 2024-04-12T03:32:02.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dctEuwqLzl7xXpPqqqwg9P4wuXc=', 'Content-Length': '0'} 2024-04-12T03:32:02.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e6349bcf76f580a-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.606 DEBUG:boto:path=/dgbpxg-56/a 2024-04-12T03:32:02.607 DEBUG:boto:auth_path=/dgbpxg-56/a 2024-04-12T03:32:02.607 DEBUG:boto:Method: GET 2024-04-12T03:32:02.607 DEBUG:boto:Path: /dgbpxg-56/a 2024-04-12T03:32:02.607 DEBUG:boto:Data: 2024-04-12T03:32:02.607 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.607 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.607 DEBUG:boto:Params: {} 2024-04-12T03:32:02.607 DEBUG:boto:Token: None 2024-04-12T03:32:02.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/a 2024-04-12T03:32:02.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dctEuwqLzl7xXpPqqqwg9P4wuXc= 2024-04-12T03:32:02.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dctEuwqLzl7xXpPqqqwg9P4wuXc=', 'Content-Length': '0'} 2024-04-12T03:32:02.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000944483bb6b396a53-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.610 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.610 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.610 DEBUG:boto:path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.610 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.610 DEBUG:boto:Method: GET 2024-04-12T03:32:02.611 DEBUG:boto:Path: /dgbpxg-56/?prefix=a 2024-04-12T03:32:02.611 DEBUG:boto:Data: 2024-04-12T03:32:02.611 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.611 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.611 DEBUG:boto:Params: {} 2024-04-12T03:32:02.611 DEBUG:boto:Token: None 2024-04-12T03:32:02.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bdde19338921dc8-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.613 DEBUG:boto:b'dgbpxg-56a1000falsea2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.614 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.614 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.614 DEBUG:boto:path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.615 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=a 2024-04-12T03:32:02.615 DEBUG:boto:Method: GET 2024-04-12T03:32:02.615 DEBUG:boto:Path: /dgbpxg-56/?prefix=a 2024-04-12T03:32:02.615 DEBUG:boto:Data: 2024-04-12T03:32:02.615 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.615 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.615 DEBUG:boto:Params: {} 2024-04-12T03:32:02.615 DEBUG:boto:Token: None 2024-04-12T03:32:02.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eca69b45a9b4317-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.618 DEBUG:boto:b'dgbpxg-56a1000falsea2024-04-12T03:31:41.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:02.618 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.618 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.618 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.618 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.619 DEBUG:boto:Method: GET 2024-04-12T03:32:02.619 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.619 DEBUG:boto:Data: 2024-04-12T03:32:02.619 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.619 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.619 DEBUG:boto:Params: {} 2024-04-12T03:32:02.619 DEBUG:boto:Token: None 2024-04-12T03:32:02.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ= 2024-04-12T03:32:02.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9308acd5f12130c-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.621 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.621 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.621 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.622 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.622 DEBUG:boto:Method: GET 2024-04-12T03:32:02.622 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.622 DEBUG:boto:Data: 2024-04-12T03:32:02.622 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.622 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.622 DEBUG:boto:Params: {} 2024-04-12T03:32:02.622 DEBUG:boto:Token: None 2024-04-12T03:32:02.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ= 2024-04-12T03:32:02.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e15a74926f0c52-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.625 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.625 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.625 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.625 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.625 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.625 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.625 DEBUG:boto:Data: 2024-04-12T03:32:02.626 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.626 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.626 DEBUG:boto:Params: {} 2024-04-12T03:32:02.626 DEBUG:boto:Token: None 2024-04-12T03:32:02.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+1M5s9yg9PNMeCNYTC9nETVLyhs= 2024-04-12T03:32:02.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+1M5s9yg9PNMeCNYTC9nETVLyhs=', 'Content-Length': '0'} 2024-04-12T03:32:02.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006abf7723d5519225-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.628 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.628 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.628 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.628 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.628 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.629 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.629 DEBUG:boto:Data: 2024-04-12T03:32:02.629 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.629 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.629 DEBUG:boto:Params: {} 2024-04-12T03:32:02.629 DEBUG:boto:Token: None 2024-04-12T03:32:02.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+1M5s9yg9PNMeCNYTC9nETVLyhs= 2024-04-12T03:32:02.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+1M5s9yg9PNMeCNYTC9nETVLyhs=', 'Content-Length': '0'} 2024-04-12T03:32:02.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000250aca823abfb8de-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:02.631 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.632 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.632 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.632 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.632 DEBUG:boto:Method: GET 2024-04-12T03:32:02.632 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.632 DEBUG:boto:Data: 2024-04-12T03:32:02.632 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.632 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.632 DEBUG:boto:Params: {} 2024-04-12T03:32:02.632 DEBUG:boto:Token: None 2024-04-12T03:32:02.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ= 2024-04-12T03:32:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8942825cb622991-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.635 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.635 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.635 DEBUG:boto:path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.635 DEBUG:boto:auth_path=/dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.635 DEBUG:boto:Method: GET 2024-04-12T03:32:02.635 DEBUG:boto:Path: /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.635 DEBUG:boto:Data: 2024-04-12T03:32:02.635 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.636 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.636 DEBUG:boto:Params: {} 2024-04-12T03:32:02.636 DEBUG:boto:Token: None 2024-04-12T03:32:02.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b?versionId=null 2024-04-12T03:32:02.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ= 2024-04-12T03:32:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DmDEtfrViJCOyADmW8on481pxLQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a21a2d0040b45a4-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.638 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.639 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.639 DEBUG:boto:path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.639 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.639 DEBUG:boto:Method: GET 2024-04-12T03:32:02.639 DEBUG:boto:Path: /dgbpxg-56/?prefix=b 2024-04-12T03:32:02.639 DEBUG:boto:Data: 2024-04-12T03:32:02.639 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.639 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.639 DEBUG:boto:Params: {} 2024-04-12T03:32:02.639 DEBUG:boto:Token: None 2024-04-12T03:32:02.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b537b999e8cb97-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.642 DEBUG:boto:b'dgbpxg-56b1000falseb2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.643 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.643 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.643 DEBUG:boto:path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.643 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.643 DEBUG:boto:Method: GET 2024-04-12T03:32:02.644 DEBUG:boto:Path: /dgbpxg-56/?prefix=b 2024-04-12T03:32:02.644 DEBUG:boto:Data: 2024-04-12T03:32:02.644 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.644 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.644 DEBUG:boto:Params: {} 2024-04-12T03:32:02.644 DEBUG:boto:Token: None 2024-04-12T03:32:02.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be014c2b165d1729-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.647 DEBUG:boto:b'dgbpxg-56b1000falseb2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.647 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.647 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.648 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.648 DEBUG:boto:Path: /dgbpxg-56/b 2024-04-12T03:32:02.648 DEBUG:boto:Data: 2024-04-12T03:32:02.648 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.648 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.648 DEBUG:boto:Params: {} 2024-04-12T03:32:02.648 DEBUG:boto:Token: None 2024-04-12T03:32:02.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b 2024-04-12T03:32:02.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1zYCDn344TbnwpTGkNoyrsDtdb0= 2024-04-12T03:32:02.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1zYCDn344TbnwpTGkNoyrsDtdb0=', 'Content-Length': '0'} 2024-04-12T03:32:02.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084fff8f86e67754c-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.650 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.651 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.651 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.651 DEBUG:boto:Path: /dgbpxg-56/b 2024-04-12T03:32:02.651 DEBUG:boto:Data: 2024-04-12T03:32:02.651 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.651 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.651 DEBUG:boto:Params: {} 2024-04-12T03:32:02.651 DEBUG:boto:Token: None 2024-04-12T03:32:02.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b 2024-04-12T03:32:02.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1zYCDn344TbnwpTGkNoyrsDtdb0= 2024-04-12T03:32:02.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1zYCDn344TbnwpTGkNoyrsDtdb0=', 'Content-Length': '0'} 2024-04-12T03:32:02.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fb7d9b5ffcb1f3-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:02.654 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.654 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.654 DEBUG:boto:Method: GET 2024-04-12T03:32:02.654 DEBUG:boto:Path: /dgbpxg-56/b 2024-04-12T03:32:02.654 DEBUG:boto:Data: 2024-04-12T03:32:02.654 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.654 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.654 DEBUG:boto:Params: {} 2024-04-12T03:32:02.655 DEBUG:boto:Token: None 2024-04-12T03:32:02.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b 2024-04-12T03:32:02.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6WcNAPiRmbdBTn3fTg972FPbKIE= 2024-04-12T03:32:02.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6WcNAPiRmbdBTn3fTg972FPbKIE=', 'Content-Length': '0'} 2024-04-12T03:32:02.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c65d12faf7b1b30-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.657 DEBUG:boto:path=/dgbpxg-56/b 2024-04-12T03:32:02.657 DEBUG:boto:auth_path=/dgbpxg-56/b 2024-04-12T03:32:02.658 DEBUG:boto:Method: GET 2024-04-12T03:32:02.658 DEBUG:boto:Path: /dgbpxg-56/b 2024-04-12T03:32:02.658 DEBUG:boto:Data: 2024-04-12T03:32:02.658 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.658 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.658 DEBUG:boto:Params: {} 2024-04-12T03:32:02.658 DEBUG:boto:Token: None 2024-04-12T03:32:02.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/b 2024-04-12T03:32:02.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6WcNAPiRmbdBTn3fTg972FPbKIE= 2024-04-12T03:32:02.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6WcNAPiRmbdBTn3fTg972FPbKIE=', 'Content-Length': '0'} 2024-04-12T03:32:02.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf23a986001163fb-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.702 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.703 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.703 DEBUG:boto:path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.703 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.703 DEBUG:boto:Method: GET 2024-04-12T03:32:02.703 DEBUG:boto:Path: /dgbpxg-56/?prefix=b 2024-04-12T03:32:02.703 DEBUG:boto:Data: 2024-04-12T03:32:02.703 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.703 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.703 DEBUG:boto:Params: {} 2024-04-12T03:32:02.704 DEBUG:boto:Token: None 2024-04-12T03:32:02.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da655ceafff217e-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.706 DEBUG:boto:b'dgbpxg-56b1000falseb2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.707 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.707 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.707 DEBUG:boto:path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.707 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=b 2024-04-12T03:32:02.707 DEBUG:boto:Method: GET 2024-04-12T03:32:02.707 DEBUG:boto:Path: /dgbpxg-56/?prefix=b 2024-04-12T03:32:02.707 DEBUG:boto:Data: 2024-04-12T03:32:02.707 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.707 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.708 DEBUG:boto:Params: {} 2024-04-12T03:32:02.708 DEBUG:boto:Token: None 2024-04-12T03:32:02.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cb99717c5fc750-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.710 DEBUG:boto:b'dgbpxg-56b1000falseb2024-04-12T03:31:41.978Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:02.711 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.711 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.711 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.711 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.711 DEBUG:boto:Method: GET 2024-04-12T03:32:02.711 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.712 DEBUG:boto:Data: 2024-04-12T03:32:02.712 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.712 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.712 DEBUG:boto:Params: {} 2024-04-12T03:32:02.712 DEBUG:boto:Token: None 2024-04-12T03:32:02.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ= 2024-04-12T03:32:02.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7a3821bc106d52-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.714 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.714 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.715 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.715 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.715 DEBUG:boto:Method: GET 2024-04-12T03:32:02.715 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.715 DEBUG:boto:Data: 2024-04-12T03:32:02.715 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.715 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.715 DEBUG:boto:Params: {} 2024-04-12T03:32:02.715 DEBUG:boto:Token: None 2024-04-12T03:32:02.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ= 2024-04-12T03:32:02.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c506453b29408ea3-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.718 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.718 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.718 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.718 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.718 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.718 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.718 DEBUG:boto:Data: 2024-04-12T03:32:02.718 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.719 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.719 DEBUG:boto:Params: {} 2024-04-12T03:32:02.719 DEBUG:boto:Token: None 2024-04-12T03:32:02.719 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wS832jgW8iwYFbrKqCxXG5aaoX8= 2024-04-12T03:32:02.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wS832jgW8iwYFbrKqCxXG5aaoX8=', 'Content-Length': '0'} 2024-04-12T03:32:02.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002503b9969b0c5244-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.722 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.722 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.722 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.722 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.722 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.722 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.722 DEBUG:boto:Data: 2024-04-12T03:32:02.722 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.722 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.722 DEBUG:boto:Params: {} 2024-04-12T03:32:02.723 DEBUG:boto:Token: None 2024-04-12T03:32:02.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.723 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wS832jgW8iwYFbrKqCxXG5aaoX8= 2024-04-12T03:32:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wS832jgW8iwYFbrKqCxXG5aaoX8=', 'Content-Length': '0'} 2024-04-12T03:32:02.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a72e1afaeb30c20-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:02.725 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.725 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.725 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.725 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.725 DEBUG:boto:Method: GET 2024-04-12T03:32:02.725 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.725 DEBUG:boto:Data: 2024-04-12T03:32:02.726 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.726 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.726 DEBUG:boto:Params: {} 2024-04-12T03:32:02.726 DEBUG:boto:Token: None 2024-04-12T03:32:02.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ= 2024-04-12T03:32:02.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4b001b6ebe75e5-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.728 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.728 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.728 DEBUG:boto:path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.728 DEBUG:boto:auth_path=/dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.729 DEBUG:boto:Method: GET 2024-04-12T03:32:02.729 DEBUG:boto:Path: /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.729 DEBUG:boto:Data: 2024-04-12T03:32:02.729 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.729 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.729 DEBUG:boto:Params: {} 2024-04-12T03:32:02.729 DEBUG:boto:Token: None 2024-04-12T03:32:02.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c?versionId=null 2024-04-12T03:32:02.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ= 2024-04-12T03:32:02.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZIdpJjBUAA9krtQVAUWPy64vsQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ade22b34ff7ed89-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.731 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.731 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.732 DEBUG:boto:path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.732 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.732 DEBUG:boto:Method: GET 2024-04-12T03:32:02.732 DEBUG:boto:Path: /dgbpxg-56/?prefix=c 2024-04-12T03:32:02.732 DEBUG:boto:Data: 2024-04-12T03:32:02.732 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.732 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.732 DEBUG:boto:Params: {} 2024-04-12T03:32:02.732 DEBUG:boto:Token: None 2024-04-12T03:32:02.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089f77f4887af3d84-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.735 DEBUG:boto:b'dgbpxg-56c1000falsec2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.735 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.736 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.736 DEBUG:boto:path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.736 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.736 DEBUG:boto:Method: GET 2024-04-12T03:32:02.736 DEBUG:boto:Path: /dgbpxg-56/?prefix=c 2024-04-12T03:32:02.736 DEBUG:boto:Data: 2024-04-12T03:32:02.736 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.736 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.736 DEBUG:boto:Params: {} 2024-04-12T03:32:02.736 DEBUG:boto:Token: None 2024-04-12T03:32:02.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a434f7e1122a7fa9-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.782 DEBUG:boto:b'dgbpxg-56c1000falsec2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.783 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.783 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.783 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.783 DEBUG:boto:Path: /dgbpxg-56/c 2024-04-12T03:32:02.783 DEBUG:boto:Data: 2024-04-12T03:32:02.783 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.783 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.783 DEBUG:boto:Params: {} 2024-04-12T03:32:02.783 DEBUG:boto:Token: None 2024-04-12T03:32:02.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c 2024-04-12T03:32:02.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:epXOiBsmIVWmV0HUZbxgfKDK0Pw= 2024-04-12T03:32:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:epXOiBsmIVWmV0HUZbxgfKDK0Pw=', 'Content-Length': '0'} 2024-04-12T03:32:02.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a419284a4efcc9f-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.786 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.786 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.786 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.786 DEBUG:boto:Path: /dgbpxg-56/c 2024-04-12T03:32:02.786 DEBUG:boto:Data: 2024-04-12T03:32:02.786 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.787 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.787 DEBUG:boto:Params: {} 2024-04-12T03:32:02.787 DEBUG:boto:Token: None 2024-04-12T03:32:02.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c 2024-04-12T03:32:02.787 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:epXOiBsmIVWmV0HUZbxgfKDK0Pw= 2024-04-12T03:32:02.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:epXOiBsmIVWmV0HUZbxgfKDK0Pw=', 'Content-Length': '0'} 2024-04-12T03:32:02.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7da0ea8ba8b81a0-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:02.789 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.789 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.789 DEBUG:boto:Method: GET 2024-04-12T03:32:02.789 DEBUG:boto:Path: /dgbpxg-56/c 2024-04-12T03:32:02.789 DEBUG:boto:Data: 2024-04-12T03:32:02.789 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.789 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.790 DEBUG:boto:Params: {} 2024-04-12T03:32:02.790 DEBUG:boto:Token: None 2024-04-12T03:32:02.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c 2024-04-12T03:32:02.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lstOjixFgZGRpwTYKU7Esp6S2l0= 2024-04-12T03:32:02.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lstOjixFgZGRpwTYKU7Esp6S2l0=', 'Content-Length': '0'} 2024-04-12T03:32:02.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e96ef643421b39f1-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.792 DEBUG:boto:path=/dgbpxg-56/c 2024-04-12T03:32:02.792 DEBUG:boto:auth_path=/dgbpxg-56/c 2024-04-12T03:32:02.792 DEBUG:boto:Method: GET 2024-04-12T03:32:02.792 DEBUG:boto:Path: /dgbpxg-56/c 2024-04-12T03:32:02.792 DEBUG:boto:Data: 2024-04-12T03:32:02.792 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.792 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.793 DEBUG:boto:Params: {} 2024-04-12T03:32:02.793 DEBUG:boto:Token: None 2024-04-12T03:32:02.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/c 2024-04-12T03:32:02.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lstOjixFgZGRpwTYKU7Esp6S2l0= 2024-04-12T03:32:02.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lstOjixFgZGRpwTYKU7Esp6S2l0=', 'Content-Length': '0'} 2024-04-12T03:32:02.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f5830bf30769f3-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.795 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.795 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.795 DEBUG:boto:path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.795 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.795 DEBUG:boto:Method: GET 2024-04-12T03:32:02.795 DEBUG:boto:Path: /dgbpxg-56/?prefix=c 2024-04-12T03:32:02.795 DEBUG:boto:Data: 2024-04-12T03:32:02.795 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.795 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.796 DEBUG:boto:Params: {} 2024-04-12T03:32:02.796 DEBUG:boto:Token: None 2024-04-12T03:32:02.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fdced614356eb9c-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.798 DEBUG:boto:b'dgbpxg-56c1000falsec2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.798 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.799 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.799 DEBUG:boto:path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.799 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=c 2024-04-12T03:32:02.799 DEBUG:boto:Method: GET 2024-04-12T03:32:02.799 DEBUG:boto:Path: /dgbpxg-56/?prefix=c 2024-04-12T03:32:02.799 DEBUG:boto:Data: 2024-04-12T03:32:02.799 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.799 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.799 DEBUG:boto:Params: {} 2024-04-12T03:32:02.799 DEBUG:boto:Token: None 2024-04-12T03:32:02.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd90f0ab372610a-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.802 DEBUG:boto:b'dgbpxg-56c1000falsec2024-04-12T03:31:42.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:02.803 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.803 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.803 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.803 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.803 DEBUG:boto:Method: GET 2024-04-12T03:32:02.803 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.803 DEBUG:boto:Data: 2024-04-12T03:32:02.803 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.803 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.803 DEBUG:boto:Params: {} 2024-04-12T03:32:02.804 DEBUG:boto:Token: None 2024-04-12T03:32:02.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.804 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4= 2024-04-12T03:32:02.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4=', 'Content-Length': '0'} 2024-04-12T03:32:02.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9ac111c18b23dd-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.805 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.806 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.806 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.806 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.806 DEBUG:boto:Method: GET 2024-04-12T03:32:02.806 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.806 DEBUG:boto:Data: 2024-04-12T03:32:02.806 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.807 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.807 DEBUG:boto:Params: {} 2024-04-12T03:32:02.807 DEBUG:boto:Token: None 2024-04-12T03:32:02.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4= 2024-04-12T03:32:02.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4=', 'Content-Length': '0'} 2024-04-12T03:32:02.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d81cd93f3ba95e2-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.809 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.809 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.809 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.809 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.809 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.809 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.810 DEBUG:boto:Data: 2024-04-12T03:32:02.810 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.810 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.810 DEBUG:boto:Params: {} 2024-04-12T03:32:02.810 DEBUG:boto:Token: None 2024-04-12T03:32:02.810 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z9D6IiIE69Ygu/4BmemdycIEVTw= 2024-04-12T03:32:02.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z9D6IiIE69Ygu/4BmemdycIEVTw=', 'Content-Length': '0'} 2024-04-12T03:32:02.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b745c64f2ea8caa9-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.812 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.812 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.812 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.812 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.813 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.813 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.813 DEBUG:boto:Data: 2024-04-12T03:32:02.813 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.813 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.813 DEBUG:boto:Params: {} 2024-04-12T03:32:02.813 DEBUG:boto:Token: None 2024-04-12T03:32:02.813 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.814 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z9D6IiIE69Ygu/4BmemdycIEVTw= 2024-04-12T03:32:02.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z9D6IiIE69Ygu/4BmemdycIEVTw=', 'Content-Length': '0'} 2024-04-12T03:32:02.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006b9893062d5806b-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:02.816 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.816 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.816 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.816 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.816 DEBUG:boto:Method: GET 2024-04-12T03:32:02.816 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.816 DEBUG:boto:Data: 2024-04-12T03:32:02.816 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.816 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.817 DEBUG:boto:Params: {} 2024-04-12T03:32:02.817 DEBUG:boto:Token: None 2024-04-12T03:32:02.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.817 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4= 2024-04-12T03:32:02.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4=', 'Content-Length': '0'} 2024-04-12T03:32:02.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be688560868cfb8-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.819 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.819 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.819 DEBUG:boto:path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.819 DEBUG:boto:auth_path=/dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.819 DEBUG:boto:Method: GET 2024-04-12T03:32:02.819 DEBUG:boto:Path: /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.819 DEBUG:boto:Data: 2024-04-12T03:32:02.820 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.820 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.820 DEBUG:boto:Params: {} 2024-04-12T03:32:02.820 DEBUG:boto:Token: None 2024-04-12T03:32:02.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d?versionId=null 2024-04-12T03:32:02.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4= 2024-04-12T03:32:02.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNsROXn2P3GTDNY23VaFCmwCEj4=', 'Content-Length': '0'} 2024-04-12T03:32:02.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1ceccc4d8b323a-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.862 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.863 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.863 DEBUG:boto:path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.863 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.863 DEBUG:boto:Method: GET 2024-04-12T03:32:02.863 DEBUG:boto:Path: /dgbpxg-56/?prefix=d 2024-04-12T03:32:02.863 DEBUG:boto:Data: 2024-04-12T03:32:02.863 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.863 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.863 DEBUG:boto:Params: {} 2024-04-12T03:32:02.864 DEBUG:boto:Token: None 2024-04-12T03:32:02.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021893aaa6ab098d5-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.866 DEBUG:boto:b'dgbpxg-56d1000falsed2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.867 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.867 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.867 DEBUG:boto:path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.867 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.867 DEBUG:boto:Method: GET 2024-04-12T03:32:02.868 DEBUG:boto:Path: /dgbpxg-56/?prefix=d 2024-04-12T03:32:02.868 DEBUG:boto:Data: 2024-04-12T03:32:02.868 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.868 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.868 DEBUG:boto:Params: {} 2024-04-12T03:32:02.868 DEBUG:boto:Token: None 2024-04-12T03:32:02.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f86e5d8049e5733-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.870 DEBUG:boto:b'dgbpxg-56d1000falsed2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.871 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.871 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.871 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.871 DEBUG:boto:Path: /dgbpxg-56/d 2024-04-12T03:32:02.871 DEBUG:boto:Data: 2024-04-12T03:32:02.871 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.871 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.871 DEBUG:boto:Params: {} 2024-04-12T03:32:02.872 DEBUG:boto:Token: None 2024-04-12T03:32:02.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d 2024-04-12T03:32:02.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EfLBiLGRFm8YCMVLYNcsbXetw1w= 2024-04-12T03:32:02.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EfLBiLGRFm8YCMVLYNcsbXetw1w=', 'Content-Length': '0'} 2024-04-12T03:32:02.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aeb982fb6fb9fac-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.874 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.874 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.874 DEBUG:boto:Method: HEAD 2024-04-12T03:32:02.874 DEBUG:boto:Path: /dgbpxg-56/d 2024-04-12T03:32:02.874 DEBUG:boto:Data: 2024-04-12T03:32:02.874 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.874 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.874 DEBUG:boto:Params: {} 2024-04-12T03:32:02.875 DEBUG:boto:Token: None 2024-04-12T03:32:02.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d 2024-04-12T03:32:02.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EfLBiLGRFm8YCMVLYNcsbXetw1w= 2024-04-12T03:32:02.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EfLBiLGRFm8YCMVLYNcsbXetw1w=', 'Content-Length': '0'} 2024-04-12T03:32:02.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b86b9e0861152ee-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:02.877 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.877 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.877 DEBUG:boto:Method: GET 2024-04-12T03:32:02.877 DEBUG:boto:Path: /dgbpxg-56/d 2024-04-12T03:32:02.877 DEBUG:boto:Data: 2024-04-12T03:32:02.877 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.877 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.877 DEBUG:boto:Params: {} 2024-04-12T03:32:02.877 DEBUG:boto:Token: None 2024-04-12T03:32:02.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d 2024-04-12T03:32:02.878 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+NvBu+x4xjmiBI66yD+3Huv0znQ= 2024-04-12T03:32:02.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+NvBu+x4xjmiBI66yD+3Huv0znQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-56:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d61ed1e9adbe39-006618ab32-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.926 DEBUG:boto:path=/dgbpxg-56/d 2024-04-12T03:32:02.926 DEBUG:boto:auth_path=/dgbpxg-56/d 2024-04-12T03:32:02.927 DEBUG:boto:Method: GET 2024-04-12T03:32:02.927 DEBUG:boto:Path: /dgbpxg-56/d 2024-04-12T03:32:02.927 DEBUG:boto:Data: 2024-04-12T03:32:02.927 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.927 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.927 DEBUG:boto:Params: {} 2024-04-12T03:32:02.929 DEBUG:boto:Token: None 2024-04-12T03:32:02.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/d 2024-04-12T03:32:02.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+NvBu+x4xjmiBI66yD+3Huv0znQ= 2024-04-12T03:32:02.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+NvBu+x4xjmiBI66yD+3Huv0znQ=', 'Content-Length': '0'} 2024-04-12T03:32:02.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ce7a3a803e0d1c-006618ab32-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.932 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.932 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.932 DEBUG:boto:path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.932 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.932 DEBUG:boto:Method: GET 2024-04-12T03:32:02.932 DEBUG:boto:Path: /dgbpxg-56/?prefix=d 2024-04-12T03:32:02.932 DEBUG:boto:Data: 2024-04-12T03:32:02.932 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:02.933 DEBUG:boto:Port: 8001 2024-04-12T03:32:02.933 DEBUG:boto:Params: {} 2024-04-12T03:32:02.933 DEBUG:boto:Token: None 2024-04-12T03:32:02.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b454a0cb6ee2782-006618ab32-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.935 DEBUG:boto:b'dgbpxg-56d1000falsed2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.936 DEBUG:boto:path=/dgbpxg-56/ 2024-04-12T03:32:02.936 DEBUG:boto:auth_path=/dgbpxg-56/ 2024-04-12T03:32:02.936 DEBUG:boto:path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.936 DEBUG:boto:auth_path=/dgbpxg-56/?prefix=d 2024-04-12T03:32:02.936 DEBUG:boto:Method: GET 2024-04-12T03:32:02.936 DEBUG:boto:Path: /dgbpxg-56/?prefix=d 2024-04-12T03:32:02.936 DEBUG:boto:Data: 2024-04-12T03:32:02.936 DEBUG:boto:Headers: {} 2024-04-12T03:32:02.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:02.937 DEBUG:boto:Port: 8000 2024-04-12T03:32:02.937 DEBUG:boto:Params: {} 2024-04-12T03:32:02.937 DEBUG:boto:Token: None 2024-04-12T03:32:02.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:02 GMT /dgbpxg-56/ 2024-04-12T03:32:02.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE= 2024-04-12T03:32:02.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yj7AJmr61O2sFecxQu1Q5hkMjBE=', 'Content-Length': '0'} 2024-04-12T03:32:02.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000928a15d7051c892-006618ab32-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:02.939 DEBUG:boto:b'dgbpxg-56d1000falsed2024-04-12T03:31:42.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:02.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-56 zones={a2, a1} 2024-04-12T03:32:02.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:02.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:02.940 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:32:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:03.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.036 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-56 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:03.123 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2024-04-12T03:32:03.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.126 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:03.212 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:32:03.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.213 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "1#00000000001.20.6", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1533110112687624154", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "a", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:03.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:15.874222970Z", 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:03.215 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:03.216 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:03.216 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:03.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.216 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18" 2024-04-12T03:32:03.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "5#00000000001.55.6", 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.12956827797752121081", 2024-04-12T03:32:03.217 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:03.218 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "d", 2024-04-12T03:32:03.218 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:03.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:03.218 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:03.218 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:15.944069074Z", 2024-04-12T03:32:03.219 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:03.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:03.219 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:32:03.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.220 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18" 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "8#00000000001.42.6", 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15622470857301418623", 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:03.221 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "c", 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:15.933652831Z", 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18" 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.223 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "10#00000000001.24.6", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.6074947972349319715", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "b", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:31:15.884104247Z", 2024-04-12T03:32:03.224 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:03.225 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-55:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.18" 2024-04-12T03:32:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:03.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:03.226 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:32:03.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:03.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:03.227 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:32:03.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:03.273 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:03.319 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.316+0000 7fda6eefe700 -1 req 10050594159242224058 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.322 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.321+0000 7fda7cf1a700 -1 req 6907348102731716918 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.325 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.324+0000 7fda23e68700 -1 req 18162414343695051353 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.325 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.324+0000 7fdaeaff6700 -1 req 10919108222772317890 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.367 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.366+0000 7fdad2fc6700 -1 req 6311772913560366125 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.369 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.368+0000 7fda9ef5e700 -1 req 3149367843709724492 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.369 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.368+0000 7fda56ece700 -1 req 9663556818415743710 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.413 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.412+0000 7fda86f2e700 -1 req 8387550624715599436 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.459 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.458+0000 7fda36e8e700 -1 req 1055011040974220685 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.459 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.459+0000 7fdaf780f700 -1 req 11880915963802748998 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.504 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.503+0000 7fdabef9e700 -1 req 5988033910952603778 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.551 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.550+0000 7fdb0b837700 -1 req 12329768183894938165 0.002999930s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.551 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.550+0000 7fda2d67b700 -1 req 11755225921254128213 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.596 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.595+0000 7fda89f34700 -1 req 17392709468979122549 0.002999930s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.601 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.600+0000 7fdaa776f700 -1 req 16002961751304069068 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-42:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.5,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-42:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.5,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.642 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.642+0000 7fda52ec6700 -1 req 3493470235894259993 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-34:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.34,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-34:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.34,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.643 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.642+0000 7fdab1f84700 -1 req 2018874410240906448 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-41:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.4,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-41:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.4,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.686 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.685+0000 7fda9df5c700 -1 req 2993728809234244272 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-38:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.1,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-38:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.1,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.686 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.686+0000 7fda78711700 -1 req 3185179507578642753 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-53:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.16,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-53:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.16,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.690 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.690+0000 7fda9f75f700 -1 req 4633522954095146416 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-29:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.29,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-29:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.29,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.730 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.730+0000 7fda3a695700 -1 req 7713366692562484941 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-44:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.7,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-44:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.7,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.732 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.731+0000 7fdb0602c700 -1 req 17917964547505125259 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-22:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.22,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-22:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.22,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.774 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.773+0000 7fda4fec0700 -1 req 13533557929559580975 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-21:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.21,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-21:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.21,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.817 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.816+0000 7fda67ef0700 -1 req 875119972493966852 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-45:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.8,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-45:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.8,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.827 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.826+0000 7fdac77af700 -1 req 8617060236630592023 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-24:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.24,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-24:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.24,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.868 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.867+0000 7fda4a6b5700 -1 req 16183559257238391293 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-43:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.6,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-43:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.6,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.871 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.871+0000 7fdab2f86700 -1 req 5841818726945589426 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-32:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.32,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-32:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.32,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.871 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:32:03.871+0000 7fda21663700 -1 req 14011349202180539602 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-31:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.31,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-31:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.31,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:32:03.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:03.929 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:03.977 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:03.977 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:04.017 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2024-04-12T03:32:04.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2024-04-12T03:32:04.021 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-57 2024-04-12T03:32:04.021 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:04.021 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:04.021 DEBUG:boto:Method: PUT 2024-04-12T03:32:04.021 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:04.021 DEBUG:boto:Data: 2024-04-12T03:32:04.021 DEBUG:boto:Headers: {} 2024-04-12T03:32:04.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.022 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.022 DEBUG:boto:Params: {} 2024-04-12T03:32:04.022 DEBUG:boto:Token: None 2024-04-12T03:32:04.022 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/ 2024-04-12T03:32:04.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e4v1XOijxVgXmNgpq4SJIxGogoA= 2024-04-12T03:32:04.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e4v1XOijxVgXmNgpq4SJIxGogoA=', 'Content-Length': '0'} 2024-04-12T03:32:04.037 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004037f59eb2de63c8-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.037 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:04.037 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:04.037 DEBUG:boto:Method: HEAD 2024-04-12T03:32:04.038 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:04.038 DEBUG:boto:Data: 2024-04-12T03:32:04.038 DEBUG:boto:Headers: {} 2024-04-12T03:32:04.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.038 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.038 DEBUG:boto:Params: {} 2024-04-12T03:32:04.038 DEBUG:boto:Token: None 2024-04-12T03:32:04.038 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/ 2024-04-12T03:32:04.038 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T03:32:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T03:32:04.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071cb7fc40735f79f-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.041 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:04.041 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:04.041 DEBUG:boto:Method: PUT 2024-04-12T03:32:04.041 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:32:04.042 DEBUG:boto:Data: 2024-04-12T03:32:04.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:04.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.042 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.042 DEBUG:boto:Params: {} 2024-04-12T03:32:04.042 DEBUG:boto:Token: None 2024-04-12T03:32:04.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/a 2024-04-12T03:32:04.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tj7A4q8PdxeKfm50lbwG82ab+Yo= 2024-04-12T03:32:04.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tj7A4q8PdxeKfm50lbwG82ab+Yo='} 2024-04-12T03:32:04.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca03e5c4e3b799a6-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.087 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:04.087 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:04.087 DEBUG:boto:Method: HEAD 2024-04-12T03:32:04.087 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:04.087 DEBUG:boto:Data: 2024-04-12T03:32:04.087 DEBUG:boto:Headers: {} 2024-04-12T03:32:04.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.087 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.087 DEBUG:boto:Params: {} 2024-04-12T03:32:04.088 DEBUG:boto:Token: None 2024-04-12T03:32:04.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/ 2024-04-12T03:32:04.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T03:32:04.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T03:32:04.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ac10aa68442b1e8-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.090 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:04.090 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:04.090 DEBUG:boto:Method: PUT 2024-04-12T03:32:04.091 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:32:04.091 DEBUG:boto:Data: 2024-04-12T03:32:04.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:04.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.091 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.091 DEBUG:boto:Params: {} 2024-04-12T03:32:04.091 DEBUG:boto:Token: None 2024-04-12T03:32:04.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/b 2024-04-12T03:32:04.091 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FS1BTjo61tBhy5GqPGSMQiVDIAA= 2024-04-12T03:32:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FS1BTjo61tBhy5GqPGSMQiVDIAA='} 2024-04-12T03:32:04.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a84969b8365784c-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.096 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:04.097 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:04.097 DEBUG:boto:Method: HEAD 2024-04-12T03:32:04.097 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:04.097 DEBUG:boto:Data: 2024-04-12T03:32:04.097 DEBUG:boto:Headers: {} 2024-04-12T03:32:04.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.097 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.097 DEBUG:boto:Params: {} 2024-04-12T03:32:04.097 DEBUG:boto:Token: None 2024-04-12T03:32:04.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/ 2024-04-12T03:32:04.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T03:32:04.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T03:32:04.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e89639259aae760c-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.100 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:04.100 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:04.100 DEBUG:boto:Method: PUT 2024-04-12T03:32:04.100 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:32:04.100 DEBUG:boto:Data: 2024-04-12T03:32:04.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:04.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.101 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.101 DEBUG:boto:Params: {} 2024-04-12T03:32:04.101 DEBUG:boto:Token: None 2024-04-12T03:32:04.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/c 2024-04-12T03:32:04.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SmZWzRkhC4ECskLe/idaxx0zfQg= 2024-04-12T03:32:04.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SmZWzRkhC4ECskLe/idaxx0zfQg='} 2024-04-12T03:32:04.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626a93277c1d6d1c-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.106 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:04.106 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:04.106 DEBUG:boto:Method: HEAD 2024-04-12T03:32:04.106 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:04.106 DEBUG:boto:Data: 2024-04-12T03:32:04.107 DEBUG:boto:Headers: {} 2024-04-12T03:32:04.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.107 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.107 DEBUG:boto:Params: {} 2024-04-12T03:32:04.107 DEBUG:boto:Token: None 2024-04-12T03:32:04.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/ 2024-04-12T03:32:04.107 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T03:32:04.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T03:32:04.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c69473375de3dc4-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.109 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:04.109 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:04.110 DEBUG:boto:Method: PUT 2024-04-12T03:32:04.110 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:32:04.110 DEBUG:boto:Data: 2024-04-12T03:32:04.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:04.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:04.110 DEBUG:boto:Port: 8000 2024-04-12T03:32:04.110 DEBUG:boto:Params: {} 2024-04-12T03:32:04.110 DEBUG:boto:Token: None 2024-04-12T03:32:04.110 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:04 GMT /dgbpxg-57/d 2024-04-12T03:32:04.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pi4gbSvwkhU2zDS3gSHQsumjjkg= 2024-04-12T03:32:04.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pi4gbSvwkhU2zDS3gSHQsumjjkg='} 2024-04-12T03:32:04.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e05ce567e441feb0-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:04.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:04.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:04.155 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:04.237 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:32:04.249 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:32:04.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892724.035697_1004.1", 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:32:04.035697Z" 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.251 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:32:04.252 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T03:32:04.253 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:32:04.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:04.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:04.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:04.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:04.301 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:04.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:04.328+0000 7f6f715f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:32:04.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:04.329+0000 7f6f715f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:32:04.342 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:32:04.343 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.344 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892724.029550_1001.1", 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.029550Z", 2024-04-12T03:32:04.347 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:32:04Z" 2024-04-12T03:32:04.348 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:32:04.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892724.029550_1001.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:32:04.029550Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:32:04Z" } 2024-04-12T03:32:04.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892724.029550_1001.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:32:04.029550Z', 'realm_epoch': 2}}] 2024-04-12T03:32:04.350 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T03:32:04.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.029550_1001.1'} 2024-04-12T03:32:04.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892724.035697_1004.1 target=1_1712892724.029550_1001.1 2024-04-12T03:32:09.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:09.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:09.351 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:09.439+0000 7f145cfd9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:32:09.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:09.440+0000 7f145cfd9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:32:09.454 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892724.035697_1004.1", 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.035697Z", 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:09.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:32:09Z" 2024-04-12T03:32:09.460 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:32:09.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892724.035697_1004.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:32:04.035697Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:32:09Z" } 2024-04-12T03:32:09.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892724.035697_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:32:04.035697Z', 'realm_epoch': 2}}] 2024-04-12T03:32:09.461 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T03:32:09.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T03:32:09.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:32:09.462 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T03:32:09.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:09.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:09.462 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:32:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:09.519 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:09.552+0000 7f422433b780 1 waiting to reach incremental sync.. 2024-04-12T03:32:14.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:14.553+0000 7f422433b780 1 waiting to reach incremental sync.. 2024-04-12T03:32:19.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:19.555+0000 7f422433b780 1 bucket sync caught up with source: 2024-04-12T03:32:19.556 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.51.6, , , , 00000000001.28.6, , , 00000000001.26.6, , 00000000001.61.6] 2024-04-12T03:32:19.556 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.51.6, , , , 00000000001.28.6, , , 00000000001.26.6, , 00000000001.61.6] 2024-04-12T03:32:19.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:19.555+0000 7f422433b780 0 bucket checkpoint complete 2024-04-12T03:32:19.573 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T03:32:19.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:19.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:19.574 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:32:19.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:19.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:19.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:19.675+0000 7f22f2dfd780 1 waiting to reach incremental sync.. 2024-04-12T03:32:24.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:24.677+0000 7f22f2dfd780 1 bucket sync caught up with source: 2024-04-12T03:32:24.679 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1211.6, , , , 00000000001.610.6, , , 00000000001.1248.6, , 00000000001.1327.6] 2024-04-12T03:32:24.679 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1211.6, , , , 00000000001.610.6, , , 00000000001.1248.6, , 00000000001.1327.6] 2024-04-12T03:32:24.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:24.677+0000 7f22f2dfd780 0 bucket checkpoint complete 2024-04-12T03:32:24.695 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:32:24.695 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.695 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.695 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.696 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:24.696 DEBUG:boto:Data: 2024-04-12T03:32:24.696 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.696 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.696 DEBUG:boto:Params: {} 2024-04-12T03:32:24.696 DEBUG:boto:Token: None 2024-04-12T03:32:24.697 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU= 2024-04-12T03:32:24.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU=', 'Content-Length': '0'} 2024-04-12T03:32:24.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea23468e3b50d158-006618ab48-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.700 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.700 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.700 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.700 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:24.700 DEBUG:boto:Data: 2024-04-12T03:32:24.700 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.700 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.700 DEBUG:boto:Params: {} 2024-04-12T03:32:24.701 DEBUG:boto:Token: None 2024-04-12T03:32:24.701 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU= 2024-04-12T03:32:24.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU=', 'Content-Length': '0'} 2024-04-12T03:32:24.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000deb92efba495fdca-006618ab48-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:32:24.704 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.704 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.704 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.704 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.704 DEBUG:boto:Method: GET 2024-04-12T03:32:24.704 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.704 DEBUG:boto:Data: 2024-04-12T03:32:24.704 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.704 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.705 DEBUG:boto:Params: {} 2024-04-12T03:32:24.705 DEBUG:boto:Token: None 2024-04-12T03:32:24.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/?versions 2024-04-12T03:32:24.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T03:32:24.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T03:32:24.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6e1984a2e5e417-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.707 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.708 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:32:24.708 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:32:24.708 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:32:24.708 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:32:24.709 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:32:24.709 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.709 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.709 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.709 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.709 DEBUG:boto:Method: GET 2024-04-12T03:32:24.709 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.709 DEBUG:boto:Data: 2024-04-12T03:32:24.709 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.709 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.710 DEBUG:boto:Params: {} 2024-04-12T03:32:24.710 DEBUG:boto:Token: None 2024-04-12T03:32:24.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/?versions 2024-04-12T03:32:24.710 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T03:32:24.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T03:32:24.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f51e48855d3913-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.713 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.713 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:32:24.713 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:32:24.714 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:32:24.714 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:32:24.714 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.714 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.714 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.714 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.714 DEBUG:boto:Method: GET 2024-04-12T03:32:24.714 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.714 DEBUG:boto:Data: 2024-04-12T03:32:24.714 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.715 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.715 DEBUG:boto:Params: {} 2024-04-12T03:32:24.715 DEBUG:boto:Token: None 2024-04-12T03:32:24.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/?versions 2024-04-12T03:32:24.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T03:32:24.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T03:32:24.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600047c6e4d0fd97-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.718 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.718 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.718 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.718 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.719 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.719 DEBUG:boto:Method: GET 2024-04-12T03:32:24.719 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:32:24.719 DEBUG:boto:Data: 2024-04-12T03:32:24.719 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.719 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.719 DEBUG:boto:Params: {} 2024-04-12T03:32:24.719 DEBUG:boto:Token: None 2024-04-12T03:32:24.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/?versions 2024-04-12T03:32:24.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T03:32:24.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T03:32:24.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e55892b09eae75e7-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.722 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:24.723 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.723 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.723 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.723 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.723 DEBUG:boto:Method: GET 2024-04-12T03:32:24.723 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.723 DEBUG:boto:Data: 2024-04-12T03:32:24.723 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.724 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.724 DEBUG:boto:Params: {} 2024-04-12T03:32:24.724 DEBUG:boto:Token: None 2024-04-12T03:32:24.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T03:32:24.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae26717403340ee-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.770 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.771 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.771 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.771 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.771 DEBUG:boto:Method: GET 2024-04-12T03:32:24.771 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.771 DEBUG:boto:Data: 2024-04-12T03:32:24.771 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.771 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.771 DEBUG:boto:Params: {} 2024-04-12T03:32:24.772 DEBUG:boto:Token: None 2024-04-12T03:32:24.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T03:32:24.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb31f4248c7daed-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.774 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.774 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.774 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.775 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.775 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.775 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.775 DEBUG:boto:Data: 2024-04-12T03:32:24.775 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.775 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.775 DEBUG:boto:Params: {} 2024-04-12T03:32:24.775 DEBUG:boto:Token: None 2024-04-12T03:32:24.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI= 2024-04-12T03:32:24.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI=', 'Content-Length': '0'} 2024-04-12T03:32:24.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62ceb95058c0fb6-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.777 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.778 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.778 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.778 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.778 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.778 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.778 DEBUG:boto:Data: 2024-04-12T03:32:24.778 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.778 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.778 DEBUG:boto:Params: {} 2024-04-12T03:32:24.779 DEBUG:boto:Token: None 2024-04-12T03:32:24.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI= 2024-04-12T03:32:24.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI=', 'Content-Length': '0'} 2024-04-12T03:32:24.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7c80d1fdea682c-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:24.781 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.781 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.781 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.781 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.781 DEBUG:boto:Method: GET 2024-04-12T03:32:24.781 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.781 DEBUG:boto:Data: 2024-04-12T03:32:24.782 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.782 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.782 DEBUG:boto:Params: {} 2024-04-12T03:32:24.782 DEBUG:boto:Token: None 2024-04-12T03:32:24.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T03:32:24.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead69c97b42e8d4f-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.784 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.784 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.784 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.784 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.785 DEBUG:boto:Method: GET 2024-04-12T03:32:24.785 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.785 DEBUG:boto:Data: 2024-04-12T03:32:24.785 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.785 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.785 DEBUG:boto:Params: {} 2024-04-12T03:32:24.785 DEBUG:boto:Token: None 2024-04-12T03:32:24.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:32:24.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T03:32:24.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ef6ea554d61764-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.787 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.788 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.788 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.788 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.788 DEBUG:boto:Method: GET 2024-04-12T03:32:24.788 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:32:24.788 DEBUG:boto:Data: 2024-04-12T03:32:24.788 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.788 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.788 DEBUG:boto:Params: {} 2024-04-12T03:32:24.788 DEBUG:boto:Token: None 2024-04-12T03:32:24.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776bd00bb0a20a35-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.791 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.791 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.792 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.792 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.792 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.792 DEBUG:boto:Method: GET 2024-04-12T03:32:24.792 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:32:24.792 DEBUG:boto:Data: 2024-04-12T03:32:24.792 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.792 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.792 DEBUG:boto:Params: {} 2024-04-12T03:32:24.793 DEBUG:boto:Token: None 2024-04-12T03:32:24.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5db47dd672c3851-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.796 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.796 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.796 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.796 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.796 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:32:24.796 DEBUG:boto:Data: 2024-04-12T03:32:24.797 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.797 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.797 DEBUG:boto:Params: {} 2024-04-12T03:32:24.797 DEBUG:boto:Token: None 2024-04-12T03:32:24.797 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a 2024-04-12T03:32:24.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI= 2024-04-12T03:32:24.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI=', 'Content-Length': '0'} 2024-04-12T03:32:24.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbda03e8dd2b8db-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.799 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.799 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.799 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.800 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:32:24.800 DEBUG:boto:Data: 2024-04-12T03:32:24.800 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.800 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.800 DEBUG:boto:Params: {} 2024-04-12T03:32:24.800 DEBUG:boto:Token: None 2024-04-12T03:32:24.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a 2024-04-12T03:32:24.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI= 2024-04-12T03:32:24.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI=', 'Content-Length': '0'} 2024-04-12T03:32:24.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4458528ef67e43-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:32:24.802 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.802 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.802 DEBUG:boto:Method: GET 2024-04-12T03:32:24.803 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:32:24.803 DEBUG:boto:Data: 2024-04-12T03:32:24.803 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.803 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.803 DEBUG:boto:Params: {} 2024-04-12T03:32:24.803 DEBUG:boto:Token: None 2024-04-12T03:32:24.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a 2024-04-12T03:32:24.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU= 2024-04-12T03:32:24.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU=', 'Content-Length': '0'} 2024-04-12T03:32:24.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048845d2331ed2618-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.846 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:32:24.847 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:32:24.847 DEBUG:boto:Method: GET 2024-04-12T03:32:24.847 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:32:24.847 DEBUG:boto:Data: 2024-04-12T03:32:24.847 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.847 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.847 DEBUG:boto:Params: {} 2024-04-12T03:32:24.848 DEBUG:boto:Token: None 2024-04-12T03:32:24.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/a 2024-04-12T03:32:24.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU= 2024-04-12T03:32:24.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU=', 'Content-Length': '0'} 2024-04-12T03:32:24.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001131951757a07294-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.850 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.850 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.850 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.851 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.851 DEBUG:boto:Method: GET 2024-04-12T03:32:24.851 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:32:24.851 DEBUG:boto:Data: 2024-04-12T03:32:24.851 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.851 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.851 DEBUG:boto:Params: {} 2024-04-12T03:32:24.851 DEBUG:boto:Token: None 2024-04-12T03:32:24.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcaaa9f7de51d866-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.854 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.855 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.855 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.855 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.855 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:32:24.855 DEBUG:boto:Method: GET 2024-04-12T03:32:24.855 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:32:24.855 DEBUG:boto:Data: 2024-04-12T03:32:24.855 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.856 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.856 DEBUG:boto:Params: {} 2024-04-12T03:32:24.856 DEBUG:boto:Token: None 2024-04-12T03:32:24.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c24909a70910970c-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.858 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:24.859 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.859 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.859 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.859 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.859 DEBUG:boto:Method: GET 2024-04-12T03:32:24.859 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.859 DEBUG:boto:Data: 2024-04-12T03:32:24.859 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.860 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.860 DEBUG:boto:Params: {} 2024-04-12T03:32:24.860 DEBUG:boto:Token: None 2024-04-12T03:32:24.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T03:32:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T03:32:24.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd13413821fb967-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.862 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.862 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.862 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.863 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.863 DEBUG:boto:Method: GET 2024-04-12T03:32:24.863 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.863 DEBUG:boto:Data: 2024-04-12T03:32:24.863 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.863 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.863 DEBUG:boto:Params: {} 2024-04-12T03:32:24.863 DEBUG:boto:Token: None 2024-04-12T03:32:24.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T03:32:24.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T03:32:24.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029878ae7dd438cf5-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.866 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.866 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.866 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.866 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.866 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.866 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.866 DEBUG:boto:Data: 2024-04-12T03:32:24.866 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.866 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.867 DEBUG:boto:Params: {} 2024-04-12T03:32:24.867 DEBUG:boto:Token: None 2024-04-12T03:32:24.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY= 2024-04-12T03:32:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY=', 'Content-Length': '0'} 2024-04-12T03:32:24.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e7d60bb78dc4a7-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.869 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.869 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.869 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.869 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.869 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.869 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.869 DEBUG:boto:Data: 2024-04-12T03:32:24.869 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.870 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.870 DEBUG:boto:Params: {} 2024-04-12T03:32:24.870 DEBUG:boto:Token: None 2024-04-12T03:32:24.870 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY= 2024-04-12T03:32:24.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY=', 'Content-Length': '0'} 2024-04-12T03:32:24.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004611e8246af91685-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:24.872 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.872 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.872 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.872 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.872 DEBUG:boto:Method: GET 2024-04-12T03:32:24.873 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.873 DEBUG:boto:Data: 2024-04-12T03:32:24.873 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.873 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.873 DEBUG:boto:Params: {} 2024-04-12T03:32:24.873 DEBUG:boto:Token: None 2024-04-12T03:32:24.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T03:32:24.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T03:32:24.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256bb3466ddf8f8f-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.876 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.876 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.876 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.876 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.876 DEBUG:boto:Method: GET 2024-04-12T03:32:24.876 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.876 DEBUG:boto:Data: 2024-04-12T03:32:24.876 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.876 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.876 DEBUG:boto:Params: {} 2024-04-12T03:32:24.877 DEBUG:boto:Token: None 2024-04-12T03:32:24.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:32:24.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T03:32:24.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T03:32:24.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f918c983b04c9dcd-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.879 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.879 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.879 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.879 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.879 DEBUG:boto:Method: GET 2024-04-12T03:32:24.879 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:32:24.879 DEBUG:boto:Data: 2024-04-12T03:32:24.879 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.880 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.880 DEBUG:boto:Params: {} 2024-04-12T03:32:24.880 DEBUG:boto:Token: None 2024-04-12T03:32:24.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273fcfad7fb87c35-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.882 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.883 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.883 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.883 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.883 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.883 DEBUG:boto:Method: GET 2024-04-12T03:32:24.883 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:32:24.883 DEBUG:boto:Data: 2024-04-12T03:32:24.884 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.884 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.884 DEBUG:boto:Params: {} 2024-04-12T03:32:24.884 DEBUG:boto:Token: None 2024-04-12T03:32:24.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2690f1f351defbd-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.887 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.887 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.887 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.887 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.887 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:32:24.887 DEBUG:boto:Data: 2024-04-12T03:32:24.888 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.888 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.888 DEBUG:boto:Params: {} 2024-04-12T03:32:24.888 DEBUG:boto:Token: None 2024-04-12T03:32:24.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b 2024-04-12T03:32:24.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk= 2024-04-12T03:32:24.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk=', 'Content-Length': '0'} 2024-04-12T03:32:24.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b4444aa70e31de-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.890 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.890 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.890 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.890 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:32:24.890 DEBUG:boto:Data: 2024-04-12T03:32:24.890 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.891 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.891 DEBUG:boto:Params: {} 2024-04-12T03:32:24.891 DEBUG:boto:Token: None 2024-04-12T03:32:24.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b 2024-04-12T03:32:24.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk= 2024-04-12T03:32:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk=', 'Content-Length': '0'} 2024-04-12T03:32:24.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1bc833657a69b59-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:32:24.893 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.893 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.893 DEBUG:boto:Method: GET 2024-04-12T03:32:24.893 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:32:24.893 DEBUG:boto:Data: 2024-04-12T03:32:24.893 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.894 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.894 DEBUG:boto:Params: {} 2024-04-12T03:32:24.894 DEBUG:boto:Token: None 2024-04-12T03:32:24.894 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b 2024-04-12T03:32:24.894 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0= 2024-04-12T03:32:24.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0=', 'Content-Length': '0'} 2024-04-12T03:32:24.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcec7687dd054e69-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.942 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:32:24.943 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:32:24.943 DEBUG:boto:Method: GET 2024-04-12T03:32:24.943 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:32:24.943 DEBUG:boto:Data: 2024-04-12T03:32:24.943 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.943 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.943 DEBUG:boto:Params: {} 2024-04-12T03:32:24.943 DEBUG:boto:Token: None 2024-04-12T03:32:24.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/b 2024-04-12T03:32:24.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0= 2024-04-12T03:32:24.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0=', 'Content-Length': '0'} 2024-04-12T03:32:24.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005918ac9461734269-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.946 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.946 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.946 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.946 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.947 DEBUG:boto:Method: GET 2024-04-12T03:32:24.947 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:32:24.947 DEBUG:boto:Data: 2024-04-12T03:32:24.947 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.947 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.947 DEBUG:boto:Params: {} 2024-04-12T03:32:24.947 DEBUG:boto:Token: None 2024-04-12T03:32:24.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009606439912693aa7-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.950 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.951 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.951 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.951 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.951 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:32:24.951 DEBUG:boto:Method: GET 2024-04-12T03:32:24.951 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:32:24.951 DEBUG:boto:Data: 2024-04-12T03:32:24.951 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.952 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.952 DEBUG:boto:Params: {} 2024-04-12T03:32:24.952 DEBUG:boto:Token: None 2024-04-12T03:32:24.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544a6bb7f2a3a83e-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.954 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:24.955 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.955 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.955 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.955 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.955 DEBUG:boto:Method: GET 2024-04-12T03:32:24.955 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.955 DEBUG:boto:Data: 2024-04-12T03:32:24.955 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.956 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.956 DEBUG:boto:Params: {} 2024-04-12T03:32:24.956 DEBUG:boto:Token: None 2024-04-12T03:32:24.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T03:32:24.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T03:32:24.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43fca59beff5218-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.958 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.958 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.958 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.958 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.958 DEBUG:boto:Method: GET 2024-04-12T03:32:24.959 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.959 DEBUG:boto:Data: 2024-04-12T03:32:24.959 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.959 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.959 DEBUG:boto:Params: {} 2024-04-12T03:32:24.959 DEBUG:boto:Token: None 2024-04-12T03:32:24.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T03:32:24.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T03:32:24.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015fa2d505f81a2f6-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.961 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.961 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.962 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.962 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.962 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.962 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.962 DEBUG:boto:Data: 2024-04-12T03:32:24.962 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.962 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.962 DEBUG:boto:Params: {} 2024-04-12T03:32:24.962 DEBUG:boto:Token: None 2024-04-12T03:32:24.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI= 2024-04-12T03:32:24.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI=', 'Content-Length': '0'} 2024-04-12T03:32:24.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3158f8f2cddebd-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.964 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.964 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.965 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.965 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.965 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.965 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.965 DEBUG:boto:Data: 2024-04-12T03:32:24.965 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.965 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.965 DEBUG:boto:Params: {} 2024-04-12T03:32:24.965 DEBUG:boto:Token: None 2024-04-12T03:32:24.966 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI= 2024-04-12T03:32:24.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI=', 'Content-Length': '0'} 2024-04-12T03:32:24.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fea6091c92b54a6-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:24.968 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.968 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.968 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.968 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.968 DEBUG:boto:Method: GET 2024-04-12T03:32:24.968 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.968 DEBUG:boto:Data: 2024-04-12T03:32:24.968 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.968 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.969 DEBUG:boto:Params: {} 2024-04-12T03:32:24.969 DEBUG:boto:Token: None 2024-04-12T03:32:24.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T03:32:24.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T03:32:24.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703a1ac52c80474d-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.971 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.971 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.971 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.971 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.971 DEBUG:boto:Method: GET 2024-04-12T03:32:24.971 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.971 DEBUG:boto:Data: 2024-04-12T03:32:24.971 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.972 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.972 DEBUG:boto:Params: {} 2024-04-12T03:32:24.972 DEBUG:boto:Token: None 2024-04-12T03:32:24.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:32:24.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T03:32:24.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T03:32:24.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2043bc84006bdb-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.974 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.974 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.974 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.975 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.975 DEBUG:boto:Method: GET 2024-04-12T03:32:24.975 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:32:24.975 DEBUG:boto:Data: 2024-04-12T03:32:24.975 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.975 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.975 DEBUG:boto:Params: {} 2024-04-12T03:32:24.975 DEBUG:boto:Token: None 2024-04-12T03:32:24.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b5b07e9fd91b5a-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.978 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.979 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.979 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.979 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.979 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.979 DEBUG:boto:Method: GET 2024-04-12T03:32:24.979 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:32:24.979 DEBUG:boto:Data: 2024-04-12T03:32:24.979 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.979 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.980 DEBUG:boto:Params: {} 2024-04-12T03:32:24.980 DEBUG:boto:Token: None 2024-04-12T03:32:24.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a749daf2a6661a61-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.982 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.983 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.983 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.983 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.983 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:32:24.983 DEBUG:boto:Data: 2024-04-12T03:32:24.983 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.983 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.983 DEBUG:boto:Params: {} 2024-04-12T03:32:24.984 DEBUG:boto:Token: None 2024-04-12T03:32:24.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c 2024-04-12T03:32:24.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo= 2024-04-12T03:32:24.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo=', 'Content-Length': '0'} 2024-04-12T03:32:24.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000247103755d70877e-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.986 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.986 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.986 DEBUG:boto:Method: HEAD 2024-04-12T03:32:24.986 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:32:24.986 DEBUG:boto:Data: 2024-04-12T03:32:24.986 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.986 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.987 DEBUG:boto:Params: {} 2024-04-12T03:32:24.987 DEBUG:boto:Token: None 2024-04-12T03:32:24.987 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c 2024-04-12T03:32:24.987 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo= 2024-04-12T03:32:24.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo=', 'Content-Length': '0'} 2024-04-12T03:32:24.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf1b568a3b3379b-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:32:24.989 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.989 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.989 DEBUG:boto:Method: GET 2024-04-12T03:32:24.989 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:32:24.989 DEBUG:boto:Data: 2024-04-12T03:32:24.989 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.989 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.989 DEBUG:boto:Params: {} 2024-04-12T03:32:24.990 DEBUG:boto:Token: None 2024-04-12T03:32:24.990 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c 2024-04-12T03:32:24.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ= 2024-04-12T03:32:24.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f1504667d8b5de-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.992 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:32:24.992 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:32:24.992 DEBUG:boto:Method: GET 2024-04-12T03:32:24.992 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:32:24.992 DEBUG:boto:Data: 2024-04-12T03:32:24.992 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:24.992 DEBUG:boto:Port: 8000 2024-04-12T03:32:24.992 DEBUG:boto:Params: {} 2024-04-12T03:32:24.992 DEBUG:boto:Token: None 2024-04-12T03:32:24.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/c 2024-04-12T03:32:24.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ= 2024-04-12T03:32:24.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ=', 'Content-Length': '0'} 2024-04-12T03:32:24.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5bb3f9d39b17829-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.994 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.995 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.995 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.995 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.995 DEBUG:boto:Method: GET 2024-04-12T03:32:24.995 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:32:24.995 DEBUG:boto:Data: 2024-04-12T03:32:24.995 DEBUG:boto:Headers: {} 2024-04-12T03:32:24.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:24.995 DEBUG:boto:Port: 8001 2024-04-12T03:32:24.995 DEBUG:boto:Params: {} 2024-04-12T03:32:24.996 DEBUG:boto:Token: None 2024-04-12T03:32:24.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:24 GMT /dgbpxg-57/ 2024-04-12T03:32:24.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T03:32:24.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T03:32:24.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028fb5278f3da6f4f-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:24.998 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:24.999 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:24.999 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:24.999 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.999 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:32:24.999 DEBUG:boto:Method: GET 2024-04-12T03:32:24.999 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:32:24.999 DEBUG:boto:Data: 2024-04-12T03:32:24.999 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.000 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.000 DEBUG:boto:Params: {} 2024-04-12T03:32:25.000 DEBUG:boto:Token: None 2024-04-12T03:32:25.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T03:32:25.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T03:32:25.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83ef1ad57e1235d-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.046 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:25.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:25.047 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.047 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.047 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.047 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.048 DEBUG:boto:Method: GET 2024-04-12T03:32:25.048 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.048 DEBUG:boto:Data: 2024-04-12T03:32:25.048 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.048 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.048 DEBUG:boto:Params: {} 2024-04-12T03:32:25.048 DEBUG:boto:Token: None 2024-04-12T03:32:25.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T03:32:25.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T03:32:25.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4944adbf534503-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.051 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.051 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.051 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.051 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.051 DEBUG:boto:Method: GET 2024-04-12T03:32:25.051 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.051 DEBUG:boto:Data: 2024-04-12T03:32:25.051 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.051 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.051 DEBUG:boto:Params: {} 2024-04-12T03:32:25.052 DEBUG:boto:Token: None 2024-04-12T03:32:25.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T03:32:25.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T03:32:25.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744973b933751fa5-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.054 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.054 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.054 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.054 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.054 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.054 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.054 DEBUG:boto:Data: 2024-04-12T03:32:25.055 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.055 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.055 DEBUG:boto:Params: {} 2024-04-12T03:32:25.055 DEBUG:boto:Token: None 2024-04-12T03:32:25.055 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8= 2024-04-12T03:32:25.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8=', 'Content-Length': '0'} 2024-04-12T03:32:25.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050299d447782dc1f-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.057 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.057 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.057 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.057 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.057 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.057 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.058 DEBUG:boto:Data: 2024-04-12T03:32:25.058 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.058 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.058 DEBUG:boto:Params: {} 2024-04-12T03:32:25.058 DEBUG:boto:Token: None 2024-04-12T03:32:25.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8= 2024-04-12T03:32:25.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8=', 'Content-Length': '0'} 2024-04-12T03:32:25.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a80a22b7c2c8680-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:25.060 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.060 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.061 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.061 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.061 DEBUG:boto:Method: GET 2024-04-12T03:32:25.061 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.061 DEBUG:boto:Data: 2024-04-12T03:32:25.061 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.061 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.061 DEBUG:boto:Params: {} 2024-04-12T03:32:25.061 DEBUG:boto:Token: None 2024-04-12T03:32:25.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T03:32:25.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T03:32:25.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444135719933d3ea-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.106 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.107 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.107 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.107 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.107 DEBUG:boto:Method: GET 2024-04-12T03:32:25.107 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.107 DEBUG:boto:Data: 2024-04-12T03:32:25.107 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.107 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.107 DEBUG:boto:Params: {} 2024-04-12T03:32:25.107 DEBUG:boto:Token: None 2024-04-12T03:32:25.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:32:25.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T03:32:25.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T03:32:25.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867434c1d49899e3-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.110 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.110 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.110 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.110 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.110 DEBUG:boto:Method: GET 2024-04-12T03:32:25.111 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:32:25.111 DEBUG:boto:Data: 2024-04-12T03:32:25.111 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.111 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.111 DEBUG:boto:Params: {} 2024-04-12T03:32:25.111 DEBUG:boto:Token: None 2024-04-12T03:32:25.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T03:32:25.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T03:32:25.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53c3832942cccc6-006618ab49-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.114 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:25.114 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.114 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.114 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.115 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.115 DEBUG:boto:Method: GET 2024-04-12T03:32:25.115 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:32:25.115 DEBUG:boto:Data: 2024-04-12T03:32:25.115 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.115 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.115 DEBUG:boto:Params: {} 2024-04-12T03:32:25.115 DEBUG:boto:Token: None 2024-04-12T03:32:25.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T03:32:25.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T03:32:25.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf7bcaeaf2dee3a-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.118 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:25.118 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.119 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.119 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.119 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:32:25.119 DEBUG:boto:Data: 2024-04-12T03:32:25.119 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.119 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.119 DEBUG:boto:Params: {} 2024-04-12T03:32:25.119 DEBUG:boto:Token: None 2024-04-12T03:32:25.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d 2024-04-12T03:32:25.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE= 2024-04-12T03:32:25.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE=', 'Content-Length': '0'} 2024-04-12T03:32:25.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a46968db79a4b2-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.122 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.122 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.122 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.122 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:32:25.122 DEBUG:boto:Data: 2024-04-12T03:32:25.122 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.122 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.122 DEBUG:boto:Params: {} 2024-04-12T03:32:25.123 DEBUG:boto:Token: None 2024-04-12T03:32:25.123 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d 2024-04-12T03:32:25.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE= 2024-04-12T03:32:25.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE=', 'Content-Length': '0'} 2024-04-12T03:32:25.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faf7e18b6430971-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:32:25.125 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.125 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.125 DEBUG:boto:Method: GET 2024-04-12T03:32:25.125 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:32:25.125 DEBUG:boto:Data: 2024-04-12T03:32:25.125 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.125 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.125 DEBUG:boto:Params: {} 2024-04-12T03:32:25.125 DEBUG:boto:Token: None 2024-04-12T03:32:25.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d 2024-04-12T03:32:25.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng= 2024-04-12T03:32:25.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng=', 'Content-Length': '0'} 2024-04-12T03:32:25.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b595ce40275293-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.128 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:32:25.128 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:32:25.128 DEBUG:boto:Method: GET 2024-04-12T03:32:25.128 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:32:25.128 DEBUG:boto:Data: 2024-04-12T03:32:25.128 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.128 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.128 DEBUG:boto:Params: {} 2024-04-12T03:32:25.128 DEBUG:boto:Token: None 2024-04-12T03:32:25.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/d 2024-04-12T03:32:25.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng= 2024-04-12T03:32:25.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng=', 'Content-Length': '0'} 2024-04-12T03:32:25.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033270db4c54e4d16-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.130 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.131 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.131 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.131 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.131 DEBUG:boto:Method: GET 2024-04-12T03:32:25.131 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:32:25.131 DEBUG:boto:Data: 2024-04-12T03:32:25.131 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:32:25.131 DEBUG:boto:Port: 8001 2024-04-12T03:32:25.131 DEBUG:boto:Params: {} 2024-04-12T03:32:25.132 DEBUG:boto:Token: None 2024-04-12T03:32:25.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T03:32:25.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T03:32:25.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080fd46c0646d0359-006618ab49-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.134 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:25.134 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.135 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.135 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.135 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:32:25.135 DEBUG:boto:Method: GET 2024-04-12T03:32:25.135 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:32:25.135 DEBUG:boto:Data: 2024-04-12T03:32:25.135 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.135 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.135 DEBUG:boto:Params: {} 2024-04-12T03:32:25.135 DEBUG:boto:Token: None 2024-04-12T03:32:25.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T03:32:25.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T03:32:25.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf64ab5fd6e0f1f-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.138 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:25.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:32:25.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.139 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:32:25.185 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.185 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.224 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:32:25.224 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.224 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2024-04-12T03:32:25.224 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2024-04-12T03:32:25.224 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:32:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.227 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.227 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.227 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:32:25.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:25.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:25.228 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.318 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:32:25.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.318 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "1#00000000001.51.6", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14556731054388386214", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "a", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.044816674Z", 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:32:25.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "5#00000000001.28.6", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16167048996727160496", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "d", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.112463396Z", 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:25.321 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T03:32:25.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "8#00000000001.26.6", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.7091642361043578140", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "c", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.103008525Z", 2024-04-12T03:32:25.323 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T03:32:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "10#00000000001.61.6", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1910817736580495436", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "b", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:32:04.093278075Z", 2024-04-12T03:32:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T03:32:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.327 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:32:25.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.328 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:32:25.378 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.378 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.416 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:32:25.416 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-57 2024-04-12T03:32:25.420 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2024-04-12T03:32:25.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:32:25.458 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:32:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:32:25.549 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:32:25.552 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:32:25.553 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:32:25.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:25.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:32:25.555 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:32:25.603 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.603 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.646 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2024-04-12T03:32:25.648 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.648 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.648 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.648 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:25.648 DEBUG:boto:Data: 2024-04-12T03:32:25.649 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.649 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.649 DEBUG:boto:Params: {} 2024-04-12T03:32:25.649 DEBUG:boto:Token: None 2024-04-12T03:32:25.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T03:32:25.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T03:32:25.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7157626c7ac9333-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.652 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:32:25.652 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:32:25.652 DEBUG:boto:Method: PUT 2024-04-12T03:32:25.652 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:32:25.652 DEBUG:boto:Data: 2024-04-12T03:32:25.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:25.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.652 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.652 DEBUG:boto:Params: {} 2024-04-12T03:32:25.653 DEBUG:boto:Token: None 2024-04-12T03:32:25.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/e 2024-04-12T03:32:25.653 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R1woxug4lRbXkPJ/AKTcJZzXiiQ= 2024-04-12T03:32:25.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R1woxug4lRbXkPJ/AKTcJZzXiiQ='} 2024-04-12T03:32:25.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff475f2d5059a3a3-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.699 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.699 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.699 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.699 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:25.699 DEBUG:boto:Data: 2024-04-12T03:32:25.699 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.700 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.700 DEBUG:boto:Params: {} 2024-04-12T03:32:25.700 DEBUG:boto:Token: None 2024-04-12T03:32:25.700 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T03:32:25.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T03:32:25.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f176b4980abdc5c0-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.702 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:32:25.702 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:32:25.703 DEBUG:boto:Method: PUT 2024-04-12T03:32:25.703 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:32:25.703 DEBUG:boto:Data: 2024-04-12T03:32:25.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:25.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.703 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.703 DEBUG:boto:Params: {} 2024-04-12T03:32:25.703 DEBUG:boto:Token: None 2024-04-12T03:32:25.703 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/f 2024-04-12T03:32:25.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kqlp2zfxJjODQbUnL1urTJG60CU= 2024-04-12T03:32:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kqlp2zfxJjODQbUnL1urTJG60CU='} 2024-04-12T03:32:25.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013dd7c9e94cb7757-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.708 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.708 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.708 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.709 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:25.709 DEBUG:boto:Data: 2024-04-12T03:32:25.709 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.709 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.709 DEBUG:boto:Params: {} 2024-04-12T03:32:25.709 DEBUG:boto:Token: None 2024-04-12T03:32:25.709 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T03:32:25.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T03:32:25.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000d36cfb6de2340c-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.711 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:32:25.711 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:32:25.711 DEBUG:boto:Method: PUT 2024-04-12T03:32:25.712 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:32:25.712 DEBUG:boto:Data: 2024-04-12T03:32:25.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:25.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.712 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.712 DEBUG:boto:Params: {} 2024-04-12T03:32:25.712 DEBUG:boto:Token: None 2024-04-12T03:32:25.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/g 2024-04-12T03:32:25.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1vKVsGuUTSTLFx9+kieiNL611Xo= 2024-04-12T03:32:25.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vKVsGuUTSTLFx9+kieiNL611Xo='} 2024-04-12T03:32:25.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3a6d7ad2e9b9aa7-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.755 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:32:25.755 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:32:25.755 DEBUG:boto:Method: HEAD 2024-04-12T03:32:25.755 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:32:25.755 DEBUG:boto:Data: 2024-04-12T03:32:25.755 DEBUG:boto:Headers: {} 2024-04-12T03:32:25.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.755 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.755 DEBUG:boto:Params: {} 2024-04-12T03:32:25.756 DEBUG:boto:Token: None 2024-04-12T03:32:25.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/ 2024-04-12T03:32:25.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T03:32:25.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T03:32:25.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030ba0a654919a71e-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.758 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:32:25.758 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:32:25.758 DEBUG:boto:Method: PUT 2024-04-12T03:32:25.759 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:32:25.759 DEBUG:boto:Data: 2024-04-12T03:32:25.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:32:25.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:32:25.759 DEBUG:boto:Port: 8000 2024-04-12T03:32:25.759 DEBUG:boto:Params: {} 2024-04-12T03:32:25.759 DEBUG:boto:Token: None 2024-04-12T03:32:25.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:32:25 GMT /dgbpxg-57/h 2024-04-12T03:32:25.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GBOIFbuzRxug25lt6szcF6bwIHM= 2024-04-12T03:32:25.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GBOIFbuzRxug25lt6szcF6bwIHM='} 2024-04-12T03:32:25.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8994fff5044af7d-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:25.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T03:32:25.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:25.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:25.764 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:32:25.819 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:25.819 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:25.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:25.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:32:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:30.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:32:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:35.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:32:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:40.855+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T03:32:40.856 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.42.6, , ] 2024-04-12T03:32:40.856 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:32:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:45.855+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T03:32:45.856 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.42.6, , ] 2024-04-12T03:32:45.857 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:32:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:50.856+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T03:32:50.857 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.42.6, , ] 2024-04-12T03:32:50.857 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:32:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:32:55.857+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T03:32:55.858 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.42.6, , ] 2024-04-12T03:32:55.858 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:33:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:00.858+0000 7f6ac1450780 1 bucket sync caught up with source: 2024-04-12T03:33:00.859 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:33:00.859 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T03:33:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:00.858+0000 7f6ac1450780 0 bucket checkpoint complete 2024-04-12T03:33:00.877 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T03:33:00.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:00.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:00.878 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:00.973+0000 7fd90ca99780 1 waiting for incremental sync to catch up: 2024-04-12T03:33:00.974 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1211.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000001.1248.6, , 00000000002.1331.6] 2024-04-12T03:33:00.974 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T03:33:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:05.974+0000 7fd90ca99780 1 bucket sync caught up with source: 2024-04-12T03:33:05.975 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T03:33:05.975 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T03:33:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:05.974+0000 7fd90ca99780 0 bucket checkpoint complete 2024-04-12T03:33:05.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:33:05.992 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:05.992 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:05.992 DEBUG:boto:Method: HEAD 2024-04-12T03:33:05.992 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:33:05.993 DEBUG:boto:Data: 2024-04-12T03:33:05.993 DEBUG:boto:Headers: {} 2024-04-12T03:33:05.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:05.993 DEBUG:boto:Port: 8001 2024-04-12T03:33:05.993 DEBUG:boto:Params: {} 2024-04-12T03:33:05.993 DEBUG:boto:Token: None 2024-04-12T03:33:05.993 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:05 GMT /dgbpxg-57/ 2024-04-12T03:33:05.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss= 2024-04-12T03:33:05.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss=', 'Content-Length': '0'} 2024-04-12T03:33:05.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005883d9650e69ba-006618ab71-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:05.996 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:05.996 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:05.996 DEBUG:boto:Method: HEAD 2024-04-12T03:33:05.997 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:33:05.997 DEBUG:boto:Data: 2024-04-12T03:33:05.997 DEBUG:boto:Headers: {} 2024-04-12T03:33:05.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:05.997 DEBUG:boto:Port: 8000 2024-04-12T03:33:05.997 DEBUG:boto:Params: {} 2024-04-12T03:33:05.997 DEBUG:boto:Token: None 2024-04-12T03:33:05.997 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:05 GMT /dgbpxg-57/ 2024-04-12T03:33:05.998 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss= 2024-04-12T03:33:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss=', 'Content-Length': '0'} 2024-04-12T03:33:05.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa14c91dcceef6ac-006618ab71-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:05.999 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:33:06.000 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.000 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.000 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.000 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.000 DEBUG:boto:Method: GET 2024-04-12T03:33:06.000 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.000 DEBUG:boto:Data: 2024-04-12T03:33:06.000 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.000 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.000 DEBUG:boto:Params: {} 2024-04-12T03:33:06.001 DEBUG:boto:Token: None 2024-04-12T03:33:06.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/?versions 2024-04-12T03:33:06.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T03:33:06.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T03:33:06.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646030a0bcb1df18-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.003 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.004 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:06.004 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:06.004 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:33:06.005 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:33:06.005 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.005 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.005 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.005 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.005 DEBUG:boto:Method: GET 2024-04-12T03:33:06.006 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.006 DEBUG:boto:Data: 2024-04-12T03:33:06.006 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.006 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.006 DEBUG:boto:Params: {} 2024-04-12T03:33:06.006 DEBUG:boto:Token: None 2024-04-12T03:33:06.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/?versions 2024-04-12T03:33:06.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T03:33:06.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T03:33:06.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a227a87b2c711c-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.009 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.009 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:06.009 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:06.009 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:06.010 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:06.010 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:33:06.010 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:33:06.010 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:33:06.010 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:33:06.010 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.010 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.010 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.010 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.010 DEBUG:boto:Method: GET 2024-04-12T03:33:06.011 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.013 DEBUG:boto:Data: 2024-04-12T03:33:06.014 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.014 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.014 DEBUG:boto:Params: {} 2024-04-12T03:33:06.014 DEBUG:boto:Token: None 2024-04-12T03:33:06.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/?versions 2024-04-12T03:33:06.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T03:33:06.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T03:33:06.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f1cc5a757b2cfe-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.017 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.018 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.018 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.018 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.018 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.018 DEBUG:boto:Method: GET 2024-04-12T03:33:06.018 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:06.018 DEBUG:boto:Data: 2024-04-12T03:33:06.018 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.019 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.019 DEBUG:boto:Params: {} 2024-04-12T03:33:06.019 DEBUG:boto:Token: None 2024-04-12T03:33:06.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/?versions 2024-04-12T03:33:06.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T03:33:06.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T03:33:06.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f7e33980f0ddd5-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.021 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:06.022 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.022 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.022 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.022 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.022 DEBUG:boto:Method: GET 2024-04-12T03:33:06.022 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.023 DEBUG:boto:Data: 2024-04-12T03:33:06.023 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.023 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.023 DEBUG:boto:Params: {} 2024-04-12T03:33:06.023 DEBUG:boto:Token: None 2024-04-12T03:33:06.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T03:33:06.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T03:33:06.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7919b151280636-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.025 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.025 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.026 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.026 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.026 DEBUG:boto:Method: GET 2024-04-12T03:33:06.026 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.026 DEBUG:boto:Data: 2024-04-12T03:33:06.026 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.026 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.026 DEBUG:boto:Params: {} 2024-04-12T03:33:06.027 DEBUG:boto:Token: None 2024-04-12T03:33:06.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T03:33:06.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T03:33:06.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ce483a2898e181-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.029 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.029 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.029 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.029 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.029 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.029 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.030 DEBUG:boto:Data: 2024-04-12T03:33:06.030 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.030 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.030 DEBUG:boto:Params: {} 2024-04-12T03:33:06.030 DEBUG:boto:Token: None 2024-04-12T03:33:06.030 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.030 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY= 2024-04-12T03:33:06.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY=', 'Content-Length': '0'} 2024-04-12T03:33:06.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9e0bf7fbd00db9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.032 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.032 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.033 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.033 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.033 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.033 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.033 DEBUG:boto:Data: 2024-04-12T03:33:06.033 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.033 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.033 DEBUG:boto:Params: {} 2024-04-12T03:33:06.033 DEBUG:boto:Token: None 2024-04-12T03:33:06.033 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY= 2024-04-12T03:33:06.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY=', 'Content-Length': '0'} 2024-04-12T03:33:06.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d064a4adbd5fc8e5-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:06.035 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.036 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.036 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.036 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.036 DEBUG:boto:Method: GET 2024-04-12T03:33:06.036 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.036 DEBUG:boto:Data: 2024-04-12T03:33:06.036 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.036 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.036 DEBUG:boto:Params: {} 2024-04-12T03:33:06.036 DEBUG:boto:Token: None 2024-04-12T03:33:06.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T03:33:06.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T03:33:06.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e8c20e7097e900-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.039 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.039 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.039 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.039 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.039 DEBUG:boto:Method: GET 2024-04-12T03:33:06.039 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.039 DEBUG:boto:Data: 2024-04-12T03:33:06.039 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.039 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.040 DEBUG:boto:Params: {} 2024-04-12T03:33:06.040 DEBUG:boto:Token: None 2024-04-12T03:33:06.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:06.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T03:33:06.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T03:33:06.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225f294b21a3f73b-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.042 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.042 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.042 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.042 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.043 DEBUG:boto:Method: GET 2024-04-12T03:33:06.043 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:06.043 DEBUG:boto:Data: 2024-04-12T03:33:06.043 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.043 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.043 DEBUG:boto:Params: {} 2024-04-12T03:33:06.043 DEBUG:boto:Token: None 2024-04-12T03:33:06.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8fa941e923d26d5-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.090 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.091 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.091 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.091 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.091 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.091 DEBUG:boto:Method: GET 2024-04-12T03:33:06.091 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:06.091 DEBUG:boto:Data: 2024-04-12T03:33:06.092 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.092 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.092 DEBUG:boto:Params: {} 2024-04-12T03:33:06.092 DEBUG:boto:Token: None 2024-04-12T03:33:06.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001151f6e1f31678-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.094 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.095 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.095 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.095 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.095 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:06.095 DEBUG:boto:Data: 2024-04-12T03:33:06.095 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.095 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.095 DEBUG:boto:Params: {} 2024-04-12T03:33:06.096 DEBUG:boto:Token: None 2024-04-12T03:33:06.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a 2024-04-12T03:33:06.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y= 2024-04-12T03:33:06.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y=', 'Content-Length': '0'} 2024-04-12T03:33:06.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9100e3115e9e7c4-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.098 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.098 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.098 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.098 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:06.098 DEBUG:boto:Data: 2024-04-12T03:33:06.098 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.098 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.098 DEBUG:boto:Params: {} 2024-04-12T03:33:06.099 DEBUG:boto:Token: None 2024-04-12T03:33:06.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a 2024-04-12T03:33:06.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y= 2024-04-12T03:33:06.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y=', 'Content-Length': '0'} 2024-04-12T03:33:06.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a323f1b68d16f6c8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:06.101 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.101 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.101 DEBUG:boto:Method: GET 2024-04-12T03:33:06.101 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:06.102 DEBUG:boto:Data: 2024-04-12T03:33:06.102 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.102 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.102 DEBUG:boto:Params: {} 2024-04-12T03:33:06.102 DEBUG:boto:Token: None 2024-04-12T03:33:06.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a 2024-04-12T03:33:06.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI= 2024-04-12T03:33:06.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI=', 'Content-Length': '0'} 2024-04-12T03:33:06.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059092285e94feea8-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.104 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:06.105 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:06.105 DEBUG:boto:Method: GET 2024-04-12T03:33:06.105 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:06.105 DEBUG:boto:Data: 2024-04-12T03:33:06.105 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.105 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.105 DEBUG:boto:Params: {} 2024-04-12T03:33:06.105 DEBUG:boto:Token: None 2024-04-12T03:33:06.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/a 2024-04-12T03:33:06.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI= 2024-04-12T03:33:06.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI=', 'Content-Length': '0'} 2024-04-12T03:33:06.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a91e32f556f6452-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.150 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.151 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.151 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.151 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.151 DEBUG:boto:Method: GET 2024-04-12T03:33:06.151 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:06.151 DEBUG:boto:Data: 2024-04-12T03:33:06.151 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.151 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.151 DEBUG:boto:Params: {} 2024-04-12T03:33:06.152 DEBUG:boto:Token: None 2024-04-12T03:33:06.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d3786a25bfb6bc-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.155 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.155 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.155 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.155 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.155 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:06.156 DEBUG:boto:Method: GET 2024-04-12T03:33:06.156 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:06.156 DEBUG:boto:Data: 2024-04-12T03:33:06.156 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.156 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.156 DEBUG:boto:Params: {} 2024-04-12T03:33:06.156 DEBUG:boto:Token: None 2024-04-12T03:33:06.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebd142f8bfc6495a-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.159 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:06.159 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.159 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.159 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.159 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.159 DEBUG:boto:Method: GET 2024-04-12T03:33:06.159 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.160 DEBUG:boto:Data: 2024-04-12T03:33:06.160 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.160 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.160 DEBUG:boto:Params: {} 2024-04-12T03:33:06.160 DEBUG:boto:Token: None 2024-04-12T03:33:06.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T03:33:06.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T03:33:06.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745487cde1ef6b1d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.162 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.163 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.163 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.163 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.163 DEBUG:boto:Method: GET 2024-04-12T03:33:06.163 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.163 DEBUG:boto:Data: 2024-04-12T03:33:06.163 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.163 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.163 DEBUG:boto:Params: {} 2024-04-12T03:33:06.163 DEBUG:boto:Token: None 2024-04-12T03:33:06.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T03:33:06.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T03:33:06.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000758d036da385c6f8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.166 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.166 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.166 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.166 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.166 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.167 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.167 DEBUG:boto:Data: 2024-04-12T03:33:06.167 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.167 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.167 DEBUG:boto:Params: {} 2024-04-12T03:33:06.167 DEBUG:boto:Token: None 2024-04-12T03:33:06.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc= 2024-04-12T03:33:06.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc=', 'Content-Length': '0'} 2024-04-12T03:33:06.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e5ffb0f4537d16-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.169 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.169 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.169 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.169 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.170 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.170 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.170 DEBUG:boto:Data: 2024-04-12T03:33:06.170 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.170 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.170 DEBUG:boto:Params: {} 2024-04-12T03:33:06.170 DEBUG:boto:Token: None 2024-04-12T03:33:06.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.170 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc= 2024-04-12T03:33:06.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc=', 'Content-Length': '0'} 2024-04-12T03:33:06.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585976525169cc91-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:06.173 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.173 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.173 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.173 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.173 DEBUG:boto:Method: GET 2024-04-12T03:33:06.173 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.173 DEBUG:boto:Data: 2024-04-12T03:33:06.173 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.173 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.174 DEBUG:boto:Params: {} 2024-04-12T03:33:06.174 DEBUG:boto:Token: None 2024-04-12T03:33:06.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.174 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T03:33:06.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T03:33:06.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d0cf3da66b1aa0-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.222 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.223 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.223 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.223 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.223 DEBUG:boto:Method: GET 2024-04-12T03:33:06.223 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.223 DEBUG:boto:Data: 2024-04-12T03:33:06.223 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.223 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.223 DEBUG:boto:Params: {} 2024-04-12T03:33:06.224 DEBUG:boto:Token: None 2024-04-12T03:33:06.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:06.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T03:33:06.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T03:33:06.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437b5ee9a33c37cf-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.228 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.228 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.228 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.228 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.228 DEBUG:boto:Method: GET 2024-04-12T03:33:06.228 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:06.228 DEBUG:boto:Data: 2024-04-12T03:33:06.228 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.228 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.228 DEBUG:boto:Params: {} 2024-04-12T03:33:06.229 DEBUG:boto:Token: None 2024-04-12T03:33:06.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a324cde70169c9-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.232 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.232 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.232 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.232 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.233 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.233 DEBUG:boto:Method: GET 2024-04-12T03:33:06.233 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:06.233 DEBUG:boto:Data: 2024-04-12T03:33:06.233 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.233 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.233 DEBUG:boto:Params: {} 2024-04-12T03:33:06.233 DEBUG:boto:Token: None 2024-04-12T03:33:06.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.234 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e1449f633b7fc43-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.282 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.283 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.283 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.283 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.283 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:06.283 DEBUG:boto:Data: 2024-04-12T03:33:06.284 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.284 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.284 DEBUG:boto:Params: {} 2024-04-12T03:33:06.284 DEBUG:boto:Token: None 2024-04-12T03:33:06.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b 2024-04-12T03:33:06.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk= 2024-04-12T03:33:06.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk=', 'Content-Length': '0'} 2024-04-12T03:33:06.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baa61815f2590500-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.287 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.287 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.287 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.287 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:06.287 DEBUG:boto:Data: 2024-04-12T03:33:06.287 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.287 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.287 DEBUG:boto:Params: {} 2024-04-12T03:33:06.288 DEBUG:boto:Token: None 2024-04-12T03:33:06.288 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b 2024-04-12T03:33:06.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk= 2024-04-12T03:33:06.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk=', 'Content-Length': '0'} 2024-04-12T03:33:06.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af2604cb1584f15-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:06.290 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.290 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.290 DEBUG:boto:Method: GET 2024-04-12T03:33:06.290 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:06.290 DEBUG:boto:Data: 2024-04-12T03:33:06.290 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.291 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.291 DEBUG:boto:Params: {} 2024-04-12T03:33:06.291 DEBUG:boto:Token: None 2024-04-12T03:33:06.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b 2024-04-12T03:33:06.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI= 2024-04-12T03:33:06.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI=', 'Content-Length': '0'} 2024-04-12T03:33:06.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed0c8167ec13183-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.293 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:06.293 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:06.293 DEBUG:boto:Method: GET 2024-04-12T03:33:06.293 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:06.293 DEBUG:boto:Data: 2024-04-12T03:33:06.294 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.294 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.294 DEBUG:boto:Params: {} 2024-04-12T03:33:06.294 DEBUG:boto:Token: None 2024-04-12T03:33:06.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/b 2024-04-12T03:33:06.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI= 2024-04-12T03:33:06.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI=', 'Content-Length': '0'} 2024-04-12T03:33:06.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d4e609834308dd-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.296 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.297 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.297 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.297 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.297 DEBUG:boto:Method: GET 2024-04-12T03:33:06.297 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:06.297 DEBUG:boto:Data: 2024-04-12T03:33:06.297 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.297 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.297 DEBUG:boto:Params: {} 2024-04-12T03:33:06.297 DEBUG:boto:Token: None 2024-04-12T03:33:06.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.298 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbaa0526dfcbd3ab-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.300 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.301 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.301 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.301 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.301 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:06.301 DEBUG:boto:Method: GET 2024-04-12T03:33:06.301 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:06.301 DEBUG:boto:Data: 2024-04-12T03:33:06.301 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.301 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.301 DEBUG:boto:Params: {} 2024-04-12T03:33:06.302 DEBUG:boto:Token: None 2024-04-12T03:33:06.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000789e90160192ac53-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.304 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:06.305 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.305 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.305 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.305 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.305 DEBUG:boto:Method: GET 2024-04-12T03:33:06.305 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.305 DEBUG:boto:Data: 2024-04-12T03:33:06.305 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.305 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.306 DEBUG:boto:Params: {} 2024-04-12T03:33:06.306 DEBUG:boto:Token: None 2024-04-12T03:33:06.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T03:33:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T03:33:06.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a3fe0af8675590-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.308 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.308 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.308 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.308 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.308 DEBUG:boto:Method: GET 2024-04-12T03:33:06.308 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.309 DEBUG:boto:Data: 2024-04-12T03:33:06.309 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.309 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.309 DEBUG:boto:Params: {} 2024-04-12T03:33:06.309 DEBUG:boto:Token: None 2024-04-12T03:33:06.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T03:33:06.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T03:33:06.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb069f6ff38439ff-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.311 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.311 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.312 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.312 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.312 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.312 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.312 DEBUG:boto:Data: 2024-04-12T03:33:06.312 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.312 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.312 DEBUG:boto:Params: {} 2024-04-12T03:33:06.312 DEBUG:boto:Token: None 2024-04-12T03:33:06.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck= 2024-04-12T03:33:06.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck=', 'Content-Length': '0'} 2024-04-12T03:33:06.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af1306f25810da5e-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.315 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.315 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.315 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.315 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.315 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.315 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.316 DEBUG:boto:Data: 2024-04-12T03:33:06.316 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.316 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.316 DEBUG:boto:Params: {} 2024-04-12T03:33:06.316 DEBUG:boto:Token: None 2024-04-12T03:33:06.317 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.317 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck= 2024-04-12T03:33:06.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck=', 'Content-Length': '0'} 2024-04-12T03:33:06.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ad7e8939ea6732-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:06.319 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.319 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.319 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.319 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.319 DEBUG:boto:Method: GET 2024-04-12T03:33:06.320 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.320 DEBUG:boto:Data: 2024-04-12T03:33:06.320 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.320 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.320 DEBUG:boto:Params: {} 2024-04-12T03:33:06.320 DEBUG:boto:Token: None 2024-04-12T03:33:06.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.321 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T03:33:06.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T03:33:06.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008032eee19ef8c0b9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.366 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.367 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.367 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.367 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.367 DEBUG:boto:Method: GET 2024-04-12T03:33:06.367 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.367 DEBUG:boto:Data: 2024-04-12T03:33:06.367 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.367 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.367 DEBUG:boto:Params: {} 2024-04-12T03:33:06.368 DEBUG:boto:Token: None 2024-04-12T03:33:06.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:06.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T03:33:06.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T03:33:06.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac6dfaa41f2acf7-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.372 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.372 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.372 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.372 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.372 DEBUG:boto:Method: GET 2024-04-12T03:33:06.372 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:06.372 DEBUG:boto:Data: 2024-04-12T03:33:06.372 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.373 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.373 DEBUG:boto:Params: {} 2024-04-12T03:33:06.373 DEBUG:boto:Token: None 2024-04-12T03:33:06.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766b694c31f05753-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.376 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.377 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.377 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.377 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.377 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.377 DEBUG:boto:Method: GET 2024-04-12T03:33:06.377 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:06.377 DEBUG:boto:Data: 2024-04-12T03:33:06.377 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.377 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.378 DEBUG:boto:Params: {} 2024-04-12T03:33:06.378 DEBUG:boto:Token: None 2024-04-12T03:33:06.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e47ca4ce5460ca0c-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.422 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.423 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.423 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.423 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.423 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:06.423 DEBUG:boto:Data: 2024-04-12T03:33:06.423 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.424 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.424 DEBUG:boto:Params: {} 2024-04-12T03:33:06.424 DEBUG:boto:Token: None 2024-04-12T03:33:06.424 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c 2024-04-12T03:33:06.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA= 2024-04-12T03:33:06.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA=', 'Content-Length': '0'} 2024-04-12T03:33:06.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720dd30c8b002458-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.427 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.427 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.427 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.427 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:06.427 DEBUG:boto:Data: 2024-04-12T03:33:06.427 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.427 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.427 DEBUG:boto:Params: {} 2024-04-12T03:33:06.427 DEBUG:boto:Token: None 2024-04-12T03:33:06.428 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c 2024-04-12T03:33:06.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA= 2024-04-12T03:33:06.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA=', 'Content-Length': '0'} 2024-04-12T03:33:06.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49a4790473ab9c6-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:06.430 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.430 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.430 DEBUG:boto:Method: GET 2024-04-12T03:33:06.430 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:06.431 DEBUG:boto:Data: 2024-04-12T03:33:06.431 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.431 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.431 DEBUG:boto:Params: {} 2024-04-12T03:33:06.431 DEBUG:boto:Token: None 2024-04-12T03:33:06.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c 2024-04-12T03:33:06.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80= 2024-04-12T03:33:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80=', 'Content-Length': '0'} 2024-04-12T03:33:06.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049527890658a1d13-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.434 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:06.434 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:06.434 DEBUG:boto:Method: GET 2024-04-12T03:33:06.434 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:06.434 DEBUG:boto:Data: 2024-04-12T03:33:06.434 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.434 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.434 DEBUG:boto:Params: {} 2024-04-12T03:33:06.434 DEBUG:boto:Token: None 2024-04-12T03:33:06.435 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/c 2024-04-12T03:33:06.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80= 2024-04-12T03:33:06.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80=', 'Content-Length': '0'} 2024-04-12T03:33:06.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523c3cc098816c17-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.437 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.437 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.437 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.437 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.437 DEBUG:boto:Method: GET 2024-04-12T03:33:06.437 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:06.437 DEBUG:boto:Data: 2024-04-12T03:33:06.437 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.438 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.438 DEBUG:boto:Params: {} 2024-04-12T03:33:06.438 DEBUG:boto:Token: None 2024-04-12T03:33:06.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.438 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d3bed081da2db4-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.441 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.441 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.441 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.441 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.441 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:06.441 DEBUG:boto:Method: GET 2024-04-12T03:33:06.441 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:06.442 DEBUG:boto:Data: 2024-04-12T03:33:06.442 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.442 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.442 DEBUG:boto:Params: {} 2024-04-12T03:33:06.442 DEBUG:boto:Token: None 2024-04-12T03:33:06.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.442 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004df45d8e14840b9a-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.445 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:06.445 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.445 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.445 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.445 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.446 DEBUG:boto:Method: GET 2024-04-12T03:33:06.446 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.446 DEBUG:boto:Data: 2024-04-12T03:33:06.446 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.446 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.446 DEBUG:boto:Params: {} 2024-04-12T03:33:06.446 DEBUG:boto:Token: None 2024-04-12T03:33:06.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.446 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T03:33:06.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T03:33:06.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf936d7671d93f1-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.449 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.449 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.449 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.449 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.449 DEBUG:boto:Method: GET 2024-04-12T03:33:06.449 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.450 DEBUG:boto:Data: 2024-04-12T03:33:06.450 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.450 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.450 DEBUG:boto:Params: {} 2024-04-12T03:33:06.450 DEBUG:boto:Token: None 2024-04-12T03:33:06.450 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T03:33:06.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T03:33:06.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a561e820ea5cc4-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.453 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.453 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.453 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.453 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.454 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.454 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.454 DEBUG:boto:Data: 2024-04-12T03:33:06.454 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.454 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.454 DEBUG:boto:Params: {} 2024-04-12T03:33:06.454 DEBUG:boto:Token: None 2024-04-12T03:33:06.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE= 2024-04-12T03:33:06.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE=', 'Content-Length': '0'} 2024-04-12T03:33:06.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ebf0133210dc77-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.456 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.457 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.457 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.457 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.457 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.457 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.457 DEBUG:boto:Data: 2024-04-12T03:33:06.458 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.458 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.458 DEBUG:boto:Params: {} 2024-04-12T03:33:06.458 DEBUG:boto:Token: None 2024-04-12T03:33:06.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE= 2024-04-12T03:33:06.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE=', 'Content-Length': '0'} 2024-04-12T03:33:06.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25105dc416eef75-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:06.461 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.461 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.461 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.462 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.462 DEBUG:boto:Method: GET 2024-04-12T03:33:06.462 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.462 DEBUG:boto:Data: 2024-04-12T03:33:06.462 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.462 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.462 DEBUG:boto:Params: {} 2024-04-12T03:33:06.462 DEBUG:boto:Token: None 2024-04-12T03:33:06.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T03:33:06.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T03:33:06.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d84db5204dd337-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.465 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.465 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.465 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.465 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.465 DEBUG:boto:Method: GET 2024-04-12T03:33:06.465 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.465 DEBUG:boto:Data: 2024-04-12T03:33:06.465 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.465 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.465 DEBUG:boto:Params: {} 2024-04-12T03:33:06.466 DEBUG:boto:Token: None 2024-04-12T03:33:06.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:06.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T03:33:06.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T03:33:06.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b6107c591137fb-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.468 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.468 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.468 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.468 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.468 DEBUG:boto:Method: GET 2024-04-12T03:33:06.468 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:06.468 DEBUG:boto:Data: 2024-04-12T03:33:06.468 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.469 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.469 DEBUG:boto:Params: {} 2024-04-12T03:33:06.469 DEBUG:boto:Token: None 2024-04-12T03:33:06.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebd7599d5f43ae2-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.471 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.472 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.472 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.472 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.472 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.472 DEBUG:boto:Method: GET 2024-04-12T03:33:06.472 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:06.472 DEBUG:boto:Data: 2024-04-12T03:33:06.472 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.472 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.473 DEBUG:boto:Params: {} 2024-04-12T03:33:06.473 DEBUG:boto:Token: None 2024-04-12T03:33:06.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a707ed4156c783b-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.475 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.475 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.475 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.475 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.475 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:06.476 DEBUG:boto:Data: 2024-04-12T03:33:06.476 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.476 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.476 DEBUG:boto:Params: {} 2024-04-12T03:33:06.476 DEBUG:boto:Token: None 2024-04-12T03:33:06.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d 2024-04-12T03:33:06.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I= 2024-04-12T03:33:06.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I=', 'Content-Length': '0'} 2024-04-12T03:33:06.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517ceacc7a43decb-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.478 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.478 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.478 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.478 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:06.479 DEBUG:boto:Data: 2024-04-12T03:33:06.479 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.479 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.479 DEBUG:boto:Params: {} 2024-04-12T03:33:06.479 DEBUG:boto:Token: None 2024-04-12T03:33:06.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d 2024-04-12T03:33:06.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I= 2024-04-12T03:33:06.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I=', 'Content-Length': '0'} 2024-04-12T03:33:06.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcffdcd00ff9620-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:06.482 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.482 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.482 DEBUG:boto:Method: GET 2024-04-12T03:33:06.482 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:06.482 DEBUG:boto:Data: 2024-04-12T03:33:06.482 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.482 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.482 DEBUG:boto:Params: {} 2024-04-12T03:33:06.482 DEBUG:boto:Token: None 2024-04-12T03:33:06.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d 2024-04-12T03:33:06.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI= 2024-04-12T03:33:06.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI=', 'Content-Length': '0'} 2024-04-12T03:33:06.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec84ec7a637e182-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.485 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:06.485 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:06.485 DEBUG:boto:Method: GET 2024-04-12T03:33:06.485 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:06.485 DEBUG:boto:Data: 2024-04-12T03:33:06.485 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.485 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.486 DEBUG:boto:Params: {} 2024-04-12T03:33:06.486 DEBUG:boto:Token: None 2024-04-12T03:33:06.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/d 2024-04-12T03:33:06.486 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI= 2024-04-12T03:33:06.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI=', 'Content-Length': '0'} 2024-04-12T03:33:06.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e47ecd5eefe5558-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.488 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.489 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.489 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.489 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.489 DEBUG:boto:Method: GET 2024-04-12T03:33:06.489 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:06.489 DEBUG:boto:Data: 2024-04-12T03:33:06.489 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.489 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.489 DEBUG:boto:Params: {} 2024-04-12T03:33:06.489 DEBUG:boto:Token: None 2024-04-12T03:33:06.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9262e01eeb9288-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.493 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.493 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.493 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.493 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.493 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:06.493 DEBUG:boto:Method: GET 2024-04-12T03:33:06.493 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:06.493 DEBUG:boto:Data: 2024-04-12T03:33:06.494 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.494 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.494 DEBUG:boto:Params: {} 2024-04-12T03:33:06.494 DEBUG:boto:Token: None 2024-04-12T03:33:06.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.494 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb6d7695316018b-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.496 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:06.497 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.497 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.497 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.497 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.497 DEBUG:boto:Method: GET 2024-04-12T03:33:06.497 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.498 DEBUG:boto:Data: 2024-04-12T03:33:06.498 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.499 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.499 DEBUG:boto:Params: {} 2024-04-12T03:33:06.499 DEBUG:boto:Token: None 2024-04-12T03:33:06.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T03:33:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T03:33:06.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebfea66986cd7f7-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.502 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.502 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.502 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.502 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.503 DEBUG:boto:Method: GET 2024-04-12T03:33:06.503 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.503 DEBUG:boto:Data: 2024-04-12T03:33:06.503 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.503 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.503 DEBUG:boto:Params: {} 2024-04-12T03:33:06.503 DEBUG:boto:Token: None 2024-04-12T03:33:06.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T03:33:06.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T03:33:06.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ffdd4f3e36b648-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.506 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.506 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.506 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.506 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.506 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.506 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.507 DEBUG:boto:Data: 2024-04-12T03:33:06.507 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.507 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.507 DEBUG:boto:Params: {} 2024-04-12T03:33:06.507 DEBUG:boto:Token: None 2024-04-12T03:33:06.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0= 2024-04-12T03:33:06.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0=', 'Content-Length': '0'} 2024-04-12T03:33:06.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62ba5b97e6e9c2b-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.509 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.509 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.510 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.510 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.510 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.510 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.510 DEBUG:boto:Data: 2024-04-12T03:33:06.510 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.510 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.510 DEBUG:boto:Params: {} 2024-04-12T03:33:06.510 DEBUG:boto:Token: None 2024-04-12T03:33:06.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0= 2024-04-12T03:33:06.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0=', 'Content-Length': '0'} 2024-04-12T03:33:06.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f14f26694199b3-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:06.513 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.513 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.513 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.513 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.513 DEBUG:boto:Method: GET 2024-04-12T03:33:06.513 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.514 DEBUG:boto:Data: 2024-04-12T03:33:06.514 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.514 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.514 DEBUG:boto:Params: {} 2024-04-12T03:33:06.514 DEBUG:boto:Token: None 2024-04-12T03:33:06.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T03:33:06.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T03:33:06.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf4c064d336be4d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.517 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.517 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.517 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.518 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.518 DEBUG:boto:Method: GET 2024-04-12T03:33:06.518 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.518 DEBUG:boto:Data: 2024-04-12T03:33:06.518 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.518 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.518 DEBUG:boto:Params: {} 2024-04-12T03:33:06.518 DEBUG:boto:Token: None 2024-04-12T03:33:06.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:06.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T03:33:06.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T03:33:06.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014377a3dbb97664f-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.521 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.521 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.521 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.521 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.521 DEBUG:boto:Method: GET 2024-04-12T03:33:06.521 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:06.521 DEBUG:boto:Data: 2024-04-12T03:33:06.522 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.522 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.522 DEBUG:boto:Params: {} 2024-04-12T03:33:06.522 DEBUG:boto:Token: None 2024-04-12T03:33:06.522 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d804cda0e6b1b1b-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.525 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.525 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.525 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.525 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.525 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.525 DEBUG:boto:Method: GET 2024-04-12T03:33:06.525 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:06.526 DEBUG:boto:Data: 2024-04-12T03:33:06.526 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.526 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.526 DEBUG:boto:Params: {} 2024-04-12T03:33:06.526 DEBUG:boto:Token: None 2024-04-12T03:33:06.526 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e476e1b9dfe2e899-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.528 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.529 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.529 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.529 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.529 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:06.529 DEBUG:boto:Data: 2024-04-12T03:33:06.529 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.529 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.529 DEBUG:boto:Params: {} 2024-04-12T03:33:06.529 DEBUG:boto:Token: None 2024-04-12T03:33:06.530 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e 2024-04-12T03:33:06.530 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4= 2024-04-12T03:33:06.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4=', 'Content-Length': '0'} 2024-04-12T03:33:06.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ad2f0f0328050f-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.532 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.532 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.532 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.532 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:06.532 DEBUG:boto:Data: 2024-04-12T03:33:06.532 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.532 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.533 DEBUG:boto:Params: {} 2024-04-12T03:33:06.533 DEBUG:boto:Token: None 2024-04-12T03:33:06.533 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e 2024-04-12T03:33:06.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4= 2024-04-12T03:33:06.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4=', 'Content-Length': '0'} 2024-04-12T03:33:06.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008795f2ebeca952d6-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:06.535 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.535 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.535 DEBUG:boto:Method: GET 2024-04-12T03:33:06.535 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:06.535 DEBUG:boto:Data: 2024-04-12T03:33:06.535 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.536 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.536 DEBUG:boto:Params: {} 2024-04-12T03:33:06.536 DEBUG:boto:Token: None 2024-04-12T03:33:06.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e 2024-04-12T03:33:06.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U= 2024-04-12T03:33:06.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U=', 'Content-Length': '0'} 2024-04-12T03:33:06.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afeca178710a17f9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.538 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:06.538 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:06.539 DEBUG:boto:Method: GET 2024-04-12T03:33:06.539 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:06.539 DEBUG:boto:Data: 2024-04-12T03:33:06.539 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.539 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.539 DEBUG:boto:Params: {} 2024-04-12T03:33:06.539 DEBUG:boto:Token: None 2024-04-12T03:33:06.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/e 2024-04-12T03:33:06.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U= 2024-04-12T03:33:06.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U=', 'Content-Length': '0'} 2024-04-12T03:33:06.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e454475320dfd355-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.586 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.587 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.587 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.587 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.587 DEBUG:boto:Method: GET 2024-04-12T03:33:06.587 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:06.587 DEBUG:boto:Data: 2024-04-12T03:33:06.587 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.587 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.587 DEBUG:boto:Params: {} 2024-04-12T03:33:06.588 DEBUG:boto:Token: None 2024-04-12T03:33:06.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001273ed892810192d-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.591 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.591 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.591 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.591 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.592 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:06.592 DEBUG:boto:Method: GET 2024-04-12T03:33:06.592 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:06.592 DEBUG:boto:Data: 2024-04-12T03:33:06.592 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.592 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.592 DEBUG:boto:Params: {} 2024-04-12T03:33:06.592 DEBUG:boto:Token: None 2024-04-12T03:33:06.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e09991f7827144-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.594 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:06.595 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.595 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.595 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.595 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.595 DEBUG:boto:Method: GET 2024-04-12T03:33:06.595 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.595 DEBUG:boto:Data: 2024-04-12T03:33:06.596 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.596 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.596 DEBUG:boto:Params: {} 2024-04-12T03:33:06.596 DEBUG:boto:Token: None 2024-04-12T03:33:06.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T03:33:06.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T03:33:06.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db8d5905cb6d3f6d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.638 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.639 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.639 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.639 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.640 DEBUG:boto:Method: GET 2024-04-12T03:33:06.640 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.640 DEBUG:boto:Data: 2024-04-12T03:33:06.640 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.640 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.640 DEBUG:boto:Params: {} 2024-04-12T03:33:06.640 DEBUG:boto:Token: None 2024-04-12T03:33:06.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.641 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T03:33:06.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T03:33:06.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180aa4496525740e-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.643 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.643 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.643 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.643 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.643 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.644 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.644 DEBUG:boto:Data: 2024-04-12T03:33:06.644 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.644 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.644 DEBUG:boto:Params: {} 2024-04-12T03:33:06.644 DEBUG:boto:Token: None 2024-04-12T03:33:06.644 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA= 2024-04-12T03:33:06.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA=', 'Content-Length': '0'} 2024-04-12T03:33:06.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce75ca45f49a668-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.647 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.647 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.647 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.647 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.647 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.648 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.648 DEBUG:boto:Data: 2024-04-12T03:33:06.648 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.648 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.648 DEBUG:boto:Params: {} 2024-04-12T03:33:06.648 DEBUG:boto:Token: None 2024-04-12T03:33:06.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA= 2024-04-12T03:33:06.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA=', 'Content-Length': '0'} 2024-04-12T03:33:06.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b0d8086bac2e8d-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:06.650 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.650 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.650 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.650 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.651 DEBUG:boto:Method: GET 2024-04-12T03:33:06.651 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.651 DEBUG:boto:Data: 2024-04-12T03:33:06.651 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.651 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.651 DEBUG:boto:Params: {} 2024-04-12T03:33:06.651 DEBUG:boto:Token: None 2024-04-12T03:33:06.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T03:33:06.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T03:33:06.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ff32f29ada8357-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.653 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.654 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.654 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.654 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.654 DEBUG:boto:Method: GET 2024-04-12T03:33:06.654 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.654 DEBUG:boto:Data: 2024-04-12T03:33:06.654 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.654 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.654 DEBUG:boto:Params: {} 2024-04-12T03:33:06.655 DEBUG:boto:Token: None 2024-04-12T03:33:06.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:06.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T03:33:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T03:33:06.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468b052187d0ef23-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.657 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.657 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.657 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.657 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.657 DEBUG:boto:Method: GET 2024-04-12T03:33:06.657 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:06.657 DEBUG:boto:Data: 2024-04-12T03:33:06.657 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.657 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.658 DEBUG:boto:Params: {} 2024-04-12T03:33:06.658 DEBUG:boto:Token: None 2024-04-12T03:33:06.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db436830dfed5b2-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.660 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.661 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.661 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.661 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.661 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.661 DEBUG:boto:Method: GET 2024-04-12T03:33:06.661 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:06.661 DEBUG:boto:Data: 2024-04-12T03:33:06.661 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.661 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.661 DEBUG:boto:Params: {} 2024-04-12T03:33:06.662 DEBUG:boto:Token: None 2024-04-12T03:33:06.662 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.662 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a6cf47204983e4-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.664 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.664 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.665 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.665 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.665 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:06.665 DEBUG:boto:Data: 2024-04-12T03:33:06.665 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.665 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.665 DEBUG:boto:Params: {} 2024-04-12T03:33:06.665 DEBUG:boto:Token: None 2024-04-12T03:33:06.665 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f 2024-04-12T03:33:06.666 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM= 2024-04-12T03:33:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM=', 'Content-Length': '0'} 2024-04-12T03:33:06.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012102413fb20fdf5-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.668 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.668 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.668 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.668 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:06.668 DEBUG:boto:Data: 2024-04-12T03:33:06.668 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.668 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.668 DEBUG:boto:Params: {} 2024-04-12T03:33:06.669 DEBUG:boto:Token: None 2024-04-12T03:33:06.669 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f 2024-04-12T03:33:06.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM= 2024-04-12T03:33:06.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM=', 'Content-Length': '0'} 2024-04-12T03:33:06.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008768e9e2540497c3-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:06.671 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.671 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.671 DEBUG:boto:Method: GET 2024-04-12T03:33:06.671 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:06.671 DEBUG:boto:Data: 2024-04-12T03:33:06.671 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.671 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.671 DEBUG:boto:Params: {} 2024-04-12T03:33:06.672 DEBUG:boto:Token: None 2024-04-12T03:33:06.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f 2024-04-12T03:33:06.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY= 2024-04-12T03:33:06.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY=', 'Content-Length': '0'} 2024-04-12T03:33:06.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002651764143101c63-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.674 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:06.674 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:06.674 DEBUG:boto:Method: GET 2024-04-12T03:33:06.674 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:06.674 DEBUG:boto:Data: 2024-04-12T03:33:06.674 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.675 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.675 DEBUG:boto:Params: {} 2024-04-12T03:33:06.675 DEBUG:boto:Token: None 2024-04-12T03:33:06.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/f 2024-04-12T03:33:06.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY= 2024-04-12T03:33:06.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY=', 'Content-Length': '0'} 2024-04-12T03:33:06.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026bde55a5f42050c-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.718 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.719 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.719 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.719 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.719 DEBUG:boto:Method: GET 2024-04-12T03:33:06.719 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:06.719 DEBUG:boto:Data: 2024-04-12T03:33:06.719 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.719 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.719 DEBUG:boto:Params: {} 2024-04-12T03:33:06.720 DEBUG:boto:Token: None 2024-04-12T03:33:06.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000558319a6ce073d95-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.722 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.723 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.723 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.723 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.723 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:06.723 DEBUG:boto:Method: GET 2024-04-12T03:33:06.723 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:06.723 DEBUG:boto:Data: 2024-04-12T03:33:06.723 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.723 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.724 DEBUG:boto:Params: {} 2024-04-12T03:33:06.724 DEBUG:boto:Token: None 2024-04-12T03:33:06.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b89e746e7388a32-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.726 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:06.727 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.727 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.727 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.727 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.727 DEBUG:boto:Method: GET 2024-04-12T03:33:06.727 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.727 DEBUG:boto:Data: 2024-04-12T03:33:06.727 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.728 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.728 DEBUG:boto:Params: {} 2024-04-12T03:33:06.728 DEBUG:boto:Token: None 2024-04-12T03:33:06.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T03:33:06.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T03:33:06.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55773f4eba14f24-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.730 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.730 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.730 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.730 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.731 DEBUG:boto:Method: GET 2024-04-12T03:33:06.731 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.731 DEBUG:boto:Data: 2024-04-12T03:33:06.731 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.731 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.731 DEBUG:boto:Params: {} 2024-04-12T03:33:06.731 DEBUG:boto:Token: None 2024-04-12T03:33:06.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T03:33:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T03:33:06.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dc2b1c23b56e82-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.734 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.734 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.734 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.734 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.734 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.734 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.734 DEBUG:boto:Data: 2024-04-12T03:33:06.734 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.735 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.735 DEBUG:boto:Params: {} 2024-04-12T03:33:06.735 DEBUG:boto:Token: None 2024-04-12T03:33:06.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc= 2024-04-12T03:33:06.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc=', 'Content-Length': '0'} 2024-04-12T03:33:06.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2eb1beb677f0f8-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.737 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.737 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.737 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.737 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.737 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.737 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.737 DEBUG:boto:Data: 2024-04-12T03:33:06.737 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.738 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.738 DEBUG:boto:Params: {} 2024-04-12T03:33:06.738 DEBUG:boto:Token: None 2024-04-12T03:33:06.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc= 2024-04-12T03:33:06.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc=', 'Content-Length': '0'} 2024-04-12T03:33:06.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb206a582dda29e-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:06.740 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.740 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.741 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.741 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.741 DEBUG:boto:Method: GET 2024-04-12T03:33:06.741 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.741 DEBUG:boto:Data: 2024-04-12T03:33:06.741 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.741 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.741 DEBUG:boto:Params: {} 2024-04-12T03:33:06.741 DEBUG:boto:Token: None 2024-04-12T03:33:06.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T03:33:06.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T03:33:06.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca89cccf0fa68113-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.744 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.744 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.744 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.744 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.744 DEBUG:boto:Method: GET 2024-04-12T03:33:06.744 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.744 DEBUG:boto:Data: 2024-04-12T03:33:06.744 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.744 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.744 DEBUG:boto:Params: {} 2024-04-12T03:33:06.745 DEBUG:boto:Token: None 2024-04-12T03:33:06.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:06.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T03:33:06.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T03:33:06.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c7877173fe9b5b-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.747 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.747 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.747 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.747 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.747 DEBUG:boto:Method: GET 2024-04-12T03:33:06.747 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:06.747 DEBUG:boto:Data: 2024-04-12T03:33:06.747 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.748 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.748 DEBUG:boto:Params: {} 2024-04-12T03:33:06.748 DEBUG:boto:Token: None 2024-04-12T03:33:06.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e7d9e4a585a216-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.751 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.751 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.751 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.751 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.751 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.751 DEBUG:boto:Method: GET 2024-04-12T03:33:06.752 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:06.752 DEBUG:boto:Data: 2024-04-12T03:33:06.752 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.752 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.752 DEBUG:boto:Params: {} 2024-04-12T03:33:06.752 DEBUG:boto:Token: None 2024-04-12T03:33:06.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b5ffa9de6fd98fd-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.754 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.755 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.755 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.755 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.755 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:06.755 DEBUG:boto:Data: 2024-04-12T03:33:06.755 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.755 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.755 DEBUG:boto:Params: {} 2024-04-12T03:33:06.755 DEBUG:boto:Token: None 2024-04-12T03:33:06.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g 2024-04-12T03:33:06.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8= 2024-04-12T03:33:06.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8=', 'Content-Length': '0'} 2024-04-12T03:33:06.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d0815d9e793cbf-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.758 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.758 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.758 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.758 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:06.758 DEBUG:boto:Data: 2024-04-12T03:33:06.758 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.758 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.758 DEBUG:boto:Params: {} 2024-04-12T03:33:06.759 DEBUG:boto:Token: None 2024-04-12T03:33:06.759 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g 2024-04-12T03:33:06.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8= 2024-04-12T03:33:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8=', 'Content-Length': '0'} 2024-04-12T03:33:06.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ca4f424b0b5ba0-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:06.761 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.761 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.761 DEBUG:boto:Method: GET 2024-04-12T03:33:06.761 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:06.761 DEBUG:boto:Data: 2024-04-12T03:33:06.761 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.761 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.761 DEBUG:boto:Params: {} 2024-04-12T03:33:06.762 DEBUG:boto:Token: None 2024-04-12T03:33:06.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g 2024-04-12T03:33:06.762 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE= 2024-04-12T03:33:06.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE=', 'Content-Length': '0'} 2024-04-12T03:33:06.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041bca1d107560370-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.806 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:06.807 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:06.807 DEBUG:boto:Method: GET 2024-04-12T03:33:06.807 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:06.807 DEBUG:boto:Data: 2024-04-12T03:33:06.807 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.807 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.807 DEBUG:boto:Params: {} 2024-04-12T03:33:06.807 DEBUG:boto:Token: None 2024-04-12T03:33:06.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/g 2024-04-12T03:33:06.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE= 2024-04-12T03:33:06.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE=', 'Content-Length': '0'} 2024-04-12T03:33:06.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4b04209acee3e5-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.810 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.810 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.810 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.810 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.810 DEBUG:boto:Method: GET 2024-04-12T03:33:06.810 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:06.810 DEBUG:boto:Data: 2024-04-12T03:33:06.811 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.811 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.811 DEBUG:boto:Params: {} 2024-04-12T03:33:06.811 DEBUG:boto:Token: None 2024-04-12T03:33:06.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f7efe983702f34-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.814 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.814 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.815 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.815 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.815 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:06.815 DEBUG:boto:Method: GET 2024-04-12T03:33:06.815 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:06.815 DEBUG:boto:Data: 2024-04-12T03:33:06.815 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.815 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.815 DEBUG:boto:Params: {} 2024-04-12T03:33:06.815 DEBUG:boto:Token: None 2024-04-12T03:33:06.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e01c6a6d530c52-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.818 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:06.818 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.818 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.818 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.819 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.819 DEBUG:boto:Method: GET 2024-04-12T03:33:06.819 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.819 DEBUG:boto:Data: 2024-04-12T03:33:06.819 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.819 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.819 DEBUG:boto:Params: {} 2024-04-12T03:33:06.819 DEBUG:boto:Token: None 2024-04-12T03:33:06.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T03:33:06.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T03:33:06.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7042769e7730ce-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.822 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.822 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.822 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.822 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.822 DEBUG:boto:Method: GET 2024-04-12T03:33:06.823 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.823 DEBUG:boto:Data: 2024-04-12T03:33:06.823 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.823 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.823 DEBUG:boto:Params: {} 2024-04-12T03:33:06.823 DEBUG:boto:Token: None 2024-04-12T03:33:06.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T03:33:06.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T03:33:06.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008505cfcd6d068de4-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.825 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.825 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.825 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.826 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.826 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.826 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.826 DEBUG:boto:Data: 2024-04-12T03:33:06.826 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.826 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.826 DEBUG:boto:Params: {} 2024-04-12T03:33:06.826 DEBUG:boto:Token: None 2024-04-12T03:33:06.826 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A= 2024-04-12T03:33:06.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A=', 'Content-Length': '0'} 2024-04-12T03:33:06.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098be36d9fd0fd13b-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.828 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.829 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.829 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.829 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.829 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.829 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.829 DEBUG:boto:Data: 2024-04-12T03:33:06.829 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.829 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.829 DEBUG:boto:Params: {} 2024-04-12T03:33:06.829 DEBUG:boto:Token: None 2024-04-12T03:33:06.830 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A= 2024-04-12T03:33:06.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A=', 'Content-Length': '0'} 2024-04-12T03:33:06.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a220ed7fb70f98d8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:06.832 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.832 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.832 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.832 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.832 DEBUG:boto:Method: GET 2024-04-12T03:33:06.832 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.832 DEBUG:boto:Data: 2024-04-12T03:33:06.833 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.833 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.833 DEBUG:boto:Params: {} 2024-04-12T03:33:06.833 DEBUG:boto:Token: None 2024-04-12T03:33:06.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.833 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T03:33:06.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T03:33:06.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63ad86849c2cfd1-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.838 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.838 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.838 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.838 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.838 DEBUG:boto:Method: GET 2024-04-12T03:33:06.838 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.838 DEBUG:boto:Data: 2024-04-12T03:33:06.838 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.839 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.839 DEBUG:boto:Params: {} 2024-04-12T03:33:06.839 DEBUG:boto:Token: None 2024-04-12T03:33:06.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:06.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T03:33:06.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T03:33:06.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d764510098761911-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.841 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.841 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.841 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.841 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.842 DEBUG:boto:Method: GET 2024-04-12T03:33:06.842 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:06.842 DEBUG:boto:Data: 2024-04-12T03:33:06.842 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.842 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.842 DEBUG:boto:Params: {} 2024-04-12T03:33:06.842 DEBUG:boto:Token: None 2024-04-12T03:33:06.842 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.842 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c02d2f00ac227c1-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.845 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.845 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.845 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.845 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.845 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.846 DEBUG:boto:Method: GET 2024-04-12T03:33:06.846 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:06.846 DEBUG:boto:Data: 2024-04-12T03:33:06.846 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.846 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.846 DEBUG:boto:Params: {} 2024-04-12T03:33:06.846 DEBUG:boto:Token: None 2024-04-12T03:33:06.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004459fc344e12cfbc-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.848 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.849 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.849 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.849 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.849 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:06.849 DEBUG:boto:Data: 2024-04-12T03:33:06.849 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.849 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.849 DEBUG:boto:Params: {} 2024-04-12T03:33:06.849 DEBUG:boto:Token: None 2024-04-12T03:33:06.850 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h 2024-04-12T03:33:06.850 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM= 2024-04-12T03:33:06.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM=', 'Content-Length': '0'} 2024-04-12T03:33:06.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3d6b7703bf51f6-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.851 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.852 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.852 DEBUG:boto:Method: HEAD 2024-04-12T03:33:06.852 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:06.852 DEBUG:boto:Data: 2024-04-12T03:33:06.852 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.852 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.852 DEBUG:boto:Params: {} 2024-04-12T03:33:06.852 DEBUG:boto:Token: None 2024-04-12T03:33:06.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h 2024-04-12T03:33:06.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM= 2024-04-12T03:33:06.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM=', 'Content-Length': '0'} 2024-04-12T03:33:06.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab8a91ab7ec1d0d-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:06.854 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.854 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.855 DEBUG:boto:Method: GET 2024-04-12T03:33:06.855 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:06.855 DEBUG:boto:Data: 2024-04-12T03:33:06.855 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.855 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.855 DEBUG:boto:Params: {} 2024-04-12T03:33:06.855 DEBUG:boto:Token: None 2024-04-12T03:33:06.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h 2024-04-12T03:33:06.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI= 2024-04-12T03:33:06.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI=', 'Content-Length': '0'} 2024-04-12T03:33:06.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979ca4d87ba4d723-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.857 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:06.857 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:06.858 DEBUG:boto:Method: GET 2024-04-12T03:33:06.858 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:06.858 DEBUG:boto:Data: 2024-04-12T03:33:06.858 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.858 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.858 DEBUG:boto:Params: {} 2024-04-12T03:33:06.858 DEBUG:boto:Token: None 2024-04-12T03:33:06.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/h 2024-04-12T03:33:06.858 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI= 2024-04-12T03:33:06.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI=', 'Content-Length': '0'} 2024-04-12T03:33:06.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45543ac6249ed97-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.860 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.861 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.861 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.861 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.861 DEBUG:boto:Method: GET 2024-04-12T03:33:06.861 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:06.861 DEBUG:boto:Data: 2024-04-12T03:33:06.861 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:06.861 DEBUG:boto:Port: 8001 2024-04-12T03:33:06.861 DEBUG:boto:Params: {} 2024-04-12T03:33:06.861 DEBUG:boto:Token: None 2024-04-12T03:33:06.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db6fde09ad09ab4-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.864 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.865 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:06.865 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:06.865 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.865 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:06.865 DEBUG:boto:Method: GET 2024-04-12T03:33:06.865 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:06.865 DEBUG:boto:Data: 2024-04-12T03:33:06.865 DEBUG:boto:Headers: {} 2024-04-12T03:33:06.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:06.865 DEBUG:boto:Port: 8000 2024-04-12T03:33:06.865 DEBUG:boto:Params: {} 2024-04-12T03:33:06.866 DEBUG:boto:Token: None 2024-04-12T03:33:06.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:06 GMT /dgbpxg-57/ 2024-04-12T03:33:06.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T03:33:06.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T03:33:06.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f069bbef19bf68d2-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:06.868 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:06.868 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:33:06.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:06.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:06.869 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:06.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:06.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:06.955 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:33:06.955 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-57 2024-04-12T03:33:06.956 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 8 2024-04-12T03:33:06.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:06.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:06.985 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:07.030 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:07.030 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:07.072 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:33:07.072 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.072 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2024-04-12T03:33:07.072 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2024-04-12T03:33:07.072 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:07.073 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2024-04-12T03:33:07.074 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.075 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:07.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:07.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:07.078 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:33:07.079 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T03:33:07.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:07.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:07.080 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:07.167+0000 7f6ab427b780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:33:12.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:12.167+0000 7f6ab427b780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:33:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:17.168+0000 7f6ab427b780 1 bucket sync caught up with empty source 2024-04-12T03:33:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:17.168+0000 7f6ab427b780 0 bucket checkpoint complete 2024-04-12T03:33:17.188 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T03:33:17.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:17.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:17.189 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:17.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:17.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:17.283+0000 7fec211b6780 1 bucket sync caught up with source: 2024-04-12T03:33:17.285 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T03:33:17.285 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T03:33:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:17.283+0000 7fec211b6780 0 bucket checkpoint complete 2024-04-12T03:33:17.298 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:33:17.298 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.299 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.299 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.299 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:33:17.299 DEBUG:boto:Data: 2024-04-12T03:33:17.299 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.299 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.299 DEBUG:boto:Params: {} 2024-04-12T03:33:17.299 DEBUG:boto:Token: None 2024-04-12T03:33:17.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU= 2024-04-12T03:33:17.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU=', 'Content-Length': '0'} 2024-04-12T03:33:17.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d52ea03efb7dfa5e-006618ab7d-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.302 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.302 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.302 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.303 DEBUG:boto:Path: /dgbpxg-57/ 2024-04-12T03:33:17.303 DEBUG:boto:Data: 2024-04-12T03:33:17.303 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.303 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.303 DEBUG:boto:Params: {} 2024-04-12T03:33:17.303 DEBUG:boto:Token: None 2024-04-12T03:33:17.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU= 2024-04-12T03:33:17.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU=', 'Content-Length': '0'} 2024-04-12T03:33:17.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037b07e1c698dc628-006618ab7d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.305 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:33:17.305 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.305 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.305 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.305 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.306 DEBUG:boto:Method: GET 2024-04-12T03:33:17.306 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.306 DEBUG:boto:Data: 2024-04-12T03:33:17.306 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.306 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.306 DEBUG:boto:Params: {} 2024-04-12T03:33:17.306 DEBUG:boto:Token: None 2024-04-12T03:33:17.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/?versions 2024-04-12T03:33:17.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T03:33:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T03:33:17.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c361faa91d9a48c0-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.308 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.309 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:33:17.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:33:17.310 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.311 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.311 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.311 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.311 DEBUG:boto:Method: GET 2024-04-12T03:33:17.311 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.311 DEBUG:boto:Data: 2024-04-12T03:33:17.311 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.311 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.311 DEBUG:boto:Params: {} 2024-04-12T03:33:17.311 DEBUG:boto:Token: None 2024-04-12T03:33:17.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/?versions 2024-04-12T03:33:17.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T03:33:17.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T03:33:17.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000303592e159a68e6e-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.314 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.314 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:33:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:33:17.315 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.315 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.315 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.316 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.316 DEBUG:boto:Method: GET 2024-04-12T03:33:17.316 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.316 DEBUG:boto:Data: 2024-04-12T03:33:17.316 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.316 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.316 DEBUG:boto:Params: {} 2024-04-12T03:33:17.316 DEBUG:boto:Token: None 2024-04-12T03:33:17.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/?versions 2024-04-12T03:33:17.317 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T03:33:17.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T03:33:17.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000986b2d2ee71c8478-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.319 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.320 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.320 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.320 DEBUG:boto:path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.320 DEBUG:boto:auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.320 DEBUG:boto:Method: GET 2024-04-12T03:33:17.320 DEBUG:boto:Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T03:33:17.320 DEBUG:boto:Data: 2024-04-12T03:33:17.321 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.321 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.321 DEBUG:boto:Params: {} 2024-04-12T03:33:17.321 DEBUG:boto:Token: None 2024-04-12T03:33:17.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/?versions 2024-04-12T03:33:17.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T03:33:17.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T03:33:17.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f843ada06589db4-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.324 DEBUG:boto:b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:17.325 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.325 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.325 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.325 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.325 DEBUG:boto:Method: GET 2024-04-12T03:33:17.325 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.325 DEBUG:boto:Data: 2024-04-12T03:33:17.325 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.325 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.326 DEBUG:boto:Params: {} 2024-04-12T03:33:17.326 DEBUG:boto:Token: None 2024-04-12T03:33:17.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T03:33:17.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T03:33:17.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a147889bb997c211-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.374 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.375 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.375 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.375 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.375 DEBUG:boto:Method: GET 2024-04-12T03:33:17.375 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.375 DEBUG:boto:Data: 2024-04-12T03:33:17.375 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.375 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.375 DEBUG:boto:Params: {} 2024-04-12T03:33:17.375 DEBUG:boto:Token: None 2024-04-12T03:33:17.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T03:33:17.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T03:33:17.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b8c26f92eb10e9-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.378 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.378 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.378 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.378 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.378 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.378 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.379 DEBUG:boto:Data: 2024-04-12T03:33:17.379 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.379 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.379 DEBUG:boto:Params: {} 2024-04-12T03:33:17.379 DEBUG:boto:Token: None 2024-04-12T03:33:17.379 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI= 2024-04-12T03:33:17.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI=', 'Content-Length': '0'} 2024-04-12T03:33:17.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021daa681ac690bf4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.381 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.381 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.381 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.381 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.381 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.382 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.382 DEBUG:boto:Data: 2024-04-12T03:33:17.382 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.382 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.382 DEBUG:boto:Params: {} 2024-04-12T03:33:17.382 DEBUG:boto:Token: None 2024-04-12T03:33:17.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI= 2024-04-12T03:33:17.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI=', 'Content-Length': '0'} 2024-04-12T03:33:17.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3b49b97ddffd0d-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:17.385 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.385 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.385 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.385 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.385 DEBUG:boto:Method: GET 2024-04-12T03:33:17.385 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.385 DEBUG:boto:Data: 2024-04-12T03:33:17.385 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.386 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.386 DEBUG:boto:Params: {} 2024-04-12T03:33:17.386 DEBUG:boto:Token: None 2024-04-12T03:33:17.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T03:33:17.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T03:33:17.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a349ec900924c5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.388 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.388 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.388 DEBUG:boto:path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.388 DEBUG:boto:auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.388 DEBUG:boto:Method: GET 2024-04-12T03:33:17.388 DEBUG:boto:Path: /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.388 DEBUG:boto:Data: 2024-04-12T03:33:17.389 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.389 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.389 DEBUG:boto:Params: {} 2024-04-12T03:33:17.389 DEBUG:boto:Token: None 2024-04-12T03:33:17.389 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a?versionId=null 2024-04-12T03:33:17.389 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T03:33:17.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T03:33:17.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541747d5b87ea2b6-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.391 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.391 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.391 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.391 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.391 DEBUG:boto:Method: GET 2024-04-12T03:33:17.392 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:17.392 DEBUG:boto:Data: 2024-04-12T03:33:17.392 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.392 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.392 DEBUG:boto:Params: {} 2024-04-12T03:33:17.392 DEBUG:boto:Token: None 2024-04-12T03:33:17.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064425f5571f9fe4c-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.395 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.395 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.395 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.395 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.395 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.395 DEBUG:boto:Method: GET 2024-04-12T03:33:17.396 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:17.396 DEBUG:boto:Data: 2024-04-12T03:33:17.396 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.396 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.396 DEBUG:boto:Params: {} 2024-04-12T03:33:17.396 DEBUG:boto:Token: None 2024-04-12T03:33:17.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9b9383b778b251-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.398 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.399 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.399 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.399 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.399 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:17.399 DEBUG:boto:Data: 2024-04-12T03:33:17.399 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.399 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.399 DEBUG:boto:Params: {} 2024-04-12T03:33:17.399 DEBUG:boto:Token: None 2024-04-12T03:33:17.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a 2024-04-12T03:33:17.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4= 2024-04-12T03:33:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4=', 'Content-Length': '0'} 2024-04-12T03:33:17.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009beb913e1d6d630b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.401 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.401 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.401 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.402 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:17.402 DEBUG:boto:Data: 2024-04-12T03:33:17.402 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.402 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.402 DEBUG:boto:Params: {} 2024-04-12T03:33:17.402 DEBUG:boto:Token: None 2024-04-12T03:33:17.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a 2024-04-12T03:33:17.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4= 2024-04-12T03:33:17.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4=', 'Content-Length': '0'} 2024-04-12T03:33:17.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29458cd0855569f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:17.404 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.404 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.404 DEBUG:boto:Method: GET 2024-04-12T03:33:17.404 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:17.405 DEBUG:boto:Data: 2024-04-12T03:33:17.405 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.405 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.405 DEBUG:boto:Params: {} 2024-04-12T03:33:17.405 DEBUG:boto:Token: None 2024-04-12T03:33:17.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a 2024-04-12T03:33:17.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ= 2024-04-12T03:33:17.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c4bba6ace7dc5c-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.407 DEBUG:boto:path=/dgbpxg-57/a 2024-04-12T03:33:17.407 DEBUG:boto:auth_path=/dgbpxg-57/a 2024-04-12T03:33:17.407 DEBUG:boto:Method: GET 2024-04-12T03:33:17.407 DEBUG:boto:Path: /dgbpxg-57/a 2024-04-12T03:33:17.407 DEBUG:boto:Data: 2024-04-12T03:33:17.407 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.408 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.408 DEBUG:boto:Params: {} 2024-04-12T03:33:17.408 DEBUG:boto:Token: None 2024-04-12T03:33:17.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/a 2024-04-12T03:33:17.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ= 2024-04-12T03:33:17.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da883f52e3ee0aa-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.410 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.410 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.410 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.410 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.410 DEBUG:boto:Method: GET 2024-04-12T03:33:17.410 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:17.410 DEBUG:boto:Data: 2024-04-12T03:33:17.411 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.411 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.411 DEBUG:boto:Params: {} 2024-04-12T03:33:17.411 DEBUG:boto:Token: None 2024-04-12T03:33:17.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717c4005e2af5062-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.454 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.455 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.455 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.455 DEBUG:boto:path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.455 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=a 2024-04-12T03:33:17.455 DEBUG:boto:Method: GET 2024-04-12T03:33:17.455 DEBUG:boto:Path: /dgbpxg-57/?prefix=a 2024-04-12T03:33:17.455 DEBUG:boto:Data: 2024-04-12T03:33:17.455 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.456 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.456 DEBUG:boto:Params: {} 2024-04-12T03:33:17.456 DEBUG:boto:Token: None 2024-04-12T03:33:17.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92e1a281eb91458-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.458 DEBUG:boto:b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:17.459 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.459 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.459 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.459 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.459 DEBUG:boto:Method: GET 2024-04-12T03:33:17.459 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.459 DEBUG:boto:Data: 2024-04-12T03:33:17.459 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.459 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.459 DEBUG:boto:Params: {} 2024-04-12T03:33:17.460 DEBUG:boto:Token: None 2024-04-12T03:33:17.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T03:33:17.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T03:33:17.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002347a427dbba2a2-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.462 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.462 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.462 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.463 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.463 DEBUG:boto:Method: GET 2024-04-12T03:33:17.463 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.463 DEBUG:boto:Data: 2024-04-12T03:33:17.463 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.463 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.463 DEBUG:boto:Params: {} 2024-04-12T03:33:17.463 DEBUG:boto:Token: None 2024-04-12T03:33:17.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T03:33:17.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T03:33:17.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91d32f47386a33e-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.466 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.466 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.466 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.466 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.466 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.466 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.467 DEBUG:boto:Data: 2024-04-12T03:33:17.467 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.467 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.467 DEBUG:boto:Params: {} 2024-04-12T03:33:17.467 DEBUG:boto:Token: None 2024-04-12T03:33:17.467 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk= 2024-04-12T03:33:17.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk=', 'Content-Length': '0'} 2024-04-12T03:33:17.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4395db9538e7eb5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.469 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.469 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.469 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.469 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.469 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.469 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.469 DEBUG:boto:Data: 2024-04-12T03:33:17.470 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.470 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.470 DEBUG:boto:Params: {} 2024-04-12T03:33:17.470 DEBUG:boto:Token: None 2024-04-12T03:33:17.470 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk= 2024-04-12T03:33:17.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk=', 'Content-Length': '0'} 2024-04-12T03:33:17.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b76706c6bc45245-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:17.472 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.472 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.472 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.472 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.472 DEBUG:boto:Method: GET 2024-04-12T03:33:17.473 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.473 DEBUG:boto:Data: 2024-04-12T03:33:17.473 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.473 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.473 DEBUG:boto:Params: {} 2024-04-12T03:33:17.473 DEBUG:boto:Token: None 2024-04-12T03:33:17.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T03:33:17.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T03:33:17.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af81f775bd3569ad-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.475 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.475 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.475 DEBUG:boto:path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.475 DEBUG:boto:auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.475 DEBUG:boto:Method: GET 2024-04-12T03:33:17.476 DEBUG:boto:Path: /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.476 DEBUG:boto:Data: 2024-04-12T03:33:17.476 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.476 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.476 DEBUG:boto:Params: {} 2024-04-12T03:33:17.476 DEBUG:boto:Token: None 2024-04-12T03:33:17.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b?versionId=null 2024-04-12T03:33:17.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T03:33:17.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T03:33:17.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d00313abb402dec-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.478 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.478 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.479 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.479 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.479 DEBUG:boto:Method: GET 2024-04-12T03:33:17.479 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:17.479 DEBUG:boto:Data: 2024-04-12T03:33:17.479 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.479 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.479 DEBUG:boto:Params: {} 2024-04-12T03:33:17.479 DEBUG:boto:Token: None 2024-04-12T03:33:17.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b970153bf55c2d3-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.482 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.482 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.482 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.483 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.483 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.483 DEBUG:boto:Method: GET 2024-04-12T03:33:17.483 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:17.483 DEBUG:boto:Data: 2024-04-12T03:33:17.483 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.483 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.483 DEBUG:boto:Params: {} 2024-04-12T03:33:17.483 DEBUG:boto:Token: None 2024-04-12T03:33:17.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a01d3572bacdbc05-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.486 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.486 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.486 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.486 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.486 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:17.486 DEBUG:boto:Data: 2024-04-12T03:33:17.486 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.487 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.487 DEBUG:boto:Params: {} 2024-04-12T03:33:17.487 DEBUG:boto:Token: None 2024-04-12T03:33:17.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b 2024-04-12T03:33:17.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY= 2024-04-12T03:33:17.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY=', 'Content-Length': '0'} 2024-04-12T03:33:17.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea48cfd0fb6dadc-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.489 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.489 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.489 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.489 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:17.489 DEBUG:boto:Data: 2024-04-12T03:33:17.489 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.489 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.489 DEBUG:boto:Params: {} 2024-04-12T03:33:17.490 DEBUG:boto:Token: None 2024-04-12T03:33:17.490 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b 2024-04-12T03:33:17.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY= 2024-04-12T03:33:17.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY=', 'Content-Length': '0'} 2024-04-12T03:33:17.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194e86f1ffc1338f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:17.492 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.492 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.492 DEBUG:boto:Method: GET 2024-04-12T03:33:17.492 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:17.492 DEBUG:boto:Data: 2024-04-12T03:33:17.492 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.492 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.492 DEBUG:boto:Params: {} 2024-04-12T03:33:17.492 DEBUG:boto:Token: None 2024-04-12T03:33:17.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b 2024-04-12T03:33:17.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ= 2024-04-12T03:33:17.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3f2fb149af113d-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.494 DEBUG:boto:path=/dgbpxg-57/b 2024-04-12T03:33:17.495 DEBUG:boto:auth_path=/dgbpxg-57/b 2024-04-12T03:33:17.495 DEBUG:boto:Method: GET 2024-04-12T03:33:17.495 DEBUG:boto:Path: /dgbpxg-57/b 2024-04-12T03:33:17.495 DEBUG:boto:Data: 2024-04-12T03:33:17.495 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.495 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.495 DEBUG:boto:Params: {} 2024-04-12T03:33:17.495 DEBUG:boto:Token: None 2024-04-12T03:33:17.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/b 2024-04-12T03:33:17.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ= 2024-04-12T03:33:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001070d28cfc5b5645-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.497 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.498 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.498 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.498 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.498 DEBUG:boto:Method: GET 2024-04-12T03:33:17.498 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:17.498 DEBUG:boto:Data: 2024-04-12T03:33:17.498 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.498 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.498 DEBUG:boto:Params: {} 2024-04-12T03:33:17.499 DEBUG:boto:Token: None 2024-04-12T03:33:17.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051386634569bc295-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.542 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.543 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.543 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.543 DEBUG:boto:path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.543 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=b 2024-04-12T03:33:17.543 DEBUG:boto:Method: GET 2024-04-12T03:33:17.543 DEBUG:boto:Path: /dgbpxg-57/?prefix=b 2024-04-12T03:33:17.543 DEBUG:boto:Data: 2024-04-12T03:33:17.544 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.544 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.544 DEBUG:boto:Params: {} 2024-04-12T03:33:17.544 DEBUG:boto:Token: None 2024-04-12T03:33:17.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9531ccc39c615f6-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.546 DEBUG:boto:b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:17.547 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.547 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.547 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.547 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.547 DEBUG:boto:Method: GET 2024-04-12T03:33:17.547 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.547 DEBUG:boto:Data: 2024-04-12T03:33:17.547 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.548 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.548 DEBUG:boto:Params: {} 2024-04-12T03:33:17.548 DEBUG:boto:Token: None 2024-04-12T03:33:17.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T03:33:17.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T03:33:17.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004981884d883260e5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.550 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.550 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.550 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.550 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.550 DEBUG:boto:Method: GET 2024-04-12T03:33:17.551 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.551 DEBUG:boto:Data: 2024-04-12T03:33:17.551 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.551 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.551 DEBUG:boto:Params: {} 2024-04-12T03:33:17.551 DEBUG:boto:Token: None 2024-04-12T03:33:17.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T03:33:17.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T03:33:17.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020880f8387456b2d-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.594 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.595 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.595 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.595 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.595 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.595 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.595 DEBUG:boto:Data: 2024-04-12T03:33:17.595 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.595 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.595 DEBUG:boto:Params: {} 2024-04-12T03:33:17.596 DEBUG:boto:Token: None 2024-04-12T03:33:17.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs= 2024-04-12T03:33:17.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs=', 'Content-Length': '0'} 2024-04-12T03:33:17.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d84e8a80b79da23-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.598 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.598 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.598 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.598 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.599 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.599 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.599 DEBUG:boto:Data: 2024-04-12T03:33:17.599 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.599 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.599 DEBUG:boto:Params: {} 2024-04-12T03:33:17.599 DEBUG:boto:Token: None 2024-04-12T03:33:17.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs= 2024-04-12T03:33:17.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs=', 'Content-Length': '0'} 2024-04-12T03:33:17.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef323e55da7323c-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:17.601 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.601 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.601 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.601 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.602 DEBUG:boto:Method: GET 2024-04-12T03:33:17.602 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.602 DEBUG:boto:Data: 2024-04-12T03:33:17.602 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.602 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.602 DEBUG:boto:Params: {} 2024-04-12T03:33:17.602 DEBUG:boto:Token: None 2024-04-12T03:33:17.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T03:33:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T03:33:17.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adac000b37c607c-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.605 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.605 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.605 DEBUG:boto:path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.605 DEBUG:boto:auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.605 DEBUG:boto:Method: GET 2024-04-12T03:33:17.605 DEBUG:boto:Path: /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.605 DEBUG:boto:Data: 2024-04-12T03:33:17.605 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.605 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.606 DEBUG:boto:Params: {} 2024-04-12T03:33:17.606 DEBUG:boto:Token: None 2024-04-12T03:33:17.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c?versionId=null 2024-04-12T03:33:17.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T03:33:17.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T03:33:17.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c862322fac142e42-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.608 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.608 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.608 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.608 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.608 DEBUG:boto:Method: GET 2024-04-12T03:33:17.609 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:17.609 DEBUG:boto:Data: 2024-04-12T03:33:17.609 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.609 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.609 DEBUG:boto:Params: {} 2024-04-12T03:33:17.609 DEBUG:boto:Token: None 2024-04-12T03:33:17.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff65eda47c15b728-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.612 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.612 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.612 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.612 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.612 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.612 DEBUG:boto:Method: GET 2024-04-12T03:33:17.613 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:17.613 DEBUG:boto:Data: 2024-04-12T03:33:17.613 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.613 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.613 DEBUG:boto:Params: {} 2024-04-12T03:33:17.613 DEBUG:boto:Token: None 2024-04-12T03:33:17.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04f4985498df740-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.616 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.616 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.616 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.617 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.617 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:17.617 DEBUG:boto:Data: 2024-04-12T03:33:17.617 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.618 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.618 DEBUG:boto:Params: {} 2024-04-12T03:33:17.618 DEBUG:boto:Token: None 2024-04-12T03:33:17.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c 2024-04-12T03:33:17.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4= 2024-04-12T03:33:17.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4=', 'Content-Length': '0'} 2024-04-12T03:33:17.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ea5e045bdd37e4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.620 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.620 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.620 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.620 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:17.620 DEBUG:boto:Data: 2024-04-12T03:33:17.621 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.621 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.621 DEBUG:boto:Params: {} 2024-04-12T03:33:17.621 DEBUG:boto:Token: None 2024-04-12T03:33:17.621 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c 2024-04-12T03:33:17.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4= 2024-04-12T03:33:17.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4=', 'Content-Length': '0'} 2024-04-12T03:33:17.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b97b2e93a92b74-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:17.623 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.623 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.623 DEBUG:boto:Method: GET 2024-04-12T03:33:17.623 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:17.623 DEBUG:boto:Data: 2024-04-12T03:33:17.623 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.624 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.624 DEBUG:boto:Params: {} 2024-04-12T03:33:17.624 DEBUG:boto:Token: None 2024-04-12T03:33:17.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c 2024-04-12T03:33:17.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ= 2024-04-12T03:33:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40b6b44a334794a-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.626 DEBUG:boto:path=/dgbpxg-57/c 2024-04-12T03:33:17.626 DEBUG:boto:auth_path=/dgbpxg-57/c 2024-04-12T03:33:17.626 DEBUG:boto:Method: GET 2024-04-12T03:33:17.626 DEBUG:boto:Path: /dgbpxg-57/c 2024-04-12T03:33:17.626 DEBUG:boto:Data: 2024-04-12T03:33:17.626 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.627 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.627 DEBUG:boto:Params: {} 2024-04-12T03:33:17.627 DEBUG:boto:Token: None 2024-04-12T03:33:17.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/c 2024-04-12T03:33:17.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ= 2024-04-12T03:33:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ=', 'Content-Length': '0'} 2024-04-12T03:33:17.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fdd40e00e423d4-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.674 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.675 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.675 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.675 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.675 DEBUG:boto:Method: GET 2024-04-12T03:33:17.675 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:17.675 DEBUG:boto:Data: 2024-04-12T03:33:17.675 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.675 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.675 DEBUG:boto:Params: {} 2024-04-12T03:33:17.676 DEBUG:boto:Token: None 2024-04-12T03:33:17.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006054f88e53c22e19-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.678 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.679 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.680 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.680 DEBUG:boto:path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.680 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=c 2024-04-12T03:33:17.680 DEBUG:boto:Method: GET 2024-04-12T03:33:17.680 DEBUG:boto:Path: /dgbpxg-57/?prefix=c 2024-04-12T03:33:17.680 DEBUG:boto:Data: 2024-04-12T03:33:17.680 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.680 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.680 DEBUG:boto:Params: {} 2024-04-12T03:33:17.680 DEBUG:boto:Token: None 2024-04-12T03:33:17.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c0eba41a593021a-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.682 DEBUG:boto:b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:17.683 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.683 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.683 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.683 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.683 DEBUG:boto:Method: GET 2024-04-12T03:33:17.683 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.683 DEBUG:boto:Data: 2024-04-12T03:33:17.683 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.684 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.684 DEBUG:boto:Params: {} 2024-04-12T03:33:17.684 DEBUG:boto:Token: None 2024-04-12T03:33:17.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T03:33:17.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T03:33:17.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae976bd5f4207c7-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.726 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.726 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.726 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.727 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.727 DEBUG:boto:Method: GET 2024-04-12T03:33:17.727 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.727 DEBUG:boto:Data: 2024-04-12T03:33:17.727 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.727 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.727 DEBUG:boto:Params: {} 2024-04-12T03:33:17.727 DEBUG:boto:Token: None 2024-04-12T03:33:17.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T03:33:17.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T03:33:17.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a267a89b7d410339-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.730 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.730 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.730 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.730 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.730 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.730 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.730 DEBUG:boto:Data: 2024-04-12T03:33:17.730 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.730 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.731 DEBUG:boto:Params: {} 2024-04-12T03:33:17.731 DEBUG:boto:Token: None 2024-04-12T03:33:17.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98= 2024-04-12T03:33:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98=', 'Content-Length': '0'} 2024-04-12T03:33:17.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca3aad887bc8b1b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.733 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.733 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.733 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.733 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.733 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.733 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.733 DEBUG:boto:Data: 2024-04-12T03:33:17.733 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.734 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.734 DEBUG:boto:Params: {} 2024-04-12T03:33:17.734 DEBUG:boto:Token: None 2024-04-12T03:33:17.734 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98= 2024-04-12T03:33:17.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98=', 'Content-Length': '0'} 2024-04-12T03:33:17.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daffa953a1905f68-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:17.736 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.736 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.737 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.737 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.737 DEBUG:boto:Method: GET 2024-04-12T03:33:17.737 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.737 DEBUG:boto:Data: 2024-04-12T03:33:17.737 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.737 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.737 DEBUG:boto:Params: {} 2024-04-12T03:33:17.737 DEBUG:boto:Token: None 2024-04-12T03:33:17.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T03:33:17.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T03:33:17.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d7a2070641d9ef-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.739 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.739 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.739 DEBUG:boto:path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.740 DEBUG:boto:auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.740 DEBUG:boto:Method: GET 2024-04-12T03:33:17.740 DEBUG:boto:Path: /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.740 DEBUG:boto:Data: 2024-04-12T03:33:17.740 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.740 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.740 DEBUG:boto:Params: {} 2024-04-12T03:33:17.740 DEBUG:boto:Token: None 2024-04-12T03:33:17.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d?versionId=null 2024-04-12T03:33:17.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T03:33:17.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T03:33:17.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ac630bd8873435-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.742 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.743 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.743 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.743 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.743 DEBUG:boto:Method: GET 2024-04-12T03:33:17.743 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:17.743 DEBUG:boto:Data: 2024-04-12T03:33:17.743 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.743 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.743 DEBUG:boto:Params: {} 2024-04-12T03:33:17.744 DEBUG:boto:Token: None 2024-04-12T03:33:17.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba2e205d6c1d983-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.746 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.747 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.747 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.747 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.747 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.747 DEBUG:boto:Method: GET 2024-04-12T03:33:17.747 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:17.747 DEBUG:boto:Data: 2024-04-12T03:33:17.747 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.747 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.747 DEBUG:boto:Params: {} 2024-04-12T03:33:17.748 DEBUG:boto:Token: None 2024-04-12T03:33:17.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000150dc1281fd1a4ce-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.750 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.750 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.750 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.750 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.750 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:17.750 DEBUG:boto:Data: 2024-04-12T03:33:17.751 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.751 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.751 DEBUG:boto:Params: {} 2024-04-12T03:33:17.751 DEBUG:boto:Token: None 2024-04-12T03:33:17.751 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d 2024-04-12T03:33:17.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA= 2024-04-12T03:33:17.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA=', 'Content-Length': '0'} 2024-04-12T03:33:17.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fa03a4b823844b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.753 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.753 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.754 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.754 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:17.754 DEBUG:boto:Data: 2024-04-12T03:33:17.754 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.754 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.754 DEBUG:boto:Params: {} 2024-04-12T03:33:17.754 DEBUG:boto:Token: None 2024-04-12T03:33:17.755 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d 2024-04-12T03:33:17.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA= 2024-04-12T03:33:17.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA=', 'Content-Length': '0'} 2024-04-12T03:33:17.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eeb1ca4dfa070b6-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:17.757 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.757 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.757 DEBUG:boto:Method: GET 2024-04-12T03:33:17.757 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:17.757 DEBUG:boto:Data: 2024-04-12T03:33:17.757 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.757 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.757 DEBUG:boto:Params: {} 2024-04-12T03:33:17.758 DEBUG:boto:Token: None 2024-04-12T03:33:17.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d 2024-04-12T03:33:17.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw= 2024-04-12T03:33:17.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw=', 'Content-Length': '0'} 2024-04-12T03:33:17.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7a8b310ea96268-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.802 DEBUG:boto:path=/dgbpxg-57/d 2024-04-12T03:33:17.802 DEBUG:boto:auth_path=/dgbpxg-57/d 2024-04-12T03:33:17.803 DEBUG:boto:Method: GET 2024-04-12T03:33:17.803 DEBUG:boto:Path: /dgbpxg-57/d 2024-04-12T03:33:17.803 DEBUG:boto:Data: 2024-04-12T03:33:17.803 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.803 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.803 DEBUG:boto:Params: {} 2024-04-12T03:33:17.803 DEBUG:boto:Token: None 2024-04-12T03:33:17.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/d 2024-04-12T03:33:17.804 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw= 2024-04-12T03:33:17.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw=', 'Content-Length': '0'} 2024-04-12T03:33:17.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e2eb85c0216ff5-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.806 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.806 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.806 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.806 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.806 DEBUG:boto:Method: GET 2024-04-12T03:33:17.807 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:17.807 DEBUG:boto:Data: 2024-04-12T03:33:17.807 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.807 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.807 DEBUG:boto:Params: {} 2024-04-12T03:33:17.807 DEBUG:boto:Token: None 2024-04-12T03:33:17.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010542cc94b406125-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.810 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.810 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.810 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.810 DEBUG:boto:path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.811 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=d 2024-04-12T03:33:17.811 DEBUG:boto:Method: GET 2024-04-12T03:33:17.811 DEBUG:boto:Path: /dgbpxg-57/?prefix=d 2024-04-12T03:33:17.811 DEBUG:boto:Data: 2024-04-12T03:33:17.811 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.811 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.811 DEBUG:boto:Params: {} 2024-04-12T03:33:17.811 DEBUG:boto:Token: None 2024-04-12T03:33:17.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee54774b9d79e4ec-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.814 DEBUG:boto:b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:17.815 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.815 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.815 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.815 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.815 DEBUG:boto:Method: GET 2024-04-12T03:33:17.815 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.815 DEBUG:boto:Data: 2024-04-12T03:33:17.815 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.815 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.815 DEBUG:boto:Params: {} 2024-04-12T03:33:17.816 DEBUG:boto:Token: None 2024-04-12T03:33:17.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T03:33:17.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T03:33:17.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea5f354a77ff861-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.818 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.818 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.818 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.818 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.819 DEBUG:boto:Method: GET 2024-04-12T03:33:17.819 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.819 DEBUG:boto:Data: 2024-04-12T03:33:17.819 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.819 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.819 DEBUG:boto:Params: {} 2024-04-12T03:33:17.820 DEBUG:boto:Token: None 2024-04-12T03:33:17.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T03:33:17.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T03:33:17.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7130a3c29c0b3f7-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.822 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.822 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.822 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.822 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.822 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.822 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.822 DEBUG:boto:Data: 2024-04-12T03:33:17.822 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.823 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.823 DEBUG:boto:Params: {} 2024-04-12T03:33:17.823 DEBUG:boto:Token: None 2024-04-12T03:33:17.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI= 2024-04-12T03:33:17.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI=', 'Content-Length': '0'} 2024-04-12T03:33:17.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd4a004e3599223-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.825 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.825 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.825 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.825 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.825 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.825 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.825 DEBUG:boto:Data: 2024-04-12T03:33:17.826 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.826 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.826 DEBUG:boto:Params: {} 2024-04-12T03:33:17.826 DEBUG:boto:Token: None 2024-04-12T03:33:17.826 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI= 2024-04-12T03:33:17.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI=', 'Content-Length': '0'} 2024-04-12T03:33:17.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af9fdcad923f3a7-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:17.828 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.828 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.828 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.828 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.828 DEBUG:boto:Method: GET 2024-04-12T03:33:17.829 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.829 DEBUG:boto:Data: 2024-04-12T03:33:17.829 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.829 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.829 DEBUG:boto:Params: {} 2024-04-12T03:33:17.829 DEBUG:boto:Token: None 2024-04-12T03:33:17.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T03:33:17.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T03:33:17.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617b1e592e1ffcf5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.831 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.831 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.831 DEBUG:boto:path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.831 DEBUG:boto:auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.831 DEBUG:boto:Method: GET 2024-04-12T03:33:17.832 DEBUG:boto:Path: /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.832 DEBUG:boto:Data: 2024-04-12T03:33:17.832 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.832 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.832 DEBUG:boto:Params: {} 2024-04-12T03:33:17.832 DEBUG:boto:Token: None 2024-04-12T03:33:17.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e?versionId=null 2024-04-12T03:33:17.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T03:33:17.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T03:33:17.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6485b3bb79cdb8-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.834 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.834 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.834 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.834 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.835 DEBUG:boto:Method: GET 2024-04-12T03:33:17.835 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:17.835 DEBUG:boto:Data: 2024-04-12T03:33:17.835 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.835 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.835 DEBUG:boto:Params: {} 2024-04-12T03:33:17.835 DEBUG:boto:Token: None 2024-04-12T03:33:17.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc94ad3317b447c-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.838 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.838 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.839 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.839 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.839 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.839 DEBUG:boto:Method: GET 2024-04-12T03:33:17.839 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:17.839 DEBUG:boto:Data: 2024-04-12T03:33:17.839 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.839 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.839 DEBUG:boto:Params: {} 2024-04-12T03:33:17.839 DEBUG:boto:Token: None 2024-04-12T03:33:17.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542ad4f9bf5edb32-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.841 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.842 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.842 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.842 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.842 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:17.842 DEBUG:boto:Data: 2024-04-12T03:33:17.842 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.842 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.842 DEBUG:boto:Params: {} 2024-04-12T03:33:17.843 DEBUG:boto:Token: None 2024-04-12T03:33:17.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e 2024-04-12T03:33:17.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g= 2024-04-12T03:33:17.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g=', 'Content-Length': '0'} 2024-04-12T03:33:17.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5f2e44b2310eb9-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.844 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.845 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.845 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.845 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:17.845 DEBUG:boto:Data: 2024-04-12T03:33:17.845 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.845 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.845 DEBUG:boto:Params: {} 2024-04-12T03:33:17.845 DEBUG:boto:Token: None 2024-04-12T03:33:17.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e 2024-04-12T03:33:17.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g= 2024-04-12T03:33:17.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g=', 'Content-Length': '0'} 2024-04-12T03:33:17.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d961fd1bfadbed8f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:33:17.848 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.848 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.848 DEBUG:boto:Method: GET 2024-04-12T03:33:17.848 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:17.848 DEBUG:boto:Data: 2024-04-12T03:33:17.848 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.848 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.849 DEBUG:boto:Params: {} 2024-04-12T03:33:17.849 DEBUG:boto:Token: None 2024-04-12T03:33:17.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e 2024-04-12T03:33:17.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU= 2024-04-12T03:33:17.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU=', 'Content-Length': '0'} 2024-04-12T03:33:17.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d275bf6b18b991-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.851 DEBUG:boto:path=/dgbpxg-57/e 2024-04-12T03:33:17.851 DEBUG:boto:auth_path=/dgbpxg-57/e 2024-04-12T03:33:17.851 DEBUG:boto:Method: GET 2024-04-12T03:33:17.851 DEBUG:boto:Path: /dgbpxg-57/e 2024-04-12T03:33:17.851 DEBUG:boto:Data: 2024-04-12T03:33:17.851 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.852 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.852 DEBUG:boto:Params: {} 2024-04-12T03:33:17.852 DEBUG:boto:Token: None 2024-04-12T03:33:17.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/e 2024-04-12T03:33:17.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU= 2024-04-12T03:33:17.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU=', 'Content-Length': '0'} 2024-04-12T03:33:17.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5fd95189ecf99e-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.854 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.854 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.854 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.854 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.854 DEBUG:boto:Method: GET 2024-04-12T03:33:17.854 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:17.854 DEBUG:boto:Data: 2024-04-12T03:33:17.855 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.855 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.855 DEBUG:boto:Params: {} 2024-04-12T03:33:17.855 DEBUG:boto:Token: None 2024-04-12T03:33:17.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08b7a78e8011ad9-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.902 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.903 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.903 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.903 DEBUG:boto:path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.903 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=e 2024-04-12T03:33:17.903 DEBUG:boto:Method: GET 2024-04-12T03:33:17.903 DEBUG:boto:Path: /dgbpxg-57/?prefix=e 2024-04-12T03:33:17.904 DEBUG:boto:Data: 2024-04-12T03:33:17.904 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.904 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.904 DEBUG:boto:Params: {} 2024-04-12T03:33:17.904 DEBUG:boto:Token: None 2024-04-12T03:33:17.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041eb86a313fb9e54-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.907 DEBUG:boto:b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:17.907 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.907 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.907 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.907 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.908 DEBUG:boto:Method: GET 2024-04-12T03:33:17.908 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.908 DEBUG:boto:Data: 2024-04-12T03:33:17.908 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.908 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.908 DEBUG:boto:Params: {} 2024-04-12T03:33:17.908 DEBUG:boto:Token: None 2024-04-12T03:33:17.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.909 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T03:33:17.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T03:33:17.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f572d654b7ad12-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.910 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.911 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.911 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.911 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.911 DEBUG:boto:Method: GET 2024-04-12T03:33:17.912 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.912 DEBUG:boto:Data: 2024-04-12T03:33:17.912 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.912 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.912 DEBUG:boto:Params: {} 2024-04-12T03:33:17.912 DEBUG:boto:Token: None 2024-04-12T03:33:17.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T03:33:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T03:33:17.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d52668fe44f1e6a-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.915 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.915 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.915 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.915 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.915 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.916 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.916 DEBUG:boto:Data: 2024-04-12T03:33:17.916 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.916 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.916 DEBUG:boto:Params: {} 2024-04-12T03:33:17.916 DEBUG:boto:Token: None 2024-04-12T03:33:17.916 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU= 2024-04-12T03:33:17.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU=', 'Content-Length': '0'} 2024-04-12T03:33:17.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad539d25e3989df6-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.918 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.918 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.918 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.919 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.919 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.919 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.919 DEBUG:boto:Data: 2024-04-12T03:33:17.919 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.919 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.919 DEBUG:boto:Params: {} 2024-04-12T03:33:17.919 DEBUG:boto:Token: None 2024-04-12T03:33:17.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU= 2024-04-12T03:33:17.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU=', 'Content-Length': '0'} 2024-04-12T03:33:17.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fab76948555cb5-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:17.921 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.921 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.921 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.922 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.922 DEBUG:boto:Method: GET 2024-04-12T03:33:17.922 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.922 DEBUG:boto:Data: 2024-04-12T03:33:17.922 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.922 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.922 DEBUG:boto:Params: {} 2024-04-12T03:33:17.922 DEBUG:boto:Token: None 2024-04-12T03:33:17.922 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T03:33:17.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T03:33:17.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621cee8faa7210dc-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.924 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.925 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.925 DEBUG:boto:path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.925 DEBUG:boto:auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.925 DEBUG:boto:Method: GET 2024-04-12T03:33:17.925 DEBUG:boto:Path: /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.925 DEBUG:boto:Data: 2024-04-12T03:33:17.925 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.926 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.926 DEBUG:boto:Params: {} 2024-04-12T03:33:17.926 DEBUG:boto:Token: None 2024-04-12T03:33:17.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f?versionId=null 2024-04-12T03:33:17.926 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T03:33:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T03:33:17.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b288cfa99dd8f10-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.974 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.975 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.975 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.975 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.975 DEBUG:boto:Method: GET 2024-04-12T03:33:17.975 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:17.975 DEBUG:boto:Data: 2024-04-12T03:33:17.975 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.975 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.975 DEBUG:boto:Params: {} 2024-04-12T03:33:17.976 DEBUG:boto:Token: None 2024-04-12T03:33:17.976 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749149480a80c69f-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.978 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.979 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.979 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.979 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.979 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.979 DEBUG:boto:Method: GET 2024-04-12T03:33:17.979 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:17.979 DEBUG:boto:Data: 2024-04-12T03:33:17.979 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.979 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.979 DEBUG:boto:Params: {} 2024-04-12T03:33:17.980 DEBUG:boto:Token: None 2024-04-12T03:33:17.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfc83de3879699c-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.982 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.982 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.982 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.982 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.982 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:17.982 DEBUG:boto:Data: 2024-04-12T03:33:17.982 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.983 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.983 DEBUG:boto:Params: {} 2024-04-12T03:33:17.983 DEBUG:boto:Token: None 2024-04-12T03:33:17.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f 2024-04-12T03:33:17.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE= 2024-04-12T03:33:17.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE=', 'Content-Length': '0'} 2024-04-12T03:33:17.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7f9e2a40d3d5a4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.985 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.985 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.985 DEBUG:boto:Method: HEAD 2024-04-12T03:33:17.985 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:17.985 DEBUG:boto:Data: 2024-04-12T03:33:17.985 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.985 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.985 DEBUG:boto:Params: {} 2024-04-12T03:33:17.986 DEBUG:boto:Token: None 2024-04-12T03:33:17.986 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f 2024-04-12T03:33:17.986 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE= 2024-04-12T03:33:17.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE=', 'Content-Length': '0'} 2024-04-12T03:33:17.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5ac67dfbd08a30-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:33:17.988 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.988 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.989 DEBUG:boto:Method: GET 2024-04-12T03:33:17.989 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:17.989 DEBUG:boto:Data: 2024-04-12T03:33:17.989 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.989 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.989 DEBUG:boto:Params: {} 2024-04-12T03:33:17.989 DEBUG:boto:Token: None 2024-04-12T03:33:17.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f 2024-04-12T03:33:17.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk= 2024-04-12T03:33:17.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk=', 'Content-Length': '0'} 2024-04-12T03:33:17.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e347219318c1a8c1-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.992 DEBUG:boto:path=/dgbpxg-57/f 2024-04-12T03:33:17.992 DEBUG:boto:auth_path=/dgbpxg-57/f 2024-04-12T03:33:17.992 DEBUG:boto:Method: GET 2024-04-12T03:33:17.992 DEBUG:boto:Path: /dgbpxg-57/f 2024-04-12T03:33:17.992 DEBUG:boto:Data: 2024-04-12T03:33:17.992 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:17.993 DEBUG:boto:Port: 8000 2024-04-12T03:33:17.993 DEBUG:boto:Params: {} 2024-04-12T03:33:17.993 DEBUG:boto:Token: None 2024-04-12T03:33:17.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/f 2024-04-12T03:33:17.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk= 2024-04-12T03:33:17.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk=', 'Content-Length': '0'} 2024-04-12T03:33:17.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002067403bb3b7a83b-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.995 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.995 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:17.995 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.995 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:17.995 DEBUG:boto:Method: GET 2024-04-12T03:33:17.995 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:17.995 DEBUG:boto:Data: 2024-04-12T03:33:17.995 DEBUG:boto:Headers: {} 2024-04-12T03:33:17.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:17.996 DEBUG:boto:Port: 8001 2024-04-12T03:33:17.996 DEBUG:boto:Params: {} 2024-04-12T03:33:17.996 DEBUG:boto:Token: None 2024-04-12T03:33:17.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:17 GMT /dgbpxg-57/ 2024-04-12T03:33:17.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T03:33:17.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T03:33:17.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccf9598ed90e534-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:17.999 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:17.999 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:17.999 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.000 DEBUG:boto:path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:18.000 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=f 2024-04-12T03:33:18.000 DEBUG:boto:Method: GET 2024-04-12T03:33:18.000 DEBUG:boto:Path: /dgbpxg-57/?prefix=f 2024-04-12T03:33:18.000 DEBUG:boto:Data: 2024-04-12T03:33:18.000 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.000 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.000 DEBUG:boto:Params: {} 2024-04-12T03:33:18.000 DEBUG:boto:Token: None 2024-04-12T03:33:18.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004749028ef76b3a0d-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.003 DEBUG:boto:b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:18.004 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.004 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.004 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.004 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.004 DEBUG:boto:Method: GET 2024-04-12T03:33:18.005 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.005 DEBUG:boto:Data: 2024-04-12T03:33:18.005 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.005 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.006 DEBUG:boto:Params: {} 2024-04-12T03:33:18.006 DEBUG:boto:Token: None 2024-04-12T03:33:18.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T03:33:18.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T03:33:18.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e04de658b379076-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.054 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.055 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.055 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.055 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.055 DEBUG:boto:Method: GET 2024-04-12T03:33:18.056 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.056 DEBUG:boto:Data: 2024-04-12T03:33:18.056 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.056 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.057 DEBUG:boto:Params: {} 2024-04-12T03:33:18.057 DEBUG:boto:Token: None 2024-04-12T03:33:18.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.057 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T03:33:18.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T03:33:18.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b16330a708ae708-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.060 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.060 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.060 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.060 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.061 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.061 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.061 DEBUG:boto:Data: 2024-04-12T03:33:18.061 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.061 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.062 DEBUG:boto:Params: {} 2024-04-12T03:33:18.062 DEBUG:boto:Token: None 2024-04-12T03:33:18.062 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M= 2024-04-12T03:33:18.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M=', 'Content-Length': '0'} 2024-04-12T03:33:18.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cab4e34728ade53-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.065 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.065 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.065 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.065 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.065 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.065 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.066 DEBUG:boto:Data: 2024-04-12T03:33:18.066 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.066 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.066 DEBUG:boto:Params: {} 2024-04-12T03:33:18.066 DEBUG:boto:Token: None 2024-04-12T03:33:18.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M= 2024-04-12T03:33:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M=', 'Content-Length': '0'} 2024-04-12T03:33:18.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c25ce97dba6d4a4-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:18.069 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.070 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.070 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.070 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.070 DEBUG:boto:Method: GET 2024-04-12T03:33:18.070 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.071 DEBUG:boto:Data: 2024-04-12T03:33:18.071 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.071 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.071 DEBUG:boto:Params: {} 2024-04-12T03:33:18.072 DEBUG:boto:Token: None 2024-04-12T03:33:18.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T03:33:18.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T03:33:18.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846b0df10f16ae08-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.074 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.074 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.074 DEBUG:boto:path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.074 DEBUG:boto:auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.074 DEBUG:boto:Method: GET 2024-04-12T03:33:18.075 DEBUG:boto:Path: /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.075 DEBUG:boto:Data: 2024-04-12T03:33:18.075 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.075 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.075 DEBUG:boto:Params: {} 2024-04-12T03:33:18.075 DEBUG:boto:Token: None 2024-04-12T03:33:18.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g?versionId=null 2024-04-12T03:33:18.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T03:33:18.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T03:33:18.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379e1da1fcdf332c-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.077 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.077 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.077 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.078 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.078 DEBUG:boto:Method: GET 2024-04-12T03:33:18.078 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:18.078 DEBUG:boto:Data: 2024-04-12T03:33:18.078 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.078 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.078 DEBUG:boto:Params: {} 2024-04-12T03:33:18.078 DEBUG:boto:Token: None 2024-04-12T03:33:18.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3cf22e038bda86-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.081 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.081 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.082 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.082 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.082 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.082 DEBUG:boto:Method: GET 2024-04-12T03:33:18.082 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:18.082 DEBUG:boto:Data: 2024-04-12T03:33:18.082 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.082 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.082 DEBUG:boto:Params: {} 2024-04-12T03:33:18.083 DEBUG:boto:Token: None 2024-04-12T03:33:18.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d35eec86ac8f540-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.085 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.085 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.085 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.085 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.085 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:18.085 DEBUG:boto:Data: 2024-04-12T03:33:18.086 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.086 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.086 DEBUG:boto:Params: {} 2024-04-12T03:33:18.086 DEBUG:boto:Token: None 2024-04-12T03:33:18.086 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g 2024-04-12T03:33:18.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU= 2024-04-12T03:33:18.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU=', 'Content-Length': '0'} 2024-04-12T03:33:18.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf8e34cf2427999-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.088 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.088 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.088 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.088 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:18.088 DEBUG:boto:Data: 2024-04-12T03:33:18.088 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.089 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.089 DEBUG:boto:Params: {} 2024-04-12T03:33:18.089 DEBUG:boto:Token: None 2024-04-12T03:33:18.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g 2024-04-12T03:33:18.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU= 2024-04-12T03:33:18.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU=', 'Content-Length': '0'} 2024-04-12T03:33:18.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da70a7d14a74279c-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:33:18.091 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.091 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.091 DEBUG:boto:Method: GET 2024-04-12T03:33:18.091 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:18.091 DEBUG:boto:Data: 2024-04-12T03:33:18.091 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.091 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.091 DEBUG:boto:Params: {} 2024-04-12T03:33:18.092 DEBUG:boto:Token: None 2024-04-12T03:33:18.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g 2024-04-12T03:33:18.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU= 2024-04-12T03:33:18.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU=', 'Content-Length': '0'} 2024-04-12T03:33:18.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006027aafb97f7ca12-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.093 DEBUG:boto:path=/dgbpxg-57/g 2024-04-12T03:33:18.094 DEBUG:boto:auth_path=/dgbpxg-57/g 2024-04-12T03:33:18.094 DEBUG:boto:Method: GET 2024-04-12T03:33:18.094 DEBUG:boto:Path: /dgbpxg-57/g 2024-04-12T03:33:18.094 DEBUG:boto:Data: 2024-04-12T03:33:18.094 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.095 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.095 DEBUG:boto:Params: {} 2024-04-12T03:33:18.095 DEBUG:boto:Token: None 2024-04-12T03:33:18.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/g 2024-04-12T03:33:18.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU= 2024-04-12T03:33:18.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU=', 'Content-Length': '0'} 2024-04-12T03:33:18.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf03f4fedd86b38b-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.138 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.138 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.138 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.139 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.139 DEBUG:boto:Method: GET 2024-04-12T03:33:18.139 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:18.139 DEBUG:boto:Data: 2024-04-12T03:33:18.139 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.139 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.139 DEBUG:boto:Params: {} 2024-04-12T03:33:18.139 DEBUG:boto:Token: None 2024-04-12T03:33:18.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea855cce890e41e2-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.142 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.142 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.142 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.143 DEBUG:boto:path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.143 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=g 2024-04-12T03:33:18.143 DEBUG:boto:Method: GET 2024-04-12T03:33:18.143 DEBUG:boto:Path: /dgbpxg-57/?prefix=g 2024-04-12T03:33:18.143 DEBUG:boto:Data: 2024-04-12T03:33:18.143 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.143 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.143 DEBUG:boto:Params: {} 2024-04-12T03:33:18.143 DEBUG:boto:Token: None 2024-04-12T03:33:18.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc5dd89a78fa7a90-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.145 DEBUG:boto:b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:18.146 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.146 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.146 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.146 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.146 DEBUG:boto:Method: GET 2024-04-12T03:33:18.146 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.147 DEBUG:boto:Data: 2024-04-12T03:33:18.147 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.147 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.147 DEBUG:boto:Params: {} 2024-04-12T03:33:18.147 DEBUG:boto:Token: None 2024-04-12T03:33:18.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T03:33:18.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T03:33:18.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a67425af8ce1c09-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.150 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.150 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.150 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.150 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.150 DEBUG:boto:Method: GET 2024-04-12T03:33:18.150 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.150 DEBUG:boto:Data: 2024-04-12T03:33:18.150 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.150 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.150 DEBUG:boto:Params: {} 2024-04-12T03:33:18.151 DEBUG:boto:Token: None 2024-04-12T03:33:18.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T03:33:18.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T03:33:18.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0795c94dd786866-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.153 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.153 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.153 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.153 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.153 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.153 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.153 DEBUG:boto:Data: 2024-04-12T03:33:18.153 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.154 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.154 DEBUG:boto:Params: {} 2024-04-12T03:33:18.154 DEBUG:boto:Token: None 2024-04-12T03:33:18.154 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo= 2024-04-12T03:33:18.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo=', 'Content-Length': '0'} 2024-04-12T03:33:18.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fbc1b8e6e7a44e4-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.156 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.156 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.156 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.156 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.156 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.156 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.156 DEBUG:boto:Data: 2024-04-12T03:33:18.157 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.157 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.157 DEBUG:boto:Params: {} 2024-04-12T03:33:18.157 DEBUG:boto:Token: None 2024-04-12T03:33:18.157 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo= 2024-04-12T03:33:18.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo=', 'Content-Length': '0'} 2024-04-12T03:33:18.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002345bff6e365483a-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:18.159 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.159 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.159 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.159 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.160 DEBUG:boto:Method: GET 2024-04-12T03:33:18.160 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.160 DEBUG:boto:Data: 2024-04-12T03:33:18.160 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.160 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.160 DEBUG:boto:Params: {} 2024-04-12T03:33:18.160 DEBUG:boto:Token: None 2024-04-12T03:33:18.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T03:33:18.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T03:33:18.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb922a65e420873-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.163 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.163 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.163 DEBUG:boto:path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.163 DEBUG:boto:auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.163 DEBUG:boto:Method: GET 2024-04-12T03:33:18.163 DEBUG:boto:Path: /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.164 DEBUG:boto:Data: 2024-04-12T03:33:18.164 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.164 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.164 DEBUG:boto:Params: {} 2024-04-12T03:33:18.164 DEBUG:boto:Token: None 2024-04-12T03:33:18.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h?versionId=null 2024-04-12T03:33:18.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T03:33:18.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T03:33:18.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3036a4e7e66b198-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.166 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.166 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.166 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.167 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.167 DEBUG:boto:Method: GET 2024-04-12T03:33:18.167 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:18.167 DEBUG:boto:Data: 2024-04-12T03:33:18.167 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.167 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.167 DEBUG:boto:Params: {} 2024-04-12T03:33:18.167 DEBUG:boto:Token: None 2024-04-12T03:33:18.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8302c7944b17fae-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.170 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.170 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.170 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.170 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.171 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.171 DEBUG:boto:Method: GET 2024-04-12T03:33:18.171 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:18.171 DEBUG:boto:Data: 2024-04-12T03:33:18.171 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.171 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.171 DEBUG:boto:Params: {} 2024-04-12T03:33:18.171 DEBUG:boto:Token: None 2024-04-12T03:33:18.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736d965af19fc5f2-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.173 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.174 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.174 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.174 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.174 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:18.174 DEBUG:boto:Data: 2024-04-12T03:33:18.174 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.174 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.175 DEBUG:boto:Params: {} 2024-04-12T03:33:18.175 DEBUG:boto:Token: None 2024-04-12T03:33:18.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h 2024-04-12T03:33:18.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE= 2024-04-12T03:33:18.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE=', 'Content-Length': '0'} 2024-04-12T03:33:18.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b963cb6aa7f35948-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.177 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.177 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.177 DEBUG:boto:Method: HEAD 2024-04-12T03:33:18.177 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:18.177 DEBUG:boto:Data: 2024-04-12T03:33:18.177 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.178 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.178 DEBUG:boto:Params: {} 2024-04-12T03:33:18.178 DEBUG:boto:Token: None 2024-04-12T03:33:18.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h 2024-04-12T03:33:18.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE= 2024-04-12T03:33:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE=', 'Content-Length': '0'} 2024-04-12T03:33:18.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b62200780c26c96-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:33:18.180 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.180 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.180 DEBUG:boto:Method: GET 2024-04-12T03:33:18.180 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:18.180 DEBUG:boto:Data: 2024-04-12T03:33:18.180 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.180 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.181 DEBUG:boto:Params: {} 2024-04-12T03:33:18.181 DEBUG:boto:Token: None 2024-04-12T03:33:18.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h 2024-04-12T03:33:18.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo= 2024-04-12T03:33:18.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo=', 'Content-Length': '0'} 2024-04-12T03:33:18.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f927933d45ccd9fb-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.183 DEBUG:boto:path=/dgbpxg-57/h 2024-04-12T03:33:18.183 DEBUG:boto:auth_path=/dgbpxg-57/h 2024-04-12T03:33:18.183 DEBUG:boto:Method: GET 2024-04-12T03:33:18.183 DEBUG:boto:Path: /dgbpxg-57/h 2024-04-12T03:33:18.183 DEBUG:boto:Data: 2024-04-12T03:33:18.183 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.183 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.184 DEBUG:boto:Params: {} 2024-04-12T03:33:18.184 DEBUG:boto:Token: None 2024-04-12T03:33:18.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/h 2024-04-12T03:33:18.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo= 2024-04-12T03:33:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo=', 'Content-Length': '0'} 2024-04-12T03:33:18.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d973a7b90bc224-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.186 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.186 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.186 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.186 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.186 DEBUG:boto:Method: GET 2024-04-12T03:33:18.186 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:18.186 DEBUG:boto:Data: 2024-04-12T03:33:18.186 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:18.187 DEBUG:boto:Port: 8001 2024-04-12T03:33:18.187 DEBUG:boto:Params: {} 2024-04-12T03:33:18.187 DEBUG:boto:Token: None 2024-04-12T03:33:18.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009277745277103920-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.230 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.231 DEBUG:boto:path=/dgbpxg-57/ 2024-04-12T03:33:18.231 DEBUG:boto:auth_path=/dgbpxg-57/ 2024-04-12T03:33:18.231 DEBUG:boto:path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.231 DEBUG:boto:auth_path=/dgbpxg-57/?prefix=h 2024-04-12T03:33:18.231 DEBUG:boto:Method: GET 2024-04-12T03:33:18.231 DEBUG:boto:Path: /dgbpxg-57/?prefix=h 2024-04-12T03:33:18.231 DEBUG:boto:Data: 2024-04-12T03:33:18.232 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.232 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.232 DEBUG:boto:Params: {} 2024-04-12T03:33:18.232 DEBUG:boto:Token: None 2024-04-12T03:33:18.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-57/ 2024-04-12T03:33:18.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T03:33:18.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T03:33:18.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78589a4620bc07c-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:18.234 DEBUG:boto:b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:18.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T03:33:18.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.235 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:18.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:18.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:18.319 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.317+0000 7fdabbf98700 -1 req 13533205992890308817 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.325 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.324+0000 7fdadc7d9700 -1 req 3357766871182794852 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.331 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.329+0000 7fdacc7b9700 -1 req 18321812564828740639 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.369 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.367+0000 7fda436a7700 -1 req 12000507695627104626 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.370 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.368+0000 7fda40ea2700 -1 req 14230048013344398153 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.372 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.371+0000 7fda5ceda700 -1 req 11522024088187569265 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.411 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.409+0000 7fda73707700 -1 req 14664376769477074816 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.415 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.413+0000 7fdae9ff4700 -1 req 7644102589494978202 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.475 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:18.559 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:18.559 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:18.599 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2024-04-12T03:33:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:33:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.604 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:33:18.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.606 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:18.648 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:18.649 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:18.688 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.686+0000 7fdb0e03c700 -1 req 5212105698036019351 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.688 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.686+0000 7fda39693700 -1 req 8275274726849920892 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.694 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.693+0000 7fda1ae56700 -1 req 16794667982912389458 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.740 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.738+0000 7fdad6fce700 -1 req 15620137926626652178 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.742 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.740+0000 7fdaa1f64700 -1 req 16042163566763317451 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.785 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:33:18.783+0000 7fda86f2e700 -1 req 12496473277467897360 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T03:33:18.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:18.848 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:18.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:18.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2024-04-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2024-04-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:18.976 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.977 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:33:18.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:18.979 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:33:18.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-12T03:33:18.984 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:18.984 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:18.984 DEBUG:boto:Method: PUT 2024-04-12T03:33:18.985 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:18.985 DEBUG:boto:Data: 2024-04-12T03:33:18.985 DEBUG:boto:Headers: {} 2024-04-12T03:33:18.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:18.985 DEBUG:boto:Port: 8000 2024-04-12T03:33:18.985 DEBUG:boto:Params: {} 2024-04-12T03:33:18.985 DEBUG:boto:Token: None 2024-04-12T03:33:18.985 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:18 GMT /dgbpxg-58/ 2024-04-12T03:33:18.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hB40GzvPRbX7TDgxLD7vNLg7f6s= 2024-04-12T03:33:18.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hB40GzvPRbX7TDgxLD7vNLg7f6s=', 'Content-Length': '0'} 2024-04-12T03:33:19.001 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077d80968b1453686-006618ab7e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:19.001 DEBUG:rgw_multi.tests:created bucket=dgbpxg-58 2024-04-12T03:33:19.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:19.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:19.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:19.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:19.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:19.081 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:33:19.093 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.094 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892798.999316_1051.1", 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:33:18.999316Z" 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.095 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:33:19.096 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892798.999316_1051.1'} 2024-04-12T03:33:19.096 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:33:19.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:19.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:19.096 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:19.144 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:19.144 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:19.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:19.172+0000 7f8ce1cb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:19.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:19.173+0000 7f8ce1cb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:33:19.187 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:19.188 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:33:19.189 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:19.190 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:19.191 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892798.323643_1046.1", 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:33:18.323643Z", 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:33:19.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:33:19Z" 2024-04-12T03:33:19.193 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:33:19.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892798.323643_1046.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:18.323643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:33:19Z" } 2024-04-12T03:33:19.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892798.323643_1046.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:18.323643Z', 'realm_epoch': 2}}] 2024-04-12T03:33:19.194 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892798.999316_1051.1'} 2024-04-12T03:33:19.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892798.323643_1046.1'} 2024-04-12T03:33:19.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892798.999316_1051.1 target=1_1712892798.323643_1046.1 2024-04-12T03:33:24.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:24.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:24.196 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:24.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:24.283+0000 7f48c72f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:24.285+0000 7f48c72f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:24.299 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:33:24.299 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:33:24.299 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:33:24.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:33:24.300 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:33:24.301 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:33:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892798.999316_1051.1", 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:33:18.999316Z", 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:33:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:33:24.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:33:24.305 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:33:24Z" 2024-04-12T03:33:24.305 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:33:24.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892798.999316_1051.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:18.999316Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:33:24Z" } 2024-04-12T03:33:24.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892798.999316_1051.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:18.999316Z', 'realm_epoch': 2}}] 2024-04-12T03:33:24.306 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892798.999316_1051.1'} 2024-04-12T03:33:24.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892798.999316_1051.1'} 2024-04-12T03:33:24.306 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:33:24.306 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:24.306 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:24.306 DEBUG:boto:Method: HEAD 2024-04-12T03:33:24.307 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:24.307 DEBUG:boto:Data: 2024-04-12T03:33:24.307 DEBUG:boto:Headers: {} 2024-04-12T03:33:24.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.307 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.307 DEBUG:boto:Params: {} 2024-04-12T03:33:24.307 DEBUG:boto:Token: None 2024-04-12T03:33:24.307 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/ 2024-04-12T03:33:24.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I= 2024-04-12T03:33:24.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I=', 'Content-Length': '0'} 2024-04-12T03:33:24.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dddef28738aa48e-006618ab84-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.310 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:24.310 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:24.310 DEBUG:boto:Method: PUT 2024-04-12T03:33:24.311 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:33:24.311 DEBUG:boto:Data: 2024-04-12T03:33:24.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:24.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.311 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.311 DEBUG:boto:Params: {} 2024-04-12T03:33:24.311 DEBUG:boto:Token: None 2024-04-12T03:33:24.311 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/a 2024-04-12T03:33:24.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RI6AsNDsraraUcHvHvo8/l0aKsM= 2024-04-12T03:33:24.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RI6AsNDsraraUcHvHvo8/l0aKsM='} 2024-04-12T03:33:24.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000731f7910620e2e99-006618ab84-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.318 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:24.318 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:24.318 DEBUG:boto:Method: HEAD 2024-04-12T03:33:24.318 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:24.318 DEBUG:boto:Data: 2024-04-12T03:33:24.318 DEBUG:boto:Headers: {} 2024-04-12T03:33:24.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.318 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.318 DEBUG:boto:Params: {} 2024-04-12T03:33:24.319 DEBUG:boto:Token: None 2024-04-12T03:33:24.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/ 2024-04-12T03:33:24.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I= 2024-04-12T03:33:24.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I=', 'Content-Length': '0'} 2024-04-12T03:33:24.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052cb68758541adfc-006618ab84-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.321 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:24.321 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:24.321 DEBUG:boto:Method: PUT 2024-04-12T03:33:24.321 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:33:24.321 DEBUG:boto:Data: 2024-04-12T03:33:24.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:24.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.321 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.322 DEBUG:boto:Params: {} 2024-04-12T03:33:24.322 DEBUG:boto:Token: None 2024-04-12T03:33:24.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/b 2024-04-12T03:33:24.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jemJ4TmV2rArSxBxcn/oy66U6CY= 2024-04-12T03:33:24.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jemJ4TmV2rArSxBxcn/oy66U6CY='} 2024-04-12T03:33:24.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c892e61d264e36e-006618ab84-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.367 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:24.367 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:24.367 DEBUG:boto:Method: HEAD 2024-04-12T03:33:24.367 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:24.367 DEBUG:boto:Data: 2024-04-12T03:33:24.367 DEBUG:boto:Headers: {} 2024-04-12T03:33:24.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.367 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.367 DEBUG:boto:Params: {} 2024-04-12T03:33:24.368 DEBUG:boto:Token: None 2024-04-12T03:33:24.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/ 2024-04-12T03:33:24.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I= 2024-04-12T03:33:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I=', 'Content-Length': '0'} 2024-04-12T03:33:24.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d01261088cfacbdf-006618ab84-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.370 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:24.370 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:24.371 DEBUG:boto:Method: PUT 2024-04-12T03:33:24.371 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:33:24.371 DEBUG:boto:Data: 2024-04-12T03:33:24.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:24.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.371 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.371 DEBUG:boto:Params: {} 2024-04-12T03:33:24.371 DEBUG:boto:Token: None 2024-04-12T03:33:24.371 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/c 2024-04-12T03:33:24.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:64+yJVf6Yxz+9fQSSEE1mdDwKQs= 2024-04-12T03:33:24.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:64+yJVf6Yxz+9fQSSEE1mdDwKQs='} 2024-04-12T03:33:24.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c754628bccfaee1-006618ab84-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.377 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:24.377 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:24.377 DEBUG:boto:Method: HEAD 2024-04-12T03:33:24.377 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:24.377 DEBUG:boto:Data: 2024-04-12T03:33:24.377 DEBUG:boto:Headers: {} 2024-04-12T03:33:24.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.377 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.377 DEBUG:boto:Params: {} 2024-04-12T03:33:24.377 DEBUG:boto:Token: None 2024-04-12T03:33:24.378 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/ 2024-04-12T03:33:24.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I= 2024-04-12T03:33:24.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ii5zKGPf6C8ES+rbhBXM7n5Xu3I=', 'Content-Length': '0'} 2024-04-12T03:33:24.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cb4f54c65d56cb5-006618ab84-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.380 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:24.380 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:24.380 DEBUG:boto:Method: PUT 2024-04-12T03:33:24.380 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:33:24.380 DEBUG:boto:Data: 2024-04-12T03:33:24.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:24.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:24.381 DEBUG:boto:Port: 8000 2024-04-12T03:33:24.381 DEBUG:boto:Params: {} 2024-04-12T03:33:24.381 DEBUG:boto:Token: None 2024-04-12T03:33:24.381 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:24 GMT /dgbpxg-58/d 2024-04-12T03:33:24.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f8NGd6fuPRmF+pDNWjuU2snoAos= 2024-04-12T03:33:24.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f8NGd6fuPRmF+pDNWjuU2snoAos='} 2024-04-12T03:33:24.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c77f6b57e2a4ca-006618ab84-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:24.386 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-58 2024-04-12T03:33:24.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:24.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:24.387 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:24.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:24.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:24.476+0000 7fee9bc6b780 1 waiting to reach incremental sync.. 2024-04-12T03:33:29.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:29.476+0000 7fee9bc6b780 1 waiting to reach incremental sync.. 2024-04-12T03:33:34.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:34.477+0000 7fee9bc6b780 1 waiting to reach incremental sync.. 2024-04-12T03:33:39.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:39.479+0000 7fee9bc6b780 1 bucket sync caught up with source: 2024-04-12T03:33:39.480 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.49.6, , , , 00000000001.53.6, , , 00000000001.51.6, , 00000000001.22.6] 2024-04-12T03:33:39.480 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.49.6, , , , 00000000001.53.6, , , 00000000001.51.6, , 00000000001.22.6] 2024-04-12T03:33:39.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:39.479+0000 7fee9bc6b780 0 bucket checkpoint complete 2024-04-12T03:33:39.496 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-58 2024-04-12T03:33:39.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:39.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:39.496 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:39.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:39.552 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:39.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:39.587+0000 7f681e478780 1 waiting to reach incremental sync.. 2024-04-12T03:33:44.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:44.589+0000 7f681e478780 1 bucket sync caught up with source: 2024-04-12T03:33:44.590 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.441.6, , , , 00000000001.442.6, , , 00000000001.440.6, , 00000000001.2071.6] 2024-04-12T03:33:44.590 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.441.6, , , , 00000000001.442.6, , , 00000000001.440.6, , 00000000001.2071.6] 2024-04-12T03:33:44.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:44.589+0000 7f681e478780 0 bucket checkpoint complete 2024-04-12T03:33:44.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-58 zones={a2, a1} 2024-04-12T03:33:44.604 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.604 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.604 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.604 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:44.604 DEBUG:boto:Data: 2024-04-12T03:33:44.604 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.605 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.605 DEBUG:boto:Params: {} 2024-04-12T03:33:44.605 DEBUG:boto:Token: None 2024-04-12T03:33:44.605 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MDxnbxG2mBBrGBO1BAmMX3wIy+M= 2024-04-12T03:33:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MDxnbxG2mBBrGBO1BAmMX3wIy+M=', 'Content-Length': '0'} 2024-04-12T03:33:44.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f89d83f0f81047d0-006618ab98-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.608 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.608 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.609 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.609 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:44.609 DEBUG:boto:Data: 2024-04-12T03:33:44.609 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.609 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.609 DEBUG:boto:Params: {} 2024-04-12T03:33:44.609 DEBUG:boto:Token: None 2024-04-12T03:33:44.609 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.610 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MDxnbxG2mBBrGBO1BAmMX3wIy+M= 2024-04-12T03:33:44.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MDxnbxG2mBBrGBO1BAmMX3wIy+M=', 'Content-Length': '0'} 2024-04-12T03:33:44.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac5831a772e9cee3-006618ab98-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:33:44.612 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.612 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.612 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.612 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.612 DEBUG:boto:Method: GET 2024-04-12T03:33:44.612 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.612 DEBUG:boto:Data: 2024-04-12T03:33:44.613 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.613 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.613 DEBUG:boto:Params: {} 2024-04-12T03:33:44.613 DEBUG:boto:Token: None 2024-04-12T03:33:44.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/?versions 2024-04-12T03:33:44.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY= 2024-04-12T03:33:44.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY=', 'Content-Length': '0'} 2024-04-12T03:33:44.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07c989b461f7843-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.616 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.617 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:44.617 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:44.617 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:44.617 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:44.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:33:44.617 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.617 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.617 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.617 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.617 DEBUG:boto:Method: GET 2024-04-12T03:33:44.618 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.618 DEBUG:boto:Data: 2024-04-12T03:33:44.618 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.618 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.618 DEBUG:boto:Params: {} 2024-04-12T03:33:44.618 DEBUG:boto:Token: None 2024-04-12T03:33:44.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/?versions 2024-04-12T03:33:44.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY= 2024-04-12T03:33:44.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY=', 'Content-Length': '0'} 2024-04-12T03:33:44.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253341a77b6a5a5f-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.621 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.621 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:33:44.621 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:33:44.621 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:33:44.621 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:33:44.621 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.622 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.622 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.622 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.622 DEBUG:boto:Method: GET 2024-04-12T03:33:44.622 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.622 DEBUG:boto:Data: 2024-04-12T03:33:44.622 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.622 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.622 DEBUG:boto:Params: {} 2024-04-12T03:33:44.622 DEBUG:boto:Token: None 2024-04-12T03:33:44.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/?versions 2024-04-12T03:33:44.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY= 2024-04-12T03:33:44.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY=', 'Content-Length': '0'} 2024-04-12T03:33:44.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b69938142fff7bea-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.625 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.625 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.625 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.626 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.626 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.626 DEBUG:boto:Method: GET 2024-04-12T03:33:44.626 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:33:44.626 DEBUG:boto:Data: 2024-04-12T03:33:44.626 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.626 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.626 DEBUG:boto:Params: {} 2024-04-12T03:33:44.626 DEBUG:boto:Token: None 2024-04-12T03:33:44.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/?versions 2024-04-12T03:33:44.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY= 2024-04-12T03:33:44.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pQ2Ao0CIFbl/YWb5l047UVCRTYY=', 'Content-Length': '0'} 2024-04-12T03:33:44.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055ea8489d79e4c9d-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.629 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:44.630 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.630 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.630 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.630 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.630 DEBUG:boto:Method: GET 2024-04-12T03:33:44.630 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.630 DEBUG:boto:Data: 2024-04-12T03:33:44.631 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.631 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.631 DEBUG:boto:Params: {} 2024-04-12T03:33:44.631 DEBUG:boto:Token: None 2024-04-12T03:33:44.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.631 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw= 2024-04-12T03:33:44.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw=', 'Content-Length': '0'} 2024-04-12T03:33:44.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d779cf8ba117ad-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.633 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.633 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.633 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.634 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.634 DEBUG:boto:Method: GET 2024-04-12T03:33:44.634 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.634 DEBUG:boto:Data: 2024-04-12T03:33:44.634 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.634 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.634 DEBUG:boto:Params: {} 2024-04-12T03:33:44.634 DEBUG:boto:Token: None 2024-04-12T03:33:44.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.635 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw= 2024-04-12T03:33:44.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw=', 'Content-Length': '0'} 2024-04-12T03:33:44.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59ea9fc70dcc418-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.678 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.679 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.679 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.679 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.679 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.679 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.679 DEBUG:boto:Data: 2024-04-12T03:33:44.679 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.679 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.679 DEBUG:boto:Params: {} 2024-04-12T03:33:44.679 DEBUG:boto:Token: None 2024-04-12T03:33:44.680 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZr01Jz+MjjQMzG08IODnQO+q+c= 2024-04-12T03:33:44.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZr01Jz+MjjQMzG08IODnQO+q+c=', 'Content-Length': '0'} 2024-04-12T03:33:44.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620612a4714cfc31-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.682 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.682 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.682 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.682 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.682 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.682 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.682 DEBUG:boto:Data: 2024-04-12T03:33:44.682 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.683 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.683 DEBUG:boto:Params: {} 2024-04-12T03:33:44.683 DEBUG:boto:Token: None 2024-04-12T03:33:44.683 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.683 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TZr01Jz+MjjQMzG08IODnQO+q+c= 2024-04-12T03:33:44.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TZr01Jz+MjjQMzG08IODnQO+q+c=', 'Content-Length': '0'} 2024-04-12T03:33:44.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000434c266f6baf34b5-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:44.685 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.685 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.685 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.685 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.685 DEBUG:boto:Method: GET 2024-04-12T03:33:44.685 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.685 DEBUG:boto:Data: 2024-04-12T03:33:44.685 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.685 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.686 DEBUG:boto:Params: {} 2024-04-12T03:33:44.686 DEBUG:boto:Token: None 2024-04-12T03:33:44.686 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.686 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw= 2024-04-12T03:33:44.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw=', 'Content-Length': '0'} 2024-04-12T03:33:44.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084dd9f4b5095c081-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.688 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.688 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.688 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.688 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.688 DEBUG:boto:Method: GET 2024-04-12T03:33:44.688 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.688 DEBUG:boto:Data: 2024-04-12T03:33:44.689 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.689 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.689 DEBUG:boto:Params: {} 2024-04-12T03:33:44.689 DEBUG:boto:Token: None 2024-04-12T03:33:44.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:33:44.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw= 2024-04-12T03:33:44.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJ1bVBdo1hAILRrup9/OfVuiTxw=', 'Content-Length': '0'} 2024-04-12T03:33:44.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183b451ad57f254f-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.691 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.691 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.691 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.691 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.691 DEBUG:boto:Method: GET 2024-04-12T03:33:44.691 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:33:44.692 DEBUG:boto:Data: 2024-04-12T03:33:44.692 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.692 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.692 DEBUG:boto:Params: {} 2024-04-12T03:33:44.692 DEBUG:boto:Token: None 2024-04-12T03:33:44.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002699594c21065edb-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.695 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.695 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.695 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.695 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.695 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.695 DEBUG:boto:Method: GET 2024-04-12T03:33:44.695 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:33:44.696 DEBUG:boto:Data: 2024-04-12T03:33:44.696 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.696 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.696 DEBUG:boto:Params: {} 2024-04-12T03:33:44.696 DEBUG:boto:Token: None 2024-04-12T03:33:44.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001297c9c60c20f9f4-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.699 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.699 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.699 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.699 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.699 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:33:44.699 DEBUG:boto:Data: 2024-04-12T03:33:44.699 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.700 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.700 DEBUG:boto:Params: {} 2024-04-12T03:33:44.700 DEBUG:boto:Token: None 2024-04-12T03:33:44.700 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a 2024-04-12T03:33:44.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d4oIl6DZexgVW7wjDTNnSEsO+Rk= 2024-04-12T03:33:44.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d4oIl6DZexgVW7wjDTNnSEsO+Rk=', 'Content-Length': '0'} 2024-04-12T03:33:44.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f043067fc11119-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.702 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.702 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.702 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.702 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:33:44.702 DEBUG:boto:Data: 2024-04-12T03:33:44.702 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.702 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.702 DEBUG:boto:Params: {} 2024-04-12T03:33:44.703 DEBUG:boto:Token: None 2024-04-12T03:33:44.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a 2024-04-12T03:33:44.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d4oIl6DZexgVW7wjDTNnSEsO+Rk= 2024-04-12T03:33:44.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d4oIl6DZexgVW7wjDTNnSEsO+Rk=', 'Content-Length': '0'} 2024-04-12T03:33:44.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545a95473c941a4a-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:33:44.705 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.705 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.705 DEBUG:boto:Method: GET 2024-04-12T03:33:44.705 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:33:44.705 DEBUG:boto:Data: 2024-04-12T03:33:44.705 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.705 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.705 DEBUG:boto:Params: {} 2024-04-12T03:33:44.705 DEBUG:boto:Token: None 2024-04-12T03:33:44.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a 2024-04-12T03:33:44.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cv6MDUy740ZHHZX8wL/XTECSwSQ= 2024-04-12T03:33:44.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cv6MDUy740ZHHZX8wL/XTECSwSQ=', 'Content-Length': '0'} 2024-04-12T03:33:44.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f697f5270b4b31-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.750 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:33:44.750 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:33:44.751 DEBUG:boto:Method: GET 2024-04-12T03:33:44.751 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:33:44.751 DEBUG:boto:Data: 2024-04-12T03:33:44.751 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.751 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.751 DEBUG:boto:Params: {} 2024-04-12T03:33:44.751 DEBUG:boto:Token: None 2024-04-12T03:33:44.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/a 2024-04-12T03:33:44.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cv6MDUy740ZHHZX8wL/XTECSwSQ= 2024-04-12T03:33:44.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cv6MDUy740ZHHZX8wL/XTECSwSQ=', 'Content-Length': '0'} 2024-04-12T03:33:44.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003771121ae2f3c8f8-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.754 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.754 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.754 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.754 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.754 DEBUG:boto:Method: GET 2024-04-12T03:33:44.754 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:33:44.754 DEBUG:boto:Data: 2024-04-12T03:33:44.754 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.755 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.755 DEBUG:boto:Params: {} 2024-04-12T03:33:44.755 DEBUG:boto:Token: None 2024-04-12T03:33:44.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092eae3ac2a6935f5-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.757 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.758 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.758 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.758 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.758 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:33:44.758 DEBUG:boto:Method: GET 2024-04-12T03:33:44.758 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:33:44.758 DEBUG:boto:Data: 2024-04-12T03:33:44.759 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.759 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.759 DEBUG:boto:Params: {} 2024-04-12T03:33:44.759 DEBUG:boto:Token: None 2024-04-12T03:33:44.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648a51a857c641ee-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.762 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:44.762 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.762 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.762 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.762 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.763 DEBUG:boto:Method: GET 2024-04-12T03:33:44.763 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.763 DEBUG:boto:Data: 2024-04-12T03:33:44.763 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.763 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.763 DEBUG:boto:Params: {} 2024-04-12T03:33:44.763 DEBUG:boto:Token: None 2024-04-12T03:33:44.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o= 2024-04-12T03:33:44.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o=', 'Content-Length': '0'} 2024-04-12T03:33:44.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9b49e6bbb4f0c4-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.765 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.766 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.766 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.766 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.766 DEBUG:boto:Method: GET 2024-04-12T03:33:44.766 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.766 DEBUG:boto:Data: 2024-04-12T03:33:44.766 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.766 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.766 DEBUG:boto:Params: {} 2024-04-12T03:33:44.767 DEBUG:boto:Token: None 2024-04-12T03:33:44.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o= 2024-04-12T03:33:44.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o=', 'Content-Length': '0'} 2024-04-12T03:33:44.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000831e301bd95a4c7b-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.768 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.769 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.769 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.769 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.769 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.769 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.769 DEBUG:boto:Data: 2024-04-12T03:33:44.769 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.769 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.769 DEBUG:boto:Params: {} 2024-04-12T03:33:44.769 DEBUG:boto:Token: None 2024-04-12T03:33:44.770 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hmP9rX5oNs/Bqyn2J7RYdmIcfFY= 2024-04-12T03:33:44.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hmP9rX5oNs/Bqyn2J7RYdmIcfFY=', 'Content-Length': '0'} 2024-04-12T03:33:44.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fce546146d4b825-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.771 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.772 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.772 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.772 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.772 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.772 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.772 DEBUG:boto:Data: 2024-04-12T03:33:44.772 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.772 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.772 DEBUG:boto:Params: {} 2024-04-12T03:33:44.772 DEBUG:boto:Token: None 2024-04-12T03:33:44.773 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hmP9rX5oNs/Bqyn2J7RYdmIcfFY= 2024-04-12T03:33:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hmP9rX5oNs/Bqyn2J7RYdmIcfFY=', 'Content-Length': '0'} 2024-04-12T03:33:44.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052518b0e05bbb714-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:44.775 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.775 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.775 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.775 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.775 DEBUG:boto:Method: GET 2024-04-12T03:33:44.775 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.775 DEBUG:boto:Data: 2024-04-12T03:33:44.775 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.775 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.775 DEBUG:boto:Params: {} 2024-04-12T03:33:44.776 DEBUG:boto:Token: None 2024-04-12T03:33:44.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o= 2024-04-12T03:33:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o=', 'Content-Length': '0'} 2024-04-12T03:33:44.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad2e352ab4c9ff0-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.778 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.778 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.778 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.778 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.778 DEBUG:boto:Method: GET 2024-04-12T03:33:44.778 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.778 DEBUG:boto:Data: 2024-04-12T03:33:44.778 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.778 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.779 DEBUG:boto:Params: {} 2024-04-12T03:33:44.779 DEBUG:boto:Token: None 2024-04-12T03:33:44.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:33:44.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o= 2024-04-12T03:33:44.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YaevnUu3fkqWKv9O1sBQpucID6o=', 'Content-Length': '0'} 2024-04-12T03:33:44.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93effa1c9089608-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.781 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.781 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.781 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.781 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.781 DEBUG:boto:Method: GET 2024-04-12T03:33:44.781 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:33:44.781 DEBUG:boto:Data: 2024-04-12T03:33:44.781 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.782 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.782 DEBUG:boto:Params: {} 2024-04-12T03:33:44.782 DEBUG:boto:Token: None 2024-04-12T03:33:44.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b943744b137aad-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.784 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.785 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.785 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.785 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.785 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.785 DEBUG:boto:Method: GET 2024-04-12T03:33:44.785 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:33:44.785 DEBUG:boto:Data: 2024-04-12T03:33:44.785 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.786 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.786 DEBUG:boto:Params: {} 2024-04-12T03:33:44.786 DEBUG:boto:Token: None 2024-04-12T03:33:44.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb217b2a19b7db75-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.789 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.789 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.789 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.789 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.789 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:33:44.789 DEBUG:boto:Data: 2024-04-12T03:33:44.789 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.790 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.790 DEBUG:boto:Params: {} 2024-04-12T03:33:44.790 DEBUG:boto:Token: None 2024-04-12T03:33:44.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b 2024-04-12T03:33:44.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZw0rTd3J2IH8Vki0ufi44MYBZE= 2024-04-12T03:33:44.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZw0rTd3J2IH8Vki0ufi44MYBZE=', 'Content-Length': '0'} 2024-04-12T03:33:44.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007174ec14209c3666-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.792 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.792 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.792 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.792 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:33:44.792 DEBUG:boto:Data: 2024-04-12T03:33:44.792 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.793 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.793 DEBUG:boto:Params: {} 2024-04-12T03:33:44.793 DEBUG:boto:Token: None 2024-04-12T03:33:44.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b 2024-04-12T03:33:44.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZw0rTd3J2IH8Vki0ufi44MYBZE= 2024-04-12T03:33:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZw0rTd3J2IH8Vki0ufi44MYBZE=', 'Content-Length': '0'} 2024-04-12T03:33:44.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078064632e7be2719-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:33:44.795 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.795 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.795 DEBUG:boto:Method: GET 2024-04-12T03:33:44.795 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:33:44.795 DEBUG:boto:Data: 2024-04-12T03:33:44.795 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.796 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.796 DEBUG:boto:Params: {} 2024-04-12T03:33:44.796 DEBUG:boto:Token: None 2024-04-12T03:33:44.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b 2024-04-12T03:33:44.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ekPSQVFAw2bZY5ESUIKFgVFx+3Q= 2024-04-12T03:33:44.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ekPSQVFAw2bZY5ESUIKFgVFx+3Q=', 'Content-Length': '0'} 2024-04-12T03:33:44.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c095308c02fb027f-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.798 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:33:44.798 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:33:44.798 DEBUG:boto:Method: GET 2024-04-12T03:33:44.798 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:33:44.798 DEBUG:boto:Data: 2024-04-12T03:33:44.798 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.799 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.799 DEBUG:boto:Params: {} 2024-04-12T03:33:44.799 DEBUG:boto:Token: None 2024-04-12T03:33:44.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/b 2024-04-12T03:33:44.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ekPSQVFAw2bZY5ESUIKFgVFx+3Q= 2024-04-12T03:33:44.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ekPSQVFAw2bZY5ESUIKFgVFx+3Q=', 'Content-Length': '0'} 2024-04-12T03:33:44.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ab6e3cd0fae4d0-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.801 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.801 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.801 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.801 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.801 DEBUG:boto:Method: GET 2024-04-12T03:33:44.801 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:33:44.801 DEBUG:boto:Data: 2024-04-12T03:33:44.802 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.802 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.802 DEBUG:boto:Params: {} 2024-04-12T03:33:44.802 DEBUG:boto:Token: None 2024-04-12T03:33:44.802 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.802 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e9517c16242ac84-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.805 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.805 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.805 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.805 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.805 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:33:44.805 DEBUG:boto:Method: GET 2024-04-12T03:33:44.805 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:33:44.805 DEBUG:boto:Data: 2024-04-12T03:33:44.806 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.806 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.806 DEBUG:boto:Params: {} 2024-04-12T03:33:44.806 DEBUG:boto:Token: None 2024-04-12T03:33:44.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0be03b172d3fa5a-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.808 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:44.809 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.809 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.809 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.809 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.809 DEBUG:boto:Method: GET 2024-04-12T03:33:44.809 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.809 DEBUG:boto:Data: 2024-04-12T03:33:44.809 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.810 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.810 DEBUG:boto:Params: {} 2024-04-12T03:33:44.810 DEBUG:boto:Token: None 2024-04-12T03:33:44.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY= 2024-04-12T03:33:44.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY=', 'Content-Length': '0'} 2024-04-12T03:33:44.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d241422babbccb3a-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.812 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.812 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.812 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.812 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.812 DEBUG:boto:Method: GET 2024-04-12T03:33:44.812 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.813 DEBUG:boto:Data: 2024-04-12T03:33:44.813 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.813 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.813 DEBUG:boto:Params: {} 2024-04-12T03:33:44.813 DEBUG:boto:Token: None 2024-04-12T03:33:44.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.813 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY= 2024-04-12T03:33:44.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY=', 'Content-Length': '0'} 2024-04-12T03:33:44.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ec783261df5eeb-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.815 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.815 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.815 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.816 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.816 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.816 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.816 DEBUG:boto:Data: 2024-04-12T03:33:44.816 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.816 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.816 DEBUG:boto:Params: {} 2024-04-12T03:33:44.816 DEBUG:boto:Token: None 2024-04-12T03:33:44.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.817 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wkNe45XoyWCsLD/HMjpR2W7lkwk= 2024-04-12T03:33:44.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wkNe45XoyWCsLD/HMjpR2W7lkwk=', 'Content-Length': '0'} 2024-04-12T03:33:44.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277bd6c148b18295-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.818 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.819 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.819 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.819 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.819 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.819 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.819 DEBUG:boto:Data: 2024-04-12T03:33:44.819 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.819 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.819 DEBUG:boto:Params: {} 2024-04-12T03:33:44.820 DEBUG:boto:Token: None 2024-04-12T03:33:44.820 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wkNe45XoyWCsLD/HMjpR2W7lkwk= 2024-04-12T03:33:44.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wkNe45XoyWCsLD/HMjpR2W7lkwk=', 'Content-Length': '0'} 2024-04-12T03:33:44.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16032dfbfdabc01-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:44.822 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.822 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.822 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.822 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.822 DEBUG:boto:Method: GET 2024-04-12T03:33:44.822 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.822 DEBUG:boto:Data: 2024-04-12T03:33:44.822 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.822 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.823 DEBUG:boto:Params: {} 2024-04-12T03:33:44.823 DEBUG:boto:Token: None 2024-04-12T03:33:44.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY= 2024-04-12T03:33:44.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY=', 'Content-Length': '0'} 2024-04-12T03:33:44.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54cfe384e5c3395-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.825 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.825 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.825 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.825 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.825 DEBUG:boto:Method: GET 2024-04-12T03:33:44.825 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.825 DEBUG:boto:Data: 2024-04-12T03:33:44.825 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.826 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.826 DEBUG:boto:Params: {} 2024-04-12T03:33:44.826 DEBUG:boto:Token: None 2024-04-12T03:33:44.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:33:44.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY= 2024-04-12T03:33:44.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tTQmTrviY9OTzWRXjY0hmbiccoY=', 'Content-Length': '0'} 2024-04-12T03:33:44.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a0b3b6c66bf75a-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.870 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.871 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.871 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.871 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.871 DEBUG:boto:Method: GET 2024-04-12T03:33:44.871 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:33:44.871 DEBUG:boto:Data: 2024-04-12T03:33:44.871 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.872 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.872 DEBUG:boto:Params: {} 2024-04-12T03:33:44.872 DEBUG:boto:Token: None 2024-04-12T03:33:44.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c44e3392c30dc3d-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.875 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.875 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.875 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.875 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.876 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.876 DEBUG:boto:Method: GET 2024-04-12T03:33:44.876 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:33:44.876 DEBUG:boto:Data: 2024-04-12T03:33:44.876 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.877 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.877 DEBUG:boto:Params: {} 2024-04-12T03:33:44.877 DEBUG:boto:Token: None 2024-04-12T03:33:44.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880538df7d9f262b-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.880 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.880 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.880 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.880 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.880 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:33:44.880 DEBUG:boto:Data: 2024-04-12T03:33:44.880 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.881 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.881 DEBUG:boto:Params: {} 2024-04-12T03:33:44.881 DEBUG:boto:Token: None 2024-04-12T03:33:44.881 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c 2024-04-12T03:33:44.881 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w5Al16QwB4QsBQNMOOHXby1kodw= 2024-04-12T03:33:44.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w5Al16QwB4QsBQNMOOHXby1kodw=', 'Content-Length': '0'} 2024-04-12T03:33:44.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003805c859a732b4c7-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.883 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.883 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.883 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.883 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:33:44.883 DEBUG:boto:Data: 2024-04-12T03:33:44.883 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.883 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.883 DEBUG:boto:Params: {} 2024-04-12T03:33:44.884 DEBUG:boto:Token: None 2024-04-12T03:33:44.884 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c 2024-04-12T03:33:44.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w5Al16QwB4QsBQNMOOHXby1kodw= 2024-04-12T03:33:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w5Al16QwB4QsBQNMOOHXby1kodw=', 'Content-Length': '0'} 2024-04-12T03:33:44.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b74ab76c377e39f-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:33:44.886 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.886 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.886 DEBUG:boto:Method: GET 2024-04-12T03:33:44.886 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:33:44.886 DEBUG:boto:Data: 2024-04-12T03:33:44.886 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.886 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.886 DEBUG:boto:Params: {} 2024-04-12T03:33:44.886 DEBUG:boto:Token: None 2024-04-12T03:33:44.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c 2024-04-12T03:33:44.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CNfbeUKDtAM0cIO3B6dG1S2aAsU= 2024-04-12T03:33:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CNfbeUKDtAM0cIO3B6dG1S2aAsU=', 'Content-Length': '0'} 2024-04-12T03:33:44.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034471a89ff2302cf-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.930 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:33:44.930 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:33:44.931 DEBUG:boto:Method: GET 2024-04-12T03:33:44.931 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:33:44.931 DEBUG:boto:Data: 2024-04-12T03:33:44.931 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.931 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.931 DEBUG:boto:Params: {} 2024-04-12T03:33:44.931 DEBUG:boto:Token: None 2024-04-12T03:33:44.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/c 2024-04-12T03:33:44.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CNfbeUKDtAM0cIO3B6dG1S2aAsU= 2024-04-12T03:33:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CNfbeUKDtAM0cIO3B6dG1S2aAsU=', 'Content-Length': '0'} 2024-04-12T03:33:44.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce466bfba9c5ff9-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.933 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.934 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.934 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.934 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.934 DEBUG:boto:Method: GET 2024-04-12T03:33:44.934 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:33:44.934 DEBUG:boto:Data: 2024-04-12T03:33:44.934 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.934 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.934 DEBUG:boto:Params: {} 2024-04-12T03:33:44.935 DEBUG:boto:Token: None 2024-04-12T03:33:44.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3d00214ab4a4d4-006618ab98-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.937 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.938 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:44.938 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:44.938 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.938 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:33:44.938 DEBUG:boto:Method: GET 2024-04-12T03:33:44.938 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:33:44.938 DEBUG:boto:Data: 2024-04-12T03:33:44.938 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.938 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.939 DEBUG:boto:Params: {} 2024-04-12T03:33:44.939 DEBUG:boto:Token: None 2024-04-12T03:33:44.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/ 2024-04-12T03:33:44.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U= 2024-04-12T03:33:44.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FU1Rj55156QvpNh8rixUR9dwN1U=', 'Content-Length': '0'} 2024-04-12T03:33:44.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3acb15f2eba913-006618ab98-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.941 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:44.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:44.942 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.942 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.942 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.942 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.942 DEBUG:boto:Method: GET 2024-04-12T03:33:44.942 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.942 DEBUG:boto:Data: 2024-04-12T03:33:44.942 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.943 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.943 DEBUG:boto:Params: {} 2024-04-12T03:33:44.943 DEBUG:boto:Token: None 2024-04-12T03:33:44.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw= 2024-04-12T03:33:44.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw=', 'Content-Length': '0'} 2024-04-12T03:33:44.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c41222cbf5df83-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.947 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.948 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.948 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.948 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.948 DEBUG:boto:Method: GET 2024-04-12T03:33:44.948 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.948 DEBUG:boto:Data: 2024-04-12T03:33:44.948 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.948 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.948 DEBUG:boto:Params: {} 2024-04-12T03:33:44.949 DEBUG:boto:Token: None 2024-04-12T03:33:44.949 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.949 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw= 2024-04-12T03:33:44.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw=', 'Content-Length': '0'} 2024-04-12T03:33:44.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019922db131a365b0-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.950 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.951 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.951 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.951 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.951 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.951 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.951 DEBUG:boto:Data: 2024-04-12T03:33:44.951 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.951 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.951 DEBUG:boto:Params: {} 2024-04-12T03:33:44.952 DEBUG:boto:Token: None 2024-04-12T03:33:44.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s6oID7tIwKCmj2YTncANv4HhEk4= 2024-04-12T03:33:44.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s6oID7tIwKCmj2YTncANv4HhEk4=', 'Content-Length': '0'} 2024-04-12T03:33:44.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4811b0b9e57c6e-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.953 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.954 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.954 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.954 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.954 DEBUG:boto:Method: HEAD 2024-04-12T03:33:44.954 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.954 DEBUG:boto:Data: 2024-04-12T03:33:44.954 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.954 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.954 DEBUG:boto:Params: {} 2024-04-12T03:33:44.955 DEBUG:boto:Token: None 2024-04-12T03:33:44.955 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s6oID7tIwKCmj2YTncANv4HhEk4= 2024-04-12T03:33:44.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s6oID7tIwKCmj2YTncANv4HhEk4=', 'Content-Length': '0'} 2024-04-12T03:33:44.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b21ef07952248ec-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:44.957 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.957 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.957 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.957 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.957 DEBUG:boto:Method: GET 2024-04-12T03:33:44.957 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.957 DEBUG:boto:Data: 2024-04-12T03:33:44.957 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:44.958 DEBUG:boto:Port: 8001 2024-04-12T03:33:44.958 DEBUG:boto:Params: {} 2024-04-12T03:33:44.958 DEBUG:boto:Token: None 2024-04-12T03:33:44.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw= 2024-04-12T03:33:44.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw=', 'Content-Length': '0'} 2024-04-12T03:33:44.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f819403fa331231f-006618ab98-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:44.960 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:44.960 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:44.960 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.960 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.960 DEBUG:boto:Method: GET 2024-04-12T03:33:44.960 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.960 DEBUG:boto:Data: 2024-04-12T03:33:44.961 DEBUG:boto:Headers: {} 2024-04-12T03:33:44.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:44.961 DEBUG:boto:Port: 8000 2024-04-12T03:33:44.961 DEBUG:boto:Params: {} 2024-04-12T03:33:44.961 DEBUG:boto:Token: None 2024-04-12T03:33:44.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:44 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:33:44.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw= 2024-04-12T03:33:44.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5/JgCCFMPHhCVmFU1jOOXN05Kw=', 'Content-Length': '0'} 2024-04-12T03:33:44.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ada5e8ba1816613-006618ab98-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.006 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.007 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.007 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.007 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.007 DEBUG:boto:Method: GET 2024-04-12T03:33:45.007 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:33:45.007 DEBUG:boto:Data: 2024-04-12T03:33:45.007 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:45.007 DEBUG:boto:Port: 8001 2024-04-12T03:33:45.007 DEBUG:boto:Params: {} 2024-04-12T03:33:45.008 DEBUG:boto:Token: None 2024-04-12T03:33:45.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q= 2024-04-12T03:33:45.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q=', 'Content-Length': '0'} 2024-04-12T03:33:45.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ceff76688f84ca4-006618ab99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.010 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:45.011 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.011 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.011 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.011 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.011 DEBUG:boto:Method: GET 2024-04-12T03:33:45.011 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:33:45.011 DEBUG:boto:Data: 2024-04-12T03:33:45.011 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.012 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.012 DEBUG:boto:Params: {} 2024-04-12T03:33:45.012 DEBUG:boto:Token: None 2024-04-12T03:33:45.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q= 2024-04-12T03:33:45.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q=', 'Content-Length': '0'} 2024-04-12T03:33:45.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662968e138a52512-006618ab99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.014 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:45.015 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:45.015 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:45.015 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.015 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:33:45.015 DEBUG:boto:Data: 2024-04-12T03:33:45.015 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:45.015 DEBUG:boto:Port: 8001 2024-04-12T03:33:45.015 DEBUG:boto:Params: {} 2024-04-12T03:33:45.015 DEBUG:boto:Token: None 2024-04-12T03:33:45.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/d 2024-04-12T03:33:45.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1SiapEF+qkoFESZtFe9VIAX78Fk= 2024-04-12T03:33:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SiapEF+qkoFESZtFe9VIAX78Fk=', 'Content-Length': '0'} 2024-04-12T03:33:45.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ef7840cdc25be3-006618ab99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.017 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:45.018 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:45.018 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.018 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:33:45.018 DEBUG:boto:Data: 2024-04-12T03:33:45.018 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.018 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.018 DEBUG:boto:Params: {} 2024-04-12T03:33:45.018 DEBUG:boto:Token: None 2024-04-12T03:33:45.018 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/d 2024-04-12T03:33:45.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1SiapEF+qkoFESZtFe9VIAX78Fk= 2024-04-12T03:33:45.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SiapEF+qkoFESZtFe9VIAX78Fk=', 'Content-Length': '0'} 2024-04-12T03:33:45.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec813406fafc63c0-006618ab99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:33:45.020 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:45.020 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:45.021 DEBUG:boto:Method: GET 2024-04-12T03:33:45.021 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:33:45.021 DEBUG:boto:Data: 2024-04-12T03:33:45.021 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:45.021 DEBUG:boto:Port: 8001 2024-04-12T03:33:45.021 DEBUG:boto:Params: {} 2024-04-12T03:33:45.021 DEBUG:boto:Token: None 2024-04-12T03:33:45.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/d 2024-04-12T03:33:45.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7QmJ18DYVy2LP080C9/iwytDvoE= 2024-04-12T03:33:45.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7QmJ18DYVy2LP080C9/iwytDvoE=', 'Content-Length': '0'} 2024-04-12T03:33:45.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002245edd430eae1bb-006618ab99-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.023 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:33:45.023 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:33:45.023 DEBUG:boto:Method: GET 2024-04-12T03:33:45.024 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:33:45.024 DEBUG:boto:Data: 2024-04-12T03:33:45.024 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.024 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.024 DEBUG:boto:Params: {} 2024-04-12T03:33:45.024 DEBUG:boto:Token: None 2024-04-12T03:33:45.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/d 2024-04-12T03:33:45.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7QmJ18DYVy2LP080C9/iwytDvoE= 2024-04-12T03:33:45.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7QmJ18DYVy2LP080C9/iwytDvoE=', 'Content-Length': '0'} 2024-04-12T03:33:45.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d764e2b0e5bbf6-006618ab99-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.026 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.026 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.026 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.026 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.026 DEBUG:boto:Method: GET 2024-04-12T03:33:45.027 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:33:45.027 DEBUG:boto:Data: 2024-04-12T03:33:45.027 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:33:45.027 DEBUG:boto:Port: 8001 2024-04-12T03:33:45.027 DEBUG:boto:Params: {} 2024-04-12T03:33:45.027 DEBUG:boto:Token: None 2024-04-12T03:33:45.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q= 2024-04-12T03:33:45.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q=', 'Content-Length': '0'} 2024-04-12T03:33:45.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30362997a71d1ae-006618ab99-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.070 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:45.071 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.071 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.071 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.071 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:33:45.071 DEBUG:boto:Method: GET 2024-04-12T03:33:45.071 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:33:45.071 DEBUG:boto:Data: 2024-04-12T03:33:45.071 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.072 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.072 DEBUG:boto:Params: {} 2024-04-12T03:33:45.072 DEBUG:boto:Token: None 2024-04-12T03:33:45.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q= 2024-04-12T03:33:45.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DiljowrPmL0IgzbggjdOzuyCh/Q=', 'Content-Length': '0'} 2024-04-12T03:33:45.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056484a5feef5fed9-006618ab99-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.075 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:45.075 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-58 zones={a2, a1} 2024-04-12T03:33:45.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:45.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:45.076 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:45.124 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:45.124 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:45.174 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:33:45.175 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-58 2024-04-12T03:33:45.179 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2024-04-12T03:33:45.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:45.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:33:45.217 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:33:45.266 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:45.304 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:33:45.304 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-58 2024-04-12T03:33:45.305 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2024-04-12T03:33:45.333 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.333 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.333 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.333 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:45.333 DEBUG:boto:Data: 2024-04-12T03:33:45.333 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.333 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.333 DEBUG:boto:Params: {} 2024-04-12T03:33:45.334 DEBUG:boto:Token: None 2024-04-12T03:33:45.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4= 2024-04-12T03:33:45.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4=', 'Content-Length': '0'} 2024-04-12T03:33:45.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcbaf9b87bd90bbb-006618ab99-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.336 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:33:45.336 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:33:45.336 DEBUG:boto:Method: PUT 2024-04-12T03:33:45.337 DEBUG:boto:Path: /dgbpxg-58/e 2024-04-12T03:33:45.337 DEBUG:boto:Data: 2024-04-12T03:33:45.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:45.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.337 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.337 DEBUG:boto:Params: {} 2024-04-12T03:33:45.337 DEBUG:boto:Token: None 2024-04-12T03:33:45.337 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/e 2024-04-12T03:33:45.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uoNJG0PdpfdVWmGdyzkrK9DyTOI= 2024-04-12T03:33:45.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uoNJG0PdpfdVWmGdyzkrK9DyTOI='} 2024-04-12T03:33:45.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093048af5526e2dcf-006618ab99-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.343 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.343 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.343 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.343 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:45.343 DEBUG:boto:Data: 2024-04-12T03:33:45.343 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.343 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.343 DEBUG:boto:Params: {} 2024-04-12T03:33:45.344 DEBUG:boto:Token: None 2024-04-12T03:33:45.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4= 2024-04-12T03:33:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4=', 'Content-Length': '0'} 2024-04-12T03:33:45.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e986a340e6793576-006618ab99-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.346 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:33:45.346 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:33:45.346 DEBUG:boto:Method: PUT 2024-04-12T03:33:45.346 DEBUG:boto:Path: /dgbpxg-58/f 2024-04-12T03:33:45.346 DEBUG:boto:Data: 2024-04-12T03:33:45.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:45.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.347 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.347 DEBUG:boto:Params: {} 2024-04-12T03:33:45.347 DEBUG:boto:Token: None 2024-04-12T03:33:45.347 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/f 2024-04-12T03:33:45.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CJ08sDMEk0yaezaNS6TBSwIEXSc= 2024-04-12T03:33:45.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CJ08sDMEk0yaezaNS6TBSwIEXSc='} 2024-04-12T03:33:45.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011e229d066c61bc3-006618ab99-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.391 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.391 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.391 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.392 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:45.392 DEBUG:boto:Data: 2024-04-12T03:33:45.392 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.392 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.392 DEBUG:boto:Params: {} 2024-04-12T03:33:45.392 DEBUG:boto:Token: None 2024-04-12T03:33:45.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4= 2024-04-12T03:33:45.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4=', 'Content-Length': '0'} 2024-04-12T03:33:45.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002709cccecca38a87-006618ab99-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.395 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:33:45.395 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:33:45.395 DEBUG:boto:Method: PUT 2024-04-12T03:33:45.395 DEBUG:boto:Path: /dgbpxg-58/g 2024-04-12T03:33:45.395 DEBUG:boto:Data: 2024-04-12T03:33:45.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:45.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.395 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.396 DEBUG:boto:Params: {} 2024-04-12T03:33:45.396 DEBUG:boto:Token: None 2024-04-12T03:33:45.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/g 2024-04-12T03:33:45.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5F7M2L4HYq0N4nRt5VdLYgiIvUA= 2024-04-12T03:33:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5F7M2L4HYq0N4nRt5VdLYgiIvUA='} 2024-04-12T03:33:45.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c10dab55ccac976-006618ab99-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.401 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:33:45.401 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:33:45.401 DEBUG:boto:Method: HEAD 2024-04-12T03:33:45.402 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:33:45.402 DEBUG:boto:Data: 2024-04-12T03:33:45.402 DEBUG:boto:Headers: {} 2024-04-12T03:33:45.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.402 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.402 DEBUG:boto:Params: {} 2024-04-12T03:33:45.402 DEBUG:boto:Token: None 2024-04-12T03:33:45.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/ 2024-04-12T03:33:45.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4= 2024-04-12T03:33:45.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sts7DE24ONrbW5NbRhlAoPuxbw4=', 'Content-Length': '0'} 2024-04-12T03:33:45.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002107971c6e85a744-006618ab99-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.404 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:33:45.404 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:33:45.404 DEBUG:boto:Method: PUT 2024-04-12T03:33:45.405 DEBUG:boto:Path: /dgbpxg-58/h 2024-04-12T03:33:45.405 DEBUG:boto:Data: 2024-04-12T03:33:45.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:33:45.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:33:45.405 DEBUG:boto:Port: 8000 2024-04-12T03:33:45.405 DEBUG:boto:Params: {} 2024-04-12T03:33:45.405 DEBUG:boto:Token: None 2024-04-12T03:33:45.405 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:33:45 GMT /dgbpxg-58/h 2024-04-12T03:33:45.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmuALbUOC2vrt3IZe6EIGGCalT8= 2024-04-12T03:33:45.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmuALbUOC2vrt3IZe6EIGGCalT8='} 2024-04-12T03:33:45.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7919cf6b36cdff2-006618ab99-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:45.451 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-58 2024-04-12T03:33:45.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:45.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:45.451 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:45.508 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:45.508 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:45.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:45.548+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:33:50.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:50.549+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:33:55.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:33:55.549+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:34:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:00.550+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:05.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:05.551+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:10.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:10.552+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:15.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:15.553+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:20.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:20.553+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:25.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:25.554+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:30.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:30.554+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:35.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:35.554+0000 7f8074dc2780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:34:40.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:40.555+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:34:40.558 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2024-04-12T03:34:40.558 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:34:45.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:45.557+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:34:45.558 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2024-04-12T03:34:45.558 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:34:50.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:50.558+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:34:50.559 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2024-04-12T03:34:50.559 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:34:55.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:34:55.559+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:34:55.560 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2024-04-12T03:34:55.560 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:00.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:00.559+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:00.561 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:00.561 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:05.560+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:05.561 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:05.561 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:10.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:10.561+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:10.562 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:10.562 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:15.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:15.562+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:15.563 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:15.563 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:20.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:20.562+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:20.564 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:20.564 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:25.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:25.563+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:25.565 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:25.565 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:30.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:30.564+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:30.565 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:30.565 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:35.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:35.565+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:35.566 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.72.6] 2024-04-12T03:35:35.566 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:40.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:40.566+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:40.567 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:35:40.567 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:45.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:45.567+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:45.568 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:35:45.568 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:50.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:50.567+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:50.569 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:35:50.569 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:35:55.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:35:55.568+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:35:55.570 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:35:55.570 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:00.568+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:36:00.570 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:36:00.570 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:05.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:05.569+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:36:05.571 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:36:05.571 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:10.571+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:36:10.572 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:36:10.572 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:15.571+0000 7f8074dc2780 1 waiting for incremental sync to catch up: 2024-04-12T03:36:15.573 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, , 00000000002.72.6] 2024-04-12T03:36:15.573 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:20.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:20.572+0000 7f8074dc2780 1 bucket sync caught up with source: 2024-04-12T03:36:20.573 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:20.573 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.29.6, 00000000002.61.6, 00000000002.72.6] 2024-04-12T03:36:20.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:20.572+0000 7f8074dc2780 0 bucket checkpoint complete 2024-04-12T03:36:20.591 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-58 2024-04-12T03:36:20.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:36:20.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:36:20.592 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:36:20.655 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:20.656 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:20.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:20.691+0000 7f88e7b4d780 1 waiting for incremental sync to catch up: 2024-04-12T03:36:20.692 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.441.6, , , 00000000001.1063.6, 00000000001.442.6, , , 00000000002.483.6, , 00000000002.2086.6] 2024-04-12T03:36:20.692 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.485.6, , , 00000000001.1063.6, 00000000001.442.6, , , 00000000002.483.6, , 00000000002.2086.6] 2024-04-12T03:36:23.812 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:36:23.811+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:36:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:25.692+0000 7f88e7b4d780 1 bucket sync caught up with source: 2024-04-12T03:36:25.694 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.485.6, , , 00000000001.1063.6, 00000000001.442.6, , , 00000000002.483.6, , 00000000002.2086.6] 2024-04-12T03:36:25.694 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.485.6, , , 00000000001.1063.6, 00000000001.442.6, , , 00000000002.483.6, , 00000000002.2086.6] 2024-04-12T03:36:25.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:25.692+0000 7f88e7b4d780 0 bucket checkpoint complete 2024-04-12T03:36:25.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-58 zones={a2, a1} 2024-04-12T03:36:25.713 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.713 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.713 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.713 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:36:25.713 DEBUG:boto:Data: 2024-04-12T03:36:25.714 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.714 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.714 DEBUG:boto:Params: {} 2024-04-12T03:36:25.714 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:36:25.714 DEBUG:boto:Token: None 2024-04-12T03:36:25.715 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R0imJQQyCc+l3fUkO7tUnq7bZWE= 2024-04-12T03:36:25.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R0imJQQyCc+l3fUkO7tUnq7bZWE=', 'Content-Length': '0'} 2024-04-12T03:36:25.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009432b8794b11e32d-006618ac39-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.745 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.745 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.745 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.745 DEBUG:boto:Path: /dgbpxg-58/ 2024-04-12T03:36:25.745 DEBUG:boto:Data: 2024-04-12T03:36:25.746 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.746 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.746 DEBUG:boto:Params: {} 2024-04-12T03:36:25.746 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:36:25.746 DEBUG:boto:Token: None 2024-04-12T03:36:25.746 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R0imJQQyCc+l3fUkO7tUnq7bZWE= 2024-04-12T03:36:25.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R0imJQQyCc+l3fUkO7tUnq7bZWE=', 'Content-Length': '0'} 2024-04-12T03:36:25.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079be0ed9f28b9e22-006618ac39-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:36:25.758 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.758 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.758 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.758 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.759 DEBUG:boto:Method: GET 2024-04-12T03:36:25.759 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.759 DEBUG:boto:Data: 2024-04-12T03:36:25.759 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.759 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.759 DEBUG:boto:Params: {} 2024-04-12T03:36:25.759 DEBUG:boto:Token: None 2024-04-12T03:36:25.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/?versions 2024-04-12T03:36:25.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0= 2024-04-12T03:36:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0=', 'Content-Length': '0'} 2024-04-12T03:36:25.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060733bb858e0165c-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.763 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:36:25.764 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:36:25.764 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.765 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.765 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.765 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.765 DEBUG:boto:Method: GET 2024-04-12T03:36:25.765 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.765 DEBUG:boto:Data: 2024-04-12T03:36:25.765 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.765 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.765 DEBUG:boto:Params: {} 2024-04-12T03:36:25.765 DEBUG:boto:Token: None 2024-04-12T03:36:25.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/?versions 2024-04-12T03:36:25.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0= 2024-04-12T03:36:25.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0=', 'Content-Length': '0'} 2024-04-12T03:36:25.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda499f64c611897-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.768 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.768 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:36:25.768 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:36:25.769 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:36:25.769 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.769 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.769 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.769 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.769 DEBUG:boto:Method: GET 2024-04-12T03:36:25.770 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.770 DEBUG:boto:Data: 2024-04-12T03:36:25.770 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.770 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.770 DEBUG:boto:Params: {} 2024-04-12T03:36:25.770 DEBUG:boto:Token: None 2024-04-12T03:36:25.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/?versions 2024-04-12T03:36:25.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0= 2024-04-12T03:36:25.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0=', 'Content-Length': '0'} 2024-04-12T03:36:25.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfb5572bd6461df-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.773 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.774 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.774 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.774 DEBUG:boto:path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.774 DEBUG:boto:auth_path=/dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.774 DEBUG:boto:Method: GET 2024-04-12T03:36:25.774 DEBUG:boto:Path: /dgbpxg-58/?versions&max-keys=999 2024-04-12T03:36:25.774 DEBUG:boto:Data: 2024-04-12T03:36:25.774 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.774 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.774 DEBUG:boto:Params: {} 2024-04-12T03:36:25.775 DEBUG:boto:Token: None 2024-04-12T03:36:25.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/?versions 2024-04-12T03:36:25.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0= 2024-04-12T03:36:25.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xt67lkkxx6D+9Y/Y1kKTLVqcNI0=', 'Content-Length': '0'} 2024-04-12T03:36:25.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b897c5f2062cd87c-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.777 DEBUG:boto:b'dgbpxg-58999falseanulltrue2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:36:25.778 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.778 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.778 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.778 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.778 DEBUG:boto:Method: GET 2024-04-12T03:36:25.778 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.778 DEBUG:boto:Data: 2024-04-12T03:36:25.778 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.778 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.779 DEBUG:boto:Params: {} 2024-04-12T03:36:25.779 DEBUG:boto:Token: None 2024-04-12T03:36:25.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878= 2024-04-12T03:36:25.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878=', 'Content-Length': '0'} 2024-04-12T03:36:25.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ca8488665c8d44-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.781 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.781 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.781 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.781 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.781 DEBUG:boto:Method: GET 2024-04-12T03:36:25.781 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.781 DEBUG:boto:Data: 2024-04-12T03:36:25.781 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.782 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.782 DEBUG:boto:Params: {} 2024-04-12T03:36:25.782 DEBUG:boto:Token: None 2024-04-12T03:36:25.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878= 2024-04-12T03:36:25.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878=', 'Content-Length': '0'} 2024-04-12T03:36:25.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71b112a279e2cbc-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.784 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.784 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.784 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.785 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.785 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.785 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.785 DEBUG:boto:Data: 2024-04-12T03:36:25.785 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.785 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.785 DEBUG:boto:Params: {} 2024-04-12T03:36:25.785 DEBUG:boto:Token: None 2024-04-12T03:36:25.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/DwP9LQzCv0bJ8x0uo6dXKU7nOM= 2024-04-12T03:36:25.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/DwP9LQzCv0bJ8x0uo6dXKU7nOM=', 'Content-Length': '0'} 2024-04-12T03:36:25.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007aa280fc9afcc7d-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.787 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.787 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.787 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.788 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.788 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.788 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.788 DEBUG:boto:Data: 2024-04-12T03:36:25.788 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.788 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.788 DEBUG:boto:Params: {} 2024-04-12T03:36:25.788 DEBUG:boto:Token: None 2024-04-12T03:36:25.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/DwP9LQzCv0bJ8x0uo6dXKU7nOM= 2024-04-12T03:36:25.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/DwP9LQzCv0bJ8x0uo6dXKU7nOM=', 'Content-Length': '0'} 2024-04-12T03:36:25.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5cdbe3ab5d4aa4-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:36:25.791 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.791 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.791 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.791 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.791 DEBUG:boto:Method: GET 2024-04-12T03:36:25.791 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.791 DEBUG:boto:Data: 2024-04-12T03:36:25.791 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.792 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.792 DEBUG:boto:Params: {} 2024-04-12T03:36:25.792 DEBUG:boto:Token: None 2024-04-12T03:36:25.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878= 2024-04-12T03:36:25.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878=', 'Content-Length': '0'} 2024-04-12T03:36:25.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286a713826b50ef0-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.794 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.794 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.794 DEBUG:boto:path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.794 DEBUG:boto:auth_path=/dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.794 DEBUG:boto:Method: GET 2024-04-12T03:36:25.794 DEBUG:boto:Path: /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.794 DEBUG:boto:Data: 2024-04-12T03:36:25.795 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.795 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.795 DEBUG:boto:Params: {} 2024-04-12T03:36:25.795 DEBUG:boto:Token: None 2024-04-12T03:36:25.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a?versionId=null 2024-04-12T03:36:25.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878= 2024-04-12T03:36:25.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nouUHlwSYah5x3HynS2dR4FC878=', 'Content-Length': '0'} 2024-04-12T03:36:25.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ccd806fdc20bf28-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.797 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.797 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.797 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.797 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.798 DEBUG:boto:Method: GET 2024-04-12T03:36:25.798 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:36:25.798 DEBUG:boto:Data: 2024-04-12T03:36:25.798 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.798 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.798 DEBUG:boto:Params: {} 2024-04-12T03:36:25.798 DEBUG:boto:Token: None 2024-04-12T03:36:25.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.798 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a028e028b68836ad-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.801 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.801 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.801 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.802 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.802 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.802 DEBUG:boto:Method: GET 2024-04-12T03:36:25.802 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:36:25.802 DEBUG:boto:Data: 2024-04-12T03:36:25.802 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.802 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.802 DEBUG:boto:Params: {} 2024-04-12T03:36:25.802 DEBUG:boto:Token: None 2024-04-12T03:36:25.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d48c5eeb479196-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.805 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.805 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.805 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.805 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.805 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:36:25.805 DEBUG:boto:Data: 2024-04-12T03:36:25.805 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.805 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.806 DEBUG:boto:Params: {} 2024-04-12T03:36:25.806 DEBUG:boto:Token: None 2024-04-12T03:36:25.806 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a 2024-04-12T03:36:25.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h69o5yEehZf75HNLZqlcH41jN2s= 2024-04-12T03:36:25.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h69o5yEehZf75HNLZqlcH41jN2s=', 'Content-Length': '0'} 2024-04-12T03:36:25.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cce41edbf938ba-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.808 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.808 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.808 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.808 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:36:25.808 DEBUG:boto:Data: 2024-04-12T03:36:25.808 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.808 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.809 DEBUG:boto:Params: {} 2024-04-12T03:36:25.809 DEBUG:boto:Token: None 2024-04-12T03:36:25.809 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a 2024-04-12T03:36:25.809 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h69o5yEehZf75HNLZqlcH41jN2s= 2024-04-12T03:36:25.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h69o5yEehZf75HNLZqlcH41jN2s=', 'Content-Length': '0'} 2024-04-12T03:36:25.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af9191d53db6e7d8-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:36:25.811 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.811 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.811 DEBUG:boto:Method: GET 2024-04-12T03:36:25.811 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:36:25.811 DEBUG:boto:Data: 2024-04-12T03:36:25.811 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.811 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.811 DEBUG:boto:Params: {} 2024-04-12T03:36:25.811 DEBUG:boto:Token: None 2024-04-12T03:36:25.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a 2024-04-12T03:36:25.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GDLAXRoU9dDcOFN6Em6me1W3h1w= 2024-04-12T03:36:25.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GDLAXRoU9dDcOFN6Em6me1W3h1w=', 'Content-Length': '0'} 2024-04-12T03:36:25.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481b550e546e109a-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.814 DEBUG:boto:path=/dgbpxg-58/a 2024-04-12T03:36:25.814 DEBUG:boto:auth_path=/dgbpxg-58/a 2024-04-12T03:36:25.814 DEBUG:boto:Method: GET 2024-04-12T03:36:25.814 DEBUG:boto:Path: /dgbpxg-58/a 2024-04-12T03:36:25.814 DEBUG:boto:Data: 2024-04-12T03:36:25.814 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.814 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.814 DEBUG:boto:Params: {} 2024-04-12T03:36:25.815 DEBUG:boto:Token: None 2024-04-12T03:36:25.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/a 2024-04-12T03:36:25.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GDLAXRoU9dDcOFN6Em6me1W3h1w= 2024-04-12T03:36:25.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GDLAXRoU9dDcOFN6Em6me1W3h1w=', 'Content-Length': '0'} 2024-04-12T03:36:25.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085862e3e42756c6c-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.817 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.817 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.817 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.817 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.817 DEBUG:boto:Method: GET 2024-04-12T03:36:25.817 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:36:25.817 DEBUG:boto:Data: 2024-04-12T03:36:25.818 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.818 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.818 DEBUG:boto:Params: {} 2024-04-12T03:36:25.818 DEBUG:boto:Token: None 2024-04-12T03:36:25.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ad3c0b7fcf9175-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.820 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.821 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.821 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.821 DEBUG:boto:path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.821 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=a 2024-04-12T03:36:25.821 DEBUG:boto:Method: GET 2024-04-12T03:36:25.821 DEBUG:boto:Path: /dgbpxg-58/?prefix=a 2024-04-12T03:36:25.821 DEBUG:boto:Data: 2024-04-12T03:36:25.822 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.822 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.822 DEBUG:boto:Params: {} 2024-04-12T03:36:25.822 DEBUG:boto:Token: None 2024-04-12T03:36:25.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f98befe4c1ab3e-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.824 DEBUG:boto:b'dgbpxg-58a1000falsea2024-04-12T03:33:24.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:36:25.824 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.824 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.825 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.825 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.825 DEBUG:boto:Method: GET 2024-04-12T03:36:25.825 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.825 DEBUG:boto:Data: 2024-04-12T03:36:25.825 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.825 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.825 DEBUG:boto:Params: {} 2024-04-12T03:36:25.825 DEBUG:boto:Token: None 2024-04-12T03:36:25.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4= 2024-04-12T03:36:25.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4=', 'Content-Length': '0'} 2024-04-12T03:36:25.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b9dcfb5dce9e33d-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.827 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.827 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.828 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.828 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.828 DEBUG:boto:Method: GET 2024-04-12T03:36:25.828 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.828 DEBUG:boto:Data: 2024-04-12T03:36:25.828 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.828 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.828 DEBUG:boto:Params: {} 2024-04-12T03:36:25.828 DEBUG:boto:Token: None 2024-04-12T03:36:25.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4= 2024-04-12T03:36:25.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4=', 'Content-Length': '0'} 2024-04-12T03:36:25.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed852e6dcf80350-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.831 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.831 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.831 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.831 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.831 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.831 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.831 DEBUG:boto:Data: 2024-04-12T03:36:25.831 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.831 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.831 DEBUG:boto:Params: {} 2024-04-12T03:36:25.832 DEBUG:boto:Token: None 2024-04-12T03:36:25.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J8ANm1Gl4b08x9GHSh1JIvSXHiY= 2024-04-12T03:36:25.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J8ANm1Gl4b08x9GHSh1JIvSXHiY=', 'Content-Length': '0'} 2024-04-12T03:36:25.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8483ada3b0261d-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.834 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.834 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.834 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.834 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.834 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.834 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.834 DEBUG:boto:Data: 2024-04-12T03:36:25.834 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.834 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.834 DEBUG:boto:Params: {} 2024-04-12T03:36:25.835 DEBUG:boto:Token: None 2024-04-12T03:36:25.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J8ANm1Gl4b08x9GHSh1JIvSXHiY= 2024-04-12T03:36:25.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J8ANm1Gl4b08x9GHSh1JIvSXHiY=', 'Content-Length': '0'} 2024-04-12T03:36:25.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030a423f20dda7fa-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:36:25.837 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.837 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.837 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.837 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.837 DEBUG:boto:Method: GET 2024-04-12T03:36:25.837 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.837 DEBUG:boto:Data: 2024-04-12T03:36:25.838 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.838 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.838 DEBUG:boto:Params: {} 2024-04-12T03:36:25.838 DEBUG:boto:Token: None 2024-04-12T03:36:25.838 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.838 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4= 2024-04-12T03:36:25.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4=', 'Content-Length': '0'} 2024-04-12T03:36:25.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28917522292679c-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.840 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.840 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.840 DEBUG:boto:path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.840 DEBUG:boto:auth_path=/dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.840 DEBUG:boto:Method: GET 2024-04-12T03:36:25.841 DEBUG:boto:Path: /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.841 DEBUG:boto:Data: 2024-04-12T03:36:25.841 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.841 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.841 DEBUG:boto:Params: {} 2024-04-12T03:36:25.841 DEBUG:boto:Token: None 2024-04-12T03:36:25.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b?versionId=null 2024-04-12T03:36:25.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4= 2024-04-12T03:36:25.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tNhff/tCtzs+pyvdGwe6B7WPhv4=', 'Content-Length': '0'} 2024-04-12T03:36:25.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002809cdb297187976-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.887 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.888 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.888 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.888 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.888 DEBUG:boto:Method: GET 2024-04-12T03:36:25.888 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:36:25.888 DEBUG:boto:Data: 2024-04-12T03:36:25.888 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.888 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.888 DEBUG:boto:Params: {} 2024-04-12T03:36:25.888 DEBUG:boto:Token: None 2024-04-12T03:36:25.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2899f73c863dbfa-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.891 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.892 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.892 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.892 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.892 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.892 DEBUG:boto:Method: GET 2024-04-12T03:36:25.892 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:36:25.892 DEBUG:boto:Data: 2024-04-12T03:36:25.892 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.893 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.893 DEBUG:boto:Params: {} 2024-04-12T03:36:25.893 DEBUG:boto:Token: None 2024-04-12T03:36:25.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe2d5ade2ec9c35-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.895 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.895 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.895 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.896 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.896 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:36:25.896 DEBUG:boto:Data: 2024-04-12T03:36:25.896 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.896 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.896 DEBUG:boto:Params: {} 2024-04-12T03:36:25.896 DEBUG:boto:Token: None 2024-04-12T03:36:25.897 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b 2024-04-12T03:36:25.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EIXTDnnGzFCqJ3wVWLDu0wF1pdg= 2024-04-12T03:36:25.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EIXTDnnGzFCqJ3wVWLDu0wF1pdg=', 'Content-Length': '0'} 2024-04-12T03:36:25.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65ed911208129c1-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.899 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.899 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.899 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.899 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:36:25.899 DEBUG:boto:Data: 2024-04-12T03:36:25.899 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.899 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.900 DEBUG:boto:Params: {} 2024-04-12T03:36:25.900 DEBUG:boto:Token: None 2024-04-12T03:36:25.900 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b 2024-04-12T03:36:25.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EIXTDnnGzFCqJ3wVWLDu0wF1pdg= 2024-04-12T03:36:25.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EIXTDnnGzFCqJ3wVWLDu0wF1pdg=', 'Content-Length': '0'} 2024-04-12T03:36:25.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007602b3c67ec84c2b-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:36:25.902 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.902 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.902 DEBUG:boto:Method: GET 2024-04-12T03:36:25.902 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:36:25.902 DEBUG:boto:Data: 2024-04-12T03:36:25.902 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.902 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.903 DEBUG:boto:Params: {} 2024-04-12T03:36:25.903 DEBUG:boto:Token: None 2024-04-12T03:36:25.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b 2024-04-12T03:36:25.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fbRDi1tG0DeTHKjnQNrFhAPagAk= 2024-04-12T03:36:25.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fbRDi1tG0DeTHKjnQNrFhAPagAk=', 'Content-Length': '0'} 2024-04-12T03:36:25.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ab190a7d49689c-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.950 DEBUG:boto:path=/dgbpxg-58/b 2024-04-12T03:36:25.951 DEBUG:boto:auth_path=/dgbpxg-58/b 2024-04-12T03:36:25.951 DEBUG:boto:Method: GET 2024-04-12T03:36:25.951 DEBUG:boto:Path: /dgbpxg-58/b 2024-04-12T03:36:25.951 DEBUG:boto:Data: 2024-04-12T03:36:25.951 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.951 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.951 DEBUG:boto:Params: {} 2024-04-12T03:36:25.951 DEBUG:boto:Token: None 2024-04-12T03:36:25.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/b 2024-04-12T03:36:25.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fbRDi1tG0DeTHKjnQNrFhAPagAk= 2024-04-12T03:36:25.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fbRDi1tG0DeTHKjnQNrFhAPagAk=', 'Content-Length': '0'} 2024-04-12T03:36:25.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2363a5283afb7a-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.954 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.954 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.954 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.954 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.954 DEBUG:boto:Method: GET 2024-04-12T03:36:25.954 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:36:25.954 DEBUG:boto:Data: 2024-04-12T03:36:25.954 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.955 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.955 DEBUG:boto:Params: {} 2024-04-12T03:36:25.955 DEBUG:boto:Token: None 2024-04-12T03:36:25.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044123c0281fbeca4-006618ac39-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.958 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.958 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:25.958 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:25.958 DEBUG:boto:path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.959 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=b 2024-04-12T03:36:25.959 DEBUG:boto:Method: GET 2024-04-12T03:36:25.959 DEBUG:boto:Path: /dgbpxg-58/?prefix=b 2024-04-12T03:36:25.959 DEBUG:boto:Data: 2024-04-12T03:36:25.959 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.959 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.959 DEBUG:boto:Params: {} 2024-04-12T03:36:25.959 DEBUG:boto:Token: None 2024-04-12T03:36:25.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/ 2024-04-12T03:36:25.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8= 2024-04-12T03:36:25.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vyKVAEefspTemVYqRcWc+KxFKc8=', 'Content-Length': '0'} 2024-04-12T03:36:25.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d471aa659324de7-006618ac39-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.961 DEBUG:boto:b'dgbpxg-58b1000falseb2024-04-12T03:33:24.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:25.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:36:25.962 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.962 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.962 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.962 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.962 DEBUG:boto:Method: GET 2024-04-12T03:36:25.962 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.962 DEBUG:boto:Data: 2024-04-12T03:36:25.963 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.963 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.963 DEBUG:boto:Params: {} 2024-04-12T03:36:25.963 DEBUG:boto:Token: None 2024-04-12T03:36:25.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ= 2024-04-12T03:36:25.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ=', 'Content-Length': '0'} 2024-04-12T03:36:25.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f172e6b99a35a4-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.965 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.965 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.965 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.965 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.966 DEBUG:boto:Method: GET 2024-04-12T03:36:25.966 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.966 DEBUG:boto:Data: 2024-04-12T03:36:25.966 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.966 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.966 DEBUG:boto:Params: {} 2024-04-12T03:36:25.966 DEBUG:boto:Token: None 2024-04-12T03:36:25.966 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.967 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ= 2024-04-12T03:36:25.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ=', 'Content-Length': '0'} 2024-04-12T03:36:25.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adabb1e08faa0bb9-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.968 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.969 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.969 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.969 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.969 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.969 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.969 DEBUG:boto:Data: 2024-04-12T03:36:25.969 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.969 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.969 DEBUG:boto:Params: {} 2024-04-12T03:36:25.970 DEBUG:boto:Token: None 2024-04-12T03:36:25.970 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.970 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DBXkaG4qyml+6IGXPC5f/+SYvxM= 2024-04-12T03:36:25.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DBXkaG4qyml+6IGXPC5f/+SYvxM=', 'Content-Length': '0'} 2024-04-12T03:36:25.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63ace5544982d53-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.972 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.972 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.972 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.972 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.972 DEBUG:boto:Method: HEAD 2024-04-12T03:36:25.972 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.972 DEBUG:boto:Data: 2024-04-12T03:36:25.972 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.973 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.973 DEBUG:boto:Params: {} 2024-04-12T03:36:25.973 DEBUG:boto:Token: None 2024-04-12T03:36:25.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.973 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DBXkaG4qyml+6IGXPC5f/+SYvxM= 2024-04-12T03:36:25.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DBXkaG4qyml+6IGXPC5f/+SYvxM=', 'Content-Length': '0'} 2024-04-12T03:36:25.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0bda3138146d034-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:36:25.975 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.975 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.976 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.976 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.976 DEBUG:boto:Method: GET 2024-04-12T03:36:25.976 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.976 DEBUG:boto:Data: 2024-04-12T03:36:25.976 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:25.976 DEBUG:boto:Port: 8001 2024-04-12T03:36:25.976 DEBUG:boto:Params: {} 2024-04-12T03:36:25.976 DEBUG:boto:Token: None 2024-04-12T03:36:25.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ= 2024-04-12T03:36:25.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ=', 'Content-Length': '0'} 2024-04-12T03:36:25.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02177c989fb7e99-006618ac39-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:25.979 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:25.979 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:25.979 DEBUG:boto:path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.979 DEBUG:boto:auth_path=/dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.979 DEBUG:boto:Method: GET 2024-04-12T03:36:25.979 DEBUG:boto:Path: /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.979 DEBUG:boto:Data: 2024-04-12T03:36:25.979 DEBUG:boto:Headers: {} 2024-04-12T03:36:25.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:25.979 DEBUG:boto:Port: 8000 2024-04-12T03:36:25.980 DEBUG:boto:Params: {} 2024-04-12T03:36:25.980 DEBUG:boto:Token: None 2024-04-12T03:36:25.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:25 GMT /dgbpxg-58/c?versionId=null 2024-04-12T03:36:25.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ= 2024-04-12T03:36:25.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XEPxfp1OufrEKNU83CUXY1LcXpQ=', 'Content-Length': '0'} 2024-04-12T03:36:25.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2453470d217ffd-006618ac39-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.022 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.023 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.023 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.023 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.023 DEBUG:boto:Method: GET 2024-04-12T03:36:26.023 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:36:26.023 DEBUG:boto:Data: 2024-04-12T03:36:26.023 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.023 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.023 DEBUG:boto:Params: {} 2024-04-12T03:36:26.024 DEBUG:boto:Token: None 2024-04-12T03:36:26.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9d8bfafa2e424e-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.026 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.026 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.027 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.027 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.027 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.027 DEBUG:boto:Method: GET 2024-04-12T03:36:26.027 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:36:26.027 DEBUG:boto:Data: 2024-04-12T03:36:26.027 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.027 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.027 DEBUG:boto:Params: {} 2024-04-12T03:36:26.028 DEBUG:boto:Token: None 2024-04-12T03:36:26.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb4b9e2bd5fb258-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.030 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.030 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:26.030 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:26.030 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.030 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:36:26.031 DEBUG:boto:Data: 2024-04-12T03:36:26.031 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.031 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.031 DEBUG:boto:Params: {} 2024-04-12T03:36:26.031 DEBUG:boto:Token: None 2024-04-12T03:36:26.031 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/c 2024-04-12T03:36:26.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fcrHlhIvnJ2oYl1i6QgG20OVgoo= 2024-04-12T03:36:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fcrHlhIvnJ2oYl1i6QgG20OVgoo=', 'Content-Length': '0'} 2024-04-12T03:36:26.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d784e942cfcc23ec-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.033 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:26.033 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:26.033 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.033 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:36:26.034 DEBUG:boto:Data: 2024-04-12T03:36:26.034 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.034 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.034 DEBUG:boto:Params: {} 2024-04-12T03:36:26.034 DEBUG:boto:Token: None 2024-04-12T03:36:26.034 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/c 2024-04-12T03:36:26.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fcrHlhIvnJ2oYl1i6QgG20OVgoo= 2024-04-12T03:36:26.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fcrHlhIvnJ2oYl1i6QgG20OVgoo=', 'Content-Length': '0'} 2024-04-12T03:36:26.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e26e288bd2b0e2f-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:36:26.037 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:26.037 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:26.037 DEBUG:boto:Method: GET 2024-04-12T03:36:26.037 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:36:26.037 DEBUG:boto:Data: 2024-04-12T03:36:26.037 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.037 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.037 DEBUG:boto:Params: {} 2024-04-12T03:36:26.037 DEBUG:boto:Token: None 2024-04-12T03:36:26.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/c 2024-04-12T03:36:26.038 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T+08BXHqJanR+3YYJhw6HqHLev0= 2024-04-12T03:36:26.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T+08BXHqJanR+3YYJhw6HqHLev0=', 'Content-Length': '0'} 2024-04-12T03:36:26.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d9214bd4cd75742-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.040 DEBUG:boto:path=/dgbpxg-58/c 2024-04-12T03:36:26.040 DEBUG:boto:auth_path=/dgbpxg-58/c 2024-04-12T03:36:26.040 DEBUG:boto:Method: GET 2024-04-12T03:36:26.040 DEBUG:boto:Path: /dgbpxg-58/c 2024-04-12T03:36:26.040 DEBUG:boto:Data: 2024-04-12T03:36:26.040 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.040 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.040 DEBUG:boto:Params: {} 2024-04-12T03:36:26.040 DEBUG:boto:Token: None 2024-04-12T03:36:26.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/c 2024-04-12T03:36:26.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T+08BXHqJanR+3YYJhw6HqHLev0= 2024-04-12T03:36:26.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T+08BXHqJanR+3YYJhw6HqHLev0=', 'Content-Length': '0'} 2024-04-12T03:36:26.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041fba06755d25d0c-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.042 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.043 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.043 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.043 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.043 DEBUG:boto:Method: GET 2024-04-12T03:36:26.043 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:36:26.043 DEBUG:boto:Data: 2024-04-12T03:36:26.043 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.043 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.043 DEBUG:boto:Params: {} 2024-04-12T03:36:26.043 DEBUG:boto:Token: None 2024-04-12T03:36:26.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae788e031f875604-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.046 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.046 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.047 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.047 DEBUG:boto:path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.047 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=c 2024-04-12T03:36:26.047 DEBUG:boto:Method: GET 2024-04-12T03:36:26.047 DEBUG:boto:Path: /dgbpxg-58/?prefix=c 2024-04-12T03:36:26.047 DEBUG:boto:Data: 2024-04-12T03:36:26.047 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.047 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.047 DEBUG:boto:Params: {} 2024-04-12T03:36:26.047 DEBUG:boto:Token: None 2024-04-12T03:36:26.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d9fafefa76dc1a-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.049 DEBUG:boto:b'dgbpxg-58c1000falsec2024-04-12T03:33:24.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:36:26.050 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.050 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.050 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.050 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.050 DEBUG:boto:Method: GET 2024-04-12T03:36:26.050 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.050 DEBUG:boto:Data: 2024-04-12T03:36:26.050 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.051 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.051 DEBUG:boto:Params: {} 2024-04-12T03:36:26.051 DEBUG:boto:Token: None 2024-04-12T03:36:26.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490= 2024-04-12T03:36:26.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490=', 'Content-Length': '0'} 2024-04-12T03:36:26.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d245257c2a49e0b-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.053 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.053 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.053 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.053 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.053 DEBUG:boto:Method: GET 2024-04-12T03:36:26.054 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.054 DEBUG:boto:Data: 2024-04-12T03:36:26.054 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.054 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.054 DEBUG:boto:Params: {} 2024-04-12T03:36:26.054 DEBUG:boto:Token: None 2024-04-12T03:36:26.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.054 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490= 2024-04-12T03:36:26.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490=', 'Content-Length': '0'} 2024-04-12T03:36:26.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586029adc121778b-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.056 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.056 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.056 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.056 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.057 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.057 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.057 DEBUG:boto:Data: 2024-04-12T03:36:26.057 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.057 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.057 DEBUG:boto:Params: {} 2024-04-12T03:36:26.057 DEBUG:boto:Token: None 2024-04-12T03:36:26.057 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.057 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fqs7ri4cVNObXtzFmnz2DzsWcrM= 2024-04-12T03:36:26.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fqs7ri4cVNObXtzFmnz2DzsWcrM=', 'Content-Length': '0'} 2024-04-12T03:36:26.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16422d044c6290a-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.059 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.059 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.059 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.060 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.060 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.060 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.060 DEBUG:boto:Data: 2024-04-12T03:36:26.060 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.060 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.060 DEBUG:boto:Params: {} 2024-04-12T03:36:26.060 DEBUG:boto:Token: None 2024-04-12T03:36:26.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fqs7ri4cVNObXtzFmnz2DzsWcrM= 2024-04-12T03:36:26.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fqs7ri4cVNObXtzFmnz2DzsWcrM=', 'Content-Length': '0'} 2024-04-12T03:36:26.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccdb8ef591dd92d-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:36:26.062 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.062 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.062 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.063 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.063 DEBUG:boto:Method: GET 2024-04-12T03:36:26.063 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.063 DEBUG:boto:Data: 2024-04-12T03:36:26.063 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.063 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.063 DEBUG:boto:Params: {} 2024-04-12T03:36:26.063 DEBUG:boto:Token: None 2024-04-12T03:36:26.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490= 2024-04-12T03:36:26.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490=', 'Content-Length': '0'} 2024-04-12T03:36:26.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a111808683d6acc8-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.065 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.065 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.066 DEBUG:boto:path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.066 DEBUG:boto:auth_path=/dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.066 DEBUG:boto:Method: GET 2024-04-12T03:36:26.066 DEBUG:boto:Path: /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.066 DEBUG:boto:Data: 2024-04-12T03:36:26.066 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.066 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.066 DEBUG:boto:Params: {} 2024-04-12T03:36:26.067 DEBUG:boto:Token: None 2024-04-12T03:36:26.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d?versionId=null 2024-04-12T03:36:26.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490= 2024-04-12T03:36:26.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LgHHsEIfW7h5df0wOJaLSFn9490=', 'Content-Length': '0'} 2024-04-12T03:36:26.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371c205bd2b12a6b-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.069 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.069 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.069 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.069 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.070 DEBUG:boto:Method: GET 2024-04-12T03:36:26.070 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:36:26.070 DEBUG:boto:Data: 2024-04-12T03:36:26.070 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.070 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.070 DEBUG:boto:Params: {} 2024-04-12T03:36:26.070 DEBUG:boto:Token: None 2024-04-12T03:36:26.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75bdf7a83ed122d-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.073 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.073 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.074 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.074 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.074 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.074 DEBUG:boto:Method: GET 2024-04-12T03:36:26.074 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:36:26.074 DEBUG:boto:Data: 2024-04-12T03:36:26.074 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.074 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.074 DEBUG:boto:Params: {} 2024-04-12T03:36:26.074 DEBUG:boto:Token: None 2024-04-12T03:36:26.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d4afa05359d23ba-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.076 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.077 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.077 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.077 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.077 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:36:26.077 DEBUG:boto:Data: 2024-04-12T03:36:26.077 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.077 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.077 DEBUG:boto:Params: {} 2024-04-12T03:36:26.077 DEBUG:boto:Token: None 2024-04-12T03:36:26.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d 2024-04-12T03:36:26.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yi0D1pYsJ2O1ZTkqtgZ4m0tpOFA= 2024-04-12T03:36:26.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yi0D1pYsJ2O1ZTkqtgZ4m0tpOFA=', 'Content-Length': '0'} 2024-04-12T03:36:26.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9e50d43f22e890-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.079 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.080 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.080 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.080 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:36:26.080 DEBUG:boto:Data: 2024-04-12T03:36:26.080 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.080 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.080 DEBUG:boto:Params: {} 2024-04-12T03:36:26.080 DEBUG:boto:Token: None 2024-04-12T03:36:26.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d 2024-04-12T03:36:26.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yi0D1pYsJ2O1ZTkqtgZ4m0tpOFA= 2024-04-12T03:36:26.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yi0D1pYsJ2O1ZTkqtgZ4m0tpOFA=', 'Content-Length': '0'} 2024-04-12T03:36:26.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3408d2878f0565-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:36:26.082 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.082 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.082 DEBUG:boto:Method: GET 2024-04-12T03:36:26.083 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:36:26.083 DEBUG:boto:Data: 2024-04-12T03:36:26.083 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.083 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.083 DEBUG:boto:Params: {} 2024-04-12T03:36:26.083 DEBUG:boto:Token: None 2024-04-12T03:36:26.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d 2024-04-12T03:36:26.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hDum1d6mqHPC64G7jDxoC/CaTyo= 2024-04-12T03:36:26.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hDum1d6mqHPC64G7jDxoC/CaTyo=', 'Content-Length': '0'} 2024-04-12T03:36:26.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ad9fec1d0969e2-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.085 DEBUG:boto:path=/dgbpxg-58/d 2024-04-12T03:36:26.085 DEBUG:boto:auth_path=/dgbpxg-58/d 2024-04-12T03:36:26.085 DEBUG:boto:Method: GET 2024-04-12T03:36:26.085 DEBUG:boto:Path: /dgbpxg-58/d 2024-04-12T03:36:26.086 DEBUG:boto:Data: 2024-04-12T03:36:26.086 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.086 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.086 DEBUG:boto:Params: {} 2024-04-12T03:36:26.086 DEBUG:boto:Token: None 2024-04-12T03:36:26.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/d 2024-04-12T03:36:26.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hDum1d6mqHPC64G7jDxoC/CaTyo= 2024-04-12T03:36:26.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hDum1d6mqHPC64G7jDxoC/CaTyo=', 'Content-Length': '0'} 2024-04-12T03:36:26.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c19b08304ce672-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.088 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.088 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.088 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.088 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.089 DEBUG:boto:Method: GET 2024-04-12T03:36:26.089 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:36:26.089 DEBUG:boto:Data: 2024-04-12T03:36:26.089 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.089 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.089 DEBUG:boto:Params: {} 2024-04-12T03:36:26.089 DEBUG:boto:Token: None 2024-04-12T03:36:26.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f089aeabfc5258a-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.092 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.092 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.092 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.092 DEBUG:boto:path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.092 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=d 2024-04-12T03:36:26.093 DEBUG:boto:Method: GET 2024-04-12T03:36:26.093 DEBUG:boto:Path: /dgbpxg-58/?prefix=d 2024-04-12T03:36:26.093 DEBUG:boto:Data: 2024-04-12T03:36:26.093 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.093 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.093 DEBUG:boto:Params: {} 2024-04-12T03:36:26.093 DEBUG:boto:Token: None 2024-04-12T03:36:26.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a16f0b727cf85e5-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.095 DEBUG:boto:b'dgbpxg-58d1000falsed2024-04-12T03:33:24.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:36:26.096 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.096 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.096 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.096 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.096 DEBUG:boto:Method: GET 2024-04-12T03:36:26.096 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.096 DEBUG:boto:Data: 2024-04-12T03:36:26.096 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.096 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.096 DEBUG:boto:Params: {} 2024-04-12T03:36:26.097 DEBUG:boto:Token: None 2024-04-12T03:36:26.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk= 2024-04-12T03:36:26.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk=', 'Content-Length': '0'} 2024-04-12T03:36:26.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d56f5275ac79fc-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.146 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.147 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.147 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.147 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.147 DEBUG:boto:Method: GET 2024-04-12T03:36:26.147 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.147 DEBUG:boto:Data: 2024-04-12T03:36:26.147 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.147 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.147 DEBUG:boto:Params: {} 2024-04-12T03:36:26.148 DEBUG:boto:Token: None 2024-04-12T03:36:26.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk= 2024-04-12T03:36:26.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk=', 'Content-Length': '0'} 2024-04-12T03:36:26.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67f9dbe0e9e9b4e-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.150 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.150 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.150 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.151 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.151 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.151 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.151 DEBUG:boto:Data: 2024-04-12T03:36:26.151 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.151 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.151 DEBUG:boto:Params: {} 2024-04-12T03:36:26.151 DEBUG:boto:Token: None 2024-04-12T03:36:26.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MHT6fB+v9oywwaQyFAcQ1XlOfdg= 2024-04-12T03:36:26.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MHT6fB+v9oywwaQyFAcQ1XlOfdg=', 'Content-Length': '0'} 2024-04-12T03:36:26.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000400e5db9de209b4-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.154 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.154 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.154 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.154 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.154 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.154 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.154 DEBUG:boto:Data: 2024-04-12T03:36:26.154 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.155 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.155 DEBUG:boto:Params: {} 2024-04-12T03:36:26.155 DEBUG:boto:Token: None 2024-04-12T03:36:26.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MHT6fB+v9oywwaQyFAcQ1XlOfdg= 2024-04-12T03:36:26.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MHT6fB+v9oywwaQyFAcQ1XlOfdg=', 'Content-Length': '0'} 2024-04-12T03:36:26.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e65c3e23ccb3ae-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:36:26.157 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.157 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.157 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.157 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.157 DEBUG:boto:Method: GET 2024-04-12T03:36:26.157 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.158 DEBUG:boto:Data: 2024-04-12T03:36:26.158 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.158 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.158 DEBUG:boto:Params: {} 2024-04-12T03:36:26.158 DEBUG:boto:Token: None 2024-04-12T03:36:26.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk= 2024-04-12T03:36:26.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk=', 'Content-Length': '0'} 2024-04-12T03:36:26.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7d090f15cb02b6-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.160 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.161 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.161 DEBUG:boto:path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.161 DEBUG:boto:auth_path=/dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.161 DEBUG:boto:Method: GET 2024-04-12T03:36:26.161 DEBUG:boto:Path: /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.161 DEBUG:boto:Data: 2024-04-12T03:36:26.161 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.161 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.161 DEBUG:boto:Params: {} 2024-04-12T03:36:26.162 DEBUG:boto:Token: None 2024-04-12T03:36:26.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e?versionId=null 2024-04-12T03:36:26.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk= 2024-04-12T03:36:26.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOIFDTcarbm5jqG1wwg9AN2Z6xk=', 'Content-Length': '0'} 2024-04-12T03:36:26.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dabbadc7c43104aa-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.164 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.164 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.164 DEBUG:boto:path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.164 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.165 DEBUG:boto:Method: GET 2024-04-12T03:36:26.165 DEBUG:boto:Path: /dgbpxg-58/?prefix=e 2024-04-12T03:36:26.165 DEBUG:boto:Data: 2024-04-12T03:36:26.165 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.165 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.165 DEBUG:boto:Params: {} 2024-04-12T03:36:26.165 DEBUG:boto:Token: None 2024-04-12T03:36:26.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.166 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005802a2fa4a2aacc8-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.168 DEBUG:boto:b'dgbpxg-58e1000falsee2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.168 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.168 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.169 DEBUG:boto:path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.169 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.169 DEBUG:boto:Method: GET 2024-04-12T03:36:26.169 DEBUG:boto:Path: /dgbpxg-58/?prefix=e 2024-04-12T03:36:26.169 DEBUG:boto:Data: 2024-04-12T03:36:26.169 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.169 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.169 DEBUG:boto:Params: {} 2024-04-12T03:36:26.169 DEBUG:boto:Token: None 2024-04-12T03:36:26.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.170 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5625bd7c371e246-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.172 DEBUG:boto:b'dgbpxg-58e1000falsee2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.172 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.172 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.172 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.172 DEBUG:boto:Path: /dgbpxg-58/e 2024-04-12T03:36:26.172 DEBUG:boto:Data: 2024-04-12T03:36:26.172 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.172 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.173 DEBUG:boto:Params: {} 2024-04-12T03:36:26.173 DEBUG:boto:Token: None 2024-04-12T03:36:26.173 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e 2024-04-12T03:36:26.173 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZKwSAWdZbw75mO0KnRd2DZ5/fxU= 2024-04-12T03:36:26.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZKwSAWdZbw75mO0KnRd2DZ5/fxU=', 'Content-Length': '0'} 2024-04-12T03:36:26.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a680ef249ba78607-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.175 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.175 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.175 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.175 DEBUG:boto:Path: /dgbpxg-58/e 2024-04-12T03:36:26.175 DEBUG:boto:Data: 2024-04-12T03:36:26.175 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.175 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.175 DEBUG:boto:Params: {} 2024-04-12T03:36:26.176 DEBUG:boto:Token: None 2024-04-12T03:36:26.176 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e 2024-04-12T03:36:26.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZKwSAWdZbw75mO0KnRd2DZ5/fxU= 2024-04-12T03:36:26.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZKwSAWdZbw75mO0KnRd2DZ5/fxU=', 'Content-Length': '0'} 2024-04-12T03:36:26.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad0fd1cd1f0758b-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:36:26.178 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.178 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.178 DEBUG:boto:Method: GET 2024-04-12T03:36:26.178 DEBUG:boto:Path: /dgbpxg-58/e 2024-04-12T03:36:26.178 DEBUG:boto:Data: 2024-04-12T03:36:26.178 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.179 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.179 DEBUG:boto:Params: {} 2024-04-12T03:36:26.179 DEBUG:boto:Token: None 2024-04-12T03:36:26.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e 2024-04-12T03:36:26.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kNZBfhD/UC95OP9aUa4sCCDUNZA= 2024-04-12T03:36:26.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kNZBfhD/UC95OP9aUa4sCCDUNZA=', 'Content-Length': '0'} 2024-04-12T03:36:26.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004061ab0c96c41fbf-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.222 DEBUG:boto:path=/dgbpxg-58/e 2024-04-12T03:36:26.223 DEBUG:boto:auth_path=/dgbpxg-58/e 2024-04-12T03:36:26.223 DEBUG:boto:Method: GET 2024-04-12T03:36:26.223 DEBUG:boto:Path: /dgbpxg-58/e 2024-04-12T03:36:26.223 DEBUG:boto:Data: 2024-04-12T03:36:26.223 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.223 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.223 DEBUG:boto:Params: {} 2024-04-12T03:36:26.223 DEBUG:boto:Token: None 2024-04-12T03:36:26.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/e 2024-04-12T03:36:26.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kNZBfhD/UC95OP9aUa4sCCDUNZA= 2024-04-12T03:36:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kNZBfhD/UC95OP9aUa4sCCDUNZA=', 'Content-Length': '0'} 2024-04-12T03:36:26.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba28f3aac48f60d-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.226 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.226 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.226 DEBUG:boto:path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.226 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.226 DEBUG:boto:Method: GET 2024-04-12T03:36:26.227 DEBUG:boto:Path: /dgbpxg-58/?prefix=e 2024-04-12T03:36:26.227 DEBUG:boto:Data: 2024-04-12T03:36:26.227 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.227 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.227 DEBUG:boto:Params: {} 2024-04-12T03:36:26.227 DEBUG:boto:Token: None 2024-04-12T03:36:26.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b30923e16ebd73-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.230 DEBUG:boto:b'dgbpxg-58e1000falsee2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.230 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.230 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.230 DEBUG:boto:path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.230 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=e 2024-04-12T03:36:26.230 DEBUG:boto:Method: GET 2024-04-12T03:36:26.231 DEBUG:boto:Path: /dgbpxg-58/?prefix=e 2024-04-12T03:36:26.231 DEBUG:boto:Data: 2024-04-12T03:36:26.231 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.231 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.231 DEBUG:boto:Params: {} 2024-04-12T03:36:26.231 DEBUG:boto:Token: None 2024-04-12T03:36:26.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b34351f9471ec1-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.233 DEBUG:boto:b'dgbpxg-58e1000falsee2024-04-12T03:33:45.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:36:26.234 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.234 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.234 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.234 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.234 DEBUG:boto:Method: GET 2024-04-12T03:36:26.234 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.234 DEBUG:boto:Data: 2024-04-12T03:36:26.234 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.235 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.235 DEBUG:boto:Params: {} 2024-04-12T03:36:26.235 DEBUG:boto:Token: None 2024-04-12T03:36:26.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE= 2024-04-12T03:36:26.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE=', 'Content-Length': '0'} 2024-04-12T03:36:26.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094012b2d61712702-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.237 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.237 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.237 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.237 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.238 DEBUG:boto:Method: GET 2024-04-12T03:36:26.238 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.238 DEBUG:boto:Data: 2024-04-12T03:36:26.238 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.238 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.238 DEBUG:boto:Params: {} 2024-04-12T03:36:26.238 DEBUG:boto:Token: None 2024-04-12T03:36:26.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE= 2024-04-12T03:36:26.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE=', 'Content-Length': '0'} 2024-04-12T03:36:26.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b49d5520b82fa1e-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.242 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.242 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.242 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.242 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.242 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.242 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.242 DEBUG:boto:Data: 2024-04-12T03:36:26.242 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.243 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.243 DEBUG:boto:Params: {} 2024-04-12T03:36:26.243 DEBUG:boto:Token: None 2024-04-12T03:36:26.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9us0HEE66Nv2VcUTFVDEOzjib1k= 2024-04-12T03:36:26.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9us0HEE66Nv2VcUTFVDEOzjib1k=', 'Content-Length': '0'} 2024-04-12T03:36:26.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ce7b8cacec4888-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.245 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.245 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.245 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.245 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.245 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.245 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.245 DEBUG:boto:Data: 2024-04-12T03:36:26.246 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.246 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.246 DEBUG:boto:Params: {} 2024-04-12T03:36:26.246 DEBUG:boto:Token: None 2024-04-12T03:36:26.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9us0HEE66Nv2VcUTFVDEOzjib1k= 2024-04-12T03:36:26.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9us0HEE66Nv2VcUTFVDEOzjib1k=', 'Content-Length': '0'} 2024-04-12T03:36:26.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005fc7052e87338c8-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:36:26.248 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.248 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.248 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.249 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.249 DEBUG:boto:Method: GET 2024-04-12T03:36:26.249 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.249 DEBUG:boto:Data: 2024-04-12T03:36:26.249 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.249 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.249 DEBUG:boto:Params: {} 2024-04-12T03:36:26.249 DEBUG:boto:Token: None 2024-04-12T03:36:26.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE= 2024-04-12T03:36:26.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE=', 'Content-Length': '0'} 2024-04-12T03:36:26.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f4be87c041c63b-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.251 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.251 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.251 DEBUG:boto:path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.251 DEBUG:boto:auth_path=/dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.252 DEBUG:boto:Method: GET 2024-04-12T03:36:26.252 DEBUG:boto:Path: /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.252 DEBUG:boto:Data: 2024-04-12T03:36:26.252 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.252 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.252 DEBUG:boto:Params: {} 2024-04-12T03:36:26.252 DEBUG:boto:Token: None 2024-04-12T03:36:26.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f?versionId=null 2024-04-12T03:36:26.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE= 2024-04-12T03:36:26.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/W1g7rRb9gWfnOpGkAFEgH7QkeE=', 'Content-Length': '0'} 2024-04-12T03:36:26.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d29d0d1f3f205221-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.254 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.255 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.255 DEBUG:boto:path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.255 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.255 DEBUG:boto:Method: GET 2024-04-12T03:36:26.255 DEBUG:boto:Path: /dgbpxg-58/?prefix=f 2024-04-12T03:36:26.255 DEBUG:boto:Data: 2024-04-12T03:36:26.255 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.255 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.255 DEBUG:boto:Params: {} 2024-04-12T03:36:26.256 DEBUG:boto:Token: None 2024-04-12T03:36:26.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e53493c02943d0-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.258 DEBUG:boto:b'dgbpxg-58f1000falsef2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.258 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.259 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.259 DEBUG:boto:path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.259 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.259 DEBUG:boto:Method: GET 2024-04-12T03:36:26.259 DEBUG:boto:Path: /dgbpxg-58/?prefix=f 2024-04-12T03:36:26.259 DEBUG:boto:Data: 2024-04-12T03:36:26.259 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.259 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.259 DEBUG:boto:Params: {} 2024-04-12T03:36:26.260 DEBUG:boto:Token: None 2024-04-12T03:36:26.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008201423639411680-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.302 DEBUG:boto:b'dgbpxg-58f1000falsef2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.303 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.303 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.303 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.303 DEBUG:boto:Path: /dgbpxg-58/f 2024-04-12T03:36:26.303 DEBUG:boto:Data: 2024-04-12T03:36:26.303 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.304 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.304 DEBUG:boto:Params: {} 2024-04-12T03:36:26.304 DEBUG:boto:Token: None 2024-04-12T03:36:26.304 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f 2024-04-12T03:36:26.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G2PylNx1xsuEsfl4krb0I2s98TE= 2024-04-12T03:36:26.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G2PylNx1xsuEsfl4krb0I2s98TE=', 'Content-Length': '0'} 2024-04-12T03:36:26.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c524866caf236c-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.306 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.306 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.306 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.307 DEBUG:boto:Path: /dgbpxg-58/f 2024-04-12T03:36:26.307 DEBUG:boto:Data: 2024-04-12T03:36:26.307 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.307 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.307 DEBUG:boto:Params: {} 2024-04-12T03:36:26.307 DEBUG:boto:Token: None 2024-04-12T03:36:26.307 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f 2024-04-12T03:36:26.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G2PylNx1xsuEsfl4krb0I2s98TE= 2024-04-12T03:36:26.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G2PylNx1xsuEsfl4krb0I2s98TE=', 'Content-Length': '0'} 2024-04-12T03:36:26.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81203312cdaaeee-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:36:26.309 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.309 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.309 DEBUG:boto:Method: GET 2024-04-12T03:36:26.310 DEBUG:boto:Path: /dgbpxg-58/f 2024-04-12T03:36:26.310 DEBUG:boto:Data: 2024-04-12T03:36:26.310 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.310 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.310 DEBUG:boto:Params: {} 2024-04-12T03:36:26.310 DEBUG:boto:Token: None 2024-04-12T03:36:26.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f 2024-04-12T03:36:26.310 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjakAunlkDZtqSUfeMViBhpAOpw= 2024-04-12T03:36:26.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjakAunlkDZtqSUfeMViBhpAOpw=', 'Content-Length': '0'} 2024-04-12T03:36:26.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af537ee4683a0ee-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.312 DEBUG:boto:path=/dgbpxg-58/f 2024-04-12T03:36:26.312 DEBUG:boto:auth_path=/dgbpxg-58/f 2024-04-12T03:36:26.312 DEBUG:boto:Method: GET 2024-04-12T03:36:26.313 DEBUG:boto:Path: /dgbpxg-58/f 2024-04-12T03:36:26.313 DEBUG:boto:Data: 2024-04-12T03:36:26.313 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.313 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.313 DEBUG:boto:Params: {} 2024-04-12T03:36:26.313 DEBUG:boto:Token: None 2024-04-12T03:36:26.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/f 2024-04-12T03:36:26.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjakAunlkDZtqSUfeMViBhpAOpw= 2024-04-12T03:36:26.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjakAunlkDZtqSUfeMViBhpAOpw=', 'Content-Length': '0'} 2024-04-12T03:36:26.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007042a1848beb3931-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.315 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.315 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.315 DEBUG:boto:path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.315 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.316 DEBUG:boto:Method: GET 2024-04-12T03:36:26.316 DEBUG:boto:Path: /dgbpxg-58/?prefix=f 2024-04-12T03:36:26.316 DEBUG:boto:Data: 2024-04-12T03:36:26.316 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.316 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.316 DEBUG:boto:Params: {} 2024-04-12T03:36:26.316 DEBUG:boto:Token: None 2024-04-12T03:36:26.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7e9aca83cf120c-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.319 DEBUG:boto:b'dgbpxg-58f1000falsef2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.319 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.319 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.319 DEBUG:boto:path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.319 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=f 2024-04-12T03:36:26.320 DEBUG:boto:Method: GET 2024-04-12T03:36:26.320 DEBUG:boto:Path: /dgbpxg-58/?prefix=f 2024-04-12T03:36:26.320 DEBUG:boto:Data: 2024-04-12T03:36:26.320 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.320 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.320 DEBUG:boto:Params: {} 2024-04-12T03:36:26.320 DEBUG:boto:Token: None 2024-04-12T03:36:26.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de44d8317aa29d4-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.322 DEBUG:boto:b'dgbpxg-58f1000falsef2024-04-12T03:33:45.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:36:26.323 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.323 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.323 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.323 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.323 DEBUG:boto:Method: GET 2024-04-12T03:36:26.323 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.323 DEBUG:boto:Data: 2024-04-12T03:36:26.323 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.324 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.324 DEBUG:boto:Params: {} 2024-04-12T03:36:26.324 DEBUG:boto:Token: None 2024-04-12T03:36:26.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI= 2024-04-12T03:36:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI=', 'Content-Length': '0'} 2024-04-12T03:36:26.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f54284edef4d817-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.366 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.367 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.367 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.367 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.367 DEBUG:boto:Method: GET 2024-04-12T03:36:26.367 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.367 DEBUG:boto:Data: 2024-04-12T03:36:26.367 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.367 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.367 DEBUG:boto:Params: {} 2024-04-12T03:36:26.368 DEBUG:boto:Token: None 2024-04-12T03:36:26.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI= 2024-04-12T03:36:26.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI=', 'Content-Length': '0'} 2024-04-12T03:36:26.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4219b3f245ee16-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.370 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.370 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.370 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.370 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.370 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.370 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.371 DEBUG:boto:Data: 2024-04-12T03:36:26.371 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.371 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.371 DEBUG:boto:Params: {} 2024-04-12T03:36:26.371 DEBUG:boto:Token: None 2024-04-12T03:36:26.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IrPalVqzbRx/w/Urd029B4EmzEU= 2024-04-12T03:36:26.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IrPalVqzbRx/w/Urd029B4EmzEU=', 'Content-Length': '0'} 2024-04-12T03:36:26.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073b27bcdc6f3b0d-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.373 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.373 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.373 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.373 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.373 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.373 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.373 DEBUG:boto:Data: 2024-04-12T03:36:26.374 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.374 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.374 DEBUG:boto:Params: {} 2024-04-12T03:36:26.374 DEBUG:boto:Token: None 2024-04-12T03:36:26.374 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.374 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IrPalVqzbRx/w/Urd029B4EmzEU= 2024-04-12T03:36:26.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IrPalVqzbRx/w/Urd029B4EmzEU=', 'Content-Length': '0'} 2024-04-12T03:36:26.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004240dbbc573202e5-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:36:26.376 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.376 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.376 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.376 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.377 DEBUG:boto:Method: GET 2024-04-12T03:36:26.377 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.377 DEBUG:boto:Data: 2024-04-12T03:36:26.377 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.377 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.377 DEBUG:boto:Params: {} 2024-04-12T03:36:26.377 DEBUG:boto:Token: None 2024-04-12T03:36:26.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI= 2024-04-12T03:36:26.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI=', 'Content-Length': '0'} 2024-04-12T03:36:26.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf08bdba11dd521-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.379 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.379 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.379 DEBUG:boto:path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.380 DEBUG:boto:auth_path=/dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.380 DEBUG:boto:Method: GET 2024-04-12T03:36:26.380 DEBUG:boto:Path: /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.380 DEBUG:boto:Data: 2024-04-12T03:36:26.380 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.380 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.380 DEBUG:boto:Params: {} 2024-04-12T03:36:26.380 DEBUG:boto:Token: None 2024-04-12T03:36:26.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g?versionId=null 2024-04-12T03:36:26.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI= 2024-04-12T03:36:26.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AWayJS2rfzS8m3dtxL2bCpHRCrI=', 'Content-Length': '0'} 2024-04-12T03:36:26.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d443944bf06f2c2f-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.383 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.383 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.383 DEBUG:boto:path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.383 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.383 DEBUG:boto:Method: GET 2024-04-12T03:36:26.383 DEBUG:boto:Path: /dgbpxg-58/?prefix=g 2024-04-12T03:36:26.383 DEBUG:boto:Data: 2024-04-12T03:36:26.383 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.384 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.384 DEBUG:boto:Params: {} 2024-04-12T03:36:26.384 DEBUG:boto:Token: None 2024-04-12T03:36:26.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2f272e64b1f6d4-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.386 DEBUG:boto:b'dgbpxg-58g1000falseg2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.387 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.387 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.387 DEBUG:boto:path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.387 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.387 DEBUG:boto:Method: GET 2024-04-12T03:36:26.387 DEBUG:boto:Path: /dgbpxg-58/?prefix=g 2024-04-12T03:36:26.387 DEBUG:boto:Data: 2024-04-12T03:36:26.387 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.388 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.388 DEBUG:boto:Params: {} 2024-04-12T03:36:26.388 DEBUG:boto:Token: None 2024-04-12T03:36:26.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd1ee29d98e77cea-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.434 DEBUG:boto:b'dgbpxg-58g1000falseg2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.435 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.435 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.435 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.435 DEBUG:boto:Path: /dgbpxg-58/g 2024-04-12T03:36:26.435 DEBUG:boto:Data: 2024-04-12T03:36:26.435 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.435 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.435 DEBUG:boto:Params: {} 2024-04-12T03:36:26.436 DEBUG:boto:Token: None 2024-04-12T03:36:26.436 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g 2024-04-12T03:36:26.436 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/6WfbVyZEtLzoD/wyAZdTlg+pBE= 2024-04-12T03:36:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/6WfbVyZEtLzoD/wyAZdTlg+pBE=', 'Content-Length': '0'} 2024-04-12T03:36:26.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b1c3d028986d29-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.438 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.438 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.438 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.438 DEBUG:boto:Path: /dgbpxg-58/g 2024-04-12T03:36:26.439 DEBUG:boto:Data: 2024-04-12T03:36:26.439 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.439 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.439 DEBUG:boto:Params: {} 2024-04-12T03:36:26.439 DEBUG:boto:Token: None 2024-04-12T03:36:26.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g 2024-04-12T03:36:26.439 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/6WfbVyZEtLzoD/wyAZdTlg+pBE= 2024-04-12T03:36:26.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/6WfbVyZEtLzoD/wyAZdTlg+pBE=', 'Content-Length': '0'} 2024-04-12T03:36:26.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628f6a8160f679a0-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:36:26.441 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.441 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.442 DEBUG:boto:Method: GET 2024-04-12T03:36:26.442 DEBUG:boto:Path: /dgbpxg-58/g 2024-04-12T03:36:26.442 DEBUG:boto:Data: 2024-04-12T03:36:26.442 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.442 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.442 DEBUG:boto:Params: {} 2024-04-12T03:36:26.442 DEBUG:boto:Token: None 2024-04-12T03:36:26.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g 2024-04-12T03:36:26.442 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/O9Q20OJOm28Rt5PAWQ3ltc1a2I= 2024-04-12T03:36:26.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/O9Q20OJOm28Rt5PAWQ3ltc1a2I=', 'Content-Length': '0'} 2024-04-12T03:36:26.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e8403159ffaf21-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.444 DEBUG:boto:path=/dgbpxg-58/g 2024-04-12T03:36:26.444 DEBUG:boto:auth_path=/dgbpxg-58/g 2024-04-12T03:36:26.444 DEBUG:boto:Method: GET 2024-04-12T03:36:26.445 DEBUG:boto:Path: /dgbpxg-58/g 2024-04-12T03:36:26.445 DEBUG:boto:Data: 2024-04-12T03:36:26.445 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.445 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.445 DEBUG:boto:Params: {} 2024-04-12T03:36:26.445 DEBUG:boto:Token: None 2024-04-12T03:36:26.445 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/g 2024-04-12T03:36:26.445 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/O9Q20OJOm28Rt5PAWQ3ltc1a2I= 2024-04-12T03:36:26.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/O9Q20OJOm28Rt5PAWQ3ltc1a2I=', 'Content-Length': '0'} 2024-04-12T03:36:26.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef592aea50c02eee-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.447 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.447 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.448 DEBUG:boto:path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.448 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.448 DEBUG:boto:Method: GET 2024-04-12T03:36:26.448 DEBUG:boto:Path: /dgbpxg-58/?prefix=g 2024-04-12T03:36:26.448 DEBUG:boto:Data: 2024-04-12T03:36:26.448 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.448 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.448 DEBUG:boto:Params: {} 2024-04-12T03:36:26.448 DEBUG:boto:Token: None 2024-04-12T03:36:26.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.449 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49518f7fee2e61b-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.451 DEBUG:boto:b'dgbpxg-58g1000falseg2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.451 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.451 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.451 DEBUG:boto:path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.451 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=g 2024-04-12T03:36:26.452 DEBUG:boto:Method: GET 2024-04-12T03:36:26.452 DEBUG:boto:Path: /dgbpxg-58/?prefix=g 2024-04-12T03:36:26.452 DEBUG:boto:Data: 2024-04-12T03:36:26.452 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.452 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.452 DEBUG:boto:Params: {} 2024-04-12T03:36:26.452 DEBUG:boto:Token: None 2024-04-12T03:36:26.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcab875acff7f6f-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.454 DEBUG:boto:b'dgbpxg-58g1000falseg2024-04-12T03:33:45.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:36:26.455 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.455 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.455 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.455 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.455 DEBUG:boto:Method: GET 2024-04-12T03:36:26.455 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.455 DEBUG:boto:Data: 2024-04-12T03:36:26.456 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.456 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.456 DEBUG:boto:Params: {} 2024-04-12T03:36:26.456 DEBUG:boto:Token: None 2024-04-12T03:36:26.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik= 2024-04-12T03:36:26.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik=', 'Content-Length': '0'} 2024-04-12T03:36:26.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b138de5f54b527-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.498 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.499 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.499 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.499 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.499 DEBUG:boto:Method: GET 2024-04-12T03:36:26.499 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.499 DEBUG:boto:Data: 2024-04-12T03:36:26.499 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.499 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.499 DEBUG:boto:Params: {} 2024-04-12T03:36:26.499 DEBUG:boto:Token: None 2024-04-12T03:36:26.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik= 2024-04-12T03:36:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik=', 'Content-Length': '0'} 2024-04-12T03:36:26.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69cbdef5e83f2c4-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.502 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.502 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.502 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.502 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.502 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.502 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.503 DEBUG:boto:Data: 2024-04-12T03:36:26.503 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.503 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.503 DEBUG:boto:Params: {} 2024-04-12T03:36:26.503 DEBUG:boto:Token: None 2024-04-12T03:36:26.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W7glA/Q8gUFCbmV1SD4C6oSwuo0= 2024-04-12T03:36:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W7glA/Q8gUFCbmV1SD4C6oSwuo0=', 'Content-Length': '0'} 2024-04-12T03:36:26.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162a2926fb3f32a8-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.505 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.505 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.505 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.505 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.505 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.506 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.506 DEBUG:boto:Data: 2024-04-12T03:36:26.506 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.506 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.506 DEBUG:boto:Params: {} 2024-04-12T03:36:26.506 DEBUG:boto:Token: None 2024-04-12T03:36:26.506 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W7glA/Q8gUFCbmV1SD4C6oSwuo0= 2024-04-12T03:36:26.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W7glA/Q8gUFCbmV1SD4C6oSwuo0=', 'Content-Length': '0'} 2024-04-12T03:36:26.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2617348ce2ef25-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:36:26.508 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.508 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.508 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.509 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.509 DEBUG:boto:Method: GET 2024-04-12T03:36:26.509 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.509 DEBUG:boto:Data: 2024-04-12T03:36:26.509 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.509 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.509 DEBUG:boto:Params: {} 2024-04-12T03:36:26.509 DEBUG:boto:Token: None 2024-04-12T03:36:26.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.510 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik= 2024-04-12T03:36:26.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik=', 'Content-Length': '0'} 2024-04-12T03:36:26.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abd4029f28a56ce-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.511 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.511 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.512 DEBUG:boto:path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.512 DEBUG:boto:auth_path=/dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.512 DEBUG:boto:Method: GET 2024-04-12T03:36:26.512 DEBUG:boto:Path: /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.512 DEBUG:boto:Data: 2024-04-12T03:36:26.512 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.512 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.512 DEBUG:boto:Params: {} 2024-04-12T03:36:26.512 DEBUG:boto:Token: None 2024-04-12T03:36:26.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h?versionId=null 2024-04-12T03:36:26.513 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik= 2024-04-12T03:36:26.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wbj9tKqrd1scqxmIK586AN1VRik=', 'Content-Length': '0'} 2024-04-12T03:36:26.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043308f9d458825e5-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.515 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.515 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.515 DEBUG:boto:path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.515 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.515 DEBUG:boto:Method: GET 2024-04-12T03:36:26.515 DEBUG:boto:Path: /dgbpxg-58/?prefix=h 2024-04-12T03:36:26.515 DEBUG:boto:Data: 2024-04-12T03:36:26.515 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.515 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.516 DEBUG:boto:Params: {} 2024-04-12T03:36:26.516 DEBUG:boto:Token: None 2024-04-12T03:36:26.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003940c60eb213ab5c-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.518 DEBUG:boto:b'dgbpxg-58h1000falseh2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.519 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.519 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.519 DEBUG:boto:path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.519 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.519 DEBUG:boto:Method: GET 2024-04-12T03:36:26.519 DEBUG:boto:Path: /dgbpxg-58/?prefix=h 2024-04-12T03:36:26.520 DEBUG:boto:Data: 2024-04-12T03:36:26.520 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.520 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.520 DEBUG:boto:Params: {} 2024-04-12T03:36:26.520 DEBUG:boto:Token: None 2024-04-12T03:36:26.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805da5ea06fb02da-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.562 DEBUG:boto:b'dgbpxg-58h1000falseh2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.563 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.563 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.563 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.563 DEBUG:boto:Path: /dgbpxg-58/h 2024-04-12T03:36:26.563 DEBUG:boto:Data: 2024-04-12T03:36:26.563 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.564 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.564 DEBUG:boto:Params: {} 2024-04-12T03:36:26.564 DEBUG:boto:Token: None 2024-04-12T03:36:26.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h 2024-04-12T03:36:26.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tKctnq/PDDZyYzYUNfLo5uQSa7s= 2024-04-12T03:36:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tKctnq/PDDZyYzYUNfLo5uQSa7s=', 'Content-Length': '0'} 2024-04-12T03:36:26.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008486614d12a4dabc-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.566 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.566 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.566 DEBUG:boto:Method: HEAD 2024-04-12T03:36:26.566 DEBUG:boto:Path: /dgbpxg-58/h 2024-04-12T03:36:26.566 DEBUG:boto:Data: 2024-04-12T03:36:26.567 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.567 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.567 DEBUG:boto:Params: {} 2024-04-12T03:36:26.567 DEBUG:boto:Token: None 2024-04-12T03:36:26.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h 2024-04-12T03:36:26.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tKctnq/PDDZyYzYUNfLo5uQSa7s= 2024-04-12T03:36:26.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tKctnq/PDDZyYzYUNfLo5uQSa7s=', 'Content-Length': '0'} 2024-04-12T03:36:26.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb1c1c7218b1efc7-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:36:26.569 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.569 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.570 DEBUG:boto:Method: GET 2024-04-12T03:36:26.570 DEBUG:boto:Path: /dgbpxg-58/h 2024-04-12T03:36:26.570 DEBUG:boto:Data: 2024-04-12T03:36:26.570 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.570 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.570 DEBUG:boto:Params: {} 2024-04-12T03:36:26.570 DEBUG:boto:Token: None 2024-04-12T03:36:26.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h 2024-04-12T03:36:26.570 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vuFQP+o8rcSAwJtHNMI/dFvB4UM= 2024-04-12T03:36:26.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vuFQP+o8rcSAwJtHNMI/dFvB4UM=', 'Content-Length': '0'} 2024-04-12T03:36:26.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-58:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3df9115c5e1ac96-006618ac3a-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.572 DEBUG:boto:path=/dgbpxg-58/h 2024-04-12T03:36:26.572 DEBUG:boto:auth_path=/dgbpxg-58/h 2024-04-12T03:36:26.572 DEBUG:boto:Method: GET 2024-04-12T03:36:26.572 DEBUG:boto:Path: /dgbpxg-58/h 2024-04-12T03:36:26.573 DEBUG:boto:Data: 2024-04-12T03:36:26.573 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.573 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.573 DEBUG:boto:Params: {} 2024-04-12T03:36:26.573 DEBUG:boto:Token: None 2024-04-12T03:36:26.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/h 2024-04-12T03:36:26.573 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vuFQP+o8rcSAwJtHNMI/dFvB4UM= 2024-04-12T03:36:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vuFQP+o8rcSAwJtHNMI/dFvB4UM=', 'Content-Length': '0'} 2024-04-12T03:36:26.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:33:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970d7a728f906314-006618ac3a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.575 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.575 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.575 DEBUG:boto:path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.575 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.575 DEBUG:boto:Method: GET 2024-04-12T03:36:26.576 DEBUG:boto:Path: /dgbpxg-58/?prefix=h 2024-04-12T03:36:26.576 DEBUG:boto:Data: 2024-04-12T03:36:26.576 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:26.576 DEBUG:boto:Port: 8001 2024-04-12T03:36:26.576 DEBUG:boto:Params: {} 2024-04-12T03:36:26.576 DEBUG:boto:Token: None 2024-04-12T03:36:26.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c497d2690491e77e-006618ac3a-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.579 DEBUG:boto:b'dgbpxg-58h1000falseh2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.579 DEBUG:boto:path=/dgbpxg-58/ 2024-04-12T03:36:26.579 DEBUG:boto:auth_path=/dgbpxg-58/ 2024-04-12T03:36:26.579 DEBUG:boto:path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.580 DEBUG:boto:auth_path=/dgbpxg-58/?prefix=h 2024-04-12T03:36:26.580 DEBUG:boto:Method: GET 2024-04-12T03:36:26.580 DEBUG:boto:Path: /dgbpxg-58/?prefix=h 2024-04-12T03:36:26.580 DEBUG:boto:Data: 2024-04-12T03:36:26.580 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.580 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.580 DEBUG:boto:Params: {} 2024-04-12T03:36:26.580 DEBUG:boto:Token: None 2024-04-12T03:36:26.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-58/ 2024-04-12T03:36:26.581 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY= 2024-04-12T03:36:26.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNAssmyw9pspAN6NEmD0Rh+cNsY=', 'Content-Length': '0'} 2024-04-12T03:36:26.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4dfc8664c3f27d9-006618ac3a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.582 DEBUG:boto:b'dgbpxg-58h1000falseh2024-04-12T03:33:45.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:36:26.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-58 zones={a2, a1} 2024-04-12T03:36:26.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2024-04-12T03:36:26.584 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:26.584 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:26.584 DEBUG:boto:Method: PUT 2024-04-12T03:36:26.585 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:26.585 DEBUG:boto:Data: 2024-04-12T03:36:26.585 DEBUG:boto:Headers: {} 2024-04-12T03:36:26.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:26.585 DEBUG:boto:Port: 8000 2024-04-12T03:36:26.585 DEBUG:boto:Params: {} 2024-04-12T03:36:26.585 DEBUG:boto:Token: None 2024-04-12T03:36:26.585 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:36:26 GMT /dgbpxg-59/ 2024-04-12T03:36:26.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5QTcwoEmFzbmciplu9d8Pi3kg0M= 2024-04-12T03:36:26.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:26 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5QTcwoEmFzbmciplu9d8Pi3kg0M=', 'Content-Length': '0'} 2024-04-12T03:36:26.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dffb07fa33a8f425-006618ac3a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:26.600 DEBUG:rgw_multi.tests:created bucket=dgbpxg-59 2024-04-12T03:36:26.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:26.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:26.601 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:26.648 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:26.648 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:26.678 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:36:26.691 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.692 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892986.598501_1111.1", 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:36:26.598501Z" 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.693 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:36:26.694 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892986.598501_1111.1'} 2024-04-12T03:36:26.694 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:36:26.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:26.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:26.695 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:26.772+0000 7fb6e206f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:36:26.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:26.773+0000 7fb6e206f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:36:26.786 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:36:26.786 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:36:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:36:26.788 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892986.598501_1111.1", 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:36:26.598501Z", 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:36:26.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:36:26Z" 2024-04-12T03:36:26.792 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:36:26.793 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892986.598501_1111.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:26.598501Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:36:26Z" } 2024-04-12T03:36:26.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892986.598501_1111.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:26.598501Z', 'realm_epoch': 2}}] 2024-04-12T03:36:26.794 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892986.598501_1111.1'} 2024-04-12T03:36:26.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892986.598501_1111.1'} 2024-04-12T03:36:26.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:36:26.794 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:36:26.794 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:36:26.795 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:36:26.794+0000 7fdb3a557700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 49043) UID: 0 2024-04-12T03:36:26.795 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:36:26.794+0000 7fdb3f72c7c0 -1 shutting down 2024-04-12T03:36:31.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:36:32.896 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:36:32.896 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:32.896 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:32.896 DEBUG:boto:Method: HEAD 2024-04-12T03:36:32.896 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:32.897 DEBUG:boto:Data: 2024-04-12T03:36:32.897 DEBUG:boto:Headers: {} 2024-04-12T03:36:32.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.897 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.897 DEBUG:boto:Params: {} 2024-04-12T03:36:32.897 DEBUG:boto:Token: None 2024-04-12T03:36:32.897 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/ 2024-04-12T03:36:32.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o= 2024-04-12T03:36:32.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o=', 'Content-Length': '0'} 2024-04-12T03:36:32.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e397d2affab6b276-006618ac40-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:32.900 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:36:32.901 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:36:32.901 DEBUG:boto:Method: PUT 2024-04-12T03:36:32.901 DEBUG:boto:Path: /dgbpxg-59/a 2024-04-12T03:36:32.901 DEBUG:boto:Data: 2024-04-12T03:36:32.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:32.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.901 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.901 DEBUG:boto:Params: {} 2024-04-12T03:36:32.901 DEBUG:boto:Token: None 2024-04-12T03:36:32.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/a 2024-04-12T03:36:32.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dlY/mrmdhhgZPR6D9ZHQ4tFtklY= 2024-04-12T03:36:32.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dlY/mrmdhhgZPR6D9ZHQ4tFtklY='} 2024-04-12T03:36:32.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b24335216ba04439-006618ac40-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:32.947 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:32.947 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:32.947 DEBUG:boto:Method: HEAD 2024-04-12T03:36:32.947 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:32.947 DEBUG:boto:Data: 2024-04-12T03:36:32.947 DEBUG:boto:Headers: {} 2024-04-12T03:36:32.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.947 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.948 DEBUG:boto:Params: {} 2024-04-12T03:36:32.948 DEBUG:boto:Token: None 2024-04-12T03:36:32.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/ 2024-04-12T03:36:32.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o= 2024-04-12T03:36:32.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o=', 'Content-Length': '0'} 2024-04-12T03:36:32.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025b0871e3a2c12a0-006618ac40-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:32.951 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:36:32.952 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:36:32.952 DEBUG:boto:Method: PUT 2024-04-12T03:36:32.952 DEBUG:boto:Path: /dgbpxg-59/b 2024-04-12T03:36:32.952 DEBUG:boto:Data: 2024-04-12T03:36:32.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:32.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.952 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.952 DEBUG:boto:Params: {} 2024-04-12T03:36:32.952 DEBUG:boto:Token: None 2024-04-12T03:36:32.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/b 2024-04-12T03:36:32.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ok1pWLqUe0WOx4Y2ctZhj63SJkQ= 2024-04-12T03:36:32.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ok1pWLqUe0WOx4Y2ctZhj63SJkQ='} 2024-04-12T03:36:32.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000831fddaab5a6a4d3-006618ac40-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:32.959 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:32.959 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:32.959 DEBUG:boto:Method: HEAD 2024-04-12T03:36:32.959 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:32.959 DEBUG:boto:Data: 2024-04-12T03:36:32.959 DEBUG:boto:Headers: {} 2024-04-12T03:36:32.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.960 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.960 DEBUG:boto:Params: {} 2024-04-12T03:36:32.960 DEBUG:boto:Token: None 2024-04-12T03:36:32.960 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/ 2024-04-12T03:36:32.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o= 2024-04-12T03:36:32.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4X8hWy3YN/rFEV7JdW8MT14Yi2o=', 'Content-Length': '0'} 2024-04-12T03:36:32.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d280ad9cb20be83-006618ac40-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:32.962 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:36:32.963 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:36:32.963 DEBUG:boto:Method: PUT 2024-04-12T03:36:32.963 DEBUG:boto:Path: /dgbpxg-59/c 2024-04-12T03:36:32.963 DEBUG:boto:Data: 2024-04-12T03:36:32.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:32.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:32.963 DEBUG:boto:Port: 8000 2024-04-12T03:36:32.963 DEBUG:boto:Params: {} 2024-04-12T03:36:32.963 DEBUG:boto:Token: None 2024-04-12T03:36:32.964 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:32 GMT /dgbpxg-59/c 2024-04-12T03:36:32.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:el5qileYanLjnbRNEdIPfpZYPug= 2024-04-12T03:36:32.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:el5qileYanLjnbRNEdIPfpZYPug='} 2024-04-12T03:36:33.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084adb0cd3bb055e4-006618ac40-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.007 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:33.007 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:33.007 DEBUG:boto:Method: HEAD 2024-04-12T03:36:33.007 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:33.007 DEBUG:boto:Data: 2024-04-12T03:36:33.007 DEBUG:boto:Headers: {} 2024-04-12T03:36:33.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.007 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.008 DEBUG:boto:Params: {} 2024-04-12T03:36:33.008 DEBUG:boto:Token: None 2024-04-12T03:36:33.008 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/ 2024-04-12T03:36:33.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U= 2024-04-12T03:36:33.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U=', 'Content-Length': '0'} 2024-04-12T03:36:33.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001100cc17d2ab9a07-006618ac41-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.011 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:36:33.011 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:36:33.012 DEBUG:boto:Method: PUT 2024-04-12T03:36:33.012 DEBUG:boto:Path: /dgbpxg-59/d 2024-04-12T03:36:33.012 DEBUG:boto:Data: 2024-04-12T03:36:33.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:33.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.012 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.012 DEBUG:boto:Params: {} 2024-04-12T03:36:33.012 DEBUG:boto:Token: None 2024-04-12T03:36:33.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/d 2024-04-12T03:36:33.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2MpHkEpHEzWu5CYNwNU1E6MeIok= 2024-04-12T03:36:33.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2MpHkEpHEzWu5CYNwNU1E6MeIok='} 2024-04-12T03:36:33.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb52e9b0338ffdeb-006618ac41-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:33.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:33.019 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-59 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:36:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:36:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-59 2024-04-12T03:36:33.124 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2024-04-12T03:36:33.163 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:33.164 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:33.164 DEBUG:boto:Method: HEAD 2024-04-12T03:36:33.164 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:33.164 DEBUG:boto:Data: 2024-04-12T03:36:33.164 DEBUG:boto:Headers: {} 2024-04-12T03:36:33.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.164 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.164 DEBUG:boto:Params: {} 2024-04-12T03:36:33.165 DEBUG:boto:Token: None 2024-04-12T03:36:33.165 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/ 2024-04-12T03:36:33.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U= 2024-04-12T03:36:33.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U=', 'Content-Length': '0'} 2024-04-12T03:36:33.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c1bd30f1d0cb1ef-006618ac41-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.167 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:36:33.167 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:36:33.167 DEBUG:boto:Method: PUT 2024-04-12T03:36:33.167 DEBUG:boto:Path: /dgbpxg-59/e 2024-04-12T03:36:33.168 DEBUG:boto:Data: 2024-04-12T03:36:33.168 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:33.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.168 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.168 DEBUG:boto:Params: {} 2024-04-12T03:36:33.168 DEBUG:boto:Token: None 2024-04-12T03:36:33.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/e 2024-04-12T03:36:33.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BK/gS0bQh+n/+JLnbNH9nvNCQWc= 2024-04-12T03:36:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BK/gS0bQh+n/+JLnbNH9nvNCQWc='} 2024-04-12T03:36:33.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095f3d1689ca5670f-006618ac41-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.211 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:33.211 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:33.211 DEBUG:boto:Method: HEAD 2024-04-12T03:36:33.211 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:33.211 DEBUG:boto:Data: 2024-04-12T03:36:33.211 DEBUG:boto:Headers: {} 2024-04-12T03:36:33.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.212 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.212 DEBUG:boto:Params: {} 2024-04-12T03:36:33.212 DEBUG:boto:Token: None 2024-04-12T03:36:33.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/ 2024-04-12T03:36:33.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U= 2024-04-12T03:36:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U=', 'Content-Length': '0'} 2024-04-12T03:36:33.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019cfae7dbc53468c-006618ac41-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.214 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:36:33.214 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:36:33.214 DEBUG:boto:Method: PUT 2024-04-12T03:36:33.214 DEBUG:boto:Path: /dgbpxg-59/f 2024-04-12T03:36:33.214 DEBUG:boto:Data: 2024-04-12T03:36:33.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:33.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.215 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.215 DEBUG:boto:Params: {} 2024-04-12T03:36:33.215 DEBUG:boto:Token: None 2024-04-12T03:36:33.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/f 2024-04-12T03:36:33.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o+QjeDT+UjugcHrxJS4Dn5TC9po= 2024-04-12T03:36:33.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o+QjeDT+UjugcHrxJS4Dn5TC9po='} 2024-04-12T03:36:33.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009839515bd5360047-006618ac41-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.220 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:33.221 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:33.221 DEBUG:boto:Method: HEAD 2024-04-12T03:36:33.221 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:33.221 DEBUG:boto:Data: 2024-04-12T03:36:33.221 DEBUG:boto:Headers: {} 2024-04-12T03:36:33.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.221 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.221 DEBUG:boto:Params: {} 2024-04-12T03:36:33.221 DEBUG:boto:Token: None 2024-04-12T03:36:33.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/ 2024-04-12T03:36:33.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U= 2024-04-12T03:36:33.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U=', 'Content-Length': '0'} 2024-04-12T03:36:33.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040d545dce0675055-006618ac41-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.223 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:36:33.224 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:36:33.224 DEBUG:boto:Method: PUT 2024-04-12T03:36:33.224 DEBUG:boto:Path: /dgbpxg-59/g 2024-04-12T03:36:33.224 DEBUG:boto:Data: 2024-04-12T03:36:33.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:33.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.225 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.225 DEBUG:boto:Params: {} 2024-04-12T03:36:33.225 DEBUG:boto:Token: None 2024-04-12T03:36:33.225 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/g 2024-04-12T03:36:33.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mV9KxBwMqeZN392F34ZH/aFt06w= 2024-04-12T03:36:33.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mV9KxBwMqeZN392F34ZH/aFt06w='} 2024-04-12T03:36:33.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8d6da22dfbdc1e7-006618ac41-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.230 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:36:33.230 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:36:33.230 DEBUG:boto:Method: HEAD 2024-04-12T03:36:33.230 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:36:33.230 DEBUG:boto:Data: 2024-04-12T03:36:33.230 DEBUG:boto:Headers: {} 2024-04-12T03:36:33.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.230 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.230 DEBUG:boto:Params: {} 2024-04-12T03:36:33.231 DEBUG:boto:Token: None 2024-04-12T03:36:33.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/ 2024-04-12T03:36:33.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U= 2024-04-12T03:36:33.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jIrHV6kG+o/l6VsWSvQkMy2Sq+U=', 'Content-Length': '0'} 2024-04-12T03:36:33.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093ffb8e3a38a3f9c-006618ac41-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.233 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:36:33.233 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:36:33.233 DEBUG:boto:Method: PUT 2024-04-12T03:36:33.233 DEBUG:boto:Path: /dgbpxg-59/h 2024-04-12T03:36:33.233 DEBUG:boto:Data: 2024-04-12T03:36:33.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:36:33.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:36:33.233 DEBUG:boto:Port: 8000 2024-04-12T03:36:33.233 DEBUG:boto:Params: {} 2024-04-12T03:36:33.233 DEBUG:boto:Token: None 2024-04-12T03:36:33.234 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:36:33 GMT /dgbpxg-59/h 2024-04-12T03:36:33.234 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ADiTyeA7CpHr8m/WdHr5nHAArBM= 2024-04-12T03:36:33.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:36:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ADiTyeA7CpHr8m/WdHr5nHAArBM='} 2024-04-12T03:36:33.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c7c4605b3c15c8b-006618ac41-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:36:33.279 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:36:33.279 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:36:33.281 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:36:33.282 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:33.301 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:36:33.301 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:36:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:36:33.303 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:36:34.305 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:36:34.326 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:36:34.326 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:36:34.327 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:36:34.327 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T03:36:34.328 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-59 2024-04-12T03:36:34.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:36:34.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:36:34.328 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:36:34.425 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:34.425 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:34.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:34.459+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:36:36.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:36:39.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:39.459+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:36:44.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:44.459+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:36:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:49.460+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:36:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:54.461+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:36:59.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:36:59.461+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:04.462+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:09.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:09.463+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:14.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:14.463+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:19.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:19.464+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:24.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:24.464+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:29.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:29.464+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:34.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:34.465+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:39.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:39.466+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:44.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:44.467+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:49.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:49.467+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:54.468+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:37:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:37:59.468+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:04.469+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:09.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:09.469+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:14.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:14.469+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:19.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:19.471+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:24.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:24.471+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:29.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:29.472+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:34.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:34.473+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:39.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:39.473+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:44.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:44.474+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:49.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:49.474+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:54.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:54.475+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:38:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:38:59.475+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:39:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:04.477+0000 7f8fedcd0780 1 waiting to reach incremental sync.. 2024-04-12T03:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:09.478+0000 7f8fedcd0780 1 bucket sync caught up with source: 2024-04-12T03:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.120.6, 00000000002.31.6, 00000000002.38.6] 2024-04-12T03:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.120.6, 00000000002.31.6, 00000000002.38.6] 2024-04-12T03:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:09.478+0000 7f8fedcd0780 0 bucket checkpoint complete 2024-04-12T03:39:09.497 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-59 2024-04-12T03:39:09.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:09.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:09.498 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:39:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:09.596+0000 7f317f056780 1 waiting for incremental sync to catch up: 2024-04-12T03:39:09.597 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3044.6, , , , 00000000001.1181.6, , , 00000000002.492.6, , 00000000001.643.6] 2024-04-12T03:39:09.597 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3044.6, , , 00000000001.1639.6, 00000000001.1181.6, , , 00000000002.492.6, , 00000000002.651.6] 2024-04-12T03:39:14.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:14.598+0000 7f317f056780 1 waiting for incremental sync to catch up: 2024-04-12T03:39:14.599 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3044.6, , , , 00000000001.1181.6, , , 00000000002.492.6, , 00000000001.643.6] 2024-04-12T03:39:14.599 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3044.6, , , 00000000001.1639.6, 00000000001.1181.6, , , 00000000002.492.6, , 00000000002.651.6] 2024-04-12T03:39:19.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:19.599+0000 7f317f056780 1 bucket sync caught up with source: 2024-04-12T03:39:19.600 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3044.6, , , 00000000001.1639.6, 00000000001.1181.6, , , 00000000002.492.6, , 00000000002.651.6] 2024-04-12T03:39:19.600 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3044.6, , , 00000000001.1639.6, 00000000001.1181.6, , , 00000000002.492.6, , 00000000002.651.6] 2024-04-12T03:39:19.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:19.599+0000 7f317f056780 0 bucket checkpoint complete 2024-04-12T03:39:19.616 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-59 zones={a2, a1} 2024-04-12T03:39:19.616 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.616 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.617 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.617 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:39:19.617 DEBUG:boto:Data: 2024-04-12T03:39:19.617 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.617 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.617 DEBUG:boto:Params: {} 2024-04-12T03:39:19.618 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:19.618 DEBUG:boto:Token: None 2024-04-12T03:39:19.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gdKFtj9I2FuuG81k1B1368qdNjA= 2024-04-12T03:39:19.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gdKFtj9I2FuuG81k1B1368qdNjA=', 'Content-Length': '0'} 2024-04-12T03:39:19.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd9178573ca45f6e-006618ace7-1da1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.650 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.650 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.650 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.650 DEBUG:boto:Path: /dgbpxg-59/ 2024-04-12T03:39:19.651 DEBUG:boto:Data: 2024-04-12T03:39:19.651 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.651 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.651 DEBUG:boto:Params: {} 2024-04-12T03:39:19.651 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:39:19.651 DEBUG:boto:Token: None 2024-04-12T03:39:19.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gdKFtj9I2FuuG81k1B1368qdNjA= 2024-04-12T03:39:19.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gdKFtj9I2FuuG81k1B1368qdNjA=', 'Content-Length': '0'} 2024-04-12T03:39:19.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f935d38414cf81ea-006618ace7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:39:19.663 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.663 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.663 DEBUG:boto:path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.664 DEBUG:boto:auth_path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.664 DEBUG:boto:Method: GET 2024-04-12T03:39:19.664 DEBUG:boto:Path: /dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.664 DEBUG:boto:Data: 2024-04-12T03:39:19.664 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.664 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.664 DEBUG:boto:Params: {} 2024-04-12T03:39:19.664 DEBUG:boto:Token: None 2024-04-12T03:39:19.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/?versions 2024-04-12T03:39:19.665 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY= 2024-04-12T03:39:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY=', 'Content-Length': '0'} 2024-04-12T03:39:19.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3753ec01f2d57e-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.667 DEBUG:boto:b'dgbpxg-59999falseanulltrue2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:39:19.669 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:39:19.669 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.669 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.670 DEBUG:boto:path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.670 DEBUG:boto:auth_path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.670 DEBUG:boto:Method: GET 2024-04-12T03:39:19.670 DEBUG:boto:Path: /dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.670 DEBUG:boto:Data: 2024-04-12T03:39:19.670 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.670 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.670 DEBUG:boto:Params: {} 2024-04-12T03:39:19.670 DEBUG:boto:Token: None 2024-04-12T03:39:19.670 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/?versions 2024-04-12T03:39:19.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY= 2024-04-12T03:39:19.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY=', 'Content-Length': '0'} 2024-04-12T03:39:19.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d513d121bbc244e1-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.673 DEBUG:boto:b'dgbpxg-59999falseanulltrue2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.673 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:39:19.673 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:39:19.674 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:39:19.674 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.674 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.674 DEBUG:boto:path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.675 DEBUG:boto:auth_path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.675 DEBUG:boto:Method: GET 2024-04-12T03:39:19.675 DEBUG:boto:Path: /dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.675 DEBUG:boto:Data: 2024-04-12T03:39:19.675 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.675 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.675 DEBUG:boto:Params: {} 2024-04-12T03:39:19.675 DEBUG:boto:Token: None 2024-04-12T03:39:19.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/?versions 2024-04-12T03:39:19.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY= 2024-04-12T03:39:19.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY=', 'Content-Length': '0'} 2024-04-12T03:39:19.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057add4964c09d789-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.678 DEBUG:boto:b'dgbpxg-59999falseanulltrue2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.679 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.679 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.679 DEBUG:boto:path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.679 DEBUG:boto:auth_path=/dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.679 DEBUG:boto:Method: GET 2024-04-12T03:39:19.679 DEBUG:boto:Path: /dgbpxg-59/?versions&max-keys=999 2024-04-12T03:39:19.679 DEBUG:boto:Data: 2024-04-12T03:39:19.679 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.680 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.680 DEBUG:boto:Params: {} 2024-04-12T03:39:19.680 DEBUG:boto:Token: None 2024-04-12T03:39:19.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/?versions 2024-04-12T03:39:19.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY= 2024-04-12T03:39:19.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vORbHFKxIWr3RqXVV0dz+UELVyY=', 'Content-Length': '0'} 2024-04-12T03:39:19.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38af99103e41c44-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.682 DEBUG:boto:b'dgbpxg-59999falseanulltrue2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:39:19.683 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.683 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.683 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.683 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.683 DEBUG:boto:Method: GET 2024-04-12T03:39:19.684 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.684 DEBUG:boto:Data: 2024-04-12T03:39:19.684 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.684 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.684 DEBUG:boto:Params: {} 2024-04-12T03:39:19.684 DEBUG:boto:Token: None 2024-04-12T03:39:19.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0= 2024-04-12T03:39:19.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0=', 'Content-Length': '0'} 2024-04-12T03:39:19.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0190fc6da547868-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.687 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.687 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.687 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.687 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.687 DEBUG:boto:Method: GET 2024-04-12T03:39:19.687 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.688 DEBUG:boto:Data: 2024-04-12T03:39:19.688 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.688 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.688 DEBUG:boto:Params: {} 2024-04-12T03:39:19.688 DEBUG:boto:Token: None 2024-04-12T03:39:19.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0= 2024-04-12T03:39:19.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0=', 'Content-Length': '0'} 2024-04-12T03:39:19.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd15309e801e623-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.690 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.690 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.690 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.691 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.691 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.691 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.691 DEBUG:boto:Data: 2024-04-12T03:39:19.691 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.691 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.691 DEBUG:boto:Params: {} 2024-04-12T03:39:19.691 DEBUG:boto:Token: None 2024-04-12T03:39:19.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dhzuHRVAjvuj6Sy866PWIaZ3QqI= 2024-04-12T03:39:19.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dhzuHRVAjvuj6Sy866PWIaZ3QqI=', 'Content-Length': '0'} 2024-04-12T03:39:19.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085add73699636dea-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.694 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.694 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.694 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.694 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.694 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.694 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.694 DEBUG:boto:Data: 2024-04-12T03:39:19.694 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.694 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.695 DEBUG:boto:Params: {} 2024-04-12T03:39:19.695 DEBUG:boto:Token: None 2024-04-12T03:39:19.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dhzuHRVAjvuj6Sy866PWIaZ3QqI= 2024-04-12T03:39:19.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dhzuHRVAjvuj6Sy866PWIaZ3QqI=', 'Content-Length': '0'} 2024-04-12T03:39:19.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c9573dc9dab88d-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:39:19.697 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.697 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.697 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.697 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.697 DEBUG:boto:Method: GET 2024-04-12T03:39:19.697 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.697 DEBUG:boto:Data: 2024-04-12T03:39:19.697 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.697 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.698 DEBUG:boto:Params: {} 2024-04-12T03:39:19.698 DEBUG:boto:Token: None 2024-04-12T03:39:19.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0= 2024-04-12T03:39:19.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0=', 'Content-Length': '0'} 2024-04-12T03:39:19.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82b4bcc445006a7-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.700 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.700 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.700 DEBUG:boto:path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.700 DEBUG:boto:auth_path=/dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.700 DEBUG:boto:Method: GET 2024-04-12T03:39:19.700 DEBUG:boto:Path: /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.700 DEBUG:boto:Data: 2024-04-12T03:39:19.700 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.701 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.701 DEBUG:boto:Params: {} 2024-04-12T03:39:19.701 DEBUG:boto:Token: None 2024-04-12T03:39:19.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a?versionId=null 2024-04-12T03:39:19.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0= 2024-04-12T03:39:19.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l+HJ5gN3sxbQrjEb9kLq2UgQAE0=', 'Content-Length': '0'} 2024-04-12T03:39:19.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e89e28fc2cb48d95-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.703 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.703 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.703 DEBUG:boto:path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.703 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.703 DEBUG:boto:Method: GET 2024-04-12T03:39:19.703 DEBUG:boto:Path: /dgbpxg-59/?prefix=a 2024-04-12T03:39:19.703 DEBUG:boto:Data: 2024-04-12T03:39:19.703 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.704 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.704 DEBUG:boto:Params: {} 2024-04-12T03:39:19.704 DEBUG:boto:Token: None 2024-04-12T03:39:19.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3ba1d5610d33895-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.706 DEBUG:boto:b'dgbpxg-59a1000falsea2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.707 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.707 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.707 DEBUG:boto:path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.707 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.707 DEBUG:boto:Method: GET 2024-04-12T03:39:19.707 DEBUG:boto:Path: /dgbpxg-59/?prefix=a 2024-04-12T03:39:19.707 DEBUG:boto:Data: 2024-04-12T03:39:19.707 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.708 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.708 DEBUG:boto:Params: {} 2024-04-12T03:39:19.708 DEBUG:boto:Token: None 2024-04-12T03:39:19.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba58a75b3b5ccb6-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.710 DEBUG:boto:b'dgbpxg-59a1000falsea2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.710 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.710 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.711 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.711 DEBUG:boto:Path: /dgbpxg-59/a 2024-04-12T03:39:19.711 DEBUG:boto:Data: 2024-04-12T03:39:19.711 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.711 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.711 DEBUG:boto:Params: {} 2024-04-12T03:39:19.711 DEBUG:boto:Token: None 2024-04-12T03:39:19.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a 2024-04-12T03:39:19.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1rRH33gfIXFiVN5AcwMnTpyxJss= 2024-04-12T03:39:19.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1rRH33gfIXFiVN5AcwMnTpyxJss=', 'Content-Length': '0'} 2024-04-12T03:39:19.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e285f39e0679bf1-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.713 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.713 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.713 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.713 DEBUG:boto:Path: /dgbpxg-59/a 2024-04-12T03:39:19.713 DEBUG:boto:Data: 2024-04-12T03:39:19.713 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.714 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.714 DEBUG:boto:Params: {} 2024-04-12T03:39:19.714 DEBUG:boto:Token: None 2024-04-12T03:39:19.714 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a 2024-04-12T03:39:19.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1rRH33gfIXFiVN5AcwMnTpyxJss= 2024-04-12T03:39:19.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1rRH33gfIXFiVN5AcwMnTpyxJss=', 'Content-Length': '0'} 2024-04-12T03:39:19.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6309297854439b-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:39:19.716 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.716 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.716 DEBUG:boto:Method: GET 2024-04-12T03:39:19.716 DEBUG:boto:Path: /dgbpxg-59/a 2024-04-12T03:39:19.716 DEBUG:boto:Data: 2024-04-12T03:39:19.716 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.717 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.717 DEBUG:boto:Params: {} 2024-04-12T03:39:19.717 DEBUG:boto:Token: None 2024-04-12T03:39:19.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a 2024-04-12T03:39:19.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jYZvKYgBECYu0fK+H11n6J7BZwI= 2024-04-12T03:39:19.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jYZvKYgBECYu0fK+H11n6J7BZwI=', 'Content-Length': '0'} 2024-04-12T03:39:19.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c65612cb60993ca-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.766 DEBUG:boto:path=/dgbpxg-59/a 2024-04-12T03:39:19.766 DEBUG:boto:auth_path=/dgbpxg-59/a 2024-04-12T03:39:19.766 DEBUG:boto:Method: GET 2024-04-12T03:39:19.767 DEBUG:boto:Path: /dgbpxg-59/a 2024-04-12T03:39:19.767 DEBUG:boto:Data: 2024-04-12T03:39:19.767 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.767 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.767 DEBUG:boto:Params: {} 2024-04-12T03:39:19.767 DEBUG:boto:Token: None 2024-04-12T03:39:19.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/a 2024-04-12T03:39:19.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jYZvKYgBECYu0fK+H11n6J7BZwI= 2024-04-12T03:39:19.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jYZvKYgBECYu0fK+H11n6J7BZwI=', 'Content-Length': '0'} 2024-04-12T03:39:19.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008783c8339b8c8fee-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.769 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.769 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.769 DEBUG:boto:path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.769 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.770 DEBUG:boto:Method: GET 2024-04-12T03:39:19.770 DEBUG:boto:Path: /dgbpxg-59/?prefix=a 2024-04-12T03:39:19.770 DEBUG:boto:Data: 2024-04-12T03:39:19.770 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.770 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.770 DEBUG:boto:Params: {} 2024-04-12T03:39:19.770 DEBUG:boto:Token: None 2024-04-12T03:39:19.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a4a10b2c00ed0a-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.773 DEBUG:boto:b'dgbpxg-59a1000falsea2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.773 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.773 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.773 DEBUG:boto:path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.773 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=a 2024-04-12T03:39:19.773 DEBUG:boto:Method: GET 2024-04-12T03:39:19.773 DEBUG:boto:Path: /dgbpxg-59/?prefix=a 2024-04-12T03:39:19.774 DEBUG:boto:Data: 2024-04-12T03:39:19.774 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.774 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.774 DEBUG:boto:Params: {} 2024-04-12T03:39:19.774 DEBUG:boto:Token: None 2024-04-12T03:39:19.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.774 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aae8636a52264ec-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.776 DEBUG:boto:b'dgbpxg-59a1000falsea2024-04-12T03:36:32.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:39:19.777 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.777 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.777 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.777 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.777 DEBUG:boto:Method: GET 2024-04-12T03:39:19.777 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.777 DEBUG:boto:Data: 2024-04-12T03:39:19.777 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.777 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.778 DEBUG:boto:Params: {} 2024-04-12T03:39:19.778 DEBUG:boto:Token: None 2024-04-12T03:39:19.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.778 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0= 2024-04-12T03:39:19.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0=', 'Content-Length': '0'} 2024-04-12T03:39:19.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d5d3f7894715ee0-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.780 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.780 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.780 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.780 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.780 DEBUG:boto:Method: GET 2024-04-12T03:39:19.781 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.781 DEBUG:boto:Data: 2024-04-12T03:39:19.781 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.781 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.781 DEBUG:boto:Params: {} 2024-04-12T03:39:19.781 DEBUG:boto:Token: None 2024-04-12T03:39:19.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.781 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0= 2024-04-12T03:39:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0=', 'Content-Length': '0'} 2024-04-12T03:39:19.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f43d7894123cc0-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.784 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.784 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.784 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.784 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.784 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.784 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.784 DEBUG:boto:Data: 2024-04-12T03:39:19.784 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.785 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.785 DEBUG:boto:Params: {} 2024-04-12T03:39:19.785 DEBUG:boto:Token: None 2024-04-12T03:39:19.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xc8Vmo/LkSReypboA/QMkcWGO58= 2024-04-12T03:39:19.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xc8Vmo/LkSReypboA/QMkcWGO58=', 'Content-Length': '0'} 2024-04-12T03:39:19.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004308a3464c0d4f96-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.787 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.787 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.787 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.787 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.787 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.787 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.788 DEBUG:boto:Data: 2024-04-12T03:39:19.788 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.788 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.788 DEBUG:boto:Params: {} 2024-04-12T03:39:19.788 DEBUG:boto:Token: None 2024-04-12T03:39:19.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xc8Vmo/LkSReypboA/QMkcWGO58= 2024-04-12T03:39:19.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xc8Vmo/LkSReypboA/QMkcWGO58=', 'Content-Length': '0'} 2024-04-12T03:39:19.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2406db6fedb622-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:39:19.791 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.791 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.791 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.791 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.791 DEBUG:boto:Method: GET 2024-04-12T03:39:19.791 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.791 DEBUG:boto:Data: 2024-04-12T03:39:19.791 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.791 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.791 DEBUG:boto:Params: {} 2024-04-12T03:39:19.792 DEBUG:boto:Token: None 2024-04-12T03:39:19.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0= 2024-04-12T03:39:19.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0=', 'Content-Length': '0'} 2024-04-12T03:39:19.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb04148525b212b4-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.794 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.794 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.794 DEBUG:boto:path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.794 DEBUG:boto:auth_path=/dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.794 DEBUG:boto:Method: GET 2024-04-12T03:39:19.794 DEBUG:boto:Path: /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.794 DEBUG:boto:Data: 2024-04-12T03:39:19.795 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.795 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.795 DEBUG:boto:Params: {} 2024-04-12T03:39:19.795 DEBUG:boto:Token: None 2024-04-12T03:39:19.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b?versionId=null 2024-04-12T03:39:19.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0= 2024-04-12T03:39:19.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EoCJyCXC1KqW78DZF2X02wKeHb0=', 'Content-Length': '0'} 2024-04-12T03:39:19.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004205a089f6c80e8f-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.842 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.842 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.842 DEBUG:boto:path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.843 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.843 DEBUG:boto:Method: GET 2024-04-12T03:39:19.843 DEBUG:boto:Path: /dgbpxg-59/?prefix=b 2024-04-12T03:39:19.843 DEBUG:boto:Data: 2024-04-12T03:39:19.843 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.843 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.843 DEBUG:boto:Params: {} 2024-04-12T03:39:19.843 DEBUG:boto:Token: None 2024-04-12T03:39:19.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055acc1622ae6ac73-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.846 DEBUG:boto:b'dgbpxg-59b1000falseb2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.846 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.846 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.846 DEBUG:boto:path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.846 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.846 DEBUG:boto:Method: GET 2024-04-12T03:39:19.847 DEBUG:boto:Path: /dgbpxg-59/?prefix=b 2024-04-12T03:39:19.847 DEBUG:boto:Data: 2024-04-12T03:39:19.847 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.847 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.847 DEBUG:boto:Params: {} 2024-04-12T03:39:19.847 DEBUG:boto:Token: None 2024-04-12T03:39:19.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009536b2c79f4e28c3-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.849 DEBUG:boto:b'dgbpxg-59b1000falseb2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.849 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.849 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.850 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.850 DEBUG:boto:Path: /dgbpxg-59/b 2024-04-12T03:39:19.850 DEBUG:boto:Data: 2024-04-12T03:39:19.850 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.850 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.850 DEBUG:boto:Params: {} 2024-04-12T03:39:19.850 DEBUG:boto:Token: None 2024-04-12T03:39:19.850 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b 2024-04-12T03:39:19.850 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dg3mRpgk8NjRkNoqNiho7Maf/I8= 2024-04-12T03:39:19.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dg3mRpgk8NjRkNoqNiho7Maf/I8=', 'Content-Length': '0'} 2024-04-12T03:39:19.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739a6f59f1a0fcd6-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.852 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.852 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.852 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.852 DEBUG:boto:Path: /dgbpxg-59/b 2024-04-12T03:39:19.852 DEBUG:boto:Data: 2024-04-12T03:39:19.852 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.853 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.853 DEBUG:boto:Params: {} 2024-04-12T03:39:19.853 DEBUG:boto:Token: None 2024-04-12T03:39:19.853 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b 2024-04-12T03:39:19.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dg3mRpgk8NjRkNoqNiho7Maf/I8= 2024-04-12T03:39:19.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dg3mRpgk8NjRkNoqNiho7Maf/I8=', 'Content-Length': '0'} 2024-04-12T03:39:19.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3228e57f6d7f8f7-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:39:19.855 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.855 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.855 DEBUG:boto:Method: GET 2024-04-12T03:39:19.855 DEBUG:boto:Path: /dgbpxg-59/b 2024-04-12T03:39:19.855 DEBUG:boto:Data: 2024-04-12T03:39:19.855 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.856 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.856 DEBUG:boto:Params: {} 2024-04-12T03:39:19.856 DEBUG:boto:Token: None 2024-04-12T03:39:19.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b 2024-04-12T03:39:19.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qG3yDlXO7XRGHpZ+tOqASAPacRI= 2024-04-12T03:39:19.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qG3yDlXO7XRGHpZ+tOqASAPacRI=', 'Content-Length': '0'} 2024-04-12T03:39:19.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc2be3875a4a74e-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.858 DEBUG:boto:path=/dgbpxg-59/b 2024-04-12T03:39:19.858 DEBUG:boto:auth_path=/dgbpxg-59/b 2024-04-12T03:39:19.858 DEBUG:boto:Method: GET 2024-04-12T03:39:19.858 DEBUG:boto:Path: /dgbpxg-59/b 2024-04-12T03:39:19.858 DEBUG:boto:Data: 2024-04-12T03:39:19.858 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.859 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.859 DEBUG:boto:Params: {} 2024-04-12T03:39:19.859 DEBUG:boto:Token: None 2024-04-12T03:39:19.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/b 2024-04-12T03:39:19.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qG3yDlXO7XRGHpZ+tOqASAPacRI= 2024-04-12T03:39:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qG3yDlXO7XRGHpZ+tOqASAPacRI=', 'Content-Length': '0'} 2024-04-12T03:39:19.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e4a4a20256a2abe-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.861 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.861 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.861 DEBUG:boto:path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.861 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.861 DEBUG:boto:Method: GET 2024-04-12T03:39:19.861 DEBUG:boto:Path: /dgbpxg-59/?prefix=b 2024-04-12T03:39:19.862 DEBUG:boto:Data: 2024-04-12T03:39:19.862 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.862 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.862 DEBUG:boto:Params: {} 2024-04-12T03:39:19.862 DEBUG:boto:Token: None 2024-04-12T03:39:19.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f213b5a98fac09f0-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.864 DEBUG:boto:b'dgbpxg-59b1000falseb2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.865 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.865 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.865 DEBUG:boto:path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.865 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=b 2024-04-12T03:39:19.865 DEBUG:boto:Method: GET 2024-04-12T03:39:19.865 DEBUG:boto:Path: /dgbpxg-59/?prefix=b 2024-04-12T03:39:19.865 DEBUG:boto:Data: 2024-04-12T03:39:19.865 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.866 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.866 DEBUG:boto:Params: {} 2024-04-12T03:39:19.866 DEBUG:boto:Token: None 2024-04-12T03:39:19.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25954f6a4506d10-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.868 DEBUG:boto:b'dgbpxg-59b1000falseb2024-04-12T03:36:32.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:39:19.868 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.868 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.868 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.868 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.868 DEBUG:boto:Method: GET 2024-04-12T03:39:19.869 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.869 DEBUG:boto:Data: 2024-04-12T03:39:19.869 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.869 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.869 DEBUG:boto:Params: {} 2024-04-12T03:39:19.869 DEBUG:boto:Token: None 2024-04-12T03:39:19.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.869 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8= 2024-04-12T03:39:19.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8=', 'Content-Length': '0'} 2024-04-12T03:39:19.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001600cb4644402e19-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.871 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.871 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.871 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.871 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.871 DEBUG:boto:Method: GET 2024-04-12T03:39:19.872 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.872 DEBUG:boto:Data: 2024-04-12T03:39:19.872 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.872 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.872 DEBUG:boto:Params: {} 2024-04-12T03:39:19.872 DEBUG:boto:Token: None 2024-04-12T03:39:19.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8= 2024-04-12T03:39:19.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8=', 'Content-Length': '0'} 2024-04-12T03:39:19.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c31187a65d6b60-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.874 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.874 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.875 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.875 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.875 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.875 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.875 DEBUG:boto:Data: 2024-04-12T03:39:19.875 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.875 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.875 DEBUG:boto:Params: {} 2024-04-12T03:39:19.875 DEBUG:boto:Token: None 2024-04-12T03:39:19.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Mf7zU7NxlRvqzc78TnWaw459i/w= 2024-04-12T03:39:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Mf7zU7NxlRvqzc78TnWaw459i/w=', 'Content-Length': '0'} 2024-04-12T03:39:19.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2413674c5e523b-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.877 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.878 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.878 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.878 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.878 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.878 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.878 DEBUG:boto:Data: 2024-04-12T03:39:19.878 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.878 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.878 DEBUG:boto:Params: {} 2024-04-12T03:39:19.879 DEBUG:boto:Token: None 2024-04-12T03:39:19.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Mf7zU7NxlRvqzc78TnWaw459i/w= 2024-04-12T03:39:19.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Mf7zU7NxlRvqzc78TnWaw459i/w=', 'Content-Length': '0'} 2024-04-12T03:39:19.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007899c6350ce22e44-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:39:19.881 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.881 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.881 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.881 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.881 DEBUG:boto:Method: GET 2024-04-12T03:39:19.881 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.882 DEBUG:boto:Data: 2024-04-12T03:39:19.882 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.882 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.882 DEBUG:boto:Params: {} 2024-04-12T03:39:19.882 DEBUG:boto:Token: None 2024-04-12T03:39:19.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8= 2024-04-12T03:39:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8=', 'Content-Length': '0'} 2024-04-12T03:39:19.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59736a47eaf5437-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.926 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.927 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.927 DEBUG:boto:path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.927 DEBUG:boto:auth_path=/dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.927 DEBUG:boto:Method: GET 2024-04-12T03:39:19.927 DEBUG:boto:Path: /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.927 DEBUG:boto:Data: 2024-04-12T03:39:19.927 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.927 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.927 DEBUG:boto:Params: {} 2024-04-12T03:39:19.928 DEBUG:boto:Token: None 2024-04-12T03:39:19.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c?versionId=null 2024-04-12T03:39:19.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8= 2024-04-12T03:39:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/b8vTp0QKqDUamqvYVB+HbARcA8=', 'Content-Length': '0'} 2024-04-12T03:39:19.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c1ec8aba1bf8cc-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.930 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.930 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.930 DEBUG:boto:path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.930 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.930 DEBUG:boto:Method: GET 2024-04-12T03:39:19.930 DEBUG:boto:Path: /dgbpxg-59/?prefix=c 2024-04-12T03:39:19.930 DEBUG:boto:Data: 2024-04-12T03:39:19.930 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.931 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.931 DEBUG:boto:Params: {} 2024-04-12T03:39:19.931 DEBUG:boto:Token: None 2024-04-12T03:39:19.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8fa365a2227a5f-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.934 DEBUG:boto:b'dgbpxg-59c1000falsec2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.934 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.934 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.934 DEBUG:boto:path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.934 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.935 DEBUG:boto:Method: GET 2024-04-12T03:39:19.935 DEBUG:boto:Path: /dgbpxg-59/?prefix=c 2024-04-12T03:39:19.935 DEBUG:boto:Data: 2024-04-12T03:39:19.935 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.935 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.935 DEBUG:boto:Params: {} 2024-04-12T03:39:19.935 DEBUG:boto:Token: None 2024-04-12T03:39:19.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7be2d5cce333214-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.937 DEBUG:boto:b'dgbpxg-59c1000falsec2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.937 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.938 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.938 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.938 DEBUG:boto:Path: /dgbpxg-59/c 2024-04-12T03:39:19.938 DEBUG:boto:Data: 2024-04-12T03:39:19.938 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.938 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.938 DEBUG:boto:Params: {} 2024-04-12T03:39:19.938 DEBUG:boto:Token: None 2024-04-12T03:39:19.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c 2024-04-12T03:39:19.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AcqMrhMjwRhPOcyctJ5pB2aghiE= 2024-04-12T03:39:19.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AcqMrhMjwRhPOcyctJ5pB2aghiE=', 'Content-Length': '0'} 2024-04-12T03:39:19.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097cb687e7747574f-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.940 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.940 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.940 DEBUG:boto:Method: HEAD 2024-04-12T03:39:19.941 DEBUG:boto:Path: /dgbpxg-59/c 2024-04-12T03:39:19.941 DEBUG:boto:Data: 2024-04-12T03:39:19.941 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.941 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.941 DEBUG:boto:Params: {} 2024-04-12T03:39:19.941 DEBUG:boto:Token: None 2024-04-12T03:39:19.941 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c 2024-04-12T03:39:19.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AcqMrhMjwRhPOcyctJ5pB2aghiE= 2024-04-12T03:39:19.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AcqMrhMjwRhPOcyctJ5pB2aghiE=', 'Content-Length': '0'} 2024-04-12T03:39:19.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a579c36aa434b335-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:39:19.943 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.943 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.943 DEBUG:boto:Method: GET 2024-04-12T03:39:19.943 DEBUG:boto:Path: /dgbpxg-59/c 2024-04-12T03:39:19.943 DEBUG:boto:Data: 2024-04-12T03:39:19.944 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.944 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.944 DEBUG:boto:Params: {} 2024-04-12T03:39:19.944 DEBUG:boto:Token: None 2024-04-12T03:39:19.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c 2024-04-12T03:39:19.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k5lXtk0mNcVne1b4phS+BOiQ/C8= 2024-04-12T03:39:19.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k5lXtk0mNcVne1b4phS+BOiQ/C8=', 'Content-Length': '0'} 2024-04-12T03:39:19.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8ca4ebcd0ede64-006618ace7-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.946 DEBUG:boto:path=/dgbpxg-59/c 2024-04-12T03:39:19.946 DEBUG:boto:auth_path=/dgbpxg-59/c 2024-04-12T03:39:19.947 DEBUG:boto:Method: GET 2024-04-12T03:39:19.947 DEBUG:boto:Path: /dgbpxg-59/c 2024-04-12T03:39:19.947 DEBUG:boto:Data: 2024-04-12T03:39:19.947 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.947 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.947 DEBUG:boto:Params: {} 2024-04-12T03:39:19.947 DEBUG:boto:Token: None 2024-04-12T03:39:19.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/c 2024-04-12T03:39:19.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k5lXtk0mNcVne1b4phS+BOiQ/C8= 2024-04-12T03:39:19.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k5lXtk0mNcVne1b4phS+BOiQ/C8=', 'Content-Length': '0'} 2024-04-12T03:39:19.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072b128c1cf14978-006618ace7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.990 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.990 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.991 DEBUG:boto:path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.991 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.991 DEBUG:boto:Method: GET 2024-04-12T03:39:19.991 DEBUG:boto:Path: /dgbpxg-59/?prefix=c 2024-04-12T03:39:19.991 DEBUG:boto:Data: 2024-04-12T03:39:19.991 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.991 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.991 DEBUG:boto:Params: {} 2024-04-12T03:39:19.991 DEBUG:boto:Token: None 2024-04-12T03:39:19.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db82b8b258502cbf-006618ace7-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.995 DEBUG:boto:b'dgbpxg-59c1000falsec2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.995 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:19.995 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:19.995 DEBUG:boto:path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.995 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=c 2024-04-12T03:39:19.995 DEBUG:boto:Method: GET 2024-04-12T03:39:19.996 DEBUG:boto:Path: /dgbpxg-59/?prefix=c 2024-04-12T03:39:19.996 DEBUG:boto:Data: 2024-04-12T03:39:19.996 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:19.996 DEBUG:boto:Port: 8000 2024-04-12T03:39:19.996 DEBUG:boto:Params: {} 2024-04-12T03:39:19.996 DEBUG:boto:Token: None 2024-04-12T03:39:19.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:19 GMT /dgbpxg-59/ 2024-04-12T03:39:19.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U= 2024-04-12T03:39:19.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yyP5Py29w0bPFq5bCyyX0bv+82U=', 'Content-Length': '0'} 2024-04-12T03:39:19.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aea446f28bd33fd-006618ace7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:19.998 DEBUG:boto:b'dgbpxg-59c1000falsec2024-04-12T03:36:32.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:19.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:39:19.999 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:19.999 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:19.999 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:19.999 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:19.999 DEBUG:boto:Method: GET 2024-04-12T03:39:19.999 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:19.999 DEBUG:boto:Data: 2024-04-12T03:39:19.999 DEBUG:boto:Headers: {} 2024-04-12T03:39:19.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:19.999 DEBUG:boto:Port: 8001 2024-04-12T03:39:19.999 DEBUG:boto:Params: {} 2024-04-12T03:39:20.000 DEBUG:boto:Token: None 2024-04-12T03:39:20.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE= 2024-04-12T03:39:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE=', 'Content-Length': '0'} 2024-04-12T03:39:20.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007140c724e2a37dc-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.002 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.002 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.002 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.002 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.002 DEBUG:boto:Method: GET 2024-04-12T03:39:20.002 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.002 DEBUG:boto:Data: 2024-04-12T03:39:20.002 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.003 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.003 DEBUG:boto:Params: {} 2024-04-12T03:39:20.003 DEBUG:boto:Token: None 2024-04-12T03:39:20.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE= 2024-04-12T03:39:20.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE=', 'Content-Length': '0'} 2024-04-12T03:39:20.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050626d191a48ad46-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.005 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.005 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.005 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.006 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.006 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.006 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.006 DEBUG:boto:Data: 2024-04-12T03:39:20.006 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.006 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.006 DEBUG:boto:Params: {} 2024-04-12T03:39:20.006 DEBUG:boto:Token: None 2024-04-12T03:39:20.006 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qCs8o1UL0Mgm6XRuC/HjmXrssGQ= 2024-04-12T03:39:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qCs8o1UL0Mgm6XRuC/HjmXrssGQ=', 'Content-Length': '0'} 2024-04-12T03:39:20.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abd06f255326643-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.008 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.008 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.008 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.008 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.009 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.009 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.009 DEBUG:boto:Data: 2024-04-12T03:39:20.009 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.009 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.009 DEBUG:boto:Params: {} 2024-04-12T03:39:20.009 DEBUG:boto:Token: None 2024-04-12T03:39:20.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.010 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qCs8o1UL0Mgm6XRuC/HjmXrssGQ= 2024-04-12T03:39:20.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qCs8o1UL0Mgm6XRuC/HjmXrssGQ=', 'Content-Length': '0'} 2024-04-12T03:39:20.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a32badf884abfa0-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:39:20.012 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.012 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.012 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.012 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.012 DEBUG:boto:Method: GET 2024-04-12T03:39:20.012 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.012 DEBUG:boto:Data: 2024-04-12T03:39:20.012 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.013 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.013 DEBUG:boto:Params: {} 2024-04-12T03:39:20.013 DEBUG:boto:Token: None 2024-04-12T03:39:20.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE= 2024-04-12T03:39:20.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE=', 'Content-Length': '0'} 2024-04-12T03:39:20.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302bd2ac5979e012-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.015 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.015 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.015 DEBUG:boto:path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.015 DEBUG:boto:auth_path=/dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.015 DEBUG:boto:Method: GET 2024-04-12T03:39:20.015 DEBUG:boto:Path: /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.016 DEBUG:boto:Data: 2024-04-12T03:39:20.016 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.016 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.016 DEBUG:boto:Params: {} 2024-04-12T03:39:20.016 DEBUG:boto:Token: None 2024-04-12T03:39:20.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d?versionId=null 2024-04-12T03:39:20.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE= 2024-04-12T03:39:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tF2F316A+RUkHiIqXcHVN9iTnLE=', 'Content-Length': '0'} 2024-04-12T03:39:20.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3ccf49c7a72801-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.058 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.058 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.059 DEBUG:boto:path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.059 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.059 DEBUG:boto:Method: GET 2024-04-12T03:39:20.059 DEBUG:boto:Path: /dgbpxg-59/?prefix=d 2024-04-12T03:39:20.059 DEBUG:boto:Data: 2024-04-12T03:39:20.059 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.059 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.059 DEBUG:boto:Params: {} 2024-04-12T03:39:20.059 DEBUG:boto:Token: None 2024-04-12T03:39:20.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df37d26796d17f84-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.062 DEBUG:boto:b'dgbpxg-59d1000falsed2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.062 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.062 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.062 DEBUG:boto:path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.063 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.063 DEBUG:boto:Method: GET 2024-04-12T03:39:20.063 DEBUG:boto:Path: /dgbpxg-59/?prefix=d 2024-04-12T03:39:20.063 DEBUG:boto:Data: 2024-04-12T03:39:20.063 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.063 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.063 DEBUG:boto:Params: {} 2024-04-12T03:39:20.063 DEBUG:boto:Token: None 2024-04-12T03:39:20.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc78079fa69c86e2-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.065 DEBUG:boto:b'dgbpxg-59d1000falsed2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.066 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.066 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.066 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.066 DEBUG:boto:Path: /dgbpxg-59/d 2024-04-12T03:39:20.066 DEBUG:boto:Data: 2024-04-12T03:39:20.066 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.067 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.067 DEBUG:boto:Params: {} 2024-04-12T03:39:20.067 DEBUG:boto:Token: None 2024-04-12T03:39:20.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d 2024-04-12T03:39:20.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZUwIMjLhnFn3PWVibH2SpjwoQc= 2024-04-12T03:39:20.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZUwIMjLhnFn3PWVibH2SpjwoQc=', 'Content-Length': '0'} 2024-04-12T03:39:20.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453ecf8128584be9-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.069 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.069 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.069 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.069 DEBUG:boto:Path: /dgbpxg-59/d 2024-04-12T03:39:20.069 DEBUG:boto:Data: 2024-04-12T03:39:20.069 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.069 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.070 DEBUG:boto:Params: {} 2024-04-12T03:39:20.070 DEBUG:boto:Token: None 2024-04-12T03:39:20.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d 2024-04-12T03:39:20.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZUwIMjLhnFn3PWVibH2SpjwoQc= 2024-04-12T03:39:20.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZUwIMjLhnFn3PWVibH2SpjwoQc=', 'Content-Length': '0'} 2024-04-12T03:39:20.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055dfbc2a4ff94e0-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:39:20.072 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.072 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.072 DEBUG:boto:Method: GET 2024-04-12T03:39:20.073 DEBUG:boto:Path: /dgbpxg-59/d 2024-04-12T03:39:20.073 DEBUG:boto:Data: 2024-04-12T03:39:20.073 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.073 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.073 DEBUG:boto:Params: {} 2024-04-12T03:39:20.073 DEBUG:boto:Token: None 2024-04-12T03:39:20.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d 2024-04-12T03:39:20.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WzpgxxH9JPkp4u8lWi8NX33m4VA= 2024-04-12T03:39:20.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WzpgxxH9JPkp4u8lWi8NX33m4VA=', 'Content-Length': '0'} 2024-04-12T03:39:20.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858a0aeabd7f7bcd-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.075 DEBUG:boto:path=/dgbpxg-59/d 2024-04-12T03:39:20.075 DEBUG:boto:auth_path=/dgbpxg-59/d 2024-04-12T03:39:20.076 DEBUG:boto:Method: GET 2024-04-12T03:39:20.076 DEBUG:boto:Path: /dgbpxg-59/d 2024-04-12T03:39:20.076 DEBUG:boto:Data: 2024-04-12T03:39:20.076 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.076 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.076 DEBUG:boto:Params: {} 2024-04-12T03:39:20.076 DEBUG:boto:Token: None 2024-04-12T03:39:20.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/d 2024-04-12T03:39:20.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WzpgxxH9JPkp4u8lWi8NX33m4VA= 2024-04-12T03:39:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WzpgxxH9JPkp4u8lWi8NX33m4VA=', 'Content-Length': '0'} 2024-04-12T03:39:20.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1a19583886c05b-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.078 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.078 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.078 DEBUG:boto:path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.078 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.079 DEBUG:boto:Method: GET 2024-04-12T03:39:20.079 DEBUG:boto:Path: /dgbpxg-59/?prefix=d 2024-04-12T03:39:20.079 DEBUG:boto:Data: 2024-04-12T03:39:20.079 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.079 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.079 DEBUG:boto:Params: {} 2024-04-12T03:39:20.079 DEBUG:boto:Token: None 2024-04-12T03:39:20.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074bab6a549713c51-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.122 DEBUG:boto:b'dgbpxg-59d1000falsed2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.123 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.123 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.123 DEBUG:boto:path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.123 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=d 2024-04-12T03:39:20.123 DEBUG:boto:Method: GET 2024-04-12T03:39:20.123 DEBUG:boto:Path: /dgbpxg-59/?prefix=d 2024-04-12T03:39:20.123 DEBUG:boto:Data: 2024-04-12T03:39:20.123 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.124 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.124 DEBUG:boto:Params: {} 2024-04-12T03:39:20.124 DEBUG:boto:Token: None 2024-04-12T03:39:20.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c400d8ea97420cd2-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.126 DEBUG:boto:b'dgbpxg-59d1000falsed2024-04-12T03:36:33.014Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:39:20.126 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.127 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.127 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.127 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.127 DEBUG:boto:Method: GET 2024-04-12T03:39:20.127 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.127 DEBUG:boto:Data: 2024-04-12T03:39:20.127 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.127 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.127 DEBUG:boto:Params: {} 2024-04-12T03:39:20.127 DEBUG:boto:Token: None 2024-04-12T03:39:20.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0= 2024-04-12T03:39:20.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0=', 'Content-Length': '0'} 2024-04-12T03:39:20.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b45b051eb61481a-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.129 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.130 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.130 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.130 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.130 DEBUG:boto:Method: GET 2024-04-12T03:39:20.130 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.130 DEBUG:boto:Data: 2024-04-12T03:39:20.130 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.130 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.130 DEBUG:boto:Params: {} 2024-04-12T03:39:20.131 DEBUG:boto:Token: None 2024-04-12T03:39:20.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0= 2024-04-12T03:39:20.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0=', 'Content-Length': '0'} 2024-04-12T03:39:20.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e5cd99d4a60863-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.133 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.133 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.133 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.133 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.133 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.133 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.133 DEBUG:boto:Data: 2024-04-12T03:39:20.133 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.133 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.133 DEBUG:boto:Params: {} 2024-04-12T03:39:20.134 DEBUG:boto:Token: None 2024-04-12T03:39:20.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tE406NoemJvRogPGMi82TQHOlI8= 2024-04-12T03:39:20.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tE406NoemJvRogPGMi82TQHOlI8=', 'Content-Length': '0'} 2024-04-12T03:39:20.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cff42b260ecfb44-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.136 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.136 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.136 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.136 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.136 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.136 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.136 DEBUG:boto:Data: 2024-04-12T03:39:20.136 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.137 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.137 DEBUG:boto:Params: {} 2024-04-12T03:39:20.137 DEBUG:boto:Token: None 2024-04-12T03:39:20.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tE406NoemJvRogPGMi82TQHOlI8= 2024-04-12T03:39:20.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tE406NoemJvRogPGMi82TQHOlI8=', 'Content-Length': '0'} 2024-04-12T03:39:20.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b36116e488d5579-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:39:20.139 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.139 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.139 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.139 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.139 DEBUG:boto:Method: GET 2024-04-12T03:39:20.139 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.140 DEBUG:boto:Data: 2024-04-12T03:39:20.140 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.140 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.140 DEBUG:boto:Params: {} 2024-04-12T03:39:20.140 DEBUG:boto:Token: None 2024-04-12T03:39:20.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0= 2024-04-12T03:39:20.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0=', 'Content-Length': '0'} 2024-04-12T03:39:20.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514ca93ebac5418d-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.142 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.142 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.142 DEBUG:boto:path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.142 DEBUG:boto:auth_path=/dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.142 DEBUG:boto:Method: GET 2024-04-12T03:39:20.142 DEBUG:boto:Path: /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.143 DEBUG:boto:Data: 2024-04-12T03:39:20.143 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.143 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.143 DEBUG:boto:Params: {} 2024-04-12T03:39:20.143 DEBUG:boto:Token: None 2024-04-12T03:39:20.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e?versionId=null 2024-04-12T03:39:20.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0= 2024-04-12T03:39:20.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0e4b0Lau+asdNCwYyK6WRUGnyj0=', 'Content-Length': '0'} 2024-04-12T03:39:20.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440b587d7991192b-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.145 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.145 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.145 DEBUG:boto:path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.145 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.145 DEBUG:boto:Method: GET 2024-04-12T03:39:20.145 DEBUG:boto:Path: /dgbpxg-59/?prefix=e 2024-04-12T03:39:20.146 DEBUG:boto:Data: 2024-04-12T03:39:20.146 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.146 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.146 DEBUG:boto:Params: {} 2024-04-12T03:39:20.146 DEBUG:boto:Token: None 2024-04-12T03:39:20.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a400b1a9c33cc72-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.149 DEBUG:boto:b'dgbpxg-59e1000falsee2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.149 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.149 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.149 DEBUG:boto:path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.150 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.150 DEBUG:boto:Method: GET 2024-04-12T03:39:20.150 DEBUG:boto:Path: /dgbpxg-59/?prefix=e 2024-04-12T03:39:20.150 DEBUG:boto:Data: 2024-04-12T03:39:20.150 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.150 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.150 DEBUG:boto:Params: {} 2024-04-12T03:39:20.150 DEBUG:boto:Token: None 2024-04-12T03:39:20.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15bc49e1a26625f-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.152 DEBUG:boto:b'dgbpxg-59e1000falsee2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.153 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.153 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.153 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.153 DEBUG:boto:Path: /dgbpxg-59/e 2024-04-12T03:39:20.153 DEBUG:boto:Data: 2024-04-12T03:39:20.153 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.153 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.153 DEBUG:boto:Params: {} 2024-04-12T03:39:20.153 DEBUG:boto:Token: None 2024-04-12T03:39:20.154 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e 2024-04-12T03:39:20.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m1yYfFMsqdHo7YMerU/ztVg0KTI= 2024-04-12T03:39:20.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m1yYfFMsqdHo7YMerU/ztVg0KTI=', 'Content-Length': '0'} 2024-04-12T03:39:20.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bcfa47b6b327507-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.155 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.155 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.155 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.155 DEBUG:boto:Path: /dgbpxg-59/e 2024-04-12T03:39:20.156 DEBUG:boto:Data: 2024-04-12T03:39:20.156 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.156 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.156 DEBUG:boto:Params: {} 2024-04-12T03:39:20.156 DEBUG:boto:Token: None 2024-04-12T03:39:20.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e 2024-04-12T03:39:20.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m1yYfFMsqdHo7YMerU/ztVg0KTI= 2024-04-12T03:39:20.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m1yYfFMsqdHo7YMerU/ztVg0KTI=', 'Content-Length': '0'} 2024-04-12T03:39:20.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051808b9ed69c1328-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:39:20.158 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.158 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.158 DEBUG:boto:Method: GET 2024-04-12T03:39:20.158 DEBUG:boto:Path: /dgbpxg-59/e 2024-04-12T03:39:20.158 DEBUG:boto:Data: 2024-04-12T03:39:20.159 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.159 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.159 DEBUG:boto:Params: {} 2024-04-12T03:39:20.159 DEBUG:boto:Token: None 2024-04-12T03:39:20.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e 2024-04-12T03:39:20.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RTZSViTxK+AVsNSKxVs8Afu0HVo= 2024-04-12T03:39:20.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RTZSViTxK+AVsNSKxVs8Afu0HVo=', 'Content-Length': '0'} 2024-04-12T03:39:20.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008974c376fbc8bdef-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.161 DEBUG:boto:path=/dgbpxg-59/e 2024-04-12T03:39:20.161 DEBUG:boto:auth_path=/dgbpxg-59/e 2024-04-12T03:39:20.161 DEBUG:boto:Method: GET 2024-04-12T03:39:20.162 DEBUG:boto:Path: /dgbpxg-59/e 2024-04-12T03:39:20.162 DEBUG:boto:Data: 2024-04-12T03:39:20.162 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.162 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.162 DEBUG:boto:Params: {} 2024-04-12T03:39:20.162 DEBUG:boto:Token: None 2024-04-12T03:39:20.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/e 2024-04-12T03:39:20.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RTZSViTxK+AVsNSKxVs8Afu0HVo= 2024-04-12T03:39:20.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RTZSViTxK+AVsNSKxVs8Afu0HVo=', 'Content-Length': '0'} 2024-04-12T03:39:20.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009387636b6d052271-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.164 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.164 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.164 DEBUG:boto:path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.165 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.165 DEBUG:boto:Method: GET 2024-04-12T03:39:20.165 DEBUG:boto:Path: /dgbpxg-59/?prefix=e 2024-04-12T03:39:20.165 DEBUG:boto:Data: 2024-04-12T03:39:20.165 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.165 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.165 DEBUG:boto:Params: {} 2024-04-12T03:39:20.165 DEBUG:boto:Token: None 2024-04-12T03:39:20.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f3b7119b228e53-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.168 DEBUG:boto:b'dgbpxg-59e1000falsee2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.168 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.168 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.168 DEBUG:boto:path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.169 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=e 2024-04-12T03:39:20.169 DEBUG:boto:Method: GET 2024-04-12T03:39:20.169 DEBUG:boto:Path: /dgbpxg-59/?prefix=e 2024-04-12T03:39:20.169 DEBUG:boto:Data: 2024-04-12T03:39:20.169 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.169 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.169 DEBUG:boto:Params: {} 2024-04-12T03:39:20.169 DEBUG:boto:Token: None 2024-04-12T03:39:20.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.170 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579bc7ca8a45c07e-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.171 DEBUG:boto:b'dgbpxg-59e1000falsee2024-04-12T03:36:33.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:39:20.172 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.172 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.172 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.172 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.172 DEBUG:boto:Method: GET 2024-04-12T03:39:20.172 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.172 DEBUG:boto:Data: 2024-04-12T03:39:20.172 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.173 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.173 DEBUG:boto:Params: {} 2024-04-12T03:39:20.173 DEBUG:boto:Token: None 2024-04-12T03:39:20.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.173 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8= 2024-04-12T03:39:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8=', 'Content-Length': '0'} 2024-04-12T03:39:20.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b063e114191a566-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.175 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.175 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.175 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.175 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.175 DEBUG:boto:Method: GET 2024-04-12T03:39:20.175 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.175 DEBUG:boto:Data: 2024-04-12T03:39:20.175 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.176 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.176 DEBUG:boto:Params: {} 2024-04-12T03:39:20.176 DEBUG:boto:Token: None 2024-04-12T03:39:20.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8= 2024-04-12T03:39:20.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8=', 'Content-Length': '0'} 2024-04-12T03:39:20.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c39ea568ba83a2d9-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.178 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.178 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.178 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.178 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.178 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.178 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.178 DEBUG:boto:Data: 2024-04-12T03:39:20.178 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.179 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.179 DEBUG:boto:Params: {} 2024-04-12T03:39:20.179 DEBUG:boto:Token: None 2024-04-12T03:39:20.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1rOpCMO7ffk8BcOjRf6NauKDafM= 2024-04-12T03:39:20.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1rOpCMO7ffk8BcOjRf6NauKDafM=', 'Content-Length': '0'} 2024-04-12T03:39:20.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a38b17f973c879-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.181 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.181 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.181 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.181 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.181 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.181 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.181 DEBUG:boto:Data: 2024-04-12T03:39:20.181 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.181 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.181 DEBUG:boto:Params: {} 2024-04-12T03:39:20.181 DEBUG:boto:Token: None 2024-04-12T03:39:20.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1rOpCMO7ffk8BcOjRf6NauKDafM= 2024-04-12T03:39:20.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1rOpCMO7ffk8BcOjRf6NauKDafM=', 'Content-Length': '0'} 2024-04-12T03:39:20.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c3e0234e516140-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:39:20.184 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.184 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.184 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.184 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.184 DEBUG:boto:Method: GET 2024-04-12T03:39:20.184 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.184 DEBUG:boto:Data: 2024-04-12T03:39:20.184 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.184 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.184 DEBUG:boto:Params: {} 2024-04-12T03:39:20.185 DEBUG:boto:Token: None 2024-04-12T03:39:20.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.185 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8= 2024-04-12T03:39:20.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8=', 'Content-Length': '0'} 2024-04-12T03:39:20.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958afa22582e5a68-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.230 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.230 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.230 DEBUG:boto:path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.231 DEBUG:boto:auth_path=/dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.231 DEBUG:boto:Method: GET 2024-04-12T03:39:20.231 DEBUG:boto:Path: /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.231 DEBUG:boto:Data: 2024-04-12T03:39:20.231 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.231 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.231 DEBUG:boto:Params: {} 2024-04-12T03:39:20.231 DEBUG:boto:Token: None 2024-04-12T03:39:20.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f?versionId=null 2024-04-12T03:39:20.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8= 2024-04-12T03:39:20.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JPu57OPgDiI3KPg7zp5uxP7/7D8=', 'Content-Length': '0'} 2024-04-12T03:39:20.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740683a6df424287-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.234 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.234 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.234 DEBUG:boto:path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.234 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.234 DEBUG:boto:Method: GET 2024-04-12T03:39:20.234 DEBUG:boto:Path: /dgbpxg-59/?prefix=f 2024-04-12T03:39:20.234 DEBUG:boto:Data: 2024-04-12T03:39:20.234 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.235 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.235 DEBUG:boto:Params: {} 2024-04-12T03:39:20.235 DEBUG:boto:Token: None 2024-04-12T03:39:20.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed4d083e8380231c-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.238 DEBUG:boto:b'dgbpxg-59f1000falsef2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.238 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.238 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.238 DEBUG:boto:path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.238 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.238 DEBUG:boto:Method: GET 2024-04-12T03:39:20.238 DEBUG:boto:Path: /dgbpxg-59/?prefix=f 2024-04-12T03:39:20.238 DEBUG:boto:Data: 2024-04-12T03:39:20.239 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.239 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.239 DEBUG:boto:Params: {} 2024-04-12T03:39:20.239 DEBUG:boto:Token: None 2024-04-12T03:39:20.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6e34301a9059d2-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.241 DEBUG:boto:b'dgbpxg-59f1000falsef2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.241 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.241 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.241 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.241 DEBUG:boto:Path: /dgbpxg-59/f 2024-04-12T03:39:20.242 DEBUG:boto:Data: 2024-04-12T03:39:20.242 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.242 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.242 DEBUG:boto:Params: {} 2024-04-12T03:39:20.242 DEBUG:boto:Token: None 2024-04-12T03:39:20.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f 2024-04-12T03:39:20.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2Up8VtVMfNCN1NOMJ6eDg6lq5i4= 2024-04-12T03:39:20.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2Up8VtVMfNCN1NOMJ6eDg6lq5i4=', 'Content-Length': '0'} 2024-04-12T03:39:20.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004225df6c8048654f-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.244 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.245 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.245 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.245 DEBUG:boto:Path: /dgbpxg-59/f 2024-04-12T03:39:20.245 DEBUG:boto:Data: 2024-04-12T03:39:20.245 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.245 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.245 DEBUG:boto:Params: {} 2024-04-12T03:39:20.245 DEBUG:boto:Token: None 2024-04-12T03:39:20.245 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f 2024-04-12T03:39:20.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2Up8VtVMfNCN1NOMJ6eDg6lq5i4= 2024-04-12T03:39:20.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2Up8VtVMfNCN1NOMJ6eDg6lq5i4=', 'Content-Length': '0'} 2024-04-12T03:39:20.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812423e052d04ec1-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:39:20.248 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.248 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.248 DEBUG:boto:Method: GET 2024-04-12T03:39:20.248 DEBUG:boto:Path: /dgbpxg-59/f 2024-04-12T03:39:20.248 DEBUG:boto:Data: 2024-04-12T03:39:20.248 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.248 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.248 DEBUG:boto:Params: {} 2024-04-12T03:39:20.249 DEBUG:boto:Token: None 2024-04-12T03:39:20.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f 2024-04-12T03:39:20.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JL3PYVlkyqWe6OborFqrR8ROHX4= 2024-04-12T03:39:20.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JL3PYVlkyqWe6OborFqrR8ROHX4=', 'Content-Length': '0'} 2024-04-12T03:39:20.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee96f616379780a-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.251 DEBUG:boto:path=/dgbpxg-59/f 2024-04-12T03:39:20.251 DEBUG:boto:auth_path=/dgbpxg-59/f 2024-04-12T03:39:20.251 DEBUG:boto:Method: GET 2024-04-12T03:39:20.251 DEBUG:boto:Path: /dgbpxg-59/f 2024-04-12T03:39:20.251 DEBUG:boto:Data: 2024-04-12T03:39:20.251 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.252 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.252 DEBUG:boto:Params: {} 2024-04-12T03:39:20.252 DEBUG:boto:Token: None 2024-04-12T03:39:20.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/f 2024-04-12T03:39:20.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JL3PYVlkyqWe6OborFqrR8ROHX4= 2024-04-12T03:39:20.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JL3PYVlkyqWe6OborFqrR8ROHX4=', 'Content-Length': '0'} 2024-04-12T03:39:20.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e5a7144ee282fb-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.254 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.254 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.254 DEBUG:boto:path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.254 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.254 DEBUG:boto:Method: GET 2024-04-12T03:39:20.255 DEBUG:boto:Path: /dgbpxg-59/?prefix=f 2024-04-12T03:39:20.255 DEBUG:boto:Data: 2024-04-12T03:39:20.255 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.255 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.255 DEBUG:boto:Params: {} 2024-04-12T03:39:20.255 DEBUG:boto:Token: None 2024-04-12T03:39:20.255 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b4bbdf6c6402d2-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.258 DEBUG:boto:b'dgbpxg-59f1000falsef2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.258 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.258 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.259 DEBUG:boto:path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.259 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=f 2024-04-12T03:39:20.259 DEBUG:boto:Method: GET 2024-04-12T03:39:20.259 DEBUG:boto:Path: /dgbpxg-59/?prefix=f 2024-04-12T03:39:20.259 DEBUG:boto:Data: 2024-04-12T03:39:20.259 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.259 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.259 DEBUG:boto:Params: {} 2024-04-12T03:39:20.259 DEBUG:boto:Token: None 2024-04-12T03:39:20.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d784b7b8b6643408-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.261 DEBUG:boto:b'dgbpxg-59f1000falsef2024-04-12T03:36:33.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:39:20.262 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.262 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.262 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.262 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.262 DEBUG:boto:Method: GET 2024-04-12T03:39:20.262 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.263 DEBUG:boto:Data: 2024-04-12T03:39:20.263 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.263 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.263 DEBUG:boto:Params: {} 2024-04-12T03:39:20.263 DEBUG:boto:Token: None 2024-04-12T03:39:20.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg= 2024-04-12T03:39:20.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg=', 'Content-Length': '0'} 2024-04-12T03:39:20.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327a418126dca95c-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.266 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.266 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.266 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.266 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.266 DEBUG:boto:Method: GET 2024-04-12T03:39:20.266 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.266 DEBUG:boto:Data: 2024-04-12T03:39:20.266 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.267 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.267 DEBUG:boto:Params: {} 2024-04-12T03:39:20.267 DEBUG:boto:Token: None 2024-04-12T03:39:20.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg= 2024-04-12T03:39:20.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg=', 'Content-Length': '0'} 2024-04-12T03:39:20.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ecc135694e4e674-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.269 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.269 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.269 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.269 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.270 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.270 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.270 DEBUG:boto:Data: 2024-04-12T03:39:20.270 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.270 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.270 DEBUG:boto:Params: {} 2024-04-12T03:39:20.270 DEBUG:boto:Token: None 2024-04-12T03:39:20.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPF5darK05LDQ3wgt8G5Iv2HVIg= 2024-04-12T03:39:20.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPF5darK05LDQ3wgt8G5Iv2HVIg=', 'Content-Length': '0'} 2024-04-12T03:39:20.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96d8462e46a5d21-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.273 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.273 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.273 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.273 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.273 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.273 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.273 DEBUG:boto:Data: 2024-04-12T03:39:20.273 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.273 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.274 DEBUG:boto:Params: {} 2024-04-12T03:39:20.274 DEBUG:boto:Token: None 2024-04-12T03:39:20.274 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.274 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VPF5darK05LDQ3wgt8G5Iv2HVIg= 2024-04-12T03:39:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VPF5darK05LDQ3wgt8G5Iv2HVIg=', 'Content-Length': '0'} 2024-04-12T03:39:20.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ff1ee614905fb5-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:39:20.276 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.276 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.276 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.276 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.277 DEBUG:boto:Method: GET 2024-04-12T03:39:20.277 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.277 DEBUG:boto:Data: 2024-04-12T03:39:20.277 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.277 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.277 DEBUG:boto:Params: {} 2024-04-12T03:39:20.277 DEBUG:boto:Token: None 2024-04-12T03:39:20.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.278 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg= 2024-04-12T03:39:20.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg=', 'Content-Length': '0'} 2024-04-12T03:39:20.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18f97350de53bc3-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.280 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.280 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.280 DEBUG:boto:path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.280 DEBUG:boto:auth_path=/dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.280 DEBUG:boto:Method: GET 2024-04-12T03:39:20.280 DEBUG:boto:Path: /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.280 DEBUG:boto:Data: 2024-04-12T03:39:20.280 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.281 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.281 DEBUG:boto:Params: {} 2024-04-12T03:39:20.281 DEBUG:boto:Token: None 2024-04-12T03:39:20.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g?versionId=null 2024-04-12T03:39:20.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg= 2024-04-12T03:39:20.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hHHRxxvX2QCqZCAUvnvCuCiAvVg=', 'Content-Length': '0'} 2024-04-12T03:39:20.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9624be12341c327-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.283 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.283 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.283 DEBUG:boto:path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.283 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.283 DEBUG:boto:Method: GET 2024-04-12T03:39:20.284 DEBUG:boto:Path: /dgbpxg-59/?prefix=g 2024-04-12T03:39:20.284 DEBUG:boto:Data: 2024-04-12T03:39:20.284 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.284 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.284 DEBUG:boto:Params: {} 2024-04-12T03:39:20.284 DEBUG:boto:Token: None 2024-04-12T03:39:20.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f306ab34a9b2137-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.287 DEBUG:boto:b'dgbpxg-59g1000falseg2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.287 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.287 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.287 DEBUG:boto:path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.287 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.287 DEBUG:boto:Method: GET 2024-04-12T03:39:20.288 DEBUG:boto:Path: /dgbpxg-59/?prefix=g 2024-04-12T03:39:20.288 DEBUG:boto:Data: 2024-04-12T03:39:20.288 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.288 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.288 DEBUG:boto:Params: {} 2024-04-12T03:39:20.288 DEBUG:boto:Token: None 2024-04-12T03:39:20.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8543c4fc3414db-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.330 DEBUG:boto:b'dgbpxg-59g1000falseg2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.331 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.331 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.331 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.331 DEBUG:boto:Path: /dgbpxg-59/g 2024-04-12T03:39:20.331 DEBUG:boto:Data: 2024-04-12T03:39:20.331 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.331 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.331 DEBUG:boto:Params: {} 2024-04-12T03:39:20.331 DEBUG:boto:Token: None 2024-04-12T03:39:20.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g 2024-04-12T03:39:20.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W2O/WrzKY1SWPD+K7iRJ4P7I8i8= 2024-04-12T03:39:20.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W2O/WrzKY1SWPD+K7iRJ4P7I8i8=', 'Content-Length': '0'} 2024-04-12T03:39:20.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118d3e9c59de8264-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.334 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.334 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.334 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.334 DEBUG:boto:Path: /dgbpxg-59/g 2024-04-12T03:39:20.334 DEBUG:boto:Data: 2024-04-12T03:39:20.334 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.334 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.334 DEBUG:boto:Params: {} 2024-04-12T03:39:20.335 DEBUG:boto:Token: None 2024-04-12T03:39:20.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g 2024-04-12T03:39:20.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W2O/WrzKY1SWPD+K7iRJ4P7I8i8= 2024-04-12T03:39:20.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W2O/WrzKY1SWPD+K7iRJ4P7I8i8=', 'Content-Length': '0'} 2024-04-12T03:39:20.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b06aee9fb710d7b-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:39:20.337 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.337 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.337 DEBUG:boto:Method: GET 2024-04-12T03:39:20.337 DEBUG:boto:Path: /dgbpxg-59/g 2024-04-12T03:39:20.337 DEBUG:boto:Data: 2024-04-12T03:39:20.337 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.337 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.337 DEBUG:boto:Params: {} 2024-04-12T03:39:20.337 DEBUG:boto:Token: None 2024-04-12T03:39:20.338 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g 2024-04-12T03:39:20.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UbWhMA5R/xa1tJd9xghTaN1Ad7w= 2024-04-12T03:39:20.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UbWhMA5R/xa1tJd9xghTaN1Ad7w=', 'Content-Length': '0'} 2024-04-12T03:39:20.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60846491a5f1821-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.340 DEBUG:boto:path=/dgbpxg-59/g 2024-04-12T03:39:20.340 DEBUG:boto:auth_path=/dgbpxg-59/g 2024-04-12T03:39:20.340 DEBUG:boto:Method: GET 2024-04-12T03:39:20.340 DEBUG:boto:Path: /dgbpxg-59/g 2024-04-12T03:39:20.340 DEBUG:boto:Data: 2024-04-12T03:39:20.340 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.340 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.340 DEBUG:boto:Params: {} 2024-04-12T03:39:20.341 DEBUG:boto:Token: None 2024-04-12T03:39:20.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/g 2024-04-12T03:39:20.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UbWhMA5R/xa1tJd9xghTaN1Ad7w= 2024-04-12T03:39:20.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UbWhMA5R/xa1tJd9xghTaN1Ad7w=', 'Content-Length': '0'} 2024-04-12T03:39:20.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eff7d0a92d7dc29-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.343 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.343 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.343 DEBUG:boto:path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.343 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.343 DEBUG:boto:Method: GET 2024-04-12T03:39:20.343 DEBUG:boto:Path: /dgbpxg-59/?prefix=g 2024-04-12T03:39:20.343 DEBUG:boto:Data: 2024-04-12T03:39:20.343 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.344 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.344 DEBUG:boto:Params: {} 2024-04-12T03:39:20.344 DEBUG:boto:Token: None 2024-04-12T03:39:20.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea77d8b71aa0419-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.346 DEBUG:boto:b'dgbpxg-59g1000falseg2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.347 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.347 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.347 DEBUG:boto:path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.347 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=g 2024-04-12T03:39:20.347 DEBUG:boto:Method: GET 2024-04-12T03:39:20.347 DEBUG:boto:Path: /dgbpxg-59/?prefix=g 2024-04-12T03:39:20.347 DEBUG:boto:Data: 2024-04-12T03:39:20.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.348 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.348 DEBUG:boto:Params: {} 2024-04-12T03:39:20.348 DEBUG:boto:Token: None 2024-04-12T03:39:20.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e311191d62d2ad2-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.350 DEBUG:boto:b'dgbpxg-59g1000falseg2024-04-12T03:36:33.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:39:20.350 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.350 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.351 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.351 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.351 DEBUG:boto:Method: GET 2024-04-12T03:39:20.351 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.351 DEBUG:boto:Data: 2024-04-12T03:39:20.351 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.351 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.351 DEBUG:boto:Params: {} 2024-04-12T03:39:20.351 DEBUG:boto:Token: None 2024-04-12T03:39:20.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98= 2024-04-12T03:39:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98=', 'Content-Length': '0'} 2024-04-12T03:39:20.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1b4a8e2b0ce669-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.394 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.394 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.394 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.395 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.395 DEBUG:boto:Method: GET 2024-04-12T03:39:20.395 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.395 DEBUG:boto:Data: 2024-04-12T03:39:20.395 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.395 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.395 DEBUG:boto:Params: {} 2024-04-12T03:39:20.395 DEBUG:boto:Token: None 2024-04-12T03:39:20.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98= 2024-04-12T03:39:20.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98=', 'Content-Length': '0'} 2024-04-12T03:39:20.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5a15db813b083f-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.398 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.398 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.398 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.398 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.398 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.398 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.398 DEBUG:boto:Data: 2024-04-12T03:39:20.399 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.399 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.399 DEBUG:boto:Params: {} 2024-04-12T03:39:20.399 DEBUG:boto:Token: None 2024-04-12T03:39:20.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XZuEsokrsNWg/JHnVPZ5dxBjszQ= 2024-04-12T03:39:20.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XZuEsokrsNWg/JHnVPZ5dxBjszQ=', 'Content-Length': '0'} 2024-04-12T03:39:20.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e89a0e0391b7983c-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.401 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.401 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.401 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.401 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.401 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.401 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.401 DEBUG:boto:Data: 2024-04-12T03:39:20.402 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.402 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.402 DEBUG:boto:Params: {} 2024-04-12T03:39:20.402 DEBUG:boto:Token: None 2024-04-12T03:39:20.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XZuEsokrsNWg/JHnVPZ5dxBjszQ= 2024-04-12T03:39:20.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XZuEsokrsNWg/JHnVPZ5dxBjszQ=', 'Content-Length': '0'} 2024-04-12T03:39:20.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f0abcb677e9dd5-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:39:20.404 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.404 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.404 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.404 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.405 DEBUG:boto:Method: GET 2024-04-12T03:39:20.405 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.405 DEBUG:boto:Data: 2024-04-12T03:39:20.405 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.405 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.405 DEBUG:boto:Params: {} 2024-04-12T03:39:20.405 DEBUG:boto:Token: None 2024-04-12T03:39:20.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98= 2024-04-12T03:39:20.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98=', 'Content-Length': '0'} 2024-04-12T03:39:20.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a974fb6dd9d34d-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.408 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.408 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.408 DEBUG:boto:path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.408 DEBUG:boto:auth_path=/dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.408 DEBUG:boto:Method: GET 2024-04-12T03:39:20.408 DEBUG:boto:Path: /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.408 DEBUG:boto:Data: 2024-04-12T03:39:20.408 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.408 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.408 DEBUG:boto:Params: {} 2024-04-12T03:39:20.409 DEBUG:boto:Token: None 2024-04-12T03:39:20.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h?versionId=null 2024-04-12T03:39:20.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98= 2024-04-12T03:39:20.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5olk7VGGYi7jVRRsT/WVYzvcM98=', 'Content-Length': '0'} 2024-04-12T03:39:20.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b54ad8f7b4b5a22-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.411 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.411 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.411 DEBUG:boto:path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.411 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.411 DEBUG:boto:Method: GET 2024-04-12T03:39:20.411 DEBUG:boto:Path: /dgbpxg-59/?prefix=h 2024-04-12T03:39:20.412 DEBUG:boto:Data: 2024-04-12T03:39:20.412 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.412 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.412 DEBUG:boto:Params: {} 2024-04-12T03:39:20.412 DEBUG:boto:Token: None 2024-04-12T03:39:20.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004726e9ecfaf53516-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.414 DEBUG:boto:b'dgbpxg-59h1000falseh2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.415 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.415 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.415 DEBUG:boto:path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.415 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.415 DEBUG:boto:Method: GET 2024-04-12T03:39:20.415 DEBUG:boto:Path: /dgbpxg-59/?prefix=h 2024-04-12T03:39:20.416 DEBUG:boto:Data: 2024-04-12T03:39:20.416 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.416 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.416 DEBUG:boto:Params: {} 2024-04-12T03:39:20.416 DEBUG:boto:Token: None 2024-04-12T03:39:20.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004867e20d96b59144-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.458 DEBUG:boto:b'dgbpxg-59h1000falseh2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.459 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.459 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.459 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.459 DEBUG:boto:Path: /dgbpxg-59/h 2024-04-12T03:39:20.459 DEBUG:boto:Data: 2024-04-12T03:39:20.459 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.459 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.459 DEBUG:boto:Params: {} 2024-04-12T03:39:20.460 DEBUG:boto:Token: None 2024-04-12T03:39:20.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h 2024-04-12T03:39:20.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fPNCZ+AjeKG8D9SxC1e6Vr90b+k= 2024-04-12T03:39:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fPNCZ+AjeKG8D9SxC1e6Vr90b+k=', 'Content-Length': '0'} 2024-04-12T03:39:20.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a16f30fe9042118-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.462 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.462 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.462 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.462 DEBUG:boto:Path: /dgbpxg-59/h 2024-04-12T03:39:20.462 DEBUG:boto:Data: 2024-04-12T03:39:20.462 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.462 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.462 DEBUG:boto:Params: {} 2024-04-12T03:39:20.463 DEBUG:boto:Token: None 2024-04-12T03:39:20.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h 2024-04-12T03:39:20.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fPNCZ+AjeKG8D9SxC1e6Vr90b+k= 2024-04-12T03:39:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fPNCZ+AjeKG8D9SxC1e6Vr90b+k=', 'Content-Length': '0'} 2024-04-12T03:39:20.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e6edcb4a6fcdc99-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:39:20.465 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.465 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.465 DEBUG:boto:Method: GET 2024-04-12T03:39:20.465 DEBUG:boto:Path: /dgbpxg-59/h 2024-04-12T03:39:20.465 DEBUG:boto:Data: 2024-04-12T03:39:20.465 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.465 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.466 DEBUG:boto:Params: {} 2024-04-12T03:39:20.466 DEBUG:boto:Token: None 2024-04-12T03:39:20.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h 2024-04-12T03:39:20.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YDNopJzf0lalxQ0vJzG+vdG7fBM= 2024-04-12T03:39:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YDNopJzf0lalxQ0vJzG+vdG7fBM=', 'Content-Length': '0'} 2024-04-12T03:39:20.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-59:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4003743445a9be6-006618ace8-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.468 DEBUG:boto:path=/dgbpxg-59/h 2024-04-12T03:39:20.468 DEBUG:boto:auth_path=/dgbpxg-59/h 2024-04-12T03:39:20.468 DEBUG:boto:Method: GET 2024-04-12T03:39:20.468 DEBUG:boto:Path: /dgbpxg-59/h 2024-04-12T03:39:20.468 DEBUG:boto:Data: 2024-04-12T03:39:20.468 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.468 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.468 DEBUG:boto:Params: {} 2024-04-12T03:39:20.469 DEBUG:boto:Token: None 2024-04-12T03:39:20.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/h 2024-04-12T03:39:20.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YDNopJzf0lalxQ0vJzG+vdG7fBM= 2024-04-12T03:39:20.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YDNopJzf0lalxQ0vJzG+vdG7fBM=', 'Content-Length': '0'} 2024-04-12T03:39:20.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:36:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090abead6cae04716-006618ace8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.471 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.471 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.471 DEBUG:boto:path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.471 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.471 DEBUG:boto:Method: GET 2024-04-12T03:39:20.471 DEBUG:boto:Path: /dgbpxg-59/?prefix=h 2024-04-12T03:39:20.471 DEBUG:boto:Data: 2024-04-12T03:39:20.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.471 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.471 DEBUG:boto:Params: {} 2024-04-12T03:39:20.472 DEBUG:boto:Token: None 2024-04-12T03:39:20.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a15b187445471f-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.474 DEBUG:boto:b'dgbpxg-59h1000falseh2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.474 DEBUG:boto:path=/dgbpxg-59/ 2024-04-12T03:39:20.474 DEBUG:boto:auth_path=/dgbpxg-59/ 2024-04-12T03:39:20.474 DEBUG:boto:path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.475 DEBUG:boto:auth_path=/dgbpxg-59/?prefix=h 2024-04-12T03:39:20.475 DEBUG:boto:Method: GET 2024-04-12T03:39:20.475 DEBUG:boto:Path: /dgbpxg-59/?prefix=h 2024-04-12T03:39:20.475 DEBUG:boto:Data: 2024-04-12T03:39:20.475 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.475 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.475 DEBUG:boto:Params: {} 2024-04-12T03:39:20.475 DEBUG:boto:Token: None 2024-04-12T03:39:20.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-59/ 2024-04-12T03:39:20.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q= 2024-04-12T03:39:20.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m3O5YklNgJP7hJnHPv2Z660UD9Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c151c144b7ab502-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.477 DEBUG:boto:b'dgbpxg-59h1000falseh2024-04-12T03:36:33.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:20.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-59 zones={a2, a1} 2024-04-12T03:39:20.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2024-04-12T03:39:20.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.480 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:20.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:20.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:20.557 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:39:20.569 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:39:20.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:39:20.569 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:39:20.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892993.128903_1126.1", 2024-04-12T03:39:20.571 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:36:33.128903Z" 2024-04-12T03:39:20.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.571 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:39:20.572 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892993.128903_1126.1'} 2024-04-12T03:39:20.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:39:20.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.572 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:20.621 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:20.621 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:20.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:20.650+0000 7f304ab7d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:39:20.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:20.651+0000 7f304ab7d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:39:20.665 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:20.665 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:39:20.665 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:39:20.665 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.668 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:39:20.669 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712892993.128903_1126.1", 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:36:33.128903Z", 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:20.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:39:20Z" 2024-04-12T03:39:20.671 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:20.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892993.128903_1126.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:33.128903Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:39:20Z" } 2024-04-12T03:39:20.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892993.128903_1126.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:33.128903Z', 'realm_epoch': 2}}] 2024-04-12T03:39:20.672 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892993.128903_1126.1'} 2024-04-12T03:39:20.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892993.128903_1126.1'} 2024-04-12T03:39:20.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:39:20.673 DEBUG:boto:path=/ 2024-04-12T03:39:20.673 DEBUG:boto:auth_path=/ 2024-04-12T03:39:20.673 DEBUG:boto:Method: GET 2024-04-12T03:39:20.673 DEBUG:boto:Path: / 2024-04-12T03:39:20.673 DEBUG:boto:Data: 2024-04-12T03:39:20.673 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.673 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.673 DEBUG:boto:Params: {} 2024-04-12T03:39:20.674 DEBUG:boto:Token: None 2024-04-12T03:39:20.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT / 2024-04-12T03:39:20.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F7bXwo/htb/hQIbxOJYHNn9JV38= 2024-04-12T03:39:20.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F7bXwo/htb/hQIbxOJYHNn9JV38=', 'Content-Length': '0'} 2024-04-12T03:39:20.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e96d14744b030719-006618ace8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.679 DEBUG:boto:path=/ 2024-04-12T03:39:20.679 DEBUG:boto:auth_path=/ 2024-04-12T03:39:20.679 DEBUG:boto:Method: GET 2024-04-12T03:39:20.679 DEBUG:boto:Path: / 2024-04-12T03:39:20.679 DEBUG:boto:Data: 2024-04-12T03:39:20.679 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:20.679 DEBUG:boto:Port: 8001 2024-04-12T03:39:20.680 DEBUG:boto:Params: {} 2024-04-12T03:39:20.680 DEBUG:boto:Token: None 2024-04-12T03:39:20.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:20 GMT / 2024-04-12T03:39:20.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F7bXwo/htb/hQIbxOJYHNn9JV38= 2024-04-12T03:39:20.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F7bXwo/htb/hQIbxOJYHNn9JV38=', 'Content-Length': '0'} 2024-04-12T03:39:20.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d87e1241194b9a-006618ace8-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-12T03:39:20.683 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:20.683 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:20.684 DEBUG:boto:Method: PUT 2024-04-12T03:39:20.684 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:20.684 DEBUG:boto:Data: 2024-04-12T03:39:20.684 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.684 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.684 DEBUG:boto:Params: {} 2024-04-12T03:39:20.684 DEBUG:boto:Token: None 2024-04-12T03:39:20.684 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-60/ 2024-04-12T03:39:20.685 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XysczcudgS4Nl4w25vn1pkvi370= 2024-04-12T03:39:20.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XysczcudgS4Nl4w25vn1pkvi370=', 'Content-Length': '0'} 2024-04-12T03:39:20.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028e923c1a0b107b1-006618ace8-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.699 DEBUG:rgw_multi.tests:created bucket=dgbpxg-60 2024-04-12T03:39:20.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:20.746 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:20.746 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:39:20.790 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:39:20.697788Z" 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.792 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:39:20.793 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:39:20.793 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:39:20.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.793 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:20.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:20.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:20.872+0000 7fec89511780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:39:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:20.873+0000 7fec89511780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:39:20.888 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:39:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:39:20Z" 2024-04-12T03:39:20.894 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:20.895 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:39:20Z" } 2024-04-12T03:39:20.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:39:20.895 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:39:20.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:39:20.896 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:39:20.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:20.896 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006" 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:20.988 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:20.989 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:20.989 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:20.990 DEBUG:boto:Method: HEAD 2024-04-12T03:39:20.990 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:20.990 DEBUG:boto:Data: 2024-04-12T03:39:20.990 DEBUG:boto:Headers: {} 2024-04-12T03:39:20.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.990 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.990 DEBUG:boto:Params: {} 2024-04-12T03:39:20.990 DEBUG:boto:Token: None 2024-04-12T03:39:20.990 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-60/ 2024-04-12T03:39:20.991 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L3sVWcAkY7SSWbfQOPNOz3TzF2Q= 2024-04-12T03:39:20.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L3sVWcAkY7SSWbfQOPNOz3TzF2Q=', 'Content-Length': '0'} 2024-04-12T03:39:20.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ccfe731b767c9a8-006618ace8-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:20.993 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:20.994 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:20.994 DEBUG:boto:Method: PUT 2024-04-12T03:39:20.994 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:20.994 DEBUG:boto:Data: 2024-04-12T03:39:20.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:20.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:20.994 DEBUG:boto:Port: 8000 2024-04-12T03:39:20.994 DEBUG:boto:Params: {} 2024-04-12T03:39:20.995 DEBUG:boto:Token: None 2024-04-12T03:39:20.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:20 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:20.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bl0F4bN0hPzsIZLUGLkyUldx210= 2024-04-12T03:39:20.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bl0F4bN0hPzsIZLUGLkyUldx210='} 2024-04-12T03:39:21.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a77750b6f6cdb2dd-006618ace8-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.001 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.001 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.001 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.001 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.001 DEBUG:boto:Data: 2024-04-12T03:39:21.001 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.001 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.001 DEBUG:boto:Params: {} 2024-04-12T03:39:21.001 DEBUG:boto:Token: None 2024-04-12T03:39:21.002 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.002 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0aa22fb01173f8b-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.004 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:21.004 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:21.004 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.004 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:21.004 DEBUG:boto:Data: 2024-04-12T03:39:21.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.005 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.005 DEBUG:boto:Params: {} 2024-04-12T03:39:21.005 DEBUG:boto:Token: None 2024-04-12T03:39:21.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:21.005 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WArBZO+ql5prQSvWWc04I7ZCY5o= 2024-04-12T03:39:21.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WArBZO+ql5prQSvWWc04I7ZCY5o='} 2024-04-12T03:39:21.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009749372532648810-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.010 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.010 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.010 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.010 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.010 DEBUG:boto:Data: 2024-04-12T03:39:21.011 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.011 DEBUG:boto:Params: {} 2024-04-12T03:39:21.011 DEBUG:boto:Token: None 2024-04-12T03:39:21.011 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bd33642dfd2b907-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:21.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:21.014 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.014 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:21.014 DEBUG:boto:Data: 2024-04-12T03:39:21.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.014 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.014 DEBUG:boto:Params: {} 2024-04-12T03:39:21.014 DEBUG:boto:Token: None 2024-04-12T03:39:21.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:21.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4I4RyapI44SLJBrUHfC8u+NGJ1g= 2024-04-12T03:39:21.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4I4RyapI44SLJBrUHfC8u+NGJ1g='} 2024-04-12T03:39:21.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb69f24e63a20a36-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.020 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.020 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.020 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.020 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.020 DEBUG:boto:Data: 2024-04-12T03:39:21.020 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.020 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.020 DEBUG:boto:Params: {} 2024-04-12T03:39:21.021 DEBUG:boto:Token: None 2024-04-12T03:39:21.021 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfc6caf56955c17d-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.023 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:21.023 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:21.023 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.023 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:21.024 DEBUG:boto:Data: 2024-04-12T03:39:21.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.024 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.024 DEBUG:boto:Params: {} 2024-04-12T03:39:21.024 DEBUG:boto:Token: None 2024-04-12T03:39:21.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:21.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QMzxddXxZyAJ2CHw+xzzbtYzzjw= 2024-04-12T03:39:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QMzxddXxZyAJ2CHw+xzzbtYzzjw='} 2024-04-12T03:39:21.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ba6aa9e077df2bf-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.070 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.071 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.071 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.071 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.071 DEBUG:boto:Data: 2024-04-12T03:39:21.071 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.071 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.071 DEBUG:boto:Params: {} 2024-04-12T03:39:21.071 DEBUG:boto:Token: None 2024-04-12T03:39:21.072 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0990d38de05c667-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.074 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:21.074 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:21.075 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.075 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:21.075 DEBUG:boto:Data: 2024-04-12T03:39:21.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.075 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.075 DEBUG:boto:Params: {} 2024-04-12T03:39:21.075 DEBUG:boto:Token: None 2024-04-12T03:39:21.075 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:21.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mjdzHmJ5rLn/D7V9Huq5nYDdHxU= 2024-04-12T03:39:21.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mjdzHmJ5rLn/D7V9Huq5nYDdHxU='} 2024-04-12T03:39:21.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebd0563e465e6bf0-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.081 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.081 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.081 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.082 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.082 DEBUG:boto:Data: 2024-04-12T03:39:21.082 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.082 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.082 DEBUG:boto:Params: {} 2024-04-12T03:39:21.082 DEBUG:boto:Token: None 2024-04-12T03:39:21.082 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016751f466cdc7690-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.085 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:21.085 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:21.085 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.085 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:21.085 DEBUG:boto:Data: 2024-04-12T03:39:21.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.085 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.085 DEBUG:boto:Params: {} 2024-04-12T03:39:21.085 DEBUG:boto:Token: None 2024-04-12T03:39:21.086 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:21.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JIjmi5au/nBXNw1gytKQm9xS+Rs= 2024-04-12T03:39:21.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JIjmi5au/nBXNw1gytKQm9xS+Rs='} 2024-04-12T03:39:21.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c61f8b2d695db81-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.090 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.091 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.091 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.091 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.091 DEBUG:boto:Data: 2024-04-12T03:39:21.091 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.091 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.091 DEBUG:boto:Params: {} 2024-04-12T03:39:21.091 DEBUG:boto:Token: None 2024-04-12T03:39:21.092 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005195830617454c03-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.094 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:21.094 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:21.094 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.095 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:21.095 DEBUG:boto:Data: 2024-04-12T03:39:21.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.095 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.095 DEBUG:boto:Params: {} 2024-04-12T03:39:21.095 DEBUG:boto:Token: None 2024-04-12T03:39:21.095 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:21.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WVq9p223h/AGeCI1k7AKyq2k/mk= 2024-04-12T03:39:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WVq9p223h/AGeCI1k7AKyq2k/mk='} 2024-04-12T03:39:21.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3734b07deeed74-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.138 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.139 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.139 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.139 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.139 DEBUG:boto:Data: 2024-04-12T03:39:21.139 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.139 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.139 DEBUG:boto:Params: {} 2024-04-12T03:39:21.139 DEBUG:boto:Token: None 2024-04-12T03:39:21.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058875dea5c3186b0-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.142 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:21.142 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:21.142 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.142 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:21.142 DEBUG:boto:Data: 2024-04-12T03:39:21.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.143 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.143 DEBUG:boto:Params: {} 2024-04-12T03:39:21.143 DEBUG:boto:Token: None 2024-04-12T03:39:21.143 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:21.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c5/3uWfrxvEe7xdckMiGVkzS8qo= 2024-04-12T03:39:21.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c5/3uWfrxvEe7xdckMiGVkzS8qo='} 2024-04-12T03:39:21.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075fa4415ab30b19e-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.148 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.148 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.148 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.148 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.148 DEBUG:boto:Data: 2024-04-12T03:39:21.148 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.149 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.149 DEBUG:boto:Params: {} 2024-04-12T03:39:21.149 DEBUG:boto:Token: None 2024-04-12T03:39:21.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e68053764bd140ad-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.152 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:21.152 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:21.152 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.152 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:21.152 DEBUG:boto:Data: 2024-04-12T03:39:21.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.152 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.152 DEBUG:boto:Params: {} 2024-04-12T03:39:21.152 DEBUG:boto:Token: None 2024-04-12T03:39:21.153 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:21.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UHrz8Ul51n0ePQZmS61u17Er5iE= 2024-04-12T03:39:21.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UHrz8Ul51n0ePQZmS61u17Er5iE='} 2024-04-12T03:39:21.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a656f17492ce915-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.195 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:21.195 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:21.195 DEBUG:boto:Method: HEAD 2024-04-12T03:39:21.195 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:21.195 DEBUG:boto:Data: 2024-04-12T03:39:21.195 DEBUG:boto:Headers: {} 2024-04-12T03:39:21.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.195 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.195 DEBUG:boto:Params: {} 2024-04-12T03:39:21.195 DEBUG:boto:Token: None 2024-04-12T03:39:21.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/ 2024-04-12T03:39:21.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8= 2024-04-12T03:39:21.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7xrZ8M2Z+MvFWSRdOOwzMlr4vZ8=', 'Content-Length': '0'} 2024-04-12T03:39:21.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000807cc815fe7767b2-006618ace9-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.198 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:21.198 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:21.198 DEBUG:boto:Method: PUT 2024-04-12T03:39:21.198 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:21.198 DEBUG:boto:Data: 2024-04-12T03:39:21.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:21.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:21.199 DEBUG:boto:Port: 8000 2024-04-12T03:39:21.199 DEBUG:boto:Params: {} 2024-04-12T03:39:21.199 DEBUG:boto:Token: None 2024-04-12T03:39:21.199 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:21 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:21.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G7R0ITUa+ndwWFSdrxT0vWeJjPU= 2024-04-12T03:39:21.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G7R0ITUa+ndwWFSdrxT0vWeJjPU='} 2024-04-12T03:39:21.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002544658cc5bce9a2-006618ace9-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:21.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-60 2024-04-12T03:39:21.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:21.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:21.205 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:39:21.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:21.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:21.295+0000 7fa8b6bf5780 1 waiting to reach incremental sync.. 2024-04-12T03:39:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:26.297+0000 7fa8b6bf5780 1 bucket sync caught up with source: 2024-04-12T03:39:26.298 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.49.6, 00000000001.123.6, 00000000001.41.6, 00000000001.47.6, , 00000000001.62.6, 00000000001.72.6, 00000000001.70.6, 00000000001.64.6, 00000000001.37.6, 00000000001.38.6] 2024-04-12T03:39:26.298 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.49.6, 00000000001.123.6, 00000000001.41.6, 00000000001.47.6, , 00000000001.62.6, 00000000001.72.6, 00000000001.70.6, 00000000001.64.6, 00000000001.37.6, 00000000001.38.6] 2024-04-12T03:39:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:26.297+0000 7fa8b6bf5780 0 bucket checkpoint complete 2024-04-12T03:39:26.316 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-60 2024-04-12T03:39:26.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:26.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:26.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:39:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:26.378 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:26.413+0000 7f8a4eeb3780 1 waiting to reach incremental sync.. 2024-04-12T03:39:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:31.415+0000 7f8a4eeb3780 1 bucket sync caught up with source: 2024-04-12T03:39:31.416 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1413.6, 00000000001.1190.6, 00000000001.654.6, 00000000001.1414.6, , 00000000001.2191.6, 00000000001.1344.6, 00000000001.1343.6, 00000000001.110.6, 00000000001.3053.6, 00000000001.1182.6] 2024-04-12T03:39:31.416 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.1413.6, 00000000001.1190.6, 00000000001.654.6, 00000000001.1414.6, , 00000000001.2191.6, 00000000001.1344.6, 00000000001.1343.6, 00000000001.110.6, 00000000001.3053.6, 00000000001.1182.6] 2024-04-12T03:39:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:31.415+0000 7f8a4eeb3780 0 bucket checkpoint complete 2024-04-12T03:39:31.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-60 zones={a2, a1} 2024-04-12T03:39:31.433 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.433 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.433 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.433 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:31.434 DEBUG:boto:Data: 2024-04-12T03:39:31.434 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.434 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.434 DEBUG:boto:Params: {} 2024-04-12T03:39:31.434 DEBUG:boto:Token: None 2024-04-12T03:39:31.434 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.434 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNNTdVjZTuPznOmPTSMRNXCw3KY= 2024-04-12T03:39:31.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNNTdVjZTuPznOmPTSMRNXCw3KY=', 'Content-Length': '0'} 2024-04-12T03:39:31.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c05aa6d9fd98dd29-006618acf3-1da1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.437 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.437 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.437 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.438 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:31.438 DEBUG:boto:Data: 2024-04-12T03:39:31.438 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.438 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.438 DEBUG:boto:Params: {} 2024-04-12T03:39:31.438 DEBUG:boto:Token: None 2024-04-12T03:39:31.438 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.438 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNNTdVjZTuPznOmPTSMRNXCw3KY= 2024-04-12T03:39:31.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNNTdVjZTuPznOmPTSMRNXCw3KY=', 'Content-Length': '0'} 2024-04-12T03:39:31.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064749e38b6ae551d-006618acf3-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.441 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:39:31.441 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.441 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.441 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.441 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.441 DEBUG:boto:Method: GET 2024-04-12T03:39:31.441 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.441 DEBUG:boto:Data: 2024-04-12T03:39:31.442 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.442 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.442 DEBUG:boto:Params: {} 2024-04-12T03:39:31.442 DEBUG:boto:Token: None 2024-04-12T03:39:31.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/?versions 2024-04-12T03:39:31.442 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk= 2024-04-12T03:39:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk=', 'Content-Length': '0'} 2024-04-12T03:39:31.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073829cf63a15417b-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.445 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:39:31.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:39:31.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:39:31.447 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:39:31.447 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.447 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.447 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.447 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.447 DEBUG:boto:Method: GET 2024-04-12T03:39:31.447 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.447 DEBUG:boto:Data: 2024-04-12T03:39:31.447 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.447 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.448 DEBUG:boto:Params: {} 2024-04-12T03:39:31.448 DEBUG:boto:Token: None 2024-04-12T03:39:31.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/?versions 2024-04-12T03:39:31.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk= 2024-04-12T03:39:31.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk=', 'Content-Length': '0'} 2024-04-12T03:39:31.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe9b51fa9412d50-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.451 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:39:31.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:39:31.453 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.453 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.453 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.453 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.453 DEBUG:boto:Method: GET 2024-04-12T03:39:31.453 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.453 DEBUG:boto:Data: 2024-04-12T03:39:31.453 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.453 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.454 DEBUG:boto:Params: {} 2024-04-12T03:39:31.454 DEBUG:boto:Token: None 2024-04-12T03:39:31.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/?versions 2024-04-12T03:39:31.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk= 2024-04-12T03:39:31.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk=', 'Content-Length': '0'} 2024-04-12T03:39:31.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035da7f93e0725751-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.457 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.457 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.457 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.458 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.458 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.458 DEBUG:boto:Method: GET 2024-04-12T03:39:31.458 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:31.458 DEBUG:boto:Data: 2024-04-12T03:39:31.458 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.458 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.458 DEBUG:boto:Params: {} 2024-04-12T03:39:31.458 DEBUG:boto:Token: None 2024-04-12T03:39:31.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/?versions 2024-04-12T03:39:31.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk= 2024-04-12T03:39:31.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFIn36JwWPKhjXwx9DoqEAjPqjk=', 'Content-Length': '0'} 2024-04-12T03:39:31.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc5408acf0bdd75f-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.461 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:31.462 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.462 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.462 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.463 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.463 DEBUG:boto:Method: GET 2024-04-12T03:39:31.463 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.463 DEBUG:boto:Data: 2024-04-12T03:39:31.463 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.463 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.463 DEBUG:boto:Params: {} 2024-04-12T03:39:31.463 DEBUG:boto:Token: None 2024-04-12T03:39:31.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w= 2024-04-12T03:39:31.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w=', 'Content-Length': '0'} 2024-04-12T03:39:31.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d041576b22a26c08-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.466 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.466 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.466 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.466 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.466 DEBUG:boto:Method: GET 2024-04-12T03:39:31.466 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.467 DEBUG:boto:Data: 2024-04-12T03:39:31.467 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.467 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.467 DEBUG:boto:Params: {} 2024-04-12T03:39:31.467 DEBUG:boto:Token: None 2024-04-12T03:39:31.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w= 2024-04-12T03:39:31.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w=', 'Content-Length': '0'} 2024-04-12T03:39:31.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce77eeb50df9f0f-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.471 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.471 DEBUG:boto:Data: 2024-04-12T03:39:31.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.471 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.471 DEBUG:boto:Params: {} 2024-04-12T03:39:31.471 DEBUG:boto:Token: None 2024-04-12T03:39:31.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4Vse3Hzde/yeU6Zub9ByBTvIETg= 2024-04-12T03:39:31.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4Vse3Hzde/yeU6Zub9ByBTvIETg=', 'Content-Length': '0'} 2024-04-12T03:39:31.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da9dac8de0e68c4-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.473 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.473 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.473 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.474 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.474 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.474 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.474 DEBUG:boto:Data: 2024-04-12T03:39:31.474 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.474 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.474 DEBUG:boto:Params: {} 2024-04-12T03:39:31.474 DEBUG:boto:Token: None 2024-04-12T03:39:31.474 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4Vse3Hzde/yeU6Zub9ByBTvIETg= 2024-04-12T03:39:31.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4Vse3Hzde/yeU6Zub9ByBTvIETg=', 'Content-Length': '0'} 2024-04-12T03:39:31.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f16b162d97893d-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:31.476 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.476 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.477 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.477 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.477 DEBUG:boto:Method: GET 2024-04-12T03:39:31.477 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.477 DEBUG:boto:Data: 2024-04-12T03:39:31.477 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.477 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.477 DEBUG:boto:Params: {} 2024-04-12T03:39:31.477 DEBUG:boto:Token: None 2024-04-12T03:39:31.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w= 2024-04-12T03:39:31.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w=', 'Content-Length': '0'} 2024-04-12T03:39:31.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085908f74953469b8-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.479 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.479 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.480 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.480 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.480 DEBUG:boto:Method: GET 2024-04-12T03:39:31.480 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.480 DEBUG:boto:Data: 2024-04-12T03:39:31.480 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.480 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.480 DEBUG:boto:Params: {} 2024-04-12T03:39:31.480 DEBUG:boto:Token: None 2024-04-12T03:39:31.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:31.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w= 2024-04-12T03:39:31.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0HdwXszRGg+onZZjID2qLamLf/w=', 'Content-Length': '0'} 2024-04-12T03:39:31.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84b3686a2a75031-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.483 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.483 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.483 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.483 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.483 DEBUG:boto:Method: GET 2024-04-12T03:39:31.483 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.483 DEBUG:boto:Data: 2024-04-12T03:39:31.483 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.483 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.483 DEBUG:boto:Params: {} 2024-04-12T03:39:31.484 DEBUG:boto:Token: None 2024-04-12T03:39:31.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d1395472d43dc9-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.486 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.486 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.487 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.487 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.487 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.487 DEBUG:boto:Method: GET 2024-04-12T03:39:31.487 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.487 DEBUG:boto:Data: 2024-04-12T03:39:31.487 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.487 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.487 DEBUG:boto:Params: {} 2024-04-12T03:39:31.487 DEBUG:boto:Token: None 2024-04-12T03:39:31.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9b13893bd3196fa-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.530 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.531 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.531 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.531 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.531 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.531 DEBUG:boto:Data: 2024-04-12T03:39:31.531 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.531 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.531 DEBUG:boto:Params: {} 2024-04-12T03:39:31.532 DEBUG:boto:Token: None 2024-04-12T03:39:31.532 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EGz4fyXMZT4N8aMnpoD36TI7swc= 2024-04-12T03:39:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EGz4fyXMZT4N8aMnpoD36TI7swc=', 'Content-Length': '0'} 2024-04-12T03:39:31.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922277bf613ffd4b-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.534 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.534 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.534 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.534 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.534 DEBUG:boto:Data: 2024-04-12T03:39:31.534 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.534 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.534 DEBUG:boto:Params: {} 2024-04-12T03:39:31.534 DEBUG:boto:Token: None 2024-04-12T03:39:31.535 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EGz4fyXMZT4N8aMnpoD36TI7swc= 2024-04-12T03:39:31.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EGz4fyXMZT4N8aMnpoD36TI7swc=', 'Content-Length': '0'} 2024-04-12T03:39:31.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490d9f521678553b-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:31.537 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.537 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.537 DEBUG:boto:Method: GET 2024-04-12T03:39:31.537 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.537 DEBUG:boto:Data: 2024-04-12T03:39:31.537 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.537 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.537 DEBUG:boto:Params: {} 2024-04-12T03:39:31.538 DEBUG:boto:Token: None 2024-04-12T03:39:31.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.538 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vSeIGQXjXZmev4oH6Bjum6XHqcc= 2024-04-12T03:39:31.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vSeIGQXjXZmev4oH6Bjum6XHqcc=', 'Content-Length': '0'} 2024-04-12T03:39:31.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037526f232cb406ce-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.540 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.540 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.540 DEBUG:boto:Method: GET 2024-04-12T03:39:31.540 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.540 DEBUG:boto:Data: 2024-04-12T03:39:31.540 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.540 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.541 DEBUG:boto:Params: {} 2024-04-12T03:39:31.541 DEBUG:boto:Token: None 2024-04-12T03:39:31.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:31.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vSeIGQXjXZmev4oH6Bjum6XHqcc= 2024-04-12T03:39:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vSeIGQXjXZmev4oH6Bjum6XHqcc=', 'Content-Length': '0'} 2024-04-12T03:39:31.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d2faa2033fb0b7-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.543 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.543 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.543 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.543 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.543 DEBUG:boto:Method: GET 2024-04-12T03:39:31.543 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.544 DEBUG:boto:Data: 2024-04-12T03:39:31.544 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.544 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.544 DEBUG:boto:Params: {} 2024-04-12T03:39:31.544 DEBUG:boto:Token: None 2024-04-12T03:39:31.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000878166be630098e-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.547 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.547 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.547 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.547 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.547 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.547 DEBUG:boto:Method: GET 2024-04-12T03:39:31.547 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:31.547 DEBUG:boto:Data: 2024-04-12T03:39:31.548 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.548 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.548 DEBUG:boto:Params: {} 2024-04-12T03:39:31.548 DEBUG:boto:Token: None 2024-04-12T03:39:31.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e582f02858c1a11d-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.551 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:21.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:31.551 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.552 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.552 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.552 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.552 DEBUG:boto:Method: GET 2024-04-12T03:39:31.552 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.552 DEBUG:boto:Data: 2024-04-12T03:39:31.552 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.552 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.552 DEBUG:boto:Params: {} 2024-04-12T03:39:31.552 DEBUG:boto:Token: None 2024-04-12T03:39:31.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY= 2024-04-12T03:39:31.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY=', 'Content-Length': '0'} 2024-04-12T03:39:31.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec21b2c2803af811-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.594 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.594 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.595 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.595 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.595 DEBUG:boto:Method: GET 2024-04-12T03:39:31.595 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.595 DEBUG:boto:Data: 2024-04-12T03:39:31.595 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.595 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.595 DEBUG:boto:Params: {} 2024-04-12T03:39:31.595 DEBUG:boto:Token: None 2024-04-12T03:39:31.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY= 2024-04-12T03:39:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY=', 'Content-Length': '0'} 2024-04-12T03:39:31.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb8735916582c4b2-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.598 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.598 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.598 DEBUG:boto:Data: 2024-04-12T03:39:31.598 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.598 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.599 DEBUG:boto:Params: {} 2024-04-12T03:39:31.599 DEBUG:boto:Token: None 2024-04-12T03:39:31.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7FTW5FW5b1+RJNiooz6051lyeAA= 2024-04-12T03:39:31.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7FTW5FW5b1+RJNiooz6051lyeAA=', 'Content-Length': '0'} 2024-04-12T03:39:31.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b114ce70d5cc9f6-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.601 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.601 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.601 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.601 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.601 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.601 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.602 DEBUG:boto:Data: 2024-04-12T03:39:31.602 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.602 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.602 DEBUG:boto:Params: {} 2024-04-12T03:39:31.602 DEBUG:boto:Token: None 2024-04-12T03:39:31.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7FTW5FW5b1+RJNiooz6051lyeAA= 2024-04-12T03:39:31.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7FTW5FW5b1+RJNiooz6051lyeAA=', 'Content-Length': '0'} 2024-04-12T03:39:31.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a561225302a71bd1-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:31.604 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.604 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.604 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.605 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.605 DEBUG:boto:Method: GET 2024-04-12T03:39:31.605 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.605 DEBUG:boto:Data: 2024-04-12T03:39:31.605 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.605 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.605 DEBUG:boto:Params: {} 2024-04-12T03:39:31.605 DEBUG:boto:Token: None 2024-04-12T03:39:31.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY= 2024-04-12T03:39:31.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY=', 'Content-Length': '0'} 2024-04-12T03:39:31.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ed6c617f178170-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.608 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.608 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.608 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.608 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.608 DEBUG:boto:Method: GET 2024-04-12T03:39:31.608 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.608 DEBUG:boto:Data: 2024-04-12T03:39:31.608 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.608 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.608 DEBUG:boto:Params: {} 2024-04-12T03:39:31.609 DEBUG:boto:Token: None 2024-04-12T03:39:31.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:31.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY= 2024-04-12T03:39:31.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzUbKLUVccrMgXKChOTZRE9VcUY=', 'Content-Length': '0'} 2024-04-12T03:39:31.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000960b9ba35002908-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.611 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.611 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.611 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.611 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.611 DEBUG:boto:Method: GET 2024-04-12T03:39:31.612 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.612 DEBUG:boto:Data: 2024-04-12T03:39:31.612 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.612 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.612 DEBUG:boto:Params: {} 2024-04-12T03:39:31.612 DEBUG:boto:Token: None 2024-04-12T03:39:31.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.612 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000848ec9e1e2971331-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.615 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.615 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.615 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.615 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.615 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.616 DEBUG:boto:Method: GET 2024-04-12T03:39:31.616 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.616 DEBUG:boto:Data: 2024-04-12T03:39:31.616 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.616 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.616 DEBUG:boto:Params: {} 2024-04-12T03:39:31.616 DEBUG:boto:Token: None 2024-04-12T03:39:31.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015de647c1fb4d7ec-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.662 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.663 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.665 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.665 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.665 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.665 DEBUG:boto:Data: 2024-04-12T03:39:31.665 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.665 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.666 DEBUG:boto:Params: {} 2024-04-12T03:39:31.666 DEBUG:boto:Token: None 2024-04-12T03:39:31.666 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.666 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gOYpQd/x8P0Cz7qudzxwQtgxSQk= 2024-04-12T03:39:31.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gOYpQd/x8P0Cz7qudzxwQtgxSQk=', 'Content-Length': '0'} 2024-04-12T03:39:31.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b005c249d43060c-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.668 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.668 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.668 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.668 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.668 DEBUG:boto:Data: 2024-04-12T03:39:31.669 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.669 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.669 DEBUG:boto:Params: {} 2024-04-12T03:39:31.669 DEBUG:boto:Token: None 2024-04-12T03:39:31.669 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gOYpQd/x8P0Cz7qudzxwQtgxSQk= 2024-04-12T03:39:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gOYpQd/x8P0Cz7qudzxwQtgxSQk=', 'Content-Length': '0'} 2024-04-12T03:39:31.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f7f6fa9f0fbffc0-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:31.671 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.671 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.672 DEBUG:boto:Method: GET 2024-04-12T03:39:31.672 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.672 DEBUG:boto:Data: 2024-04-12T03:39:31.672 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.672 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.672 DEBUG:boto:Params: {} 2024-04-12T03:39:31.672 DEBUG:boto:Token: None 2024-04-12T03:39:31.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8pbqt0JY06hnVp/B95yX2rbRj0Q= 2024-04-12T03:39:31.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8pbqt0JY06hnVp/B95yX2rbRj0Q=', 'Content-Length': '0'} 2024-04-12T03:39:31.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e329b49403bd79-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.675 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.675 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.675 DEBUG:boto:Method: GET 2024-04-12T03:39:31.675 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.675 DEBUG:boto:Data: 2024-04-12T03:39:31.675 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.676 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.676 DEBUG:boto:Params: {} 2024-04-12T03:39:31.676 DEBUG:boto:Token: None 2024-04-12T03:39:31.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:31.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8pbqt0JY06hnVp/B95yX2rbRj0Q= 2024-04-12T03:39:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8pbqt0JY06hnVp/B95yX2rbRj0Q=', 'Content-Length': '0'} 2024-04-12T03:39:31.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073bf1fec4297822-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.678 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.679 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.679 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.679 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.679 DEBUG:boto:Method: GET 2024-04-12T03:39:31.679 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.679 DEBUG:boto:Data: 2024-04-12T03:39:31.679 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.679 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.679 DEBUG:boto:Params: {} 2024-04-12T03:39:31.680 DEBUG:boto:Token: None 2024-04-12T03:39:31.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7743b7042072d82-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.722 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.723 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.723 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.723 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.723 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.723 DEBUG:boto:Method: GET 2024-04-12T03:39:31.723 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:31.723 DEBUG:boto:Data: 2024-04-12T03:39:31.723 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.724 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.724 DEBUG:boto:Params: {} 2024-04-12T03:39:31.724 DEBUG:boto:Token: None 2024-04-12T03:39:31.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d409685a1cf3d6-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.727 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:31.728 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.728 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.728 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.728 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.728 DEBUG:boto:Method: GET 2024-04-12T03:39:31.728 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.728 DEBUG:boto:Data: 2024-04-12T03:39:31.728 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.729 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.729 DEBUG:boto:Params: {} 2024-04-12T03:39:31.729 DEBUG:boto:Token: None 2024-04-12T03:39:31.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ= 2024-04-12T03:39:31.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7b2712369a9058-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.731 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.731 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.731 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.731 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.732 DEBUG:boto:Method: GET 2024-04-12T03:39:31.732 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.732 DEBUG:boto:Data: 2024-04-12T03:39:31.732 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.732 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.732 DEBUG:boto:Params: {} 2024-04-12T03:39:31.732 DEBUG:boto:Token: None 2024-04-12T03:39:31.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ= 2024-04-12T03:39:31.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016238e0697f4b9bc-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.774 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.774 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.774 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.775 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.775 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.775 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.775 DEBUG:boto:Data: 2024-04-12T03:39:31.775 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.775 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.775 DEBUG:boto:Params: {} 2024-04-12T03:39:31.775 DEBUG:boto:Token: None 2024-04-12T03:39:31.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:izQEdocZsMR5rPRK5X0gs0qsRQQ= 2024-04-12T03:39:31.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:izQEdocZsMR5rPRK5X0gs0qsRQQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fc0b6e12b5e941-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.777 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.777 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.777 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.777 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.778 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.778 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.778 DEBUG:boto:Data: 2024-04-12T03:39:31.778 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.778 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.778 DEBUG:boto:Params: {} 2024-04-12T03:39:31.778 DEBUG:boto:Token: None 2024-04-12T03:39:31.778 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.778 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:izQEdocZsMR5rPRK5X0gs0qsRQQ= 2024-04-12T03:39:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:izQEdocZsMR5rPRK5X0gs0qsRQQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5dd2771dd1fd6c0-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:31.781 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.781 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.781 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.781 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.781 DEBUG:boto:Method: GET 2024-04-12T03:39:31.781 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.781 DEBUG:boto:Data: 2024-04-12T03:39:31.781 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.782 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.782 DEBUG:boto:Params: {} 2024-04-12T03:39:31.782 DEBUG:boto:Token: None 2024-04-12T03:39:31.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ= 2024-04-12T03:39:31.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f9d42107c5e45b-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.784 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.784 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.784 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.784 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.784 DEBUG:boto:Method: GET 2024-04-12T03:39:31.784 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.784 DEBUG:boto:Data: 2024-04-12T03:39:31.785 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.785 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.785 DEBUG:boto:Params: {} 2024-04-12T03:39:31.785 DEBUG:boto:Token: None 2024-04-12T03:39:31.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:31.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ= 2024-04-12T03:39:31.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YOAfCm2iWcZhcDarQDzl2zMEWPQ=', 'Content-Length': '0'} 2024-04-12T03:39:31.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b259d457866a2c-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.787 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.787 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.787 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.788 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.788 DEBUG:boto:Method: GET 2024-04-12T03:39:31.788 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.788 DEBUG:boto:Data: 2024-04-12T03:39:31.788 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.788 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.788 DEBUG:boto:Params: {} 2024-04-12T03:39:31.788 DEBUG:boto:Token: None 2024-04-12T03:39:31.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e1febc6eb1e54f-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.791 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.792 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.792 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.792 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.792 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.792 DEBUG:boto:Method: GET 2024-04-12T03:39:31.792 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.792 DEBUG:boto:Data: 2024-04-12T03:39:31.792 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.792 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.793 DEBUG:boto:Params: {} 2024-04-12T03:39:31.793 DEBUG:boto:Token: None 2024-04-12T03:39:31.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a648098cf16efe-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.795 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.796 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.796 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.796 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.796 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.796 DEBUG:boto:Data: 2024-04-12T03:39:31.796 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.796 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.796 DEBUG:boto:Params: {} 2024-04-12T03:39:31.797 DEBUG:boto:Token: None 2024-04-12T03:39:31.797 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LCtdpxTZu+QyLb7PEK0cC3UvbTE= 2024-04-12T03:39:31.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LCtdpxTZu+QyLb7PEK0cC3UvbTE=', 'Content-Length': '0'} 2024-04-12T03:39:31.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c57728df0e832f4-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.799 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.799 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.799 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.799 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.799 DEBUG:boto:Data: 2024-04-12T03:39:31.799 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.799 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.800 DEBUG:boto:Params: {} 2024-04-12T03:39:31.800 DEBUG:boto:Token: None 2024-04-12T03:39:31.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LCtdpxTZu+QyLb7PEK0cC3UvbTE= 2024-04-12T03:39:31.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LCtdpxTZu+QyLb7PEK0cC3UvbTE=', 'Content-Length': '0'} 2024-04-12T03:39:31.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c407a37612ddfbac-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:31.802 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.802 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.802 DEBUG:boto:Method: GET 2024-04-12T03:39:31.802 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.802 DEBUG:boto:Data: 2024-04-12T03:39:31.802 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.802 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.802 DEBUG:boto:Params: {} 2024-04-12T03:39:31.803 DEBUG:boto:Token: None 2024-04-12T03:39:31.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xRK7Ar99shNrmkGf8IC3hkkS1Mw= 2024-04-12T03:39:31.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xRK7Ar99shNrmkGf8IC3hkkS1Mw=', 'Content-Length': '0'} 2024-04-12T03:39:31.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e042248aaf995d-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.846 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.846 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.847 DEBUG:boto:Method: GET 2024-04-12T03:39:31.847 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.847 DEBUG:boto:Data: 2024-04-12T03:39:31.847 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.847 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.847 DEBUG:boto:Params: {} 2024-04-12T03:39:31.847 DEBUG:boto:Token: None 2024-04-12T03:39:31.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:31.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xRK7Ar99shNrmkGf8IC3hkkS1Mw= 2024-04-12T03:39:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xRK7Ar99shNrmkGf8IC3hkkS1Mw=', 'Content-Length': '0'} 2024-04-12T03:39:31.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094991d52ce4656f5-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.850 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.850 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.850 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.850 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.850 DEBUG:boto:Method: GET 2024-04-12T03:39:31.851 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.851 DEBUG:boto:Data: 2024-04-12T03:39:31.851 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.851 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.851 DEBUG:boto:Params: {} 2024-04-12T03:39:31.851 DEBUG:boto:Token: None 2024-04-12T03:39:31.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd599a5815ebe4a-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.854 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.854 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.855 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.855 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.855 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.855 DEBUG:boto:Method: GET 2024-04-12T03:39:31.855 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:31.855 DEBUG:boto:Data: 2024-04-12T03:39:31.855 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.855 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.855 DEBUG:boto:Params: {} 2024-04-12T03:39:31.856 DEBUG:boto:Token: None 2024-04-12T03:39:31.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065427bc1ca8d7a8a-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.859 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:20.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:31.859 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.859 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.859 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.859 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.860 DEBUG:boto:Method: GET 2024-04-12T03:39:31.860 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.860 DEBUG:boto:Data: 2024-04-12T03:39:31.860 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.860 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.860 DEBUG:boto:Params: {} 2024-04-12T03:39:31.860 DEBUG:boto:Token: None 2024-04-12T03:39:31.861 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg= 2024-04-12T03:39:31.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg=', 'Content-Length': '0'} 2024-04-12T03:39:31.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd2c8776785846b-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.863 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.863 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.863 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.863 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.863 DEBUG:boto:Method: GET 2024-04-12T03:39:31.864 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.864 DEBUG:boto:Data: 2024-04-12T03:39:31.864 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.864 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.864 DEBUG:boto:Params: {} 2024-04-12T03:39:31.864 DEBUG:boto:Token: None 2024-04-12T03:39:31.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg= 2024-04-12T03:39:31.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg=', 'Content-Length': '0'} 2024-04-12T03:39:31.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e171d0dc6a9885-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.867 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.867 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.867 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.867 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.867 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.867 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.867 DEBUG:boto:Data: 2024-04-12T03:39:31.868 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.868 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.868 DEBUG:boto:Params: {} 2024-04-12T03:39:31.868 DEBUG:boto:Token: None 2024-04-12T03:39:31.868 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fIXzPfV0gw0BGVEKJLtbSt8GAyg= 2024-04-12T03:39:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fIXzPfV0gw0BGVEKJLtbSt8GAyg=', 'Content-Length': '0'} 2024-04-12T03:39:31.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3a439eef73af9a-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.870 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.870 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.871 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.871 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.871 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.871 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.871 DEBUG:boto:Data: 2024-04-12T03:39:31.871 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.871 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.871 DEBUG:boto:Params: {} 2024-04-12T03:39:31.871 DEBUG:boto:Token: None 2024-04-12T03:39:31.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fIXzPfV0gw0BGVEKJLtbSt8GAyg= 2024-04-12T03:39:31.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fIXzPfV0gw0BGVEKJLtbSt8GAyg=', 'Content-Length': '0'} 2024-04-12T03:39:31.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6e39ccfd730b55-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:31.874 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.874 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.874 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.874 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.874 DEBUG:boto:Method: GET 2024-04-12T03:39:31.874 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.875 DEBUG:boto:Data: 2024-04-12T03:39:31.875 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.875 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.875 DEBUG:boto:Params: {} 2024-04-12T03:39:31.875 DEBUG:boto:Token: None 2024-04-12T03:39:31.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg= 2024-04-12T03:39:31.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg=', 'Content-Length': '0'} 2024-04-12T03:39:31.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf4cf09a4b96a06-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.877 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.878 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.878 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.878 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.878 DEBUG:boto:Method: GET 2024-04-12T03:39:31.878 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.878 DEBUG:boto:Data: 2024-04-12T03:39:31.878 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.879 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.879 DEBUG:boto:Params: {} 2024-04-12T03:39:31.879 DEBUG:boto:Token: None 2024-04-12T03:39:31.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:31.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg= 2024-04-12T03:39:31.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0UHe26y4vVyus6pJnedNv/k45Pg=', 'Content-Length': '0'} 2024-04-12T03:39:31.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa0fe5654b39bb8-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.881 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.881 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.881 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.882 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.882 DEBUG:boto:Method: GET 2024-04-12T03:39:31.882 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.882 DEBUG:boto:Data: 2024-04-12T03:39:31.882 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.882 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.882 DEBUG:boto:Params: {} 2024-04-12T03:39:31.882 DEBUG:boto:Token: None 2024-04-12T03:39:31.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2fa192ab72f0c34-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.885 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.885 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.886 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.886 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.886 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.886 DEBUG:boto:Method: GET 2024-04-12T03:39:31.886 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.886 DEBUG:boto:Data: 2024-04-12T03:39:31.886 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.886 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.886 DEBUG:boto:Params: {} 2024-04-12T03:39:31.887 DEBUG:boto:Token: None 2024-04-12T03:39:31.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b7ef2fd490b49d-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.890 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.890 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.890 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.890 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.890 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.890 DEBUG:boto:Data: 2024-04-12T03:39:31.891 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.891 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.891 DEBUG:boto:Params: {} 2024-04-12T03:39:31.891 DEBUG:boto:Token: None 2024-04-12T03:39:31.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GeyRqBAlITSeeJi85EmmOJNmG6Y= 2024-04-12T03:39:31.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GeyRqBAlITSeeJi85EmmOJNmG6Y=', 'Content-Length': '0'} 2024-04-12T03:39:31.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27f4aac1c05542e-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.893 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.893 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.893 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.894 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.894 DEBUG:boto:Data: 2024-04-12T03:39:31.894 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.894 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.894 DEBUG:boto:Params: {} 2024-04-12T03:39:31.894 DEBUG:boto:Token: None 2024-04-12T03:39:31.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GeyRqBAlITSeeJi85EmmOJNmG6Y= 2024-04-12T03:39:31.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GeyRqBAlITSeeJi85EmmOJNmG6Y=', 'Content-Length': '0'} 2024-04-12T03:39:31.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00e6612e2684e55-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:31.897 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.897 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.897 DEBUG:boto:Method: GET 2024-04-12T03:39:31.897 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.897 DEBUG:boto:Data: 2024-04-12T03:39:31.897 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.898 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.898 DEBUG:boto:Params: {} 2024-04-12T03:39:31.898 DEBUG:boto:Token: None 2024-04-12T03:39:31.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0rFbpjSZuoUVvMViI7QuY3o5PQU= 2024-04-12T03:39:31.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0rFbpjSZuoUVvMViI7QuY3o5PQU=', 'Content-Length': '0'} 2024-04-12T03:39:31.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe665284e5d1c087-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.900 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.900 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.900 DEBUG:boto:Method: GET 2024-04-12T03:39:31.901 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.901 DEBUG:boto:Data: 2024-04-12T03:39:31.901 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.901 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.901 DEBUG:boto:Params: {} 2024-04-12T03:39:31.901 DEBUG:boto:Token: None 2024-04-12T03:39:31.901 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:31.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0rFbpjSZuoUVvMViI7QuY3o5PQU= 2024-04-12T03:39:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0rFbpjSZuoUVvMViI7QuY3o5PQU=', 'Content-Length': '0'} 2024-04-12T03:39:31.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef369909961be0c0-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.904 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.904 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.904 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.904 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.904 DEBUG:boto:Method: GET 2024-04-12T03:39:31.904 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.904 DEBUG:boto:Data: 2024-04-12T03:39:31.905 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.905 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.905 DEBUG:boto:Params: {} 2024-04-12T03:39:31.905 DEBUG:boto:Token: None 2024-04-12T03:39:31.905 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.905 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da551190a4a9277-006618acf3-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.907 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.908 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:31.908 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:31.908 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.908 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.908 DEBUG:boto:Method: GET 2024-04-12T03:39:31.908 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:31.908 DEBUG:boto:Data: 2024-04-12T03:39:31.908 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.909 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.909 DEBUG:boto:Params: {} 2024-04-12T03:39:31.909 DEBUG:boto:Token: None 2024-04-12T03:39:31.909 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/ 2024-04-12T03:39:31.909 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s= 2024-04-12T03:39:31.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/TP0MwFg1WfCX0Te2tGGz7eva+s=', 'Content-Length': '0'} 2024-04-12T03:39:31.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e047b5190669dc6a-006618acf3-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.958 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:21.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:31.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:31.959 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.959 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.959 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.959 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.959 DEBUG:boto:Method: GET 2024-04-12T03:39:31.959 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.959 DEBUG:boto:Data: 2024-04-12T03:39:31.960 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.960 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.960 DEBUG:boto:Params: {} 2024-04-12T03:39:31.960 DEBUG:boto:Token: None 2024-04-12T03:39:31.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs= 2024-04-12T03:39:31.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs=', 'Content-Length': '0'} 2024-04-12T03:39:31.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3857d4613b3bd27-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.962 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.963 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.963 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.963 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.963 DEBUG:boto:Method: GET 2024-04-12T03:39:31.963 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.963 DEBUG:boto:Data: 2024-04-12T03:39:31.963 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.963 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.963 DEBUG:boto:Params: {} 2024-04-12T03:39:31.963 DEBUG:boto:Token: None 2024-04-12T03:39:31.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs= 2024-04-12T03:39:31.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs=', 'Content-Length': '0'} 2024-04-12T03:39:31.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630c42c967b9d6f0-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.966 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.966 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.967 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.967 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.967 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.967 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.967 DEBUG:boto:Data: 2024-04-12T03:39:31.967 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.967 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.967 DEBUG:boto:Params: {} 2024-04-12T03:39:31.967 DEBUG:boto:Token: None 2024-04-12T03:39:31.967 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:96bzzzL6omuY6gSXaWAkC30Qn34= 2024-04-12T03:39:31.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:96bzzzL6omuY6gSXaWAkC30Qn34=', 'Content-Length': '0'} 2024-04-12T03:39:31.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be7d93bb530eace-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.969 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.969 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.969 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.969 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.970 DEBUG:boto:Method: HEAD 2024-04-12T03:39:31.970 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.970 DEBUG:boto:Data: 2024-04-12T03:39:31.970 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:31.970 DEBUG:boto:Port: 8000 2024-04-12T03:39:31.970 DEBUG:boto:Params: {} 2024-04-12T03:39:31.970 DEBUG:boto:Token: None 2024-04-12T03:39:31.970 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:96bzzzL6omuY6gSXaWAkC30Qn34= 2024-04-12T03:39:31.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:96bzzzL6omuY6gSXaWAkC30Qn34=', 'Content-Length': '0'} 2024-04-12T03:39:31.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4a831e5aa2992ec-006618acf3-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:31.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:31.973 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.973 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:31.973 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.973 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.973 DEBUG:boto:Method: GET 2024-04-12T03:39:31.973 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.974 DEBUG:boto:Data: 2024-04-12T03:39:31.974 DEBUG:boto:Headers: {} 2024-04-12T03:39:31.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:31.974 DEBUG:boto:Port: 8001 2024-04-12T03:39:31.974 DEBUG:boto:Params: {} 2024-04-12T03:39:31.974 DEBUG:boto:Token: None 2024-04-12T03:39:31.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:31 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:31.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs= 2024-04-12T03:39:31.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UNcGL8C6/X+xpN6VmwevmB5I/bs=', 'Content-Length': '0'} 2024-04-12T03:39:31.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009211c037ce80998f-006618acf3-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:32.019 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:32.019 DEBUG:boto:Method: GET 2024-04-12T03:39:32.019 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:32.019 DEBUG:boto:Data: 2024-04-12T03:39:32.019 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.019 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.019 DEBUG:boto:Params: {} 2024-04-12T03:39:32.019 DEBUG:boto:Token: None 2024-04-12T03:39:32.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:32.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:suX7QDdx7vFAzLkva6UIyOK6MYA= 2024-04-12T03:39:32.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:suX7QDdx7vFAzLkva6UIyOK6MYA=', 'Content-Length': '0'} 2024-04-12T03:39:32.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de8feaebfad39db-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.022 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.022 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.022 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.022 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.022 DEBUG:boto:Method: GET 2024-04-12T03:39:32.022 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.023 DEBUG:boto:Data: 2024-04-12T03:39:32.023 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.023 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.023 DEBUG:boto:Params: {} 2024-04-12T03:39:32.023 DEBUG:boto:Token: None 2024-04-12T03:39:32.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ff345915910f0c-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.026 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.026 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.026 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.026 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.026 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.027 DEBUG:boto:Method: GET 2024-04-12T03:39:32.027 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.027 DEBUG:boto:Data: 2024-04-12T03:39:32.027 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.027 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.027 DEBUG:boto:Params: {} 2024-04-12T03:39:32.027 DEBUG:boto:Token: None 2024-04-12T03:39:32.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee165ea4450e3cc-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.030 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.031 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.031 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.031 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.031 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.031 DEBUG:boto:Data: 2024-04-12T03:39:32.031 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.031 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.031 DEBUG:boto:Params: {} 2024-04-12T03:39:32.031 DEBUG:boto:Token: None 2024-04-12T03:39:32.031 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:060x1ljfjEu2fyX/xIKP8lwsew0= 2024-04-12T03:39:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:060x1ljfjEu2fyX/xIKP8lwsew0=', 'Content-Length': '0'} 2024-04-12T03:39:32.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2360e82ad0fbd35-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.033 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.033 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.033 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.033 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.034 DEBUG:boto:Data: 2024-04-12T03:39:32.034 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.034 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.034 DEBUG:boto:Params: {} 2024-04-12T03:39:32.034 DEBUG:boto:Token: None 2024-04-12T03:39:32.034 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:060x1ljfjEu2fyX/xIKP8lwsew0= 2024-04-12T03:39:32.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:060x1ljfjEu2fyX/xIKP8lwsew0=', 'Content-Length': '0'} 2024-04-12T03:39:32.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7ecdbc1ff260a6-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:32.036 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.037 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.037 DEBUG:boto:Method: GET 2024-04-12T03:39:32.037 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.037 DEBUG:boto:Data: 2024-04-12T03:39:32.037 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.037 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.037 DEBUG:boto:Params: {} 2024-04-12T03:39:32.037 DEBUG:boto:Token: None 2024-04-12T03:39:32.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.038 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zM6XnDvqaEi+RkP4RRyzKV0uipo= 2024-04-12T03:39:32.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zM6XnDvqaEi+RkP4RRyzKV0uipo=', 'Content-Length': '0'} 2024-04-12T03:39:32.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c4ef32b95499d3-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.040 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.040 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.040 DEBUG:boto:Method: GET 2024-04-12T03:39:32.040 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.040 DEBUG:boto:Data: 2024-04-12T03:39:32.040 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.040 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.040 DEBUG:boto:Params: {} 2024-04-12T03:39:32.041 DEBUG:boto:Token: None 2024-04-12T03:39:32.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:32.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zM6XnDvqaEi+RkP4RRyzKV0uipo= 2024-04-12T03:39:32.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zM6XnDvqaEi+RkP4RRyzKV0uipo=', 'Content-Length': '0'} 2024-04-12T03:39:32.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2aacd230c5c002a-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.043 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.043 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.043 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.043 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.043 DEBUG:boto:Method: GET 2024-04-12T03:39:32.043 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.043 DEBUG:boto:Data: 2024-04-12T03:39:32.043 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.043 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.043 DEBUG:boto:Params: {} 2024-04-12T03:39:32.044 DEBUG:boto:Token: None 2024-04-12T03:39:32.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d72755c753c295-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.046 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.046 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.046 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.047 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.047 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.047 DEBUG:boto:Method: GET 2024-04-12T03:39:32.047 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:32.047 DEBUG:boto:Data: 2024-04-12T03:39:32.047 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.047 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.047 DEBUG:boto:Params: {} 2024-04-12T03:39:32.048 DEBUG:boto:Token: None 2024-04-12T03:39:32.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88d528d0a1bc46e-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.050 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:21.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:32.051 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.051 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.051 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.051 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.051 DEBUG:boto:Method: GET 2024-04-12T03:39:32.051 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.051 DEBUG:boto:Data: 2024-04-12T03:39:32.051 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.051 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.052 DEBUG:boto:Params: {} 2024-04-12T03:39:32.052 DEBUG:boto:Token: None 2024-04-12T03:39:32.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY= 2024-04-12T03:39:32.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY=', 'Content-Length': '0'} 2024-04-12T03:39:32.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710c422073637b18-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.054 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.054 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.054 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.054 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.054 DEBUG:boto:Method: GET 2024-04-12T03:39:32.054 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.054 DEBUG:boto:Data: 2024-04-12T03:39:32.054 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.055 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.055 DEBUG:boto:Params: {} 2024-04-12T03:39:32.055 DEBUG:boto:Token: None 2024-04-12T03:39:32.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY= 2024-04-12T03:39:32.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY=', 'Content-Length': '0'} 2024-04-12T03:39:32.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c724c9fb66bbba-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.057 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.057 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.057 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.057 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.058 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.058 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.058 DEBUG:boto:Data: 2024-04-12T03:39:32.058 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.058 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.058 DEBUG:boto:Params: {} 2024-04-12T03:39:32.058 DEBUG:boto:Token: None 2024-04-12T03:39:32.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0H3sPTD2TcmAhxF2O4iIWvB3V1Y= 2024-04-12T03:39:32.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0H3sPTD2TcmAhxF2O4iIWvB3V1Y=', 'Content-Length': '0'} 2024-04-12T03:39:32.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004045a79d38e912e7-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.060 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.060 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.060 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.061 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.061 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.061 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.061 DEBUG:boto:Data: 2024-04-12T03:39:32.061 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.061 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.061 DEBUG:boto:Params: {} 2024-04-12T03:39:32.061 DEBUG:boto:Token: None 2024-04-12T03:39:32.061 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0H3sPTD2TcmAhxF2O4iIWvB3V1Y= 2024-04-12T03:39:32.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0H3sPTD2TcmAhxF2O4iIWvB3V1Y=', 'Content-Length': '0'} 2024-04-12T03:39:32.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040edb1b1e6a47b00-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:32.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.064 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.064 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.064 DEBUG:boto:Method: GET 2024-04-12T03:39:32.064 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.064 DEBUG:boto:Data: 2024-04-12T03:39:32.064 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.064 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.064 DEBUG:boto:Params: {} 2024-04-12T03:39:32.064 DEBUG:boto:Token: None 2024-04-12T03:39:32.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY= 2024-04-12T03:39:32.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY=', 'Content-Length': '0'} 2024-04-12T03:39:32.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3e6e21326a1980-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.066 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.067 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.067 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.067 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.067 DEBUG:boto:Method: GET 2024-04-12T03:39:32.067 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.067 DEBUG:boto:Data: 2024-04-12T03:39:32.067 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.067 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.067 DEBUG:boto:Params: {} 2024-04-12T03:39:32.067 DEBUG:boto:Token: None 2024-04-12T03:39:32.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:32.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY= 2024-04-12T03:39:32.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dTbSA1jQb89JER94EqlE3o6ehKY=', 'Content-Length': '0'} 2024-04-12T03:39:32.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887c55c013d4610a-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.110 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.110 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.110 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.111 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.111 DEBUG:boto:Method: GET 2024-04-12T03:39:32.111 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.111 DEBUG:boto:Data: 2024-04-12T03:39:32.111 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.111 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.111 DEBUG:boto:Params: {} 2024-04-12T03:39:32.111 DEBUG:boto:Token: None 2024-04-12T03:39:32.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb3915f054d592d-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.114 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.114 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.114 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.114 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.114 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.115 DEBUG:boto:Method: GET 2024-04-12T03:39:32.115 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.115 DEBUG:boto:Data: 2024-04-12T03:39:32.115 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.115 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.115 DEBUG:boto:Params: {} 2024-04-12T03:39:32.115 DEBUG:boto:Token: None 2024-04-12T03:39:32.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a907578c9deeeeae-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.118 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.118 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.118 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.118 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.118 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.119 DEBUG:boto:Data: 2024-04-12T03:39:32.119 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.119 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.119 DEBUG:boto:Params: {} 2024-04-12T03:39:32.119 DEBUG:boto:Token: None 2024-04-12T03:39:32.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r5Lmf1PyAveNACmsTFL2ZJo90GE= 2024-04-12T03:39:32.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r5Lmf1PyAveNACmsTFL2ZJo90GE=', 'Content-Length': '0'} 2024-04-12T03:39:32.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950e30f6592aaf30-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.121 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.121 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.121 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.121 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.121 DEBUG:boto:Data: 2024-04-12T03:39:32.121 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.122 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.122 DEBUG:boto:Params: {} 2024-04-12T03:39:32.122 DEBUG:boto:Token: None 2024-04-12T03:39:32.122 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r5Lmf1PyAveNACmsTFL2ZJo90GE= 2024-04-12T03:39:32.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r5Lmf1PyAveNACmsTFL2ZJo90GE=', 'Content-Length': '0'} 2024-04-12T03:39:32.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ac19ce4af5e11c-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:32.124 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.124 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.124 DEBUG:boto:Method: GET 2024-04-12T03:39:32.124 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.124 DEBUG:boto:Data: 2024-04-12T03:39:32.124 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.125 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.125 DEBUG:boto:Params: {} 2024-04-12T03:39:32.125 DEBUG:boto:Token: None 2024-04-12T03:39:32.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7gaNJJM96jy5y2v3E+pzP6LCG8= 2024-04-12T03:39:32.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7gaNJJM96jy5y2v3E+pzP6LCG8=', 'Content-Length': '0'} 2024-04-12T03:39:32.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004242ddeedee67d00-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.127 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.127 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.127 DEBUG:boto:Method: GET 2024-04-12T03:39:32.127 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.127 DEBUG:boto:Data: 2024-04-12T03:39:32.127 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.128 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.128 DEBUG:boto:Params: {} 2024-04-12T03:39:32.128 DEBUG:boto:Token: None 2024-04-12T03:39:32.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:32.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b7gaNJJM96jy5y2v3E+pzP6LCG8= 2024-04-12T03:39:32.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b7gaNJJM96jy5y2v3E+pzP6LCG8=', 'Content-Length': '0'} 2024-04-12T03:39:32.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df0bcacb82323699-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.130 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.130 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.130 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.130 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.130 DEBUG:boto:Method: GET 2024-04-12T03:39:32.131 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.131 DEBUG:boto:Data: 2024-04-12T03:39:32.131 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.131 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.131 DEBUG:boto:Params: {} 2024-04-12T03:39:32.131 DEBUG:boto:Token: None 2024-04-12T03:39:32.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf182c7f3bc8526-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.134 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.134 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.134 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.134 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.134 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.134 DEBUG:boto:Method: GET 2024-04-12T03:39:32.134 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:32.135 DEBUG:boto:Data: 2024-04-12T03:39:32.135 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.135 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.135 DEBUG:boto:Params: {} 2024-04-12T03:39:32.135 DEBUG:boto:Token: None 2024-04-12T03:39:32.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d3a58e86e5d5e65-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.137 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:21.077Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:32.138 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.138 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.138 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.138 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.138 DEBUG:boto:Method: GET 2024-04-12T03:39:32.138 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.138 DEBUG:boto:Data: 2024-04-12T03:39:32.139 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.139 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.139 DEBUG:boto:Params: {} 2024-04-12T03:39:32.139 DEBUG:boto:Token: None 2024-04-12T03:39:32.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI= 2024-04-12T03:39:32.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI=', 'Content-Length': '0'} 2024-04-12T03:39:32.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e4a675cd48f1b6-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.141 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.142 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.142 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.142 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.142 DEBUG:boto:Method: GET 2024-04-12T03:39:32.142 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.142 DEBUG:boto:Data: 2024-04-12T03:39:32.142 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.142 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.142 DEBUG:boto:Params: {} 2024-04-12T03:39:32.142 DEBUG:boto:Token: None 2024-04-12T03:39:32.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI= 2024-04-12T03:39:32.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI=', 'Content-Length': '0'} 2024-04-12T03:39:32.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ba57fa04e41a30-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.145 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.145 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.145 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.145 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.145 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.145 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.145 DEBUG:boto:Data: 2024-04-12T03:39:32.145 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.146 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.146 DEBUG:boto:Params: {} 2024-04-12T03:39:32.146 DEBUG:boto:Token: None 2024-04-12T03:39:32.146 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:THTEinCp/NYSw1xEX8IMri/zwQY= 2024-04-12T03:39:32.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:THTEinCp/NYSw1xEX8IMri/zwQY=', 'Content-Length': '0'} 2024-04-12T03:39:32.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c54c13def7fde7-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.148 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.148 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.148 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.148 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.148 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.148 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.149 DEBUG:boto:Data: 2024-04-12T03:39:32.149 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.149 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.149 DEBUG:boto:Params: {} 2024-04-12T03:39:32.149 DEBUG:boto:Token: None 2024-04-12T03:39:32.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:THTEinCp/NYSw1xEX8IMri/zwQY= 2024-04-12T03:39:32.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:THTEinCp/NYSw1xEX8IMri/zwQY=', 'Content-Length': '0'} 2024-04-12T03:39:32.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae35df4eecdab9b8-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:32.151 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.152 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.152 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.152 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.152 DEBUG:boto:Method: GET 2024-04-12T03:39:32.152 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.152 DEBUG:boto:Data: 2024-04-12T03:39:32.152 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.152 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.152 DEBUG:boto:Params: {} 2024-04-12T03:39:32.153 DEBUG:boto:Token: None 2024-04-12T03:39:32.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI= 2024-04-12T03:39:32.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI=', 'Content-Length': '0'} 2024-04-12T03:39:32.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dac83cd3b504fee-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.155 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.155 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.155 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.155 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.155 DEBUG:boto:Method: GET 2024-04-12T03:39:32.155 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.155 DEBUG:boto:Data: 2024-04-12T03:39:32.155 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.155 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.155 DEBUG:boto:Params: {} 2024-04-12T03:39:32.156 DEBUG:boto:Token: None 2024-04-12T03:39:32.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:32.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI= 2024-04-12T03:39:32.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jcHERldo4+pKAY0VuKLENEC9NgI=', 'Content-Length': '0'} 2024-04-12T03:39:32.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb4389a23123dfd-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.157 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.158 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.158 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.158 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.158 DEBUG:boto:Method: GET 2024-04-12T03:39:32.158 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.158 DEBUG:boto:Data: 2024-04-12T03:39:32.158 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.158 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.158 DEBUG:boto:Params: {} 2024-04-12T03:39:32.158 DEBUG:boto:Token: None 2024-04-12T03:39:32.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605ddc6862ec9926-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.202 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.203 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.203 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.203 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.203 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.203 DEBUG:boto:Method: GET 2024-04-12T03:39:32.203 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.203 DEBUG:boto:Data: 2024-04-12T03:39:32.203 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.203 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.204 DEBUG:boto:Params: {} 2024-04-12T03:39:32.204 DEBUG:boto:Token: None 2024-04-12T03:39:32.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be204c5a7c6b31de-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.207 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.207 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.207 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.207 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.207 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.207 DEBUG:boto:Data: 2024-04-12T03:39:32.207 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.208 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.208 DEBUG:boto:Params: {} 2024-04-12T03:39:32.208 DEBUG:boto:Token: None 2024-04-12T03:39:32.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hrTBxrVlhYKcqoTNxrbasA8qIPs= 2024-04-12T03:39:32.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hrTBxrVlhYKcqoTNxrbasA8qIPs=', 'Content-Length': '0'} 2024-04-12T03:39:32.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c9bcad3bddbd1d-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.210 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.210 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.210 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.210 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.210 DEBUG:boto:Data: 2024-04-12T03:39:32.210 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.210 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.210 DEBUG:boto:Params: {} 2024-04-12T03:39:32.211 DEBUG:boto:Token: None 2024-04-12T03:39:32.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hrTBxrVlhYKcqoTNxrbasA8qIPs= 2024-04-12T03:39:32.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hrTBxrVlhYKcqoTNxrbasA8qIPs=', 'Content-Length': '0'} 2024-04-12T03:39:32.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fcb827a949a73e2-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:32.213 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.213 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.213 DEBUG:boto:Method: GET 2024-04-12T03:39:32.213 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.213 DEBUG:boto:Data: 2024-04-12T03:39:32.213 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.213 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.213 DEBUG:boto:Params: {} 2024-04-12T03:39:32.214 DEBUG:boto:Token: None 2024-04-12T03:39:32.214 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.214 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A6mdm3Nl76hmn2WX7m9rqpyCZXY= 2024-04-12T03:39:32.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A6mdm3Nl76hmn2WX7m9rqpyCZXY=', 'Content-Length': '0'} 2024-04-12T03:39:32.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6df6c2010ec599b-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.216 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.216 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.216 DEBUG:boto:Method: GET 2024-04-12T03:39:32.216 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.216 DEBUG:boto:Data: 2024-04-12T03:39:32.216 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.216 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.217 DEBUG:boto:Params: {} 2024-04-12T03:39:32.217 DEBUG:boto:Token: None 2024-04-12T03:39:32.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:32.217 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A6mdm3Nl76hmn2WX7m9rqpyCZXY= 2024-04-12T03:39:32.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A6mdm3Nl76hmn2WX7m9rqpyCZXY=', 'Content-Length': '0'} 2024-04-12T03:39:32.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2b2e82b34b480b-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.262 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.262 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.263 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.263 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.263 DEBUG:boto:Method: GET 2024-04-12T03:39:32.263 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.263 DEBUG:boto:Data: 2024-04-12T03:39:32.263 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.263 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.263 DEBUG:boto:Params: {} 2024-04-12T03:39:32.263 DEBUG:boto:Token: None 2024-04-12T03:39:32.264 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6018973e5e0ab8-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.266 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.267 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.267 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.267 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.267 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.267 DEBUG:boto:Method: GET 2024-04-12T03:39:32.267 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:32.267 DEBUG:boto:Data: 2024-04-12T03:39:32.267 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.268 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.268 DEBUG:boto:Params: {} 2024-04-12T03:39:32.268 DEBUG:boto:Token: None 2024-04-12T03:39:32.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2ab25afb9dd993-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.270 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:21.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:32.271 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.271 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.271 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.271 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.271 DEBUG:boto:Method: GET 2024-04-12T03:39:32.271 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.271 DEBUG:boto:Data: 2024-04-12T03:39:32.271 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.271 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.272 DEBUG:boto:Params: {} 2024-04-12T03:39:32.272 DEBUG:boto:Token: None 2024-04-12T03:39:32.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo= 2024-04-12T03:39:32.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo=', 'Content-Length': '0'} 2024-04-12T03:39:32.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c70b48a4531d5c-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.274 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.274 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.274 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.275 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.275 DEBUG:boto:Method: GET 2024-04-12T03:39:32.275 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.275 DEBUG:boto:Data: 2024-04-12T03:39:32.275 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.275 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.275 DEBUG:boto:Params: {} 2024-04-12T03:39:32.275 DEBUG:boto:Token: None 2024-04-12T03:39:32.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo= 2024-04-12T03:39:32.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo=', 'Content-Length': '0'} 2024-04-12T03:39:32.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005056441c5dcb9ebd-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.280 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.280 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.280 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.280 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.280 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.280 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.280 DEBUG:boto:Data: 2024-04-12T03:39:32.280 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.281 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.281 DEBUG:boto:Params: {} 2024-04-12T03:39:32.281 DEBUG:boto:Token: None 2024-04-12T03:39:32.281 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oCvaE54nXVHimwcqcGOqC0s2cdQ= 2024-04-12T03:39:32.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oCvaE54nXVHimwcqcGOqC0s2cdQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdd502eb60153113-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.283 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.283 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.283 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.283 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.283 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.284 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.284 DEBUG:boto:Data: 2024-04-12T03:39:32.284 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.284 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.284 DEBUG:boto:Params: {} 2024-04-12T03:39:32.284 DEBUG:boto:Token: None 2024-04-12T03:39:32.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oCvaE54nXVHimwcqcGOqC0s2cdQ= 2024-04-12T03:39:32.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oCvaE54nXVHimwcqcGOqC0s2cdQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69b1ae0679a1303-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:32.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.287 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.287 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.287 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.287 DEBUG:boto:Method: GET 2024-04-12T03:39:32.287 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.287 DEBUG:boto:Data: 2024-04-12T03:39:32.287 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.287 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.287 DEBUG:boto:Params: {} 2024-04-12T03:39:32.288 DEBUG:boto:Token: None 2024-04-12T03:39:32.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo= 2024-04-12T03:39:32.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo=', 'Content-Length': '0'} 2024-04-12T03:39:32.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5145a75222fc1ee-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.330 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.330 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.330 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.331 DEBUG:boto:Method: GET 2024-04-12T03:39:32.331 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.331 DEBUG:boto:Data: 2024-04-12T03:39:32.331 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.331 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.331 DEBUG:boto:Params: {} 2024-04-12T03:39:32.331 DEBUG:boto:Token: None 2024-04-12T03:39:32.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:32.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo= 2024-04-12T03:39:32.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pYtjflybeJLsgmcoVH6a0mqJIyo=', 'Content-Length': '0'} 2024-04-12T03:39:32.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477b7f28dbc5cb1f-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.333 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.333 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.333 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.334 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.334 DEBUG:boto:Method: GET 2024-04-12T03:39:32.334 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.334 DEBUG:boto:Data: 2024-04-12T03:39:32.334 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.334 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.334 DEBUG:boto:Params: {} 2024-04-12T03:39:32.334 DEBUG:boto:Token: None 2024-04-12T03:39:32.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960dc8f96dd62a14-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.337 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.337 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.338 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.338 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.338 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.338 DEBUG:boto:Method: GET 2024-04-12T03:39:32.338 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.338 DEBUG:boto:Data: 2024-04-12T03:39:32.338 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.338 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.338 DEBUG:boto:Params: {} 2024-04-12T03:39:32.339 DEBUG:boto:Token: None 2024-04-12T03:39:32.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007853c434259f5858-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.341 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.342 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.342 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.342 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.342 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.342 DEBUG:boto:Data: 2024-04-12T03:39:32.342 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.343 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.343 DEBUG:boto:Params: {} 2024-04-12T03:39:32.343 DEBUG:boto:Token: None 2024-04-12T03:39:32.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2gsSHfS/8owDCpcRlMfO9DKv6wo= 2024-04-12T03:39:32.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2gsSHfS/8owDCpcRlMfO9DKv6wo=', 'Content-Length': '0'} 2024-04-12T03:39:32.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1308aa3aefcd2c-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.345 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.345 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.345 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.345 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.346 DEBUG:boto:Data: 2024-04-12T03:39:32.346 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.346 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.346 DEBUG:boto:Params: {} 2024-04-12T03:39:32.346 DEBUG:boto:Token: None 2024-04-12T03:39:32.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.346 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2gsSHfS/8owDCpcRlMfO9DKv6wo= 2024-04-12T03:39:32.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2gsSHfS/8owDCpcRlMfO9DKv6wo=', 'Content-Length': '0'} 2024-04-12T03:39:32.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b43874f4fdf0c5-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:32.348 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.348 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.348 DEBUG:boto:Method: GET 2024-04-12T03:39:32.348 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.348 DEBUG:boto:Data: 2024-04-12T03:39:32.349 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.349 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.349 DEBUG:boto:Params: {} 2024-04-12T03:39:32.349 DEBUG:boto:Token: None 2024-04-12T03:39:32.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WSDiZz+qtzVX0VhN8OFwu8HJkuQ= 2024-04-12T03:39:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WSDiZz+qtzVX0VhN8OFwu8HJkuQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000b458cc7d941ce-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.351 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.351 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.351 DEBUG:boto:Method: GET 2024-04-12T03:39:32.351 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.351 DEBUG:boto:Data: 2024-04-12T03:39:32.351 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.352 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.352 DEBUG:boto:Params: {} 2024-04-12T03:39:32.352 DEBUG:boto:Token: None 2024-04-12T03:39:32.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:32.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WSDiZz+qtzVX0VhN8OFwu8HJkuQ= 2024-04-12T03:39:32.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WSDiZz+qtzVX0VhN8OFwu8HJkuQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001299a13bfdeaa412-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.354 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.354 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.354 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.354 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.354 DEBUG:boto:Method: GET 2024-04-12T03:39:32.354 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.355 DEBUG:boto:Data: 2024-04-12T03:39:32.355 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.355 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.355 DEBUG:boto:Params: {} 2024-04-12T03:39:32.355 DEBUG:boto:Token: None 2024-04-12T03:39:32.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151ea4c6c92dcdd6-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.358 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.358 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.358 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.358 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.358 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.358 DEBUG:boto:Method: GET 2024-04-12T03:39:32.358 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:32.359 DEBUG:boto:Data: 2024-04-12T03:39:32.359 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.359 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.359 DEBUG:boto:Params: {} 2024-04-12T03:39:32.359 DEBUG:boto:Token: None 2024-04-12T03:39:32.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b20ef26f21e3a4-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.362 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:21.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:32.362 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.362 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.362 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.362 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.362 DEBUG:boto:Method: GET 2024-04-12T03:39:32.362 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.363 DEBUG:boto:Data: 2024-04-12T03:39:32.363 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.363 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.363 DEBUG:boto:Params: {} 2024-04-12T03:39:32.363 DEBUG:boto:Token: None 2024-04-12T03:39:32.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8= 2024-04-12T03:39:32.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8=', 'Content-Length': '0'} 2024-04-12T03:39:32.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a162f7644da1f0-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.365 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.365 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.366 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.366 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.366 DEBUG:boto:Method: GET 2024-04-12T03:39:32.366 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.366 DEBUG:boto:Data: 2024-04-12T03:39:32.366 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.366 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.366 DEBUG:boto:Params: {} 2024-04-12T03:39:32.366 DEBUG:boto:Token: None 2024-04-12T03:39:32.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.367 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8= 2024-04-12T03:39:32.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8=', 'Content-Length': '0'} 2024-04-12T03:39:32.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052902b2ba3c5185b-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.369 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.369 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.369 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.369 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.369 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.369 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.369 DEBUG:boto:Data: 2024-04-12T03:39:32.369 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.369 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.370 DEBUG:boto:Params: {} 2024-04-12T03:39:32.370 DEBUG:boto:Token: None 2024-04-12T03:39:32.370 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gFHHUekhZSwh5ZaxSumoo13q9WQ= 2024-04-12T03:39:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gFHHUekhZSwh5ZaxSumoo13q9WQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c9ba323a991b27-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.372 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.372 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.372 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.372 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.372 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.372 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.372 DEBUG:boto:Data: 2024-04-12T03:39:32.372 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.373 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.373 DEBUG:boto:Params: {} 2024-04-12T03:39:32.373 DEBUG:boto:Token: None 2024-04-12T03:39:32.373 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gFHHUekhZSwh5ZaxSumoo13q9WQ= 2024-04-12T03:39:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gFHHUekhZSwh5ZaxSumoo13q9WQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a6deafb03230d3-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:32.375 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.375 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.375 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.375 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.375 DEBUG:boto:Method: GET 2024-04-12T03:39:32.376 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.376 DEBUG:boto:Data: 2024-04-12T03:39:32.376 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.376 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.376 DEBUG:boto:Params: {} 2024-04-12T03:39:32.376 DEBUG:boto:Token: None 2024-04-12T03:39:32.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8= 2024-04-12T03:39:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8=', 'Content-Length': '0'} 2024-04-12T03:39:32.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f88f09ff0cc74f-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.378 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.379 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.379 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.379 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.379 DEBUG:boto:Method: GET 2024-04-12T03:39:32.379 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.379 DEBUG:boto:Data: 2024-04-12T03:39:32.379 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.379 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.379 DEBUG:boto:Params: {} 2024-04-12T03:39:32.379 DEBUG:boto:Token: None 2024-04-12T03:39:32.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:32.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8= 2024-04-12T03:39:32.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EQ2m2UeJERj5nH2PL5K/sVDgKc8=', 'Content-Length': '0'} 2024-04-12T03:39:32.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a523efbb20392ae-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.382 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.382 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.382 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.382 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.382 DEBUG:boto:Method: GET 2024-04-12T03:39:32.382 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.382 DEBUG:boto:Data: 2024-04-12T03:39:32.382 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.383 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.383 DEBUG:boto:Params: {} 2024-04-12T03:39:32.383 DEBUG:boto:Token: None 2024-04-12T03:39:32.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001460b60b611cd860-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.385 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.386 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.386 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.386 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.386 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.386 DEBUG:boto:Method: GET 2024-04-12T03:39:32.386 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.386 DEBUG:boto:Data: 2024-04-12T03:39:32.386 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.387 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.387 DEBUG:boto:Params: {} 2024-04-12T03:39:32.387 DEBUG:boto:Token: None 2024-04-12T03:39:32.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158ce6bb54ccfc74-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.389 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.390 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.390 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.390 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.390 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.390 DEBUG:boto:Data: 2024-04-12T03:39:32.390 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.391 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.391 DEBUG:boto:Params: {} 2024-04-12T03:39:32.391 DEBUG:boto:Token: None 2024-04-12T03:39:32.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tWsu3RUeBqstRo2c/8vLRGz9wwQ= 2024-04-12T03:39:32.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tWsu3RUeBqstRo2c/8vLRGz9wwQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851ec990bc0d895b-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.393 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.394 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.394 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.394 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.394 DEBUG:boto:Data: 2024-04-12T03:39:32.394 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.394 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.394 DEBUG:boto:Params: {} 2024-04-12T03:39:32.394 DEBUG:boto:Token: None 2024-04-12T03:39:32.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tWsu3RUeBqstRo2c/8vLRGz9wwQ= 2024-04-12T03:39:32.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tWsu3RUeBqstRo2c/8vLRGz9wwQ=', 'Content-Length': '0'} 2024-04-12T03:39:32.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006157956674c60d92-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:32.396 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.396 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.397 DEBUG:boto:Method: GET 2024-04-12T03:39:32.397 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.397 DEBUG:boto:Data: 2024-04-12T03:39:32.397 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.397 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.397 DEBUG:boto:Params: {} 2024-04-12T03:39:32.397 DEBUG:boto:Token: None 2024-04-12T03:39:32.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qJiezBWZJ1YR0bNd9GtCELzPhXg= 2024-04-12T03:39:32.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qJiezBWZJ1YR0bNd9GtCELzPhXg=', 'Content-Length': '0'} 2024-04-12T03:39:32.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e809cc969689e726-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.399 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.400 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.400 DEBUG:boto:Method: GET 2024-04-12T03:39:32.400 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.400 DEBUG:boto:Data: 2024-04-12T03:39:32.400 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.400 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.400 DEBUG:boto:Params: {} 2024-04-12T03:39:32.400 DEBUG:boto:Token: None 2024-04-12T03:39:32.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:32.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qJiezBWZJ1YR0bNd9GtCELzPhXg= 2024-04-12T03:39:32.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qJiezBWZJ1YR0bNd9GtCELzPhXg=', 'Content-Length': '0'} 2024-04-12T03:39:32.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e9d8de29927a4ad-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.403 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.403 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.403 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.403 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.403 DEBUG:boto:Method: GET 2024-04-12T03:39:32.403 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.403 DEBUG:boto:Data: 2024-04-12T03:39:32.403 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.404 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.404 DEBUG:boto:Params: {} 2024-04-12T03:39:32.404 DEBUG:boto:Token: None 2024-04-12T03:39:32.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b70b5413b8aae8-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.450 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.451 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.451 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.451 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.451 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.451 DEBUG:boto:Method: GET 2024-04-12T03:39:32.451 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:32.451 DEBUG:boto:Data: 2024-04-12T03:39:32.451 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.451 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.452 DEBUG:boto:Params: {} 2024-04-12T03:39:32.452 DEBUG:boto:Token: None 2024-04-12T03:39:32.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f22eea92039c4ec1-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.455 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:21.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:32.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.456 DEBUG:boto:Method: GET 2024-04-12T03:39:32.456 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.456 DEBUG:boto:Data: 2024-04-12T03:39:32.456 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.456 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.456 DEBUG:boto:Params: {} 2024-04-12T03:39:32.456 DEBUG:boto:Token: None 2024-04-12T03:39:32.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.457 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM= 2024-04-12T03:39:32.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM=', 'Content-Length': '0'} 2024-04-12T03:39:32.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006578a978939b56a8-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.458 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.459 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.459 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.459 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.459 DEBUG:boto:Method: GET 2024-04-12T03:39:32.459 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.459 DEBUG:boto:Data: 2024-04-12T03:39:32.459 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.459 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.460 DEBUG:boto:Params: {} 2024-04-12T03:39:32.460 DEBUG:boto:Token: None 2024-04-12T03:39:32.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM= 2024-04-12T03:39:32.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM=', 'Content-Length': '0'} 2024-04-12T03:39:32.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff74dbe6a4529df1-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.462 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.462 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.462 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.462 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.463 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.463 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.463 DEBUG:boto:Data: 2024-04-12T03:39:32.463 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.463 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.463 DEBUG:boto:Params: {} 2024-04-12T03:39:32.463 DEBUG:boto:Token: None 2024-04-12T03:39:32.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YeNbtf0XwMCcTjPtU5vXxge7DfE= 2024-04-12T03:39:32.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YeNbtf0XwMCcTjPtU5vXxge7DfE=', 'Content-Length': '0'} 2024-04-12T03:39:32.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000585ada600806b4c-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.465 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.465 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.465 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.465 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.465 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.466 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.466 DEBUG:boto:Data: 2024-04-12T03:39:32.466 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.466 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.466 DEBUG:boto:Params: {} 2024-04-12T03:39:32.466 DEBUG:boto:Token: None 2024-04-12T03:39:32.466 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YeNbtf0XwMCcTjPtU5vXxge7DfE= 2024-04-12T03:39:32.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YeNbtf0XwMCcTjPtU5vXxge7DfE=', 'Content-Length': '0'} 2024-04-12T03:39:32.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc9e05baa2cbe75-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:32.468 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.469 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.469 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.469 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.469 DEBUG:boto:Method: GET 2024-04-12T03:39:32.469 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.469 DEBUG:boto:Data: 2024-04-12T03:39:32.469 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.469 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.469 DEBUG:boto:Params: {} 2024-04-12T03:39:32.469 DEBUG:boto:Token: None 2024-04-12T03:39:32.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM= 2024-04-12T03:39:32.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM=', 'Content-Length': '0'} 2024-04-12T03:39:32.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022451a85c48ca189-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.472 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.472 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.472 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.472 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.472 DEBUG:boto:Method: GET 2024-04-12T03:39:32.472 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.472 DEBUG:boto:Data: 2024-04-12T03:39:32.472 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.473 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.473 DEBUG:boto:Params: {} 2024-04-12T03:39:32.473 DEBUG:boto:Token: None 2024-04-12T03:39:32.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:32.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM= 2024-04-12T03:39:32.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FPA3ZfvvC/vnlv0Nk0WhrTovxaM=', 'Content-Length': '0'} 2024-04-12T03:39:32.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b105d21f265184d-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.475 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.475 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.475 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.476 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.476 DEBUG:boto:Method: GET 2024-04-12T03:39:32.476 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.476 DEBUG:boto:Data: 2024-04-12T03:39:32.476 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.476 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.476 DEBUG:boto:Params: {} 2024-04-12T03:39:32.476 DEBUG:boto:Token: None 2024-04-12T03:39:32.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bff8e977dd87550-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.479 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.479 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.479 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.479 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.479 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.479 DEBUG:boto:Method: GET 2024-04-12T03:39:32.480 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.480 DEBUG:boto:Data: 2024-04-12T03:39:32.480 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.480 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.480 DEBUG:boto:Params: {} 2024-04-12T03:39:32.480 DEBUG:boto:Token: None 2024-04-12T03:39:32.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b43b84868e2b8d9-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.526 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.527 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.527 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.527 DEBUG:boto:Data: 2024-04-12T03:39:32.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.527 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.527 DEBUG:boto:Params: {} 2024-04-12T03:39:32.527 DEBUG:boto:Token: None 2024-04-12T03:39:32.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j/fVMaEE5E+ZY65Dwn79Y0YOzzE= 2024-04-12T03:39:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j/fVMaEE5E+ZY65Dwn79Y0YOzzE=', 'Content-Length': '0'} 2024-04-12T03:39:32.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928e2ef09f2104e3-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.530 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.530 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.530 DEBUG:boto:Method: HEAD 2024-04-12T03:39:32.530 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.530 DEBUG:boto:Data: 2024-04-12T03:39:32.530 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.530 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.530 DEBUG:boto:Params: {} 2024-04-12T03:39:32.531 DEBUG:boto:Token: None 2024-04-12T03:39:32.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j/fVMaEE5E+ZY65Dwn79Y0YOzzE= 2024-04-12T03:39:32.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j/fVMaEE5E+ZY65Dwn79Y0YOzzE=', 'Content-Length': '0'} 2024-04-12T03:39:32.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c78fb03436bc8c7-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:32.533 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.533 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.533 DEBUG:boto:Method: GET 2024-04-12T03:39:32.533 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.533 DEBUG:boto:Data: 2024-04-12T03:39:32.534 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.534 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.534 DEBUG:boto:Params: {} 2024-04-12T03:39:32.534 DEBUG:boto:Token: None 2024-04-12T03:39:32.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VU9kR4wP+bYZFNn+0C1Y3YfLjeo= 2024-04-12T03:39:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VU9kR4wP+bYZFNn+0C1Y3YfLjeo=', 'Content-Length': '0'} 2024-04-12T03:39:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4839763649a70f-006618acf4-1da1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.536 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.536 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.536 DEBUG:boto:Method: GET 2024-04-12T03:39:32.537 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.537 DEBUG:boto:Data: 2024-04-12T03:39:32.537 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.537 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.537 DEBUG:boto:Params: {} 2024-04-12T03:39:32.537 DEBUG:boto:Token: None 2024-04-12T03:39:32.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:32.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VU9kR4wP+bYZFNn+0C1Y3YfLjeo= 2024-04-12T03:39:32.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VU9kR4wP+bYZFNn+0C1Y3YfLjeo=', 'Content-Length': '0'} 2024-04-12T03:39:32.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75eaae312d2eb9b-006618acf4-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.539 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.539 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.540 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.540 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.540 DEBUG:boto:Method: GET 2024-04-12T03:39:32.540 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.540 DEBUG:boto:Data: 2024-04-12T03:39:32.540 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:32.540 DEBUG:boto:Port: 8001 2024-04-12T03:39:32.540 DEBUG:boto:Params: {} 2024-04-12T03:39:32.540 DEBUG:boto:Token: None 2024-04-12T03:39:32.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bead5c4b545a2119-006618acf4-1da1-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.543 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.543 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:32.543 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:32.544 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.544 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.544 DEBUG:boto:Method: GET 2024-04-12T03:39:32.544 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:32.544 DEBUG:boto:Data: 2024-04-12T03:39:32.544 DEBUG:boto:Headers: {} 2024-04-12T03:39:32.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:32.544 DEBUG:boto:Port: 8000 2024-04-12T03:39:32.544 DEBUG:boto:Params: {} 2024-04-12T03:39:32.544 DEBUG:boto:Token: None 2024-04-12T03:39:32.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:32 GMT /dgbpxg-60/ 2024-04-12T03:39:32.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8= 2024-04-12T03:39:32.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ixfe9dn5nC5LlSdbhwWer+yE+q8=', 'Content-Length': '0'} 2024-04-12T03:39:32.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004587c380924d64b-006618acf4-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:32.547 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:21.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:32.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-60 zones={a2, a1} 2024-04-12T03:39:32.547 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:39:32.548 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:39:32.548 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:39:32.546+0000 7fb5e42e4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 93111) UID: 0 2024-04-12T03:39:32.548 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:39:32.546+0000 7fb5e94b97c0 -1 shutting down 2024-04-12T03:39:33.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:39:38.648 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:39:38.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:38.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:38.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:38.707 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:38.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:38.754 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:38.754 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:38.758 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2024-04-12T03:39:38.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:38.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:38.797 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:38.847 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:38.847 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:38.887 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2024-04-12T03:39:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2024-04-12T03:39:38.887 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000024", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000028", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000032", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2024-04-12T03:39:38.888 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000076" 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:38.889 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:38.890 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:38.890 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:38.890 DEBUG:boto:Method: HEAD 2024-04-12T03:39:38.890 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:38.891 DEBUG:boto:Data: 2024-04-12T03:39:38.891 DEBUG:boto:Headers: {} 2024-04-12T03:39:38.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.891 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.891 DEBUG:boto:Params: {} 2024-04-12T03:39:38.891 DEBUG:boto:Token: None 2024-04-12T03:39:38.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/ 2024-04-12T03:39:38.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E= 2024-04-12T03:39:38.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E=', 'Content-Length': '0'} 2024-04-12T03:39:38.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f546c0ee1b315c85-006618acfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.894 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:38.894 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:38.894 DEBUG:boto:Method: PUT 2024-04-12T03:39:38.895 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:38.895 DEBUG:boto:Data: 2024-04-12T03:39:38.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:38.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.895 DEBUG:boto:Params: {} 2024-04-12T03:39:38.895 DEBUG:boto:Token: None 2024-04-12T03:39:38.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:38.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mSvo8MflpQkkUjhC5iiNUJ652VQ= 2024-04-12T03:39:38.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mSvo8MflpQkkUjhC5iiNUJ652VQ='} 2024-04-12T03:39:38.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fb9b7ef14d0d70d-006618acfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.902 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:38.902 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:38.902 DEBUG:boto:Method: HEAD 2024-04-12T03:39:38.902 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:38.902 DEBUG:boto:Data: 2024-04-12T03:39:38.902 DEBUG:boto:Headers: {} 2024-04-12T03:39:38.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.902 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.903 DEBUG:boto:Params: {} 2024-04-12T03:39:38.903 DEBUG:boto:Token: None 2024-04-12T03:39:38.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/ 2024-04-12T03:39:38.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E= 2024-04-12T03:39:38.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E=', 'Content-Length': '0'} 2024-04-12T03:39:38.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000909ae1448e0d8b2b-006618acfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.905 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:38.906 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:38.906 DEBUG:boto:Method: PUT 2024-04-12T03:39:38.906 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:38.906 DEBUG:boto:Data: 2024-04-12T03:39:38.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:38.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.906 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.906 DEBUG:boto:Params: {} 2024-04-12T03:39:38.906 DEBUG:boto:Token: None 2024-04-12T03:39:38.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:38.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AMdiVgKSAP7F9xmWLQ44XKZtmmo= 2024-04-12T03:39:38.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AMdiVgKSAP7F9xmWLQ44XKZtmmo='} 2024-04-12T03:39:38.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4d1e2ee2bcf300d-006618acfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.951 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:38.951 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:38.951 DEBUG:boto:Method: HEAD 2024-04-12T03:39:38.951 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:38.951 DEBUG:boto:Data: 2024-04-12T03:39:38.951 DEBUG:boto:Headers: {} 2024-04-12T03:39:38.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.951 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.951 DEBUG:boto:Params: {} 2024-04-12T03:39:38.951 DEBUG:boto:Token: None 2024-04-12T03:39:38.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/ 2024-04-12T03:39:38.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E= 2024-04-12T03:39:38.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E=', 'Content-Length': '0'} 2024-04-12T03:39:38.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b2905a2f7beb9af-006618acfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:38.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:38.955 DEBUG:boto:Method: PUT 2024-04-12T03:39:38.955 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:38.955 DEBUG:boto:Data: 2024-04-12T03:39:38.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:38.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.955 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.955 DEBUG:boto:Params: {} 2024-04-12T03:39:38.955 DEBUG:boto:Token: None 2024-04-12T03:39:38.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:38.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V/EUn0XhAdSxSdkySljh1fCvoKI= 2024-04-12T03:39:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V/EUn0XhAdSxSdkySljh1fCvoKI='} 2024-04-12T03:39:38.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ace554c354ad498c-006618acfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.961 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:38.961 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:38.961 DEBUG:boto:Method: HEAD 2024-04-12T03:39:38.961 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:38.961 DEBUG:boto:Data: 2024-04-12T03:39:38.961 DEBUG:boto:Headers: {} 2024-04-12T03:39:38.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.961 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.962 DEBUG:boto:Params: {} 2024-04-12T03:39:38.962 DEBUG:boto:Token: None 2024-04-12T03:39:38.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/ 2024-04-12T03:39:38.962 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E= 2024-04-12T03:39:38.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D7F+n7RrAPEx3X1YvkURIhQXh0E=', 'Content-Length': '0'} 2024-04-12T03:39:38.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca3370a8c5b1737-006618acfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:38.965 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:38.965 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:38.965 DEBUG:boto:Method: PUT 2024-04-12T03:39:38.965 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:38.965 DEBUG:boto:Data: 2024-04-12T03:39:38.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:38.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:38.965 DEBUG:boto:Port: 8000 2024-04-12T03:39:38.965 DEBUG:boto:Params: {} 2024-04-12T03:39:38.965 DEBUG:boto:Token: None 2024-04-12T03:39:38.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:38 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:38.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:snXuwLStNA2JEf1thA+NGBInGCY= 2024-04-12T03:39:38.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:snXuwLStNA2JEf1thA+NGBInGCY='} 2024-04-12T03:39:39.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe2a9871c4563dcc-006618acfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.011 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.011 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.011 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.011 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.011 DEBUG:boto:Data: 2024-04-12T03:39:39.011 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.011 DEBUG:boto:Params: {} 2024-04-12T03:39:39.011 DEBUG:boto:Token: None 2024-04-12T03:39:39.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff3c2b3c11884cee-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:39.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:39.015 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.015 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:39.015 DEBUG:boto:Data: 2024-04-12T03:39:39.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.015 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.015 DEBUG:boto:Params: {} 2024-04-12T03:39:39.015 DEBUG:boto:Token: None 2024-04-12T03:39:39.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:39.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IoeUOvTzKdEp4azuUba+uEZ3BBs= 2024-04-12T03:39:39.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IoeUOvTzKdEp4azuUba+uEZ3BBs='} 2024-04-12T03:39:39.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eea274ba9cb4f452-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.021 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.021 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.021 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.021 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.021 DEBUG:boto:Data: 2024-04-12T03:39:39.021 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.021 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.021 DEBUG:boto:Params: {} 2024-04-12T03:39:39.022 DEBUG:boto:Token: None 2024-04-12T03:39:39.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017662d908cda76eb-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.024 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.025 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.025 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.025 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.025 DEBUG:boto:Data: 2024-04-12T03:39:39.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.025 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.025 DEBUG:boto:Params: {} 2024-04-12T03:39:39.025 DEBUG:boto:Token: None 2024-04-12T03:39:39.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.026 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ASeYCaZNH7Zz5sw+HecPSr0RN8= 2024-04-12T03:39:39.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ASeYCaZNH7Zz5sw+HecPSr0RN8='} 2024-04-12T03:39:39.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000293e6ff4df57696c-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.031 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.032 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.032 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.032 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.032 DEBUG:boto:Data: 2024-04-12T03:39:39.032 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.032 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.032 DEBUG:boto:Params: {} 2024-04-12T03:39:39.032 DEBUG:boto:Token: None 2024-04-12T03:39:39.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.033 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f382349404a3803-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.035 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:39.035 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:39.036 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.036 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:39.036 DEBUG:boto:Data: 2024-04-12T03:39:39.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.036 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.036 DEBUG:boto:Params: {} 2024-04-12T03:39:39.036 DEBUG:boto:Token: None 2024-04-12T03:39:39.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:39.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H8YRWe96icrSfgZoHyeajgAFHHs= 2024-04-12T03:39:39.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H8YRWe96icrSfgZoHyeajgAFHHs='} 2024-04-12T03:39:39.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ac3e35e99f9c3fa-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.079 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.079 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.079 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.079 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.079 DEBUG:boto:Data: 2024-04-12T03:39:39.079 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.079 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.079 DEBUG:boto:Params: {} 2024-04-12T03:39:39.079 DEBUG:boto:Token: None 2024-04-12T03:39:39.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028aee4ccc1a375a3-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.082 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.082 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.082 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.082 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.083 DEBUG:boto:Data: 2024-04-12T03:39:39.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.083 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.083 DEBUG:boto:Params: {} 2024-04-12T03:39:39.083 DEBUG:boto:Token: None 2024-04-12T03:39:39.083 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/f+F2paEPFb9YDodYtKwREoLEa8= 2024-04-12T03:39:39.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/f+F2paEPFb9YDodYtKwREoLEa8='} 2024-04-12T03:39:39.089 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b523c886abf7d304-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.089 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.089 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.089 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.089 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.089 DEBUG:boto:Data: 2024-04-12T03:39:39.090 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.090 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.090 DEBUG:boto:Params: {} 2024-04-12T03:39:39.090 DEBUG:boto:Token: None 2024-04-12T03:39:39.090 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d085493efb05cf96-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.093 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.093 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.093 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.093 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.093 DEBUG:boto:Data: 2024-04-12T03:39:39.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.094 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.094 DEBUG:boto:Params: {} 2024-04-12T03:39:39.094 DEBUG:boto:Token: None 2024-04-12T03:39:39.094 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VyicmfNeIsGwLi+s1aa34bl+jQo= 2024-04-12T03:39:39.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VyicmfNeIsGwLi+s1aa34bl+jQo='} 2024-04-12T03:39:39.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbe567fab4895412-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.099 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.099 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.099 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.100 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.100 DEBUG:boto:Data: 2024-04-12T03:39:39.100 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.100 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.100 DEBUG:boto:Params: {} 2024-04-12T03:39:39.100 DEBUG:boto:Token: None 2024-04-12T03:39:39.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009715516714d3a349-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.103 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:39.103 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:39.103 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.103 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:39.103 DEBUG:boto:Data: 2024-04-12T03:39:39.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.104 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.104 DEBUG:boto:Params: {} 2024-04-12T03:39:39.104 DEBUG:boto:Token: None 2024-04-12T03:39:39.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:39.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uIs3qA3eIgCJ41NA0LqMGuAmtgQ= 2024-04-12T03:39:39.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uIs3qA3eIgCJ41NA0LqMGuAmtgQ='} 2024-04-12T03:39:39.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc7365fe268fd79f-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.147 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.147 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.147 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.147 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.147 DEBUG:boto:Data: 2024-04-12T03:39:39.147 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.147 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.147 DEBUG:boto:Params: {} 2024-04-12T03:39:39.147 DEBUG:boto:Token: None 2024-04-12T03:39:39.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035263e136ed2a5f3-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.150 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:39.150 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:39.150 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.150 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:39.151 DEBUG:boto:Data: 2024-04-12T03:39:39.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.151 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.151 DEBUG:boto:Params: {} 2024-04-12T03:39:39.151 DEBUG:boto:Token: None 2024-04-12T03:39:39.151 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:39.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+qg7fuq4cM2ux7kxl5oftC4Q8Cc= 2024-04-12T03:39:39.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+qg7fuq4cM2ux7kxl5oftC4Q8Cc='} 2024-04-12T03:39:39.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daf6e8dea48cde8d-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.157 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.157 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.157 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.157 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.157 DEBUG:boto:Data: 2024-04-12T03:39:39.157 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.157 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.158 DEBUG:boto:Params: {} 2024-04-12T03:39:39.158 DEBUG:boto:Token: None 2024-04-12T03:39:39.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af600fa9d9cb4638-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.160 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.160 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.161 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.161 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.161 DEBUG:boto:Data: 2024-04-12T03:39:39.161 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.161 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.161 DEBUG:boto:Params: {} 2024-04-12T03:39:39.161 DEBUG:boto:Token: None 2024-04-12T03:39:39.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NAAyZxM3rMEk1+wvShJhDdQbugc= 2024-04-12T03:39:39.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NAAyZxM3rMEk1+wvShJhDdQbugc='} 2024-04-12T03:39:39.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adc9b79f7548720e-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.166 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.166 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.166 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.166 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.167 DEBUG:boto:Data: 2024-04-12T03:39:39.167 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.167 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.167 DEBUG:boto:Params: {} 2024-04-12T03:39:39.167 DEBUG:boto:Token: None 2024-04-12T03:39:39.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbed1c98482f6d99-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.170 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:39.170 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:39.170 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.170 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:39.170 DEBUG:boto:Data: 2024-04-12T03:39:39.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.171 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.171 DEBUG:boto:Params: {} 2024-04-12T03:39:39.171 DEBUG:boto:Token: None 2024-04-12T03:39:39.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:39.171 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qJkr9g6v3xUHZ6ngqJ0MwJYNuOI= 2024-04-12T03:39:39.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qJkr9g6v3xUHZ6ngqJ0MwJYNuOI='} 2024-04-12T03:39:39.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:39:39.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a66852267c342a82-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.176 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.176 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.176 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.176 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.177 DEBUG:boto:Data: 2024-04-12T03:39:39.177 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.177 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.177 DEBUG:boto:Params: {} 2024-04-12T03:39:39.177 DEBUG:boto:Token: None 2024-04-12T03:39:39.177 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.177 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852d10da0ea8c129-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.180 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.180 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.180 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.180 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.181 DEBUG:boto:Data: 2024-04-12T03:39:39.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.181 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.181 DEBUG:boto:Params: {} 2024-04-12T03:39:39.181 DEBUG:boto:Token: None 2024-04-12T03:39:39.181 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCpXRFlUH2y/T9BFhHhQRcRP5Pc= 2024-04-12T03:39:39.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCpXRFlUH2y/T9BFhHhQRcRP5Pc='} 2024-04-12T03:39:39.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e270e0e137b8a36-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.227 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.227 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.227 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.227 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.227 DEBUG:boto:Data: 2024-04-12T03:39:39.227 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.227 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.227 DEBUG:boto:Params: {} 2024-04-12T03:39:39.227 DEBUG:boto:Token: None 2024-04-12T03:39:39.228 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b27cc1a96f4e970-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.230 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:39.230 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:39.231 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.231 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:39.231 DEBUG:boto:Data: 2024-04-12T03:39:39.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.231 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.231 DEBUG:boto:Params: {} 2024-04-12T03:39:39.231 DEBUG:boto:Token: None 2024-04-12T03:39:39.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:39.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RyxZdP2Y5fl6uIJQ3Oi0BBebKZM= 2024-04-12T03:39:39.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RyxZdP2Y5fl6uIJQ3Oi0BBebKZM='} 2024-04-12T03:39:39.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b839eb68e4379182-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.236 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.236 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.237 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.237 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.237 DEBUG:boto:Data: 2024-04-12T03:39:39.237 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.237 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.237 DEBUG:boto:Params: {} 2024-04-12T03:39:39.237 DEBUG:boto:Token: None 2024-04-12T03:39:39.237 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009897f0afdc180af0-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.240 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:39.241 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:39.241 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.241 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:39.241 DEBUG:boto:Data: 2024-04-12T03:39:39.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.241 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.241 DEBUG:boto:Params: {} 2024-04-12T03:39:39.241 DEBUG:boto:Token: None 2024-04-12T03:39:39.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:39.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GaVrSM/WUCr2HcSj4rX7tfqEmvc= 2024-04-12T03:39:39.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GaVrSM/WUCr2HcSj4rX7tfqEmvc='} 2024-04-12T03:39:39.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3a35051df51a118-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:39.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:39.287 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:39.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:39.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:39.387 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:39.388 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:39.394 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 21 2024-04-12T03:39:39.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:39.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:39.440 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:39.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:39.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:39.532 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000025", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2024-04-12T03:39:39.533 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007" 2024-04-12T03:39:39.534 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:39.535 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:39.535 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.536 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.536 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.536 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.536 DEBUG:boto:Data: 2024-04-12T03:39:39.536 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.536 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.536 DEBUG:boto:Params: {} 2024-04-12T03:39:39.536 DEBUG:boto:Token: None 2024-04-12T03:39:39.537 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e3d68108a02e3b6-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.540 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.540 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.540 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.540 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.540 DEBUG:boto:Data: 2024-04-12T03:39:39.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.540 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.540 DEBUG:boto:Params: {} 2024-04-12T03:39:39.540 DEBUG:boto:Token: None 2024-04-12T03:39:39.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:39.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/f+F2paEPFb9YDodYtKwREoLEa8= 2024-04-12T03:39:39.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/f+F2paEPFb9YDodYtKwREoLEa8='} 2024-04-12T03:39:39.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a01c3858b9004480-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.547 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.547 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.547 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.547 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.547 DEBUG:boto:Data: 2024-04-12T03:39:39.548 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.548 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.548 DEBUG:boto:Params: {} 2024-04-12T03:39:39.548 DEBUG:boto:Token: None 2024-04-12T03:39:39.548 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b90fdad11b90fe6d-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.551 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:39.551 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:39.551 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.551 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:39.551 DEBUG:boto:Data: 2024-04-12T03:39:39.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.552 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.552 DEBUG:boto:Params: {} 2024-04-12T03:39:39.552 DEBUG:boto:Token: None 2024-04-12T03:39:39.552 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:39.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FJOx5B/qyqdq5+tZutAfaErPlzw= 2024-04-12T03:39:39.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FJOx5B/qyqdq5+tZutAfaErPlzw='} 2024-04-12T03:39:39.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb247982e2bc89ba-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.594 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.595 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.595 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.595 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.595 DEBUG:boto:Data: 2024-04-12T03:39:39.595 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.595 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.595 DEBUG:boto:Params: {} 2024-04-12T03:39:39.595 DEBUG:boto:Token: None 2024-04-12T03:39:39.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ffc67a287fed0fd-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.598 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.598 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.599 DEBUG:boto:Data: 2024-04-12T03:39:39.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.599 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.599 DEBUG:boto:Params: {} 2024-04-12T03:39:39.599 DEBUG:boto:Token: None 2024-04-12T03:39:39.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:39.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NAAyZxM3rMEk1+wvShJhDdQbugc= 2024-04-12T03:39:39.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NAAyZxM3rMEk1+wvShJhDdQbugc='} 2024-04-12T03:39:39.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0a66c4a99881f02-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.605 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.606 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.606 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.606 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.606 DEBUG:boto:Data: 2024-04-12T03:39:39.606 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.606 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.606 DEBUG:boto:Params: {} 2024-04-12T03:39:39.606 DEBUG:boto:Token: None 2024-04-12T03:39:39.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9fe64d051235646-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.609 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:39.609 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:39.610 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.610 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:39.610 DEBUG:boto:Data: 2024-04-12T03:39:39.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.610 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.610 DEBUG:boto:Params: {} 2024-04-12T03:39:39.610 DEBUG:boto:Token: None 2024-04-12T03:39:39.610 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:39.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UGzOIGuEixawPGYK2SDBqDjWf3Q= 2024-04-12T03:39:39.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UGzOIGuEixawPGYK2SDBqDjWf3Q='} 2024-04-12T03:39:39.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9cffc42e438a446-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.615 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.615 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.615 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.616 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.616 DEBUG:boto:Data: 2024-04-12T03:39:39.616 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.616 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.616 DEBUG:boto:Params: {} 2024-04-12T03:39:39.616 DEBUG:boto:Token: None 2024-04-12T03:39:39.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000553b8df29cdb7d9e-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.619 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:39.619 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:39.619 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.619 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:39.619 DEBUG:boto:Data: 2024-04-12T03:39:39.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.620 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.620 DEBUG:boto:Params: {} 2024-04-12T03:39:39.620 DEBUG:boto:Token: None 2024-04-12T03:39:39.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:39.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I0DqiKKyPhVhYLNpUlLzU+HU4jI= 2024-04-12T03:39:39.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I0DqiKKyPhVhYLNpUlLzU+HU4jI='} 2024-04-12T03:39:39.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015788df5e57e42c8-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.663 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.663 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.663 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.663 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.663 DEBUG:boto:Data: 2024-04-12T03:39:39.663 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.663 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.663 DEBUG:boto:Params: {} 2024-04-12T03:39:39.663 DEBUG:boto:Token: None 2024-04-12T03:39:39.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007948eb40df780870-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.667 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:39.667 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:39.667 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.667 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:39.667 DEBUG:boto:Data: 2024-04-12T03:39:39.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.667 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.667 DEBUG:boto:Params: {} 2024-04-12T03:39:39.668 DEBUG:boto:Token: None 2024-04-12T03:39:39.668 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:39.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hgwet2EWqBSlgPE8sPvYtDR1gg8= 2024-04-12T03:39:39.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hgwet2EWqBSlgPE8sPvYtDR1gg8='} 2024-04-12T03:39:39.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095df4c6316a522e3-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.675 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.675 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.676 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.676 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.676 DEBUG:boto:Data: 2024-04-12T03:39:39.676 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.676 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.676 DEBUG:boto:Params: {} 2024-04-12T03:39:39.676 DEBUG:boto:Token: None 2024-04-12T03:39:39.676 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000261f474d8687ef4d-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.680 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:39.680 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:39.680 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.680 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:39.680 DEBUG:boto:Data: 2024-04-12T03:39:39.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.680 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.681 DEBUG:boto:Params: {} 2024-04-12T03:39:39.681 DEBUG:boto:Token: None 2024-04-12T03:39:39.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:39.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i8KAN0nr3PMKl7kRZ7OJI7RMoNk= 2024-04-12T03:39:39.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i8KAN0nr3PMKl7kRZ7OJI7RMoNk='} 2024-04-12T03:39:39.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007853746f7cb67a65-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.723 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.723 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.723 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.723 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.723 DEBUG:boto:Data: 2024-04-12T03:39:39.723 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.723 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.723 DEBUG:boto:Params: {} 2024-04-12T03:39:39.723 DEBUG:boto:Token: None 2024-04-12T03:39:39.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000412b08275b9d99b6-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.727 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:39.727 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:39.728 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.728 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:39.728 DEBUG:boto:Data: 2024-04-12T03:39:39.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.728 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.728 DEBUG:boto:Params: {} 2024-04-12T03:39:39.728 DEBUG:boto:Token: None 2024-04-12T03:39:39.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:39.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IFuHKWczG7m9GEu9hacG93AkpWc= 2024-04-12T03:39:39.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IFuHKWczG7m9GEu9hacG93AkpWc='} 2024-04-12T03:39:39.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e071ce4b205c04-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.734 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.735 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.735 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.735 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.735 DEBUG:boto:Data: 2024-04-12T03:39:39.735 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.735 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.735 DEBUG:boto:Params: {} 2024-04-12T03:39:39.735 DEBUG:boto:Token: None 2024-04-12T03:39:39.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c404fec337e957f-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.738 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.738 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.738 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.738 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.738 DEBUG:boto:Data: 2024-04-12T03:39:39.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.739 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.739 DEBUG:boto:Params: {} 2024-04-12T03:39:39.739 DEBUG:boto:Token: None 2024-04-12T03:39:39.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:39.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCpXRFlUH2y/T9BFhHhQRcRP5Pc= 2024-04-12T03:39:39.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCpXRFlUH2y/T9BFhHhQRcRP5Pc='} 2024-04-12T03:39:39.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eeff6a87495be8f4-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.787 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.787 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.787 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.787 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.787 DEBUG:boto:Data: 2024-04-12T03:39:39.787 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.787 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.788 DEBUG:boto:Params: {} 2024-04-12T03:39:39.788 DEBUG:boto:Token: None 2024-04-12T03:39:39.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b1a845a6765f5b4-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.791 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:39.791 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:39.791 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.791 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:39.791 DEBUG:boto:Data: 2024-04-12T03:39:39.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.792 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.792 DEBUG:boto:Params: {} 2024-04-12T03:39:39.792 DEBUG:boto:Token: None 2024-04-12T03:39:39.792 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:39.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jv40/u8/5tY0Rjk6YQ8KPabJjXE= 2024-04-12T03:39:39.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jv40/u8/5tY0Rjk6YQ8KPabJjXE='} 2024-04-12T03:39:39.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000801e91dead7d54c6-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.797 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.797 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.798 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.798 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.798 DEBUG:boto:Data: 2024-04-12T03:39:39.798 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.798 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.798 DEBUG:boto:Params: {} 2024-04-12T03:39:39.798 DEBUG:boto:Token: None 2024-04-12T03:39:39.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b41bab6638ba973-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.801 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:39.801 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:39.801 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.801 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:39.801 DEBUG:boto:Data: 2024-04-12T03:39:39.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.802 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.802 DEBUG:boto:Params: {} 2024-04-12T03:39:39.802 DEBUG:boto:Token: None 2024-04-12T03:39:39.802 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:39.802 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AYOlv/xSZkxCTb8i0aJtvWI2Fgg= 2024-04-12T03:39:39.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AYOlv/xSZkxCTb8i0aJtvWI2Fgg='} 2024-04-12T03:39:39.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fd2fa55e9182a37-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.847 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.847 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.847 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.847 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.847 DEBUG:boto:Data: 2024-04-12T03:39:39.847 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.848 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.848 DEBUG:boto:Params: {} 2024-04-12T03:39:39.848 DEBUG:boto:Token: None 2024-04-12T03:39:39.848 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aa3314a75661e7a-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.851 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.851 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.851 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.852 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.852 DEBUG:boto:Data: 2024-04-12T03:39:39.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.852 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.852 DEBUG:boto:Params: {} 2024-04-12T03:39:39.852 DEBUG:boto:Token: None 2024-04-12T03:39:39.852 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:39.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+ASeYCaZNH7Zz5sw+HecPSr0RN8= 2024-04-12T03:39:39.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+ASeYCaZNH7Zz5sw+HecPSr0RN8='} 2024-04-12T03:39:39.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006707c49341d7bd6a-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.858 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.859 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.859 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.859 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.859 DEBUG:boto:Data: 2024-04-12T03:39:39.859 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.859 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.859 DEBUG:boto:Params: {} 2024-04-12T03:39:39.859 DEBUG:boto:Token: None 2024-04-12T03:39:39.860 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b094abcb9ee58da3-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.862 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.863 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.863 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.863 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.863 DEBUG:boto:Data: 2024-04-12T03:39:39.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.863 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.863 DEBUG:boto:Params: {} 2024-04-12T03:39:39.863 DEBUG:boto:Token: None 2024-04-12T03:39:39.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:39.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VyicmfNeIsGwLi+s1aa34bl+jQo= 2024-04-12T03:39:39.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VyicmfNeIsGwLi+s1aa34bl+jQo='} 2024-04-12T03:39:39.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc57808bb8159ad9-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.907 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.907 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.907 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.907 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.907 DEBUG:boto:Data: 2024-04-12T03:39:39.907 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.907 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.907 DEBUG:boto:Params: {} 2024-04-12T03:39:39.907 DEBUG:boto:Token: None 2024-04-12T03:39:39.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dfa792996694d9b-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.910 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:39.911 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:39.911 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.911 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:39.911 DEBUG:boto:Data: 2024-04-12T03:39:39.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.911 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.911 DEBUG:boto:Params: {} 2024-04-12T03:39:39.911 DEBUG:boto:Token: None 2024-04-12T03:39:39.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:39.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9ohEsup2HxIfim5T37rKLulJ0ws= 2024-04-12T03:39:39.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9ohEsup2HxIfim5T37rKLulJ0ws='} 2024-04-12T03:39:39.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da8b145857b8750-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.917 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.917 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.917 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.917 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.917 DEBUG:boto:Data: 2024-04-12T03:39:39.918 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.918 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.918 DEBUG:boto:Params: {} 2024-04-12T03:39:39.918 DEBUG:boto:Token: None 2024-04-12T03:39:39.918 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.918 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44071bf74d89586-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.921 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:39.921 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:39.921 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.921 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:39.921 DEBUG:boto:Data: 2024-04-12T03:39:39.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.922 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.922 DEBUG:boto:Params: {} 2024-04-12T03:39:39.922 DEBUG:boto:Token: None 2024-04-12T03:39:39.922 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:39.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:txFBXFQZ8pAFOrlnK8xob6ukimQ= 2024-04-12T03:39:39.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:txFBXFQZ8pAFOrlnK8xob6ukimQ='} 2024-04-12T03:39:39.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de557da50405353d-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.967 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.967 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.967 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.967 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.967 DEBUG:boto:Data: 2024-04-12T03:39:39.967 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.967 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.967 DEBUG:boto:Params: {} 2024-04-12T03:39:39.967 DEBUG:boto:Token: None 2024-04-12T03:39:39.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbd24139222ab917-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.971 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:39.971 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:39.971 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.971 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:39.971 DEBUG:boto:Data: 2024-04-12T03:39:39.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.971 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.971 DEBUG:boto:Params: {} 2024-04-12T03:39:39.972 DEBUG:boto:Token: None 2024-04-12T03:39:39.972 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:39.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V/2Xb43/mlpz4P0+oPIJLqhU7+w= 2024-04-12T03:39:39.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V/2Xb43/mlpz4P0+oPIJLqhU7+w='} 2024-04-12T03:39:39.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016d48c105d1d4756-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.977 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:39.977 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:39.977 DEBUG:boto:Method: HEAD 2024-04-12T03:39:39.977 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:39.977 DEBUG:boto:Data: 2024-04-12T03:39:39.977 DEBUG:boto:Headers: {} 2024-04-12T03:39:39.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.978 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.978 DEBUG:boto:Params: {} 2024-04-12T03:39:39.978 DEBUG:boto:Token: None 2024-04-12T03:39:39.978 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/ 2024-04-12T03:39:39.978 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M= 2024-04-12T03:39:39.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jzywdt0fbmUYJ6+2AB9oaJsEo7M=', 'Content-Length': '0'} 2024-04-12T03:39:39.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e7d4136a894c24b-006618acfb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:39.981 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:39.981 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:39.981 DEBUG:boto:Method: PUT 2024-04-12T03:39:39.981 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:39.981 DEBUG:boto:Data: 2024-04-12T03:39:39.981 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:39.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:39.982 DEBUG:boto:Port: 8000 2024-04-12T03:39:39.982 DEBUG:boto:Params: {} 2024-04-12T03:39:39.982 DEBUG:boto:Token: None 2024-04-12T03:39:39.982 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:39 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:39.982 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yo0kYr5AADL34hXzOUG3+ZlxKMY= 2024-04-12T03:39:39.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yo0kYr5AADL34hXzOUG3+ZlxKMY='} 2024-04-12T03:39:40.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e907034d72bb5e10-006618acfb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.027 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.027 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.027 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.027 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.027 DEBUG:boto:Data: 2024-04-12T03:39:40.027 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.027 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.027 DEBUG:boto:Params: {} 2024-04-12T03:39:40.027 DEBUG:boto:Token: None 2024-04-12T03:39:40.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b1846cfa140f382-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.031 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:40.031 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:40.031 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.031 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:40.031 DEBUG:boto:Data: 2024-04-12T03:39:40.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.031 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.031 DEBUG:boto:Params: {} 2024-04-12T03:39:40.031 DEBUG:boto:Token: None 2024-04-12T03:39:40.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:40.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gpNlanXSa3cD8iDbxoiYrXiUu9w= 2024-04-12T03:39:40.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gpNlanXSa3cD8iDbxoiYrXiUu9w='} 2024-04-12T03:39:40.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c95fe2bc46dc4c1-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:40.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:40.038 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:40.136 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:40.136 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:40.143 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 27 2024-04-12T03:39:40.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:40.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:40.191 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:40.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:40.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:40.282 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:40.282 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:40.282 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000058", 2024-04-12T03:39:40.282 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2024-04-12T03:39:40.282 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000046", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000050", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000032", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000039", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000040", 2024-04-12T03:39:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000045", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000055", 2024-04-12T03:39:40.284 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008" 2024-04-12T03:39:40.285 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:40.285 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:40.286 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.286 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.286 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.286 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.286 DEBUG:boto:Data: 2024-04-12T03:39:40.286 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.286 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.286 DEBUG:boto:Params: {} 2024-04-12T03:39:40.287 DEBUG:boto:Token: None 2024-04-12T03:39:40.287 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.287 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d58ae427cce9d717-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.290 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:40.291 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:40.291 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.291 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:40.291 DEBUG:boto:Data: 2024-04-12T03:39:40.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.291 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.291 DEBUG:boto:Params: {} 2024-04-12T03:39:40.291 DEBUG:boto:Token: None 2024-04-12T03:39:40.291 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:40.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GW8py74Wjjl8C6QIuouqGZKYPbc= 2024-04-12T03:39:40.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GW8py74Wjjl8C6QIuouqGZKYPbc='} 2024-04-12T03:39:40.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004023e8226a2bc836-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.297 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.298 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.298 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.298 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.298 DEBUG:boto:Data: 2024-04-12T03:39:40.298 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.298 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.298 DEBUG:boto:Params: {} 2024-04-12T03:39:40.298 DEBUG:boto:Token: None 2024-04-12T03:39:40.298 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065de31ea25c6dc5f-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.302 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:40.302 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:40.302 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.302 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:40.302 DEBUG:boto:Data: 2024-04-12T03:39:40.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.302 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.302 DEBUG:boto:Params: {} 2024-04-12T03:39:40.303 DEBUG:boto:Token: None 2024-04-12T03:39:40.303 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:40.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tBxeEEOeVtIqEfRJT/B5X03ykoo= 2024-04-12T03:39:40.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tBxeEEOeVtIqEfRJT/B5X03ykoo='} 2024-04-12T03:39:40.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bbfdc56b3369443-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.347 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.347 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.347 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.347 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.347 DEBUG:boto:Data: 2024-04-12T03:39:40.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.347 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.347 DEBUG:boto:Params: {} 2024-04-12T03:39:40.348 DEBUG:boto:Token: None 2024-04-12T03:39:40.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcdfab1f0d3b6488-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.351 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:40.351 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:40.351 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.351 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:40.351 DEBUG:boto:Data: 2024-04-12T03:39:40.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.352 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.352 DEBUG:boto:Params: {} 2024-04-12T03:39:40.352 DEBUG:boto:Token: None 2024-04-12T03:39:40.352 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:40.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MYMT4+6slkoyZIeqv/Xg9JcgK+U= 2024-04-12T03:39:40.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MYMT4+6slkoyZIeqv/Xg9JcgK+U='} 2024-04-12T03:39:40.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3c4d0196d649d94-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.358 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.358 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.358 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.358 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.358 DEBUG:boto:Data: 2024-04-12T03:39:40.359 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.359 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.359 DEBUG:boto:Params: {} 2024-04-12T03:39:40.359 DEBUG:boto:Token: None 2024-04-12T03:39:40.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae43e1ec1c3dd228-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.362 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:40.362 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:40.363 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.363 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:40.363 DEBUG:boto:Data: 2024-04-12T03:39:40.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.363 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.363 DEBUG:boto:Params: {} 2024-04-12T03:39:40.363 DEBUG:boto:Token: None 2024-04-12T03:39:40.363 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:40.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v8G6eHSygmaKaKktabKsE4bmR1M= 2024-04-12T03:39:40.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v8G6eHSygmaKaKktabKsE4bmR1M='} 2024-04-12T03:39:40.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7e478201186e9a8-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.407 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.407 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.407 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.407 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.407 DEBUG:boto:Data: 2024-04-12T03:39:40.407 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.407 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.407 DEBUG:boto:Params: {} 2024-04-12T03:39:40.408 DEBUG:boto:Token: None 2024-04-12T03:39:40.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca92c3d8a8a7c65b-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.411 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:40.411 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:40.411 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.411 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:40.412 DEBUG:boto:Data: 2024-04-12T03:39:40.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.412 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.412 DEBUG:boto:Params: {} 2024-04-12T03:39:40.412 DEBUG:boto:Token: None 2024-04-12T03:39:40.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:40.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cckGjk6+sKgRt9aMZ1yNlN1tHpE= 2024-04-12T03:39:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cckGjk6+sKgRt9aMZ1yNlN1tHpE='} 2024-04-12T03:39:40.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c45da1edacf0b10f-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.417 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.417 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.417 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.418 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.418 DEBUG:boto:Data: 2024-04-12T03:39:40.418 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.418 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.418 DEBUG:boto:Params: {} 2024-04-12T03:39:40.418 DEBUG:boto:Token: None 2024-04-12T03:39:40.418 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b5ff9b6e6892e67-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.421 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:40.422 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:40.422 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.422 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:40.422 DEBUG:boto:Data: 2024-04-12T03:39:40.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.422 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.422 DEBUG:boto:Params: {} 2024-04-12T03:39:40.422 DEBUG:boto:Token: None 2024-04-12T03:39:40.422 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:40.423 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IcZcZ8yHsU1nUB4wWheYheizzIk= 2024-04-12T03:39:40.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IcZcZ8yHsU1nUB4wWheYheizzIk='} 2024-04-12T03:39:40.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3e6c648182eee88-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.467 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.467 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.467 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.467 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.467 DEBUG:boto:Data: 2024-04-12T03:39:40.467 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.467 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.467 DEBUG:boto:Params: {} 2024-04-12T03:39:40.467 DEBUG:boto:Token: None 2024-04-12T03:39:40.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cb7b82f305b9947-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.471 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:40.471 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:40.471 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:40.471 DEBUG:boto:Data: 2024-04-12T03:39:40.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.471 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.471 DEBUG:boto:Params: {} 2024-04-12T03:39:40.472 DEBUG:boto:Token: None 2024-04-12T03:39:40.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:40.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9okMCYrOO2jWE7dloomUiiDtyyw= 2024-04-12T03:39:40.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9okMCYrOO2jWE7dloomUiiDtyyw='} 2024-04-12T03:39:40.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb69a3715b2fb472-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.477 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.477 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.477 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.477 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.477 DEBUG:boto:Data: 2024-04-12T03:39:40.478 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.478 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.478 DEBUG:boto:Params: {} 2024-04-12T03:39:40.478 DEBUG:boto:Token: None 2024-04-12T03:39:40.478 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.478 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008844f95daf8d44d8-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.481 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:40.482 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:40.482 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.482 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:40.482 DEBUG:boto:Data: 2024-04-12T03:39:40.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.482 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.482 DEBUG:boto:Params: {} 2024-04-12T03:39:40.482 DEBUG:boto:Token: None 2024-04-12T03:39:40.483 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:40.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x+c6JOe/rY/7/UA/TrreH3BuCHo= 2024-04-12T03:39:40.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x+c6JOe/rY/7/UA/TrreH3BuCHo='} 2024-04-12T03:39:40.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb5c73c757fc51b9-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.526 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.527 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.527 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.527 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.527 DEBUG:boto:Data: 2024-04-12T03:39:40.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.527 DEBUG:boto:Params: {} 2024-04-12T03:39:40.527 DEBUG:boto:Token: None 2024-04-12T03:39:40.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecdbe662fe7906b5-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.531 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:40.531 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:40.531 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.531 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:40.531 DEBUG:boto:Data: 2024-04-12T03:39:40.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.532 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.532 DEBUG:boto:Params: {} 2024-04-12T03:39:40.532 DEBUG:boto:Token: None 2024-04-12T03:39:40.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:40.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HMeLnhUcZkhCe1hcF8uMA/Plbtg= 2024-04-12T03:39:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HMeLnhUcZkhCe1hcF8uMA/Plbtg='} 2024-04-12T03:39:40.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a41189b06691c71-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.539 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.539 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.539 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.539 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.539 DEBUG:boto:Data: 2024-04-12T03:39:40.539 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.539 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.539 DEBUG:boto:Params: {} 2024-04-12T03:39:40.539 DEBUG:boto:Token: None 2024-04-12T03:39:40.540 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cac8d154612e206b-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.543 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:40.543 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:40.543 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.543 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:40.543 DEBUG:boto:Data: 2024-04-12T03:39:40.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.543 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.543 DEBUG:boto:Params: {} 2024-04-12T03:39:40.543 DEBUG:boto:Token: None 2024-04-12T03:39:40.544 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:40.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IujgYfQFRls6zOKnng5tDfvC4G4= 2024-04-12T03:39:40.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IujgYfQFRls6zOKnng5tDfvC4G4='} 2024-04-12T03:39:40.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039b55d14fd0f658f-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.586 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.587 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.587 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.587 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.587 DEBUG:boto:Data: 2024-04-12T03:39:40.587 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.587 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.587 DEBUG:boto:Params: {} 2024-04-12T03:39:40.587 DEBUG:boto:Token: None 2024-04-12T03:39:40.588 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009af0425e3b6bf549-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.591 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:40.591 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:40.591 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.591 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:40.591 DEBUG:boto:Data: 2024-04-12T03:39:40.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.591 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.591 DEBUG:boto:Params: {} 2024-04-12T03:39:40.591 DEBUG:boto:Token: None 2024-04-12T03:39:40.592 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:40.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wo6u3dlYuuQtU1Cx4dRP6K6V1yg= 2024-04-12T03:39:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wo6u3dlYuuQtU1Cx4dRP6K6V1yg='} 2024-04-12T03:39:40.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000123ddec96ee3d71b-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.597 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.597 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.597 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.597 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.597 DEBUG:boto:Data: 2024-04-12T03:39:40.597 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.597 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.597 DEBUG:boto:Params: {} 2024-04-12T03:39:40.597 DEBUG:boto:Token: None 2024-04-12T03:39:40.598 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa1ebbfbe7170688-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.601 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:40.601 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:40.601 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.601 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:40.601 DEBUG:boto:Data: 2024-04-12T03:39:40.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.601 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.602 DEBUG:boto:Params: {} 2024-04-12T03:39:40.602 DEBUG:boto:Token: None 2024-04-12T03:39:40.602 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:40.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:raCY26xi0k8bS5wXczzU8QR1/xY= 2024-04-12T03:39:40.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:raCY26xi0k8bS5wXczzU8QR1/xY='} 2024-04-12T03:39:40.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e619adeda779455-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.647 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.647 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.647 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.647 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.647 DEBUG:boto:Data: 2024-04-12T03:39:40.648 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.648 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.648 DEBUG:boto:Params: {} 2024-04-12T03:39:40.648 DEBUG:boto:Token: None 2024-04-12T03:39:40.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096610e24a24d29d7-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.653 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:40.654 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:40.654 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.654 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:40.654 DEBUG:boto:Data: 2024-04-12T03:39:40.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.654 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.654 DEBUG:boto:Params: {} 2024-04-12T03:39:40.654 DEBUG:boto:Token: None 2024-04-12T03:39:40.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:40.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cnjZUrgOYABHcPkSrI6vz9lw7V4= 2024-04-12T03:39:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cnjZUrgOYABHcPkSrI6vz9lw7V4='} 2024-04-12T03:39:40.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d1401c504b0a102-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.659 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.659 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.660 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.660 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.660 DEBUG:boto:Data: 2024-04-12T03:39:40.660 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.660 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.660 DEBUG:boto:Params: {} 2024-04-12T03:39:40.660 DEBUG:boto:Token: None 2024-04-12T03:39:40.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048adda78f90e631f-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.663 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:40.663 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:40.663 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.664 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:40.664 DEBUG:boto:Data: 2024-04-12T03:39:40.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.664 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.664 DEBUG:boto:Params: {} 2024-04-12T03:39:40.664 DEBUG:boto:Token: None 2024-04-12T03:39:40.664 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:40.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s83aXMabUd3Q+CANsCnNISitpcA= 2024-04-12T03:39:40.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s83aXMabUd3Q+CANsCnNISitpcA='} 2024-04-12T03:39:40.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b75d47cdbcd199a-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.707 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.707 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.707 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.707 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.707 DEBUG:boto:Data: 2024-04-12T03:39:40.707 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.707 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.707 DEBUG:boto:Params: {} 2024-04-12T03:39:40.707 DEBUG:boto:Token: None 2024-04-12T03:39:40.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cc7495a2c25975d-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.711 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:40.711 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:40.711 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.711 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:40.711 DEBUG:boto:Data: 2024-04-12T03:39:40.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.712 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.712 DEBUG:boto:Params: {} 2024-04-12T03:39:40.712 DEBUG:boto:Token: None 2024-04-12T03:39:40.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:40.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qCkst/oaDEkAgQPoTSeUsNbzYqo= 2024-04-12T03:39:40.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qCkst/oaDEkAgQPoTSeUsNbzYqo='} 2024-04-12T03:39:40.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000741a641110d75f3f-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.718 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.719 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.719 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.719 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.719 DEBUG:boto:Data: 2024-04-12T03:39:40.719 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.719 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.719 DEBUG:boto:Params: {} 2024-04-12T03:39:40.719 DEBUG:boto:Token: None 2024-04-12T03:39:40.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da1b255eeffbdaab-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.722 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:40.723 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:40.723 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.723 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:40.723 DEBUG:boto:Data: 2024-04-12T03:39:40.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.723 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.723 DEBUG:boto:Params: {} 2024-04-12T03:39:40.723 DEBUG:boto:Token: None 2024-04-12T03:39:40.723 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:40.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SbfTkwnM1YoqkTZMWwv+e2YFSpE= 2024-04-12T03:39:40.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SbfTkwnM1YoqkTZMWwv+e2YFSpE='} 2024-04-12T03:39:40.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61398dfe8e83cd3-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.770 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.771 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.771 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.771 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.771 DEBUG:boto:Data: 2024-04-12T03:39:40.771 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.771 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.771 DEBUG:boto:Params: {} 2024-04-12T03:39:40.771 DEBUG:boto:Token: None 2024-04-12T03:39:40.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5dbcd698ba924b9-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.774 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:40.774 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:40.775 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.775 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:40.775 DEBUG:boto:Data: 2024-04-12T03:39:40.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.775 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.775 DEBUG:boto:Params: {} 2024-04-12T03:39:40.775 DEBUG:boto:Token: None 2024-04-12T03:39:40.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:40.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RPxgLBi92uY21qzaS+qsB7WMvCc= 2024-04-12T03:39:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RPxgLBi92uY21qzaS+qsB7WMvCc='} 2024-04-12T03:39:40.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072eb223e8a7f9faf-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.780 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.780 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.781 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.781 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.781 DEBUG:boto:Data: 2024-04-12T03:39:40.781 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.781 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.781 DEBUG:boto:Params: {} 2024-04-12T03:39:40.781 DEBUG:boto:Token: None 2024-04-12T03:39:40.781 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fff0d68eb96d0b00-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.784 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:40.785 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:40.785 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.785 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:40.785 DEBUG:boto:Data: 2024-04-12T03:39:40.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.785 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.785 DEBUG:boto:Params: {} 2024-04-12T03:39:40.785 DEBUG:boto:Token: None 2024-04-12T03:39:40.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:40.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qk3Uo87j0PHgHapb1dCxwsxE24c= 2024-04-12T03:39:40.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qk3Uo87j0PHgHapb1dCxwsxE24c='} 2024-04-12T03:39:40.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f236aa807d10d53a-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.831 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.831 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.831 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.831 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.831 DEBUG:boto:Data: 2024-04-12T03:39:40.831 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.831 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.831 DEBUG:boto:Params: {} 2024-04-12T03:39:40.831 DEBUG:boto:Token: None 2024-04-12T03:39:40.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090d15b9c7e67f0d3-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.835 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:40.835 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:40.835 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.835 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:40.835 DEBUG:boto:Data: 2024-04-12T03:39:40.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.835 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.835 DEBUG:boto:Params: {} 2024-04-12T03:39:40.836 DEBUG:boto:Token: None 2024-04-12T03:39:40.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:40.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pjPpMzY9y/yT+q3l5nt9bqaVfEI= 2024-04-12T03:39:40.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pjPpMzY9y/yT+q3l5nt9bqaVfEI='} 2024-04-12T03:39:40.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be11dde395b8a00f-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.841 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.841 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.841 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.841 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.841 DEBUG:boto:Data: 2024-04-12T03:39:40.841 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.842 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.842 DEBUG:boto:Params: {} 2024-04-12T03:39:40.842 DEBUG:boto:Token: None 2024-04-12T03:39:40.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.842 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000737a5ee9fff2ec55-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.845 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:40.845 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:40.845 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.846 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:40.846 DEBUG:boto:Data: 2024-04-12T03:39:40.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.846 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.846 DEBUG:boto:Params: {} 2024-04-12T03:39:40.846 DEBUG:boto:Token: None 2024-04-12T03:39:40.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:40.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3bD2iuJUmovppjEpDPrjqYdqHCc= 2024-04-12T03:39:40.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3bD2iuJUmovppjEpDPrjqYdqHCc='} 2024-04-12T03:39:40.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b39dec30bb861e98-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.891 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:40.891 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:40.891 DEBUG:boto:Method: HEAD 2024-04-12T03:39:40.891 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:40.891 DEBUG:boto:Data: 2024-04-12T03:39:40.891 DEBUG:boto:Headers: {} 2024-04-12T03:39:40.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.891 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.891 DEBUG:boto:Params: {} 2024-04-12T03:39:40.891 DEBUG:boto:Token: None 2024-04-12T03:39:40.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/ 2024-04-12T03:39:40.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4= 2024-04-12T03:39:40.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NwNWyKnsH1jN+hJEMu9UEeNNk4=', 'Content-Length': '0'} 2024-04-12T03:39:40.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c9513f60c500cab-006618acfc-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.895 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:40.895 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:40.895 DEBUG:boto:Method: PUT 2024-04-12T03:39:40.895 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:40.895 DEBUG:boto:Data: 2024-04-12T03:39:40.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:40.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:40.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:40.896 DEBUG:boto:Params: {} 2024-04-12T03:39:40.896 DEBUG:boto:Token: None 2024-04-12T03:39:40.896 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:40 GMT /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:40.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jCmZzJgWXzfMaXtW73otppVp5go= 2024-04-12T03:39:40.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jCmZzJgWXzfMaXtW73otppVp5go='} 2024-04-12T03:39:40.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd9e42448babeac2-006618acfc-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:40.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:40.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:40.902 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:40.999 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:40.999 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:41.011 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 35 2024-04-12T03:39:41.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:41.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:41.065 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000059", 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000027", 2024-04-12T03:39:41.154 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000049", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000031", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000022", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2024-04-12T03:39:41.155 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000037", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000042", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2024-04-12T03:39:41.156 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000025", 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021" 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:41.157 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:41.158 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.158 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.158 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.158 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.159 DEBUG:boto:Data: 2024-04-12T03:39:41.159 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.159 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.159 DEBUG:boto:Params: {} 2024-04-12T03:39:41.159 DEBUG:boto:Token: None 2024-04-12T03:39:41.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df75b39e52f37d04-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.163 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:41.163 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:41.163 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.163 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:41.163 DEBUG:boto:Data: 2024-04-12T03:39:41.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.163 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.164 DEBUG:boto:Params: {} 2024-04-12T03:39:41.164 DEBUG:boto:Token: None 2024-04-12T03:39:41.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:41.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:w95y6OHGG7gSk/8T8Lx+1JJfUfI= 2024-04-12T03:39:41.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w95y6OHGG7gSk/8T8Lx+1JJfUfI='} 2024-04-12T03:39:41.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a12efcc2a2ef4a3e-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.207 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.207 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.207 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.207 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.207 DEBUG:boto:Data: 2024-04-12T03:39:41.207 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.207 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.207 DEBUG:boto:Params: {} 2024-04-12T03:39:41.208 DEBUG:boto:Token: None 2024-04-12T03:39:41.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3fe838504e27d62-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.211 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:41.211 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:41.211 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.211 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:41.211 DEBUG:boto:Data: 2024-04-12T03:39:41.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.212 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.212 DEBUG:boto:Params: {} 2024-04-12T03:39:41.212 DEBUG:boto:Token: None 2024-04-12T03:39:41.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:41.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:myKDne0M4VF1Arv2kbqA0LADi/o= 2024-04-12T03:39:41.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:myKDne0M4VF1Arv2kbqA0LADi/o='} 2024-04-12T03:39:41.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008aaaa0256547d51e-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.217 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.217 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.217 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.218 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.218 DEBUG:boto:Data: 2024-04-12T03:39:41.218 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.218 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.218 DEBUG:boto:Params: {} 2024-04-12T03:39:41.218 DEBUG:boto:Token: None 2024-04-12T03:39:41.218 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c72d6f2dd32d6d9-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.222 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:41.222 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:41.222 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.222 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:41.222 DEBUG:boto:Data: 2024-04-12T03:39:41.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.222 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.222 DEBUG:boto:Params: {} 2024-04-12T03:39:41.223 DEBUG:boto:Token: None 2024-04-12T03:39:41.223 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:41.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:auTWSKBY5llnxkCaFOcCCLcOS3A= 2024-04-12T03:39:41.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:auTWSKBY5llnxkCaFOcCCLcOS3A='} 2024-04-12T03:39:41.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0e4b692eb8d6640-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.267 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.267 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.267 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.267 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.267 DEBUG:boto:Data: 2024-04-12T03:39:41.267 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.267 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.267 DEBUG:boto:Params: {} 2024-04-12T03:39:41.267 DEBUG:boto:Token: None 2024-04-12T03:39:41.267 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f421e585ea67dfb-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.271 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:41.271 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:41.271 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.271 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:41.271 DEBUG:boto:Data: 2024-04-12T03:39:41.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.271 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.271 DEBUG:boto:Params: {} 2024-04-12T03:39:41.271 DEBUG:boto:Token: None 2024-04-12T03:39:41.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:41.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:se7EAYoqAeB0SZ7TqZ5i0W9a944= 2024-04-12T03:39:41.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:se7EAYoqAeB0SZ7TqZ5i0W9a944='} 2024-04-12T03:39:41.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b41236f15f86577-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.278 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.278 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.278 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.278 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.278 DEBUG:boto:Data: 2024-04-12T03:39:41.279 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.279 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.279 DEBUG:boto:Params: {} 2024-04-12T03:39:41.279 DEBUG:boto:Token: None 2024-04-12T03:39:41.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce68ab2739d9d27b-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.282 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:41.283 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:41.283 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.283 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:41.283 DEBUG:boto:Data: 2024-04-12T03:39:41.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.283 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.283 DEBUG:boto:Params: {} 2024-04-12T03:39:41.283 DEBUG:boto:Token: None 2024-04-12T03:39:41.283 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:41.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7eX+mXDNwsVGUkt2oYR/1eXxxAw= 2024-04-12T03:39:41.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7eX+mXDNwsVGUkt2oYR/1eXxxAw='} 2024-04-12T03:39:41.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000021b3a980adafbbf-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.327 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.327 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.327 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.327 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.327 DEBUG:boto:Data: 2024-04-12T03:39:41.327 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.328 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.328 DEBUG:boto:Params: {} 2024-04-12T03:39:41.328 DEBUG:boto:Token: None 2024-04-12T03:39:41.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f584a7e9da6a2a4-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:41.332 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:41.332 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.332 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:41.332 DEBUG:boto:Data: 2024-04-12T03:39:41.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.332 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.332 DEBUG:boto:Params: {} 2024-04-12T03:39:41.332 DEBUG:boto:Token: None 2024-04-12T03:39:41.333 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:41.333 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+OhoLaNUaHQrw7f6QdfH1Q/Mrd0= 2024-04-12T03:39:41.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+OhoLaNUaHQrw7f6QdfH1Q/Mrd0='} 2024-04-12T03:39:41.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad5193efd84b5258-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.339 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.339 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.340 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.340 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.340 DEBUG:boto:Data: 2024-04-12T03:39:41.340 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.340 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.340 DEBUG:boto:Params: {} 2024-04-12T03:39:41.341 DEBUG:boto:Token: None 2024-04-12T03:39:41.341 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001befc6abc5fb9b9-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.344 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:41.344 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:41.344 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.344 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:41.345 DEBUG:boto:Data: 2024-04-12T03:39:41.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.345 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.345 DEBUG:boto:Params: {} 2024-04-12T03:39:41.345 DEBUG:boto:Token: None 2024-04-12T03:39:41.345 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:41.345 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:g1Omae6zkkAk+RTSdIraRl0hSxI= 2024-04-12T03:39:41.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:g1Omae6zkkAk+RTSdIraRl0hSxI='} 2024-04-12T03:39:41.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c54bcbb6d51a2d-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.391 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.391 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.391 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.391 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.391 DEBUG:boto:Data: 2024-04-12T03:39:41.391 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.391 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.391 DEBUG:boto:Params: {} 2024-04-12T03:39:41.392 DEBUG:boto:Token: None 2024-04-12T03:39:41.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d763972e392c3b3-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.396 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:41.396 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:41.396 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.396 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:41.396 DEBUG:boto:Data: 2024-04-12T03:39:41.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.396 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.396 DEBUG:boto:Params: {} 2024-04-12T03:39:41.397 DEBUG:boto:Token: None 2024-04-12T03:39:41.397 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:41.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QSMfJmlIirwk95MfRwQpe8ozXXc= 2024-04-12T03:39:41.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QSMfJmlIirwk95MfRwQpe8ozXXc='} 2024-04-12T03:39:41.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee847ad0bd27f88f-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.403 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.403 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.403 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.403 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.403 DEBUG:boto:Data: 2024-04-12T03:39:41.403 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.403 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.403 DEBUG:boto:Params: {} 2024-04-12T03:39:41.404 DEBUG:boto:Token: None 2024-04-12T03:39:41.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad7f610dd57ef733-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.407 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:41.407 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:41.407 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.407 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:41.407 DEBUG:boto:Data: 2024-04-12T03:39:41.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.408 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.408 DEBUG:boto:Params: {} 2024-04-12T03:39:41.408 DEBUG:boto:Token: None 2024-04-12T03:39:41.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:41.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJjgtfL7p7ZadVMvd7gR+OA3jkA= 2024-04-12T03:39:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJjgtfL7p7ZadVMvd7gR+OA3jkA='} 2024-04-12T03:39:41.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000349317bdf7abd025-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.450 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.451 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.451 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.451 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.451 DEBUG:boto:Data: 2024-04-12T03:39:41.451 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.451 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.451 DEBUG:boto:Params: {} 2024-04-12T03:39:41.451 DEBUG:boto:Token: None 2024-04-12T03:39:41.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ceaf2a29613eb6e-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:41.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:41.455 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.455 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:41.455 DEBUG:boto:Data: 2024-04-12T03:39:41.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.456 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.456 DEBUG:boto:Params: {} 2024-04-12T03:39:41.456 DEBUG:boto:Token: None 2024-04-12T03:39:41.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:41.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kqaS/5kIKxAg4DTaZ9JUHmm7ZV8= 2024-04-12T03:39:41.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kqaS/5kIKxAg4DTaZ9JUHmm7ZV8='} 2024-04-12T03:39:41.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ba76ccc2b1276a1-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.461 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.461 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.461 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.461 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.462 DEBUG:boto:Data: 2024-04-12T03:39:41.462 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.462 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.462 DEBUG:boto:Params: {} 2024-04-12T03:39:41.462 DEBUG:boto:Token: None 2024-04-12T03:39:41.462 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d96119ad863c7cd-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.465 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:41.465 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:41.465 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.465 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:41.465 DEBUG:boto:Data: 2024-04-12T03:39:41.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.466 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.466 DEBUG:boto:Params: {} 2024-04-12T03:39:41.466 DEBUG:boto:Token: None 2024-04-12T03:39:41.466 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:41.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EkhotPiGeefWIf+HIDMeaXgjUdI= 2024-04-12T03:39:41.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EkhotPiGeefWIf+HIDMeaXgjUdI='} 2024-04-12T03:39:41.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afd2c6052e448dfb-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.511 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.511 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.511 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.511 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.511 DEBUG:boto:Data: 2024-04-12T03:39:41.511 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.511 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.511 DEBUG:boto:Params: {} 2024-04-12T03:39:41.512 DEBUG:boto:Token: None 2024-04-12T03:39:41.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000982663e5a039fa18-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.515 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:41.515 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:41.515 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.515 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:41.515 DEBUG:boto:Data: 2024-04-12T03:39:41.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.516 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.516 DEBUG:boto:Params: {} 2024-04-12T03:39:41.516 DEBUG:boto:Token: None 2024-04-12T03:39:41.516 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:41.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:njEIzrO+EAod6GJ9YM0ZGnlKieI= 2024-04-12T03:39:41.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:njEIzrO+EAod6GJ9YM0ZGnlKieI='} 2024-04-12T03:39:41.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4902ffd3a40024d-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.521 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.521 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.521 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.521 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.521 DEBUG:boto:Data: 2024-04-12T03:39:41.521 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.522 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.522 DEBUG:boto:Params: {} 2024-04-12T03:39:41.522 DEBUG:boto:Token: None 2024-04-12T03:39:41.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016e61a38ddef189d-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.525 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:41.525 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:41.525 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.525 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:41.526 DEBUG:boto:Data: 2024-04-12T03:39:41.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.526 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.526 DEBUG:boto:Params: {} 2024-04-12T03:39:41.526 DEBUG:boto:Token: None 2024-04-12T03:39:41.526 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:41.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EGXZZLXbClzsdiK0jk892wrpJKc= 2024-04-12T03:39:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EGXZZLXbClzsdiK0jk892wrpJKc='} 2024-04-12T03:39:41.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f900d3ad66b7344-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.571 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.571 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.571 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.571 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.571 DEBUG:boto:Data: 2024-04-12T03:39:41.571 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.571 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.572 DEBUG:boto:Params: {} 2024-04-12T03:39:41.572 DEBUG:boto:Token: None 2024-04-12T03:39:41.572 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b7f59761d99eacc-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.575 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:41.575 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:41.575 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.575 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:41.575 DEBUG:boto:Data: 2024-04-12T03:39:41.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.576 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.576 DEBUG:boto:Params: {} 2024-04-12T03:39:41.576 DEBUG:boto:Token: None 2024-04-12T03:39:41.576 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:41.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5oCchSQAoqPYH23NqNUAmItG/x8= 2024-04-12T03:39:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5oCchSQAoqPYH23NqNUAmItG/x8='} 2024-04-12T03:39:41.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025359cf130cf5a20-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.581 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.581 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.581 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.581 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.582 DEBUG:boto:Data: 2024-04-12T03:39:41.582 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.582 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.582 DEBUG:boto:Params: {} 2024-04-12T03:39:41.582 DEBUG:boto:Token: None 2024-04-12T03:39:41.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ca6cbb3c2746d7b-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.585 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:41.585 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:41.585 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.585 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:41.585 DEBUG:boto:Data: 2024-04-12T03:39:41.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.586 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.586 DEBUG:boto:Params: {} 2024-04-12T03:39:41.586 DEBUG:boto:Token: None 2024-04-12T03:39:41.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:41.586 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:87YzocVS1ghsJ68rd/gorM34uAE= 2024-04-12T03:39:41.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:87YzocVS1ghsJ68rd/gorM34uAE='} 2024-04-12T03:39:41.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050822dede32369f9-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.631 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.631 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.631 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.631 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.631 DEBUG:boto:Data: 2024-04-12T03:39:41.631 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.631 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.631 DEBUG:boto:Params: {} 2024-04-12T03:39:41.631 DEBUG:boto:Token: None 2024-04-12T03:39:41.632 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c80703b168ee4379-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.635 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:41.635 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:41.635 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.635 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:41.635 DEBUG:boto:Data: 2024-04-12T03:39:41.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.635 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.635 DEBUG:boto:Params: {} 2024-04-12T03:39:41.636 DEBUG:boto:Token: None 2024-04-12T03:39:41.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:41.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8QvpBm/4MEAqGjhZAi6c+IpLojw= 2024-04-12T03:39:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8QvpBm/4MEAqGjhZAi6c+IpLojw='} 2024-04-12T03:39:41.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079d20180ad7b377d-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.642 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.642 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.642 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.642 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.642 DEBUG:boto:Data: 2024-04-12T03:39:41.642 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.642 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.642 DEBUG:boto:Params: {} 2024-04-12T03:39:41.643 DEBUG:boto:Token: None 2024-04-12T03:39:41.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bc2c63f11c1e61c-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.646 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:41.646 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:41.646 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.646 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:41.646 DEBUG:boto:Data: 2024-04-12T03:39:41.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.647 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.647 DEBUG:boto:Params: {} 2024-04-12T03:39:41.647 DEBUG:boto:Token: None 2024-04-12T03:39:41.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:41.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dNdbrD2wNnSUl7sb3MNAj48xCUY= 2024-04-12T03:39:41.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dNdbrD2wNnSUl7sb3MNAj48xCUY='} 2024-04-12T03:39:41.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018dd0ec0ce7361f1-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.691 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.691 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.691 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.691 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.691 DEBUG:boto:Data: 2024-04-12T03:39:41.691 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.691 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.691 DEBUG:boto:Params: {} 2024-04-12T03:39:41.691 DEBUG:boto:Token: None 2024-04-12T03:39:41.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005129172f7cbf5c13-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.694 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:41.695 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:41.695 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.695 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:41.695 DEBUG:boto:Data: 2024-04-12T03:39:41.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.695 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.695 DEBUG:boto:Params: {} 2024-04-12T03:39:41.695 DEBUG:boto:Token: None 2024-04-12T03:39:41.695 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:41.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dj5cyoXCW0AZq59XI8dw0b3sGjI= 2024-04-12T03:39:41.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dj5cyoXCW0AZq59XI8dw0b3sGjI='} 2024-04-12T03:39:41.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bee886a5ac20aeff-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.702 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.702 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.702 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.702 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.703 DEBUG:boto:Data: 2024-04-12T03:39:41.703 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.703 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.703 DEBUG:boto:Params: {} 2024-04-12T03:39:41.703 DEBUG:boto:Token: None 2024-04-12T03:39:41.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002964a8baf97124cf-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.707 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:41.707 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:41.707 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.707 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:41.707 DEBUG:boto:Data: 2024-04-12T03:39:41.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.707 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.707 DEBUG:boto:Params: {} 2024-04-12T03:39:41.707 DEBUG:boto:Token: None 2024-04-12T03:39:41.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:41.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FHwMpb9RHN/IGRWJR+fMp6g64bw= 2024-04-12T03:39:41.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FHwMpb9RHN/IGRWJR+fMp6g64bw='} 2024-04-12T03:39:41.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4154a0c730f999d-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.751 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.751 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.751 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.751 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.751 DEBUG:boto:Data: 2024-04-12T03:39:41.751 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.752 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.752 DEBUG:boto:Params: {} 2024-04-12T03:39:41.752 DEBUG:boto:Token: None 2024-04-12T03:39:41.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7226b67ccecb139-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.755 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:41.755 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:41.755 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.756 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:41.756 DEBUG:boto:Data: 2024-04-12T03:39:41.756 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.756 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.756 DEBUG:boto:Params: {} 2024-04-12T03:39:41.756 DEBUG:boto:Token: None 2024-04-12T03:39:41.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:41.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rt/uhm0bKmTcd9wpAqvVZ19KleI= 2024-04-12T03:39:41.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rt/uhm0bKmTcd9wpAqvVZ19KleI='} 2024-04-12T03:39:41.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd6cb04fffd0d55-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.762 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.763 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.763 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.763 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.763 DEBUG:boto:Data: 2024-04-12T03:39:41.763 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.763 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.763 DEBUG:boto:Params: {} 2024-04-12T03:39:41.763 DEBUG:boto:Token: None 2024-04-12T03:39:41.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a6aa1c9939e33ac-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.767 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:41.767 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:41.767 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.767 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:41.767 DEBUG:boto:Data: 2024-04-12T03:39:41.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.767 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.768 DEBUG:boto:Params: {} 2024-04-12T03:39:41.768 DEBUG:boto:Token: None 2024-04-12T03:39:41.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:41.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NVBFO7OWvP+IC5RJdpkLNNazUrc= 2024-04-12T03:39:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NVBFO7OWvP+IC5RJdpkLNNazUrc='} 2024-04-12T03:39:41.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037e24331efcabf12-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.811 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.811 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.811 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.811 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.811 DEBUG:boto:Data: 2024-04-12T03:39:41.811 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.811 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.811 DEBUG:boto:Params: {} 2024-04-12T03:39:41.812 DEBUG:boto:Token: None 2024-04-12T03:39:41.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025a86c611225a0f6-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.815 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:41.815 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:41.815 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.815 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:41.815 DEBUG:boto:Data: 2024-04-12T03:39:41.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.816 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.816 DEBUG:boto:Params: {} 2024-04-12T03:39:41.816 DEBUG:boto:Token: None 2024-04-12T03:39:41.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:41.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ufmTZd95gkIjZ29Mziem25i9TnE= 2024-04-12T03:39:41.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ufmTZd95gkIjZ29Mziem25i9TnE='} 2024-04-12T03:39:41.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b66db9d3e1d1fb98-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.821 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.821 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.821 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.821 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.821 DEBUG:boto:Data: 2024-04-12T03:39:41.821 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.822 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.822 DEBUG:boto:Params: {} 2024-04-12T03:39:41.822 DEBUG:boto:Token: None 2024-04-12T03:39:41.822 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa1afffc1cd0072b-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.826 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:41.826 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:41.826 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.826 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:41.826 DEBUG:boto:Data: 2024-04-12T03:39:41.826 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.826 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.826 DEBUG:boto:Params: {} 2024-04-12T03:39:41.827 DEBUG:boto:Token: None 2024-04-12T03:39:41.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:41.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V0Oy+2TrN3hn8njT3g8rm/DjDoQ= 2024-04-12T03:39:41.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V0Oy+2TrN3hn8njT3g8rm/DjDoQ='} 2024-04-12T03:39:41.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003bc6fc6e1ab2440e-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.871 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.871 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.871 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.871 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.871 DEBUG:boto:Data: 2024-04-12T03:39:41.871 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.871 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.871 DEBUG:boto:Params: {} 2024-04-12T03:39:41.871 DEBUG:boto:Token: None 2024-04-12T03:39:41.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e97947440c8a2746-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.875 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:41.875 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:41.875 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.875 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:41.875 DEBUG:boto:Data: 2024-04-12T03:39:41.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.875 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.875 DEBUG:boto:Params: {} 2024-04-12T03:39:41.876 DEBUG:boto:Token: None 2024-04-12T03:39:41.876 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:41.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1x4iee8/Ry/U353Yia1TcAxnAiA= 2024-04-12T03:39:41.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1x4iee8/Ry/U353Yia1TcAxnAiA='} 2024-04-12T03:39:41.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b72420aeed2dc8a1-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.882 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.882 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.882 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.882 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.882 DEBUG:boto:Data: 2024-04-12T03:39:41.882 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.882 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.883 DEBUG:boto:Params: {} 2024-04-12T03:39:41.883 DEBUG:boto:Token: None 2024-04-12T03:39:41.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001629dd4cd458b7a4-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.886 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:41.886 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:41.886 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.886 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:41.886 DEBUG:boto:Data: 2024-04-12T03:39:41.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.887 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.887 DEBUG:boto:Params: {} 2024-04-12T03:39:41.887 DEBUG:boto:Token: None 2024-04-12T03:39:41.887 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:41.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eACsoio4MXrqgPNkEeVs2lWkoCk= 2024-04-12T03:39:41.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eACsoio4MXrqgPNkEeVs2lWkoCk='} 2024-04-12T03:39:41.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003043d28fa85b00c8-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.931 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.931 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.931 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.931 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.931 DEBUG:boto:Data: 2024-04-12T03:39:41.931 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.931 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.931 DEBUG:boto:Params: {} 2024-04-12T03:39:41.931 DEBUG:boto:Token: None 2024-04-12T03:39:41.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000776a45af6662b5f6-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.934 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:41.935 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:41.935 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.935 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:41.935 DEBUG:boto:Data: 2024-04-12T03:39:41.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.935 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.935 DEBUG:boto:Params: {} 2024-04-12T03:39:41.935 DEBUG:boto:Token: None 2024-04-12T03:39:41.935 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:41.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PAco/3ofkg2JWedZQkb0b24m0cw= 2024-04-12T03:39:41.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PAco/3ofkg2JWedZQkb0b24m0cw='} 2024-04-12T03:39:41.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000452e81794bb3a588-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.942 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:41.942 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:41.942 DEBUG:boto:Method: HEAD 2024-04-12T03:39:41.942 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:41.942 DEBUG:boto:Data: 2024-04-12T03:39:41.942 DEBUG:boto:Headers: {} 2024-04-12T03:39:41.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.942 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.942 DEBUG:boto:Params: {} 2024-04-12T03:39:41.942 DEBUG:boto:Token: None 2024-04-12T03:39:41.943 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/ 2024-04-12T03:39:41.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4= 2024-04-12T03:39:41.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGChGMODjK6wkj1lPQ4qTFprU+4=', 'Content-Length': '0'} 2024-04-12T03:39:41.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdf355c80d3f878d-006618acfd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.946 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:41.946 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:41.946 DEBUG:boto:Method: PUT 2024-04-12T03:39:41.946 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:41.947 DEBUG:boto:Data: 2024-04-12T03:39:41.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:41.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:41.947 DEBUG:boto:Port: 8000 2024-04-12T03:39:41.947 DEBUG:boto:Params: {} 2024-04-12T03:39:41.947 DEBUG:boto:Token: None 2024-04-12T03:39:41.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:41 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:41.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9cvH16lsKkAP6Miu79+Gz0KixpM= 2024-04-12T03:39:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9cvH16lsKkAP6Miu79+Gz0KixpM='} 2024-04-12T03:39:41.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f511ecef0393362-006618acfd-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:41.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:41.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:41.954 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:42.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:42.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:42.060 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:42.060 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:42.071 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 43 2024-04-12T03:39:42.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:42.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:42.129 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:42.178 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:42.179 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000467", 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2024-04-12T03:39:42.218 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000056", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000023", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000317", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000022", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000271", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000018", 2024-04-12T03:39:42.219 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000162", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000423", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:42.220 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000283", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000061", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000050", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000029", 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000133" 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:42.221 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:42.222 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.222 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.222 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.222 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.223 DEBUG:boto:Data: 2024-04-12T03:39:42.223 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.223 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.223 DEBUG:boto:Params: {} 2024-04-12T03:39:42.223 DEBUG:boto:Token: None 2024-04-12T03:39:42.223 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000313f86c93e56c88f-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.227 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:42.227 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:42.227 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.228 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:42.228 DEBUG:boto:Data: 2024-04-12T03:39:42.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.228 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.228 DEBUG:boto:Params: {} 2024-04-12T03:39:42.228 DEBUG:boto:Token: None 2024-04-12T03:39:42.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:42.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JnVwkmHX0fcfbxUL/l0MgWvADQ4= 2024-04-12T03:39:42.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JnVwkmHX0fcfbxUL/l0MgWvADQ4='} 2024-04-12T03:39:42.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b947bb3d43b38bf-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.271 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.271 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.271 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.271 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.271 DEBUG:boto:Data: 2024-04-12T03:39:42.271 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.271 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.271 DEBUG:boto:Params: {} 2024-04-12T03:39:42.271 DEBUG:boto:Token: None 2024-04-12T03:39:42.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e39e717de0082c7-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.275 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:42.275 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:42.275 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.275 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:42.275 DEBUG:boto:Data: 2024-04-12T03:39:42.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.275 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.275 DEBUG:boto:Params: {} 2024-04-12T03:39:42.276 DEBUG:boto:Token: None 2024-04-12T03:39:42.276 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:42.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:plnv6uLrmH4qc27e/PbXwfHPbWw= 2024-04-12T03:39:42.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:plnv6uLrmH4qc27e/PbXwfHPbWw='} 2024-04-12T03:39:42.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc6e18eeee98e444-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.282 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.282 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.282 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.282 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.283 DEBUG:boto:Data: 2024-04-12T03:39:42.283 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.283 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.283 DEBUG:boto:Params: {} 2024-04-12T03:39:42.283 DEBUG:boto:Token: None 2024-04-12T03:39:42.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000145d12890363cd23-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:42.287 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:42.287 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.287 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:42.287 DEBUG:boto:Data: 2024-04-12T03:39:42.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.287 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.287 DEBUG:boto:Params: {} 2024-04-12T03:39:42.287 DEBUG:boto:Token: None 2024-04-12T03:39:42.288 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:42.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jbIQNQxf1cImaLbZyQjE9Fnlwns= 2024-04-12T03:39:42.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jbIQNQxf1cImaLbZyQjE9Fnlwns='} 2024-04-12T03:39:42.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f18f3e2fa03e1224-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.331 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.331 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.331 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.331 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.331 DEBUG:boto:Data: 2024-04-12T03:39:42.331 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.331 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.331 DEBUG:boto:Params: {} 2024-04-12T03:39:42.331 DEBUG:boto:Token: None 2024-04-12T03:39:42.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e4a9966ee127aae-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.335 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:42.335 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:42.335 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.335 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:42.335 DEBUG:boto:Data: 2024-04-12T03:39:42.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.336 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.336 DEBUG:boto:Params: {} 2024-04-12T03:39:42.336 DEBUG:boto:Token: None 2024-04-12T03:39:42.336 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:42.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4G72Q3F1Ta+PRZRFBVPYjopXUsc= 2024-04-12T03:39:42.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4G72Q3F1Ta+PRZRFBVPYjopXUsc='} 2024-04-12T03:39:42.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f319e6628664cb77-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.342 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.342 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.342 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.343 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.343 DEBUG:boto:Data: 2024-04-12T03:39:42.343 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.343 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.343 DEBUG:boto:Params: {} 2024-04-12T03:39:42.343 DEBUG:boto:Token: None 2024-04-12T03:39:42.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001061b4843b9e0c9b-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.347 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:42.347 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:42.347 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.347 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:42.347 DEBUG:boto:Data: 2024-04-12T03:39:42.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.347 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.348 DEBUG:boto:Params: {} 2024-04-12T03:39:42.348 DEBUG:boto:Token: None 2024-04-12T03:39:42.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:42.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ltxmbFh3ON3aTmQY502h3/0b8p8= 2024-04-12T03:39:42.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ltxmbFh3ON3aTmQY502h3/0b8p8='} 2024-04-12T03:39:42.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000487958960b69ae4b-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.395 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.395 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.395 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.395 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.395 DEBUG:boto:Data: 2024-04-12T03:39:42.395 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.396 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.396 DEBUG:boto:Params: {} 2024-04-12T03:39:42.396 DEBUG:boto:Token: None 2024-04-12T03:39:42.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000721e259ae9f6ed1a-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.399 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:42.399 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:42.400 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.400 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:42.400 DEBUG:boto:Data: 2024-04-12T03:39:42.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.400 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.400 DEBUG:boto:Params: {} 2024-04-12T03:39:42.400 DEBUG:boto:Token: None 2024-04-12T03:39:42.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:42.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j5DmuxvCrhH139cN63wgCc6nZkU= 2024-04-12T03:39:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j5DmuxvCrhH139cN63wgCc6nZkU='} 2024-04-12T03:39:42.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cdc572f184ee17c-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.408 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.408 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.408 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.408 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.408 DEBUG:boto:Data: 2024-04-12T03:39:42.409 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.409 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.409 DEBUG:boto:Params: {} 2024-04-12T03:39:42.409 DEBUG:boto:Token: None 2024-04-12T03:39:42.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003596765b7c0fc336-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.413 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:42.413 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:42.413 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.413 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:42.413 DEBUG:boto:Data: 2024-04-12T03:39:42.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.413 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.413 DEBUG:boto:Params: {} 2024-04-12T03:39:42.414 DEBUG:boto:Token: None 2024-04-12T03:39:42.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:42.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s0TEu6XRv3WLgB6XFvpCl4C3wYQ= 2024-04-12T03:39:42.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s0TEu6XRv3WLgB6XFvpCl4C3wYQ='} 2024-04-12T03:39:42.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d13154b4b69e40bd-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.459 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.459 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.459 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.459 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.459 DEBUG:boto:Data: 2024-04-12T03:39:42.459 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.459 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.459 DEBUG:boto:Params: {} 2024-04-12T03:39:42.459 DEBUG:boto:Token: None 2024-04-12T03:39:42.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c020a0883bb9b1c-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.463 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:42.463 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:42.463 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.463 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:42.463 DEBUG:boto:Data: 2024-04-12T03:39:42.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.464 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.464 DEBUG:boto:Params: {} 2024-04-12T03:39:42.464 DEBUG:boto:Token: None 2024-04-12T03:39:42.464 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:42.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iWUtX4+/cwr2iqLFIbhjepxUv2g= 2024-04-12T03:39:42.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iWUtX4+/cwr2iqLFIbhjepxUv2g='} 2024-04-12T03:39:42.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b28d32305d8217db-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.470 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.470 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.470 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.471 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.471 DEBUG:boto:Data: 2024-04-12T03:39:42.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.471 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.471 DEBUG:boto:Params: {} 2024-04-12T03:39:42.471 DEBUG:boto:Token: None 2024-04-12T03:39:42.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf31d34503b1eb4-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.475 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:42.475 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:42.475 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.475 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:42.475 DEBUG:boto:Data: 2024-04-12T03:39:42.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.475 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.476 DEBUG:boto:Params: {} 2024-04-12T03:39:42.476 DEBUG:boto:Token: None 2024-04-12T03:39:42.476 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:42.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qea8hgpWztEVp2E4MF9ak7jshKg= 2024-04-12T03:39:42.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qea8hgpWztEVp2E4MF9ak7jshKg='} 2024-04-12T03:39:42.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e2f6e865f461d2c-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.519 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.519 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.519 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.519 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.519 DEBUG:boto:Data: 2024-04-12T03:39:42.519 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.519 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.519 DEBUG:boto:Params: {} 2024-04-12T03:39:42.519 DEBUG:boto:Token: None 2024-04-12T03:39:42.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d50c90f383d1ebe8-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.523 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:42.523 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:42.523 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.523 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:42.523 DEBUG:boto:Data: 2024-04-12T03:39:42.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.523 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.524 DEBUG:boto:Params: {} 2024-04-12T03:39:42.524 DEBUG:boto:Token: None 2024-04-12T03:39:42.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:42.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W14m2k8oWA+dSz5n1knkEH1iozc= 2024-04-12T03:39:42.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W14m2k8oWA+dSz5n1knkEH1iozc='} 2024-04-12T03:39:42.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002552d4e1ace19743-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.530 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.530 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.531 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.531 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.531 DEBUG:boto:Data: 2024-04-12T03:39:42.531 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.531 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.531 DEBUG:boto:Params: {} 2024-04-12T03:39:42.531 DEBUG:boto:Token: None 2024-04-12T03:39:42.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d384ea0aa8326ffc-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.535 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:42.535 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:42.535 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.535 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:42.535 DEBUG:boto:Data: 2024-04-12T03:39:42.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.535 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.535 DEBUG:boto:Params: {} 2024-04-12T03:39:42.535 DEBUG:boto:Token: None 2024-04-12T03:39:42.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:42.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p3X3MYhr0VSuLtlz4f52/gU5TLU= 2024-04-12T03:39:42.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p3X3MYhr0VSuLtlz4f52/gU5TLU='} 2024-04-12T03:39:42.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085c3c559d8962d1e-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.579 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.579 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.579 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.579 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.579 DEBUG:boto:Data: 2024-04-12T03:39:42.579 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.579 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.579 DEBUG:boto:Params: {} 2024-04-12T03:39:42.579 DEBUG:boto:Token: None 2024-04-12T03:39:42.580 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d7f84a967c0ce0e-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.583 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:39:42.583 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:39:42.583 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.583 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467 2024-04-12T03:39:42.583 DEBUG:boto:Data: 2024-04-12T03:39:42.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.584 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.584 DEBUG:boto:Params: {} 2024-04-12T03:39:42.584 DEBUG:boto:Token: None 2024-04-12T03:39:42.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000467 2024-04-12T03:39:42.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ltNpiC/M4FAqPoftaFTRUBis+p4= 2024-04-12T03:39:42.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ltNpiC/M4FAqPoftaFTRUBis+p4='} 2024-04-12T03:39:42.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3915394a5cae1fc-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.589 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.589 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.589 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.590 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.590 DEBUG:boto:Data: 2024-04-12T03:39:42.590 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.590 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.590 DEBUG:boto:Params: {} 2024-04-12T03:39:42.590 DEBUG:boto:Token: None 2024-04-12T03:39:42.590 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8eaa327655420f7-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.594 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:42.594 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:42.594 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.594 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:42.594 DEBUG:boto:Data: 2024-04-12T03:39:42.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.594 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.594 DEBUG:boto:Params: {} 2024-04-12T03:39:42.594 DEBUG:boto:Token: None 2024-04-12T03:39:42.595 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:42.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rfdKePXNx4DrSABJDQRJNp0dtDI= 2024-04-12T03:39:42.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rfdKePXNx4DrSABJDQRJNp0dtDI='} 2024-04-12T03:39:42.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b53b747ccfce6a38-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.639 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.639 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.639 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.639 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.639 DEBUG:boto:Data: 2024-04-12T03:39:42.639 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.639 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.639 DEBUG:boto:Params: {} 2024-04-12T03:39:42.639 DEBUG:boto:Token: None 2024-04-12T03:39:42.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cc3951f25da01d3-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.643 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:42.643 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:42.643 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.643 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:42.643 DEBUG:boto:Data: 2024-04-12T03:39:42.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.644 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.644 DEBUG:boto:Params: {} 2024-04-12T03:39:42.644 DEBUG:boto:Token: None 2024-04-12T03:39:42.644 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:42.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KZEiP/LcptBnBAzUp1np8CG2Qy8= 2024-04-12T03:39:42.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KZEiP/LcptBnBAzUp1np8CG2Qy8='} 2024-04-12T03:39:42.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd735e1fcc876b5c-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.649 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.649 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.649 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.650 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.650 DEBUG:boto:Data: 2024-04-12T03:39:42.650 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.650 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.650 DEBUG:boto:Params: {} 2024-04-12T03:39:42.650 DEBUG:boto:Token: None 2024-04-12T03:39:42.650 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.650 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a315c67348fc410-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.654 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:42.654 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:42.654 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.654 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:42.654 DEBUG:boto:Data: 2024-04-12T03:39:42.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.655 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.655 DEBUG:boto:Params: {} 2024-04-12T03:39:42.655 DEBUG:boto:Token: None 2024-04-12T03:39:42.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:42.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8mCsn6pv5Bz7kPnanMp6wF2MiSk= 2024-04-12T03:39:42.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8mCsn6pv5Bz7kPnanMp6wF2MiSk='} 2024-04-12T03:39:42.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c8faab1e5e2250d-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.698 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.699 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.699 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.699 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.699 DEBUG:boto:Data: 2024-04-12T03:39:42.699 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.699 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.699 DEBUG:boto:Params: {} 2024-04-12T03:39:42.699 DEBUG:boto:Token: None 2024-04-12T03:39:42.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b09a7892186dc2a-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.703 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:42.703 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:42.703 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.703 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:42.703 DEBUG:boto:Data: 2024-04-12T03:39:42.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.703 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.703 DEBUG:boto:Params: {} 2024-04-12T03:39:42.703 DEBUG:boto:Token: None 2024-04-12T03:39:42.704 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:42.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e47gon4nBXWJRggkdZL4ROtxjUM= 2024-04-12T03:39:42.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e47gon4nBXWJRggkdZL4ROtxjUM='} 2024-04-12T03:39:42.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7b7a8d8568105d7-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.709 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.709 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.709 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.709 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.709 DEBUG:boto:Data: 2024-04-12T03:39:42.709 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.709 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.709 DEBUG:boto:Params: {} 2024-04-12T03:39:42.709 DEBUG:boto:Token: None 2024-04-12T03:39:42.710 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.710 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e61a1341db6856e-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.713 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:42.713 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:42.713 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.713 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:42.713 DEBUG:boto:Data: 2024-04-12T03:39:42.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.713 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.714 DEBUG:boto:Params: {} 2024-04-12T03:39:42.714 DEBUG:boto:Token: None 2024-04-12T03:39:42.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:42.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ULUz4xhWv9RnGR5FUyXtjd7Gm+c= 2024-04-12T03:39:42.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ULUz4xhWv9RnGR5FUyXtjd7Gm+c='} 2024-04-12T03:39:42.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063dfcd0a649f1193-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.759 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.759 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.759 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.759 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.759 DEBUG:boto:Data: 2024-04-12T03:39:42.759 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.759 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.759 DEBUG:boto:Params: {} 2024-04-12T03:39:42.759 DEBUG:boto:Token: None 2024-04-12T03:39:42.760 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b4fe04244da3efb-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.763 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:42.763 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:42.763 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.763 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:42.763 DEBUG:boto:Data: 2024-04-12T03:39:42.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.764 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.764 DEBUG:boto:Params: {} 2024-04-12T03:39:42.764 DEBUG:boto:Token: None 2024-04-12T03:39:42.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:42.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Edzpd5k0XeEJOigEIDczXVbcdRU= 2024-04-12T03:39:42.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Edzpd5k0XeEJOigEIDczXVbcdRU='} 2024-04-12T03:39:42.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb654b2180bdb02f-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.770 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.770 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.770 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.770 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.770 DEBUG:boto:Data: 2024-04-12T03:39:42.771 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.771 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.771 DEBUG:boto:Params: {} 2024-04-12T03:39:42.771 DEBUG:boto:Token: None 2024-04-12T03:39:42.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4eb8b60d46cba66-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.774 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:42.774 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:42.775 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.775 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:42.775 DEBUG:boto:Data: 2024-04-12T03:39:42.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.775 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.775 DEBUG:boto:Params: {} 2024-04-12T03:39:42.775 DEBUG:boto:Token: None 2024-04-12T03:39:42.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:42.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sNwu/ANPSOg2igKFBhyqpv6kQrM= 2024-04-12T03:39:42.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sNwu/ANPSOg2igKFBhyqpv6kQrM='} 2024-04-12T03:39:42.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afe19fd84c06bfae-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.819 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.819 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.819 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.819 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.819 DEBUG:boto:Data: 2024-04-12T03:39:42.819 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.819 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.819 DEBUG:boto:Params: {} 2024-04-12T03:39:42.819 DEBUG:boto:Token: None 2024-04-12T03:39:42.820 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fca09f9b38242ef-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.823 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:42.823 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:42.823 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.823 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:42.824 DEBUG:boto:Data: 2024-04-12T03:39:42.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.824 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.824 DEBUG:boto:Params: {} 2024-04-12T03:39:42.824 DEBUG:boto:Token: None 2024-04-12T03:39:42.825 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:42.825 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLXbfhyhwqqZ9f17BSC3dcO4W+I= 2024-04-12T03:39:42.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLXbfhyhwqqZ9f17BSC3dcO4W+I='} 2024-04-12T03:39:42.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000639a68d7ce4a4333-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.831 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.831 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.832 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.832 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.832 DEBUG:boto:Data: 2024-04-12T03:39:42.832 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.832 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.832 DEBUG:boto:Params: {} 2024-04-12T03:39:42.832 DEBUG:boto:Token: None 2024-04-12T03:39:42.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6a302ef75a2d461-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:42.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:42.836 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.836 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:42.836 DEBUG:boto:Data: 2024-04-12T03:39:42.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.836 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.836 DEBUG:boto:Params: {} 2024-04-12T03:39:42.836 DEBUG:boto:Token: None 2024-04-12T03:39:42.837 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:42.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UPBD7SztFvZXqHTZKRsSzFzZCD0= 2024-04-12T03:39:42.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UPBD7SztFvZXqHTZKRsSzFzZCD0='} 2024-04-12T03:39:42.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d7334e3975e662d-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.883 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.883 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.883 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.883 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.883 DEBUG:boto:Data: 2024-04-12T03:39:42.883 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.883 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.883 DEBUG:boto:Params: {} 2024-04-12T03:39:42.883 DEBUG:boto:Token: None 2024-04-12T03:39:42.884 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd358fdf1a1dc460-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.887 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:42.887 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:42.887 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.887 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:42.887 DEBUG:boto:Data: 2024-04-12T03:39:42.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.888 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.888 DEBUG:boto:Params: {} 2024-04-12T03:39:42.888 DEBUG:boto:Token: None 2024-04-12T03:39:42.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:42.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XzzDZB3NUn4bwpMM4gbdQOREvEM= 2024-04-12T03:39:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XzzDZB3NUn4bwpMM4gbdQOREvEM='} 2024-04-12T03:39:42.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078b71fba1a2d1dd4-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.894 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.894 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.894 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.894 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.894 DEBUG:boto:Data: 2024-04-12T03:39:42.895 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.895 DEBUG:boto:Params: {} 2024-04-12T03:39:42.895 DEBUG:boto:Token: None 2024-04-12T03:39:42.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fee39794245ea603-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.900 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:42.900 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:42.900 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.900 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:42.901 DEBUG:boto:Data: 2024-04-12T03:39:42.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.901 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.901 DEBUG:boto:Params: {} 2024-04-12T03:39:42.901 DEBUG:boto:Token: None 2024-04-12T03:39:42.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:42.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wp2J4tu7nbjLqgXuTQWLuEkeFns= 2024-04-12T03:39:42.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wp2J4tu7nbjLqgXuTQWLuEkeFns='} 2024-04-12T03:39:42.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022308e5a3ae1da61-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.947 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.947 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.947 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.947 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.947 DEBUG:boto:Data: 2024-04-12T03:39:42.947 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.947 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.947 DEBUG:boto:Params: {} 2024-04-12T03:39:42.947 DEBUG:boto:Token: None 2024-04-12T03:39:42.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0b05115cc797b61-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.951 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:42.951 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:42.951 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.951 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:42.951 DEBUG:boto:Data: 2024-04-12T03:39:42.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.952 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.952 DEBUG:boto:Params: {} 2024-04-12T03:39:42.952 DEBUG:boto:Token: None 2024-04-12T03:39:42.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:42.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h06xlc9GGH0CjMEDT3ohRkL2Ats= 2024-04-12T03:39:42.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h06xlc9GGH0CjMEDT3ohRkL2Ats='} 2024-04-12T03:39:42.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000843e45d717bf8856-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.958 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:42.958 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:42.958 DEBUG:boto:Method: HEAD 2024-04-12T03:39:42.958 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:42.959 DEBUG:boto:Data: 2024-04-12T03:39:42.959 DEBUG:boto:Headers: {} 2024-04-12T03:39:42.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.959 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.959 DEBUG:boto:Params: {} 2024-04-12T03:39:42.959 DEBUG:boto:Token: None 2024-04-12T03:39:42.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/ 2024-04-12T03:39:42.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ= 2024-04-12T03:39:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH/0qGcTEy/HxEta2QHc+kQMOWQ=', 'Content-Length': '0'} 2024-04-12T03:39:42.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8fcf576eabeaaee-006618acfe-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:42.962 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:42.962 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:42.962 DEBUG:boto:Method: PUT 2024-04-12T03:39:42.963 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:42.963 DEBUG:boto:Data: 2024-04-12T03:39:42.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:42.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:42.963 DEBUG:boto:Port: 8000 2024-04-12T03:39:42.963 DEBUG:boto:Params: {} 2024-04-12T03:39:42.963 DEBUG:boto:Token: None 2024-04-12T03:39:42.963 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:42 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:42.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gRYLZHJhXl59+7NCAeTulPkfd1c= 2024-04-12T03:39:42.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gRYLZHJhXl59+7NCAeTulPkfd1c='} 2024-04-12T03:39:43.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084e72740b3e9ae02-006618acfe-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.007 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.007 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.007 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.007 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.007 DEBUG:boto:Data: 2024-04-12T03:39:43.007 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.007 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.007 DEBUG:boto:Params: {} 2024-04-12T03:39:43.007 DEBUG:boto:Token: None 2024-04-12T03:39:43.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d6d09204d4b502-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.010 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.011 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.011 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.011 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.011 DEBUG:boto:Data: 2024-04-12T03:39:43.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.011 DEBUG:boto:Params: {} 2024-04-12T03:39:43.011 DEBUG:boto:Token: None 2024-04-12T03:39:43.011 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rPRIRTsMGZi5Cz0B5m7oNvgogRg= 2024-04-12T03:39:43.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rPRIRTsMGZi5Cz0B5m7oNvgogRg='} 2024-04-12T03:39:43.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008096de605f69b730-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.017 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.018 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.018 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.018 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.018 DEBUG:boto:Data: 2024-04-12T03:39:43.018 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.018 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.018 DEBUG:boto:Params: {} 2024-04-12T03:39:43.018 DEBUG:boto:Token: None 2024-04-12T03:39:43.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efda8f0aac6380b2-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.022 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:43.022 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:43.022 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.022 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:43.022 DEBUG:boto:Data: 2024-04-12T03:39:43.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.022 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.022 DEBUG:boto:Params: {} 2024-04-12T03:39:43.023 DEBUG:boto:Token: None 2024-04-12T03:39:43.023 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:43.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3lMND0W9v68qKIW9BKM9Eat9hF0= 2024-04-12T03:39:43.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3lMND0W9v68qKIW9BKM9Eat9hF0='} 2024-04-12T03:39:43.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000220287169fdc362f-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.067 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.067 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.067 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.067 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.067 DEBUG:boto:Data: 2024-04-12T03:39:43.067 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.067 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.067 DEBUG:boto:Params: {} 2024-04-12T03:39:43.067 DEBUG:boto:Token: None 2024-04-12T03:39:43.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b720dc12c0310c5-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.071 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:43.071 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:43.071 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.071 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:43.071 DEBUG:boto:Data: 2024-04-12T03:39:43.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.072 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.072 DEBUG:boto:Params: {} 2024-04-12T03:39:43.072 DEBUG:boto:Token: None 2024-04-12T03:39:43.072 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:43.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qRoFUAfz/WJ47z4dvKH6qAnmpXw= 2024-04-12T03:39:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qRoFUAfz/WJ47z4dvKH6qAnmpXw='} 2024-04-12T03:39:43.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df0c30b83677f41c-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:43.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:43.077 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:43.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:43.129 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:43.178 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:39:43.179 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-60 2024-04-12T03:39:43.191 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 54 2024-04-12T03:39:43.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:43.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:43.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:43.302 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:43.303 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:43.342 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:39:43.342 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2024-04-12T03:39:43.342 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000060", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000065", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000048", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2024-04-12T03:39:43.343 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000062", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000040", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000024", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000045", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000039", 2024-04-12T03:39:43.344 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000027", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000053", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000036", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000057", 2024-04-12T03:39:43.345 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000018", 2024-04-12T03:39:43.346 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008" 2024-04-12T03:39:43.346 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:39:43.346 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:39:43.347 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.347 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.347 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.347 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.347 DEBUG:boto:Data: 2024-04-12T03:39:43.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.347 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.347 DEBUG:boto:Params: {} 2024-04-12T03:39:43.348 DEBUG:boto:Token: None 2024-04-12T03:39:43.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c28425d7d99c887-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.352 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:43.352 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:43.352 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.352 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:43.352 DEBUG:boto:Data: 2024-04-12T03:39:43.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.352 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.352 DEBUG:boto:Params: {} 2024-04-12T03:39:43.352 DEBUG:boto:Token: None 2024-04-12T03:39:43.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:43.353 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsPzW46oISFI+yH0/Evb+QUrbaY= 2024-04-12T03:39:43.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsPzW46oISFI+yH0/Evb+QUrbaY='} 2024-04-12T03:39:43.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e281b221d7a85ca7-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.394 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.395 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.395 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.395 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.395 DEBUG:boto:Data: 2024-04-12T03:39:43.395 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.395 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.395 DEBUG:boto:Params: {} 2024-04-12T03:39:43.395 DEBUG:boto:Token: None 2024-04-12T03:39:43.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018756cd2a8a1c8bb-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.399 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:43.399 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:43.399 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.399 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:43.399 DEBUG:boto:Data: 2024-04-12T03:39:43.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.399 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.399 DEBUG:boto:Params: {} 2024-04-12T03:39:43.400 DEBUG:boto:Token: None 2024-04-12T03:39:43.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:43.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eoWya+BFj1IJSxoIgOas1wuAJ1Y= 2024-04-12T03:39:43.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eoWya+BFj1IJSxoIgOas1wuAJ1Y='} 2024-04-12T03:39:43.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000385c8b5da6122048-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.405 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.405 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.405 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.405 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.405 DEBUG:boto:Data: 2024-04-12T03:39:43.405 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.405 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.406 DEBUG:boto:Params: {} 2024-04-12T03:39:43.406 DEBUG:boto:Token: None 2024-04-12T03:39:43.406 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010b3e553625a576b-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.410 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:43.410 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:43.410 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.410 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:43.410 DEBUG:boto:Data: 2024-04-12T03:39:43.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.410 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.410 DEBUG:boto:Params: {} 2024-04-12T03:39:43.410 DEBUG:boto:Token: None 2024-04-12T03:39:43.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:43.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:83nX8P6c5wk22oozoutF6rqSeII= 2024-04-12T03:39:43.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:83nX8P6c5wk22oozoutF6rqSeII='} 2024-04-12T03:39:43.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c394dea7b6fb6c05-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.455 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.455 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.455 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.455 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.455 DEBUG:boto:Data: 2024-04-12T03:39:43.455 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.455 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.455 DEBUG:boto:Params: {} 2024-04-12T03:39:43.455 DEBUG:boto:Token: None 2024-04-12T03:39:43.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044365e758a0ebb07-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.459 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:43.460 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:43.460 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.460 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:43.460 DEBUG:boto:Data: 2024-04-12T03:39:43.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.460 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.460 DEBUG:boto:Params: {} 2024-04-12T03:39:43.460 DEBUG:boto:Token: None 2024-04-12T03:39:43.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:43.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eB0QXvgFVCkSZVrRX7mVxhQlTo4= 2024-04-12T03:39:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eB0QXvgFVCkSZVrRX7mVxhQlTo4='} 2024-04-12T03:39:43.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee05cd6a45ce64ec-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.467 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.467 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.467 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.467 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.467 DEBUG:boto:Data: 2024-04-12T03:39:43.467 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.467 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.467 DEBUG:boto:Params: {} 2024-04-12T03:39:43.468 DEBUG:boto:Token: None 2024-04-12T03:39:43.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c32469da0ab47fab-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.471 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:43.471 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:43.471 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:43.472 DEBUG:boto:Data: 2024-04-12T03:39:43.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.472 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.472 DEBUG:boto:Params: {} 2024-04-12T03:39:43.472 DEBUG:boto:Token: None 2024-04-12T03:39:43.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:43.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88jDYBKQ3q/jGKSVX7lHkYveyVo= 2024-04-12T03:39:43.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88jDYBKQ3q/jGKSVX7lHkYveyVo='} 2024-04-12T03:39:43.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fa9ff6fe4dbf191-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.515 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.515 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.515 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.515 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.515 DEBUG:boto:Data: 2024-04-12T03:39:43.515 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.515 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.515 DEBUG:boto:Params: {} 2024-04-12T03:39:43.515 DEBUG:boto:Token: None 2024-04-12T03:39:43.516 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041532985704a590b-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.519 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:43.519 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:43.519 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.519 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:43.519 DEBUG:boto:Data: 2024-04-12T03:39:43.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.520 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.520 DEBUG:boto:Params: {} 2024-04-12T03:39:43.520 DEBUG:boto:Token: None 2024-04-12T03:39:43.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:43.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c0cmxRH7Br0DD7K9dH+7t8MAKcA= 2024-04-12T03:39:43.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c0cmxRH7Br0DD7K9dH+7t8MAKcA='} 2024-04-12T03:39:43.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008137e24bf2f65c68-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.526 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.526 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.527 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.527 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.527 DEBUG:boto:Data: 2024-04-12T03:39:43.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.527 DEBUG:boto:Params: {} 2024-04-12T03:39:43.527 DEBUG:boto:Token: None 2024-04-12T03:39:43.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027ec0f8bd6318d11-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.531 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:43.531 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:43.531 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.531 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:43.531 DEBUG:boto:Data: 2024-04-12T03:39:43.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.532 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.532 DEBUG:boto:Params: {} 2024-04-12T03:39:43.532 DEBUG:boto:Token: None 2024-04-12T03:39:43.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:43.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qOGvScThVjK2bHB9F0cAIYR/FLQ= 2024-04-12T03:39:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qOGvScThVjK2bHB9F0cAIYR/FLQ='} 2024-04-12T03:39:43.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060994bf05e2d6eb3-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.575 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.575 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.575 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.575 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.575 DEBUG:boto:Data: 2024-04-12T03:39:43.575 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.575 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.575 DEBUG:boto:Params: {} 2024-04-12T03:39:43.576 DEBUG:boto:Token: None 2024-04-12T03:39:43.576 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e85d0422c415037-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.579 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:43.579 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:43.579 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.580 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:43.580 DEBUG:boto:Data: 2024-04-12T03:39:43.580 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.580 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.580 DEBUG:boto:Params: {} 2024-04-12T03:39:43.580 DEBUG:boto:Token: None 2024-04-12T03:39:43.580 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:43.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJs7vDQS6mdddhVXf5xSRaR7vBs= 2024-04-12T03:39:43.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJs7vDQS6mdddhVXf5xSRaR7vBs='} 2024-04-12T03:39:43.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097332f5066044567-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.586 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.587 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.587 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.587 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.587 DEBUG:boto:Data: 2024-04-12T03:39:43.587 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.587 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.587 DEBUG:boto:Params: {} 2024-04-12T03:39:43.587 DEBUG:boto:Token: None 2024-04-12T03:39:43.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096afc99982fc38fa-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.591 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:43.591 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:43.591 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.591 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:43.591 DEBUG:boto:Data: 2024-04-12T03:39:43.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.592 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.592 DEBUG:boto:Params: {} 2024-04-12T03:39:43.592 DEBUG:boto:Token: None 2024-04-12T03:39:43.592 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:43.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uGUj4v4MPlF8IEzPCqQWAVDsjic= 2024-04-12T03:39:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uGUj4v4MPlF8IEzPCqQWAVDsjic='} 2024-04-12T03:39:43.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be681b03cfedaba7-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.634 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.635 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.635 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.635 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.635 DEBUG:boto:Data: 2024-04-12T03:39:43.635 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.635 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.635 DEBUG:boto:Params: {} 2024-04-12T03:39:43.635 DEBUG:boto:Token: None 2024-04-12T03:39:43.635 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dfaf7bcb751c472-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.639 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:43.639 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:43.639 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.640 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:43.640 DEBUG:boto:Data: 2024-04-12T03:39:43.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.640 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.640 DEBUG:boto:Params: {} 2024-04-12T03:39:43.640 DEBUG:boto:Token: None 2024-04-12T03:39:43.640 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:43.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DY1wPBTSvdIQB/6n68uIsB/9dw8= 2024-04-12T03:39:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DY1wPBTSvdIQB/6n68uIsB/9dw8='} 2024-04-12T03:39:43.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8647377d422e514-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.646 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.647 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.647 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.647 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.647 DEBUG:boto:Data: 2024-04-12T03:39:43.647 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.647 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.647 DEBUG:boto:Params: {} 2024-04-12T03:39:43.647 DEBUG:boto:Token: None 2024-04-12T03:39:43.647 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066ad1872418d1883-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.651 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:43.651 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:43.651 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.651 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:43.652 DEBUG:boto:Data: 2024-04-12T03:39:43.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.652 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.652 DEBUG:boto:Params: {} 2024-04-12T03:39:43.652 DEBUG:boto:Token: None 2024-04-12T03:39:43.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:43.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6XeM3Os2oR5UyyUtRoHuQn8Ddjg= 2024-04-12T03:39:43.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6XeM3Os2oR5UyyUtRoHuQn8Ddjg='} 2024-04-12T03:39:43.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b7c8c598dd2abcd-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.695 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.695 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.695 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.695 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.695 DEBUG:boto:Data: 2024-04-12T03:39:43.695 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.695 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.695 DEBUG:boto:Params: {} 2024-04-12T03:39:43.696 DEBUG:boto:Token: None 2024-04-12T03:39:43.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b25311338dd3177d-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.699 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:43.699 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:43.699 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.699 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:43.699 DEBUG:boto:Data: 2024-04-12T03:39:43.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.700 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.700 DEBUG:boto:Params: {} 2024-04-12T03:39:43.700 DEBUG:boto:Token: None 2024-04-12T03:39:43.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:43.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nlLhNvJocwz8A9VGIVTlgkkEVps= 2024-04-12T03:39:43.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nlLhNvJocwz8A9VGIVTlgkkEVps='} 2024-04-12T03:39:43.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1502c8007f5c430-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.707 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.707 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.707 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.707 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.707 DEBUG:boto:Data: 2024-04-12T03:39:43.707 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.707 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.707 DEBUG:boto:Params: {} 2024-04-12T03:39:43.707 DEBUG:boto:Token: None 2024-04-12T03:39:43.707 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7ebc9733241d444-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.711 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:43.711 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:43.711 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.711 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:43.711 DEBUG:boto:Data: 2024-04-12T03:39:43.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.712 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.712 DEBUG:boto:Params: {} 2024-04-12T03:39:43.712 DEBUG:boto:Token: None 2024-04-12T03:39:43.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:43.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Alhs4dGPCZX2TwvBSDAMEb1OT5c= 2024-04-12T03:39:43.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Alhs4dGPCZX2TwvBSDAMEb1OT5c='} 2024-04-12T03:39:43.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8af7ecb30a09a8c-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.755 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.755 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.755 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.755 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.755 DEBUG:boto:Data: 2024-04-12T03:39:43.755 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.755 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.755 DEBUG:boto:Params: {} 2024-04-12T03:39:43.756 DEBUG:boto:Token: None 2024-04-12T03:39:43.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a8c8060f24f0b82-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.759 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:43.759 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:43.760 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.760 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:43.760 DEBUG:boto:Data: 2024-04-12T03:39:43.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.760 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.760 DEBUG:boto:Params: {} 2024-04-12T03:39:43.760 DEBUG:boto:Token: None 2024-04-12T03:39:43.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:43.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WaTLfzRIom4RRT1w6yy0NgYv3lM= 2024-04-12T03:39:43.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WaTLfzRIom4RRT1w6yy0NgYv3lM='} 2024-04-12T03:39:43.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000136c3aed7f70e6b9-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.767 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.767 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.767 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.767 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.767 DEBUG:boto:Data: 2024-04-12T03:39:43.767 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.767 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.767 DEBUG:boto:Params: {} 2024-04-12T03:39:43.767 DEBUG:boto:Token: None 2024-04-12T03:39:43.768 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6cbd91aa7085b7f-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.771 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:43.771 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:43.771 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.771 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:43.771 DEBUG:boto:Data: 2024-04-12T03:39:43.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.772 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.772 DEBUG:boto:Params: {} 2024-04-12T03:39:43.772 DEBUG:boto:Token: None 2024-04-12T03:39:43.772 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:43.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aUriYgkeqYRqskgatoPPbu99s6I= 2024-04-12T03:39:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aUriYgkeqYRqskgatoPPbu99s6I='} 2024-04-12T03:39:43.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000855ea3c17b29aec7-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.848 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.848 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.848 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.848 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.848 DEBUG:boto:Data: 2024-04-12T03:39:43.848 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.849 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.849 DEBUG:boto:Params: {} 2024-04-12T03:39:43.849 DEBUG:boto:Token: None 2024-04-12T03:39:43.849 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f944d5dc47700a-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.853 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:43.853 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:43.853 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.853 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:43.853 DEBUG:boto:Data: 2024-04-12T03:39:43.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.853 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.853 DEBUG:boto:Params: {} 2024-04-12T03:39:43.853 DEBUG:boto:Token: None 2024-04-12T03:39:43.854 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:43.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UZFAdOvkBubrshI4xNhJ1u2RpJo= 2024-04-12T03:39:43.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UZFAdOvkBubrshI4xNhJ1u2RpJo='} 2024-04-12T03:39:43.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d29cddf126ede32f-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.860 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.860 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.860 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.860 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.860 DEBUG:boto:Data: 2024-04-12T03:39:43.860 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.860 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.860 DEBUG:boto:Params: {} 2024-04-12T03:39:43.860 DEBUG:boto:Token: None 2024-04-12T03:39:43.861 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000755c3bb49f38604c-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.864 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.864 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.865 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.865 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.865 DEBUG:boto:Data: 2024-04-12T03:39:43.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.865 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.865 DEBUG:boto:Params: {} 2024-04-12T03:39:43.865 DEBUG:boto:Token: None 2024-04-12T03:39:43.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:43.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rPRIRTsMGZi5Cz0B5m7oNvgogRg= 2024-04-12T03:39:43.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rPRIRTsMGZi5Cz0B5m7oNvgogRg='} 2024-04-12T03:39:43.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000642dc091dec5cd8e-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.911 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.911 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.911 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.911 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.911 DEBUG:boto:Data: 2024-04-12T03:39:43.911 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.911 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.911 DEBUG:boto:Params: {} 2024-04-12T03:39:43.911 DEBUG:boto:Token: None 2024-04-12T03:39:43.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf661fb735c3351f-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.915 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:43.915 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:43.915 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.915 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:43.915 DEBUG:boto:Data: 2024-04-12T03:39:43.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.915 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.915 DEBUG:boto:Params: {} 2024-04-12T03:39:43.915 DEBUG:boto:Token: None 2024-04-12T03:39:43.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:43.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iH9y1h2nwtIaIO2/dJEV9VSXEnU= 2024-04-12T03:39:43.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iH9y1h2nwtIaIO2/dJEV9VSXEnU='} 2024-04-12T03:39:43.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004540e862bbaee19a-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.922 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.922 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.922 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.922 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.922 DEBUG:boto:Data: 2024-04-12T03:39:43.922 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.922 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.922 DEBUG:boto:Params: {} 2024-04-12T03:39:43.923 DEBUG:boto:Token: None 2024-04-12T03:39:43.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a95d6d67c408c7b-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.926 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:43.927 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:43.927 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.927 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:43.927 DEBUG:boto:Data: 2024-04-12T03:39:43.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.927 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.927 DEBUG:boto:Params: {} 2024-04-12T03:39:43.927 DEBUG:boto:Token: None 2024-04-12T03:39:43.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:43.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q42GzLeaun4oiZ5KMkPBueo5YoM= 2024-04-12T03:39:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q42GzLeaun4oiZ5KMkPBueo5YoM='} 2024-04-12T03:39:43.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005150d0205bde24e6-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.971 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.971 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.971 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.971 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.971 DEBUG:boto:Data: 2024-04-12T03:39:43.971 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.972 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.972 DEBUG:boto:Params: {} 2024-04-12T03:39:43.972 DEBUG:boto:Token: None 2024-04-12T03:39:43.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.973 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000739453e0cfce20d6-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.976 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:43.977 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:43.977 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.977 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:43.977 DEBUG:boto:Data: 2024-04-12T03:39:43.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.977 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.977 DEBUG:boto:Params: {} 2024-04-12T03:39:43.977 DEBUG:boto:Token: None 2024-04-12T03:39:43.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:43.978 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DaPkFO7648p4Q3LMxtaZYYd5Oh8= 2024-04-12T03:39:43.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DaPkFO7648p4Q3LMxtaZYYd5Oh8='} 2024-04-12T03:39:43.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000734ac631170a1baa-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.984 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:43.984 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:43.984 DEBUG:boto:Method: HEAD 2024-04-12T03:39:43.984 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:43.984 DEBUG:boto:Data: 2024-04-12T03:39:43.984 DEBUG:boto:Headers: {} 2024-04-12T03:39:43.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.985 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.985 DEBUG:boto:Params: {} 2024-04-12T03:39:43.985 DEBUG:boto:Token: None 2024-04-12T03:39:43.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/ 2024-04-12T03:39:43.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U= 2024-04-12T03:39:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PcpCH6dTcDEkiGodKVQuIuL3E7U=', 'Content-Length': '0'} 2024-04-12T03:39:43.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042ce7bd0f90d0889-006618acff-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:43.989 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:43.989 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:43.989 DEBUG:boto:Method: PUT 2024-04-12T03:39:43.989 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:43.989 DEBUG:boto:Data: 2024-04-12T03:39:43.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:43.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:43.990 DEBUG:boto:Port: 8000 2024-04-12T03:39:43.990 DEBUG:boto:Params: {} 2024-04-12T03:39:43.990 DEBUG:boto:Token: None 2024-04-12T03:39:43.990 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:43 GMT /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:43.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Nu7kRzAFYVbULtgIXtL8EFTldhI= 2024-04-12T03:39:43.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Nu7kRzAFYVbULtgIXtL8EFTldhI='} 2024-04-12T03:39:44.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a33dc179ca050a7-006618acff-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.038 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.039 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.039 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.039 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.039 DEBUG:boto:Data: 2024-04-12T03:39:44.039 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.039 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.039 DEBUG:boto:Params: {} 2024-04-12T03:39:44.040 DEBUG:boto:Token: None 2024-04-12T03:39:44.040 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000193709fd7405f74a-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.044 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:44.044 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:44.044 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.044 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:44.044 DEBUG:boto:Data: 2024-04-12T03:39:44.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.044 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.044 DEBUG:boto:Params: {} 2024-04-12T03:39:44.044 DEBUG:boto:Token: None 2024-04-12T03:39:44.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:44.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b4JUFdaEKoeyY7edz7ER2AhMOZc= 2024-04-12T03:39:44.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b4JUFdaEKoeyY7edz7ER2AhMOZc='} 2024-04-12T03:39:44.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a77e4200064ae0c-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.055 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.055 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.055 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.055 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.055 DEBUG:boto:Data: 2024-04-12T03:39:44.055 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.055 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.056 DEBUG:boto:Params: {} 2024-04-12T03:39:44.056 DEBUG:boto:Token: None 2024-04-12T03:39:44.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000009e9b47d74ebb73-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.059 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:44.059 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:44.060 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.060 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:44.060 DEBUG:boto:Data: 2024-04-12T03:39:44.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.060 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.060 DEBUG:boto:Params: {} 2024-04-12T03:39:44.060 DEBUG:boto:Token: None 2024-04-12T03:39:44.060 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:44.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O58UE+kpZYp4At9rkfqw9exvDto= 2024-04-12T03:39:44.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O58UE+kpZYp4At9rkfqw9exvDto='} 2024-04-12T03:39:44.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f139ba1044f02b6-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.103 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.103 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.103 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.103 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.103 DEBUG:boto:Data: 2024-04-12T03:39:44.103 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.103 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.103 DEBUG:boto:Params: {} 2024-04-12T03:39:44.104 DEBUG:boto:Token: None 2024-04-12T03:39:44.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092d5ec5669c8f6a9-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.107 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:44.107 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:44.107 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.107 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:44.108 DEBUG:boto:Data: 2024-04-12T03:39:44.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.108 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.108 DEBUG:boto:Params: {} 2024-04-12T03:39:44.108 DEBUG:boto:Token: None 2024-04-12T03:39:44.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:44.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+N9RdBkGj3I11/amPZXsCc9fTyc= 2024-04-12T03:39:44.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+N9RdBkGj3I11/amPZXsCc9fTyc='} 2024-04-12T03:39:44.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000749dc59771b96427-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.115 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.115 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.115 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.115 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.115 DEBUG:boto:Data: 2024-04-12T03:39:44.116 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.116 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.116 DEBUG:boto:Params: {} 2024-04-12T03:39:44.116 DEBUG:boto:Token: None 2024-04-12T03:39:44.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddca1927074dae42-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.121 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:44.122 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:44.122 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.122 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:44.122 DEBUG:boto:Data: 2024-04-12T03:39:44.122 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.122 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.122 DEBUG:boto:Params: {} 2024-04-12T03:39:44.122 DEBUG:boto:Token: None 2024-04-12T03:39:44.123 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:44.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BtWyISQ1jqRxHawAMdPlQVQGyH4= 2024-04-12T03:39:44.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BtWyISQ1jqRxHawAMdPlQVQGyH4='} 2024-04-12T03:39:44.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009370cefdd88a3233-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.166 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.167 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.167 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.167 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.167 DEBUG:boto:Data: 2024-04-12T03:39:44.167 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.167 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.167 DEBUG:boto:Params: {} 2024-04-12T03:39:44.167 DEBUG:boto:Token: None 2024-04-12T03:39:44.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000248585fc62295799-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.171 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:44.171 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:44.171 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.171 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:44.171 DEBUG:boto:Data: 2024-04-12T03:39:44.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.172 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.172 DEBUG:boto:Params: {} 2024-04-12T03:39:44.172 DEBUG:boto:Token: None 2024-04-12T03:39:44.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:44.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uwDehQT21F/HbWfB9yYYFEe0BS4= 2024-04-12T03:39:44.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uwDehQT21F/HbWfB9yYYFEe0BS4='} 2024-04-12T03:39:44.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042925a15226813f3-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.178 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.178 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.178 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.179 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.179 DEBUG:boto:Data: 2024-04-12T03:39:44.179 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.179 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.179 DEBUG:boto:Params: {} 2024-04-12T03:39:44.179 DEBUG:boto:Token: None 2024-04-12T03:39:44.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000089f08bd5f71c9c5-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.183 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:44.183 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:44.183 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.183 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:44.183 DEBUG:boto:Data: 2024-04-12T03:39:44.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.184 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.184 DEBUG:boto:Params: {} 2024-04-12T03:39:44.184 DEBUG:boto:Token: None 2024-04-12T03:39:44.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:44.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:74x/1oQ11kxy1/RIMQt/4L9cW7c= 2024-04-12T03:39:44.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:74x/1oQ11kxy1/RIMQt/4L9cW7c='} 2024-04-12T03:39:44.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036c5861568b67e31-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.227 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.227 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.227 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.227 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.227 DEBUG:boto:Data: 2024-04-12T03:39:44.227 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.227 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.227 DEBUG:boto:Params: {} 2024-04-12T03:39:44.227 DEBUG:boto:Token: None 2024-04-12T03:39:44.228 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004feb8a22a1c6ee0b-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.231 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:44.231 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:44.232 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.232 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:44.232 DEBUG:boto:Data: 2024-04-12T03:39:44.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.232 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.232 DEBUG:boto:Params: {} 2024-04-12T03:39:44.232 DEBUG:boto:Token: None 2024-04-12T03:39:44.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:44.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZCLLrTIyO9gDWPWvFEe6zZS9x3U= 2024-04-12T03:39:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZCLLrTIyO9gDWPWvFEe6zZS9x3U='} 2024-04-12T03:39:44.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4df7bbee4efed0d-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.238 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.239 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.239 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.239 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.239 DEBUG:boto:Data: 2024-04-12T03:39:44.239 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.239 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.239 DEBUG:boto:Params: {} 2024-04-12T03:39:44.239 DEBUG:boto:Token: None 2024-04-12T03:39:44.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000667c28aca8b5b8ba-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.243 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:44.243 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:44.243 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.243 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:44.243 DEBUG:boto:Data: 2024-04-12T03:39:44.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.244 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.244 DEBUG:boto:Params: {} 2024-04-12T03:39:44.244 DEBUG:boto:Token: None 2024-04-12T03:39:44.244 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:44.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bDpzJW5nAwax+JQNDzbK8d4UiLA= 2024-04-12T03:39:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bDpzJW5nAwax+JQNDzbK8d4UiLA='} 2024-04-12T03:39:44.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4d23a7458b2506f-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.287 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.287 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.287 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.287 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.287 DEBUG:boto:Data: 2024-04-12T03:39:44.287 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.287 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.287 DEBUG:boto:Params: {} 2024-04-12T03:39:44.288 DEBUG:boto:Token: None 2024-04-12T03:39:44.288 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002707137ae7043de2-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.291 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:44.291 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:44.291 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.292 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:44.292 DEBUG:boto:Data: 2024-04-12T03:39:44.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.292 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.292 DEBUG:boto:Params: {} 2024-04-12T03:39:44.292 DEBUG:boto:Token: None 2024-04-12T03:39:44.292 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:44.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f8RgjOMExlE4yTib9T7d3/DNGqg= 2024-04-12T03:39:44.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f8RgjOMExlE4yTib9T7d3/DNGqg='} 2024-04-12T03:39:44.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b6f84dd3f6139b-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.298 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.298 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.298 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.298 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.299 DEBUG:boto:Data: 2024-04-12T03:39:44.299 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.299 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.299 DEBUG:boto:Params: {} 2024-04-12T03:39:44.299 DEBUG:boto:Token: None 2024-04-12T03:39:44.299 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011a7a4216372e9f1-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.303 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:44.303 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:44.303 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.303 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:44.303 DEBUG:boto:Data: 2024-04-12T03:39:44.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.304 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.304 DEBUG:boto:Params: {} 2024-04-12T03:39:44.304 DEBUG:boto:Token: None 2024-04-12T03:39:44.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:44.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AgWOUFONQK2NTis9KTXNPs1hQWk= 2024-04-12T03:39:44.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AgWOUFONQK2NTis9KTXNPs1hQWk='} 2024-04-12T03:39:44.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8bf3510ff7078ba-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.346 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.347 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.347 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.347 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.347 DEBUG:boto:Data: 2024-04-12T03:39:44.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.347 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.347 DEBUG:boto:Params: {} 2024-04-12T03:39:44.347 DEBUG:boto:Token: None 2024-04-12T03:39:44.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53fab0a58f04594-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.351 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:44.351 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:44.351 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.351 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:44.352 DEBUG:boto:Data: 2024-04-12T03:39:44.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.352 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.352 DEBUG:boto:Params: {} 2024-04-12T03:39:44.352 DEBUG:boto:Token: None 2024-04-12T03:39:44.352 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:44.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RjB+bm/E9WjTJndbYI5Q1xHgh1E= 2024-04-12T03:39:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RjB+bm/E9WjTJndbYI5Q1xHgh1E='} 2024-04-12T03:39:44.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6bf5a0d7e685d2a-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.358 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.358 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.359 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.359 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.359 DEBUG:boto:Data: 2024-04-12T03:39:44.359 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.359 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.359 DEBUG:boto:Params: {} 2024-04-12T03:39:44.359 DEBUG:boto:Token: None 2024-04-12T03:39:44.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023c54061c6f728ab-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.363 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:44.363 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:44.363 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.363 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:44.363 DEBUG:boto:Data: 2024-04-12T03:39:44.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.363 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.363 DEBUG:boto:Params: {} 2024-04-12T03:39:44.364 DEBUG:boto:Token: None 2024-04-12T03:39:44.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:44.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IH/6HhMzN/3tHyMQQzCnJUHOPGk= 2024-04-12T03:39:44.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IH/6HhMzN/3tHyMQQzCnJUHOPGk='} 2024-04-12T03:39:44.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e5303cc9ef2158c-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.407 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:44.407 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:44.407 DEBUG:boto:Method: HEAD 2024-04-12T03:39:44.407 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:44.407 DEBUG:boto:Data: 2024-04-12T03:39:44.407 DEBUG:boto:Headers: {} 2024-04-12T03:39:44.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.407 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.407 DEBUG:boto:Params: {} 2024-04-12T03:39:44.407 DEBUG:boto:Token: None 2024-04-12T03:39:44.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/ 2024-04-12T03:39:44.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c= 2024-04-12T03:39:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EJx9vaiS3tOJi71gb/IR5Ukwh8c=', 'Content-Length': '0'} 2024-04-12T03:39:44.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1da2ba3ad689f17-006618ad00-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.411 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:44.411 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:44.411 DEBUG:boto:Method: PUT 2024-04-12T03:39:44.412 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:44.412 DEBUG:boto:Data: 2024-04-12T03:39:44.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:39:44.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:44.412 DEBUG:boto:Port: 8000 2024-04-12T03:39:44.412 DEBUG:boto:Params: {} 2024-04-12T03:39:44.412 DEBUG:boto:Token: None 2024-04-12T03:39:44.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:39:44 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:44.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OSPeITrWDaQbnBd0T0MXiRuPh5M= 2024-04-12T03:39:44.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OSPeITrWDaQbnBd0T0MXiRuPh5M='} 2024-04-12T03:39:44.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a2ad384fa9f9687-006618ad00-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:44.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:44.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:44.419 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:44.475 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:44.475 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:44.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.543+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.543+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.543+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 2 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.543+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000001.49.6 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 2 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 2 2024-04-12T03:39:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 2 2024-04-12T03:39:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.544+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000001.123.6 2024-04-12T03:39:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.545+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 2 2024-04-12T03:39:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.545+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.545+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000001.62.6 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.546+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000001.72.6 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000001.64.6 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 2 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000001.70.6 2024-04-12T03:39:44.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.547+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000001.38.6 2024-04-12T03:39:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.600+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.601+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.601+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.601+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.601+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3/obj00000000000000000020 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: entry->timestamp=2024-04-12T03:39:39.233340+0000 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.59.6 op=0 op_state=1 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000020[0] 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000001.59.6 2024-04-12T03:39:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.602+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000021 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:39.096257+0000 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.78.6 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000021 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000019 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:39.163376+0000 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-04-12T03:39:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000013 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:38.897374+0000 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000019[0] 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000001.78.6 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000013[0] 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.603+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.604+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0/obj00000000000000000007 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: entry->timestamp=2024-04-12T03:39:39.183177+0000 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000007[0] 2024-04-12T03:39:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.606+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: a sync operation returned error 2024-04-12T03:39:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:39:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000033 2024-04-12T03:39:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:38.908722+0000 2024-04-12T03:39:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.67.6 op=0 op_state=1 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000033[0] 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000001.67.6 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000004 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:39.243780+0000 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: backing out with sync_status=-5 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000004[0] 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000076 2024-04-12T03:39:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:39.017566+0000 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.76.6 op=0 op_state=1 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: : injecting data sync error on key=obj00000000000000000076 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000001.76.6 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000076 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13/obj00000000000000000026 2024-04-12T03:39:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: entry->timestamp=2024-04-12T03:39:39.173199+0000 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.50.6 op=0 op_state=1 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000026[0] 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.607+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.609+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.610+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.610+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000013]: success 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:1 marker=00000000002.69.6 timestamp=2024-04-12T03:39:38.897374+0000 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000019]: success 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:1 marker=00000000001.78.6 timestamp=2024-04-12T03:39:39.163376+0000 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000026]: success 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13:1 marker=00000000001.50.6 timestamp=2024-04-12T03:39:39.173199+0000 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000020]: success 2024-04-12T03:39:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.620+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3:1 marker=00000000001.59.6 timestamp=2024-04-12T03:39:39.233340+0000 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000024 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:38.967768+0000 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.69.6 op=0 op_state=1 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000024[0] 2024-04-12T03:39:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.640+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000001.69.6 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000028 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:39.106025+0000 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.36.6 op=0 op_state=1 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000028[0] 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000001.36.6 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000016 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:38.957766+0000 2024-04-12T03:39:44.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000001.127.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000032 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:39.153378+0000 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.127.6 op=0 op_state=1 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000032[0] 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000001.127.6 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.641+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000016 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000001 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:39.085084+0000 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.642+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000001 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8/obj00000000000000000010 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: entry->timestamp=2024-04-12T03:39:39.038140+0000 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000010[0] 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000000 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:39.027424+0000 2024-04-12T03:39:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:44.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:44.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.645+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.648+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000028]: success 2024-04-12T03:39:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.648+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4:1 marker=00000000001.36.6 timestamp=2024-04-12T03:39:39.106025+0000 2024-04-12T03:39:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.650+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: a sync operation returned error 2024-04-12T03:39:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.650+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: backing out with sync_status=-5 2024-04-12T03:39:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.650+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.650+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.650+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.651+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.651+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000021 2024-04-12T03:39:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:39.096257+0000 2024-04-12T03:39:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-04-12T03:39:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-12T03:39:44.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.78.6 2024-04-12T03:39:44.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:44.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000021 2024-04-12T03:39:44.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: success 2024-04-12T03:39:44.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.652+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:1 marker=00000000001.127.6 timestamp=2024-04-12T03:39:39.153378+0000 2024-04-12T03:39:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.658+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000004]: success 2024-04-12T03:39:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.658+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15:1 marker=00000000002.49.6 timestamp=2024-04-12T03:39:39.243780+0000 2024-04-12T03:39:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.659+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000033]: success 2024-04-12T03:39:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.659+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:1 marker=00000000001.67.6 timestamp=2024-04-12T03:39:38.908722+0000 2024-04-12T03:39:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.660+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000007]: success 2024-04-12T03:39:44.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.660+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0:1 marker=00000000001.41.6 timestamp=2024-04-12T03:39:39.183177+0000 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: a sync operation returned error 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: a sync operation returned error 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: backing out with sync_status=-5 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: backing out with sync_status=-5 2024-04-12T03:39:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.684+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.685+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.686+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000024]: success 2024-04-12T03:39:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.686+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:1 marker=00000000001.69.6 timestamp=2024-04-12T03:39:38.967768+0000 2024-04-12T03:39:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.691+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000010]: success 2024-04-12T03:39:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.691+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8:1 marker=00000000002.74.6 timestamp=2024-04-12T03:39:39.038140+0000 2024-04-12T03:39:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.692+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000000]: success 2024-04-12T03:39:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.692+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:1 marker=00000000002.68.6 timestamp=2024-04-12T03:39:39.027424+0000 2024-04-12T03:39:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-04-12T03:39:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000076 2024-04-12T03:39:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:39.017566+0000 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.76.6 op=0 op_state=1 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000076[0] 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.693+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000001.76.6 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.694+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: a sync operation returned error 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.694+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: backing out with sync_status=-5 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.694+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:44.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.694+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:44.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.696+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.696+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:44.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.696+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000001 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:39.085084+0000 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:44.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000016 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:38.957766+0000 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000016[0] 2024-04-12T03:39:44.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.728+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:44.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.734+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000016]: success 2024-04-12T03:39:44.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.734+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:1 marker=00000000001.41.6 timestamp=2024-04-12T03:39:38.957766+0000 2024-04-12T03:39:44.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.734+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000001]: success 2024-04-12T03:39:44.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.734+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:1 marker=00000000002.43.6 timestamp=2024-04-12T03:39:39.085084+0000 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000021 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:39.096257+0000 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000021[0] 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.78.6 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000076]: success 2024-04-12T03:39:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.738+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:1 marker=00000000001.76.6 timestamp=2024-04-12T03:39:39.017566+0000 2024-04-12T03:39:44.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.783+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000021]: success 2024-04-12T03:39:44.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.783+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11:1 marker=00000000001.78.6 timestamp=2024-04-12T03:39:39.096257+0000 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.788+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.788+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.788+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.788+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.788+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:44.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.789+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:44.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.790+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:44.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.791+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.792+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.793+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-12T03:39:44.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000025 2024-04-12T03:39:44.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:39.924070+0000 2024-04-12T03:39:44.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.63.6 op=0 op_state=1 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000016 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:39.622232+0000 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.29.6 op=0 op_state=1 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000030 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:39.612289+0000 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:39:44.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000015 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:39.794221+0000 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.28.6 op=0 op_state=1 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8/obj00000000000000000019 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: entry->timestamp=2024-04-12T03:39:39.601139+0000 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000025[0] 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000016[0] 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000002.29.6 2024-04-12T03:39:44.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000003 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:39.682833+0000 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000030[0] 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000015[0] 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000002.28.6 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000019[0] 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000009 2024-04-12T03:39:44.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:39.984050+0000 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.49.6 op=0 op_state=1 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000014 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:39.973767+0000 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.32.6 op=0 op_state=1 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000003[0] 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000009[0] 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000001.49.6 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000014[0] 2024-04-12T03:39:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.794+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.32.6 2024-04-12T03:39:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.795+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket shard: 0 2024-04-12T03:39:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.795+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18/obj00000000000000000007 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: entry->timestamp=2024-04-12T03:39:39.741309+0000 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000007[0] 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000019]: success 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8:2 marker=00000000002.47.6 timestamp=2024-04-12T03:39:39.601139+0000 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000016]: success 2024-04-12T03:39:44.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.799+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:2 marker=00000000002.29.6 timestamp=2024-04-12T03:39:39.622232+0000 2024-04-12T03:39:44.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.802+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000007]: success 2024-04-12T03:39:44.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.802+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18:2 marker=00000000002.46.6 timestamp=2024-04-12T03:39:39.741309+0000 2024-04-12T03:39:44.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.807+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000014]: success 2024-04-12T03:39:44.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.807+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11:2 marker=00000000001.32.6 timestamp=2024-04-12T03:39:39.973767+0000 2024-04-12T03:39:44.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.807+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000025]: success 2024-04-12T03:39:44.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.807+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:2 marker=00000000001.63.6 timestamp=2024-04-12T03:39:39.924070+0000 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000009]: success 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:2 marker=00000000001.49.6 timestamp=2024-04-12T03:39:39.984050+0000 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000015]: success 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:2 marker=00000000002.28.6 timestamp=2024-04-12T03:39:39.794221+0000 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000003]: success 2024-04-12T03:39:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.809+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:2 marker=00000000002.53.6 timestamp=2024-04-12T03:39:39.682833+0000 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000010 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:40.033778+0000 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000010[0] 2024-04-12T03:39:44.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.831+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0/obj00000000000000000005 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: entry->timestamp=2024-04-12T03:39:39.730195+0000 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000005[0] 2024-04-12T03:39:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.832+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-12T03:39:44.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.833+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000010]: success 2024-04-12T03:39:44.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.833+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:2 marker=00000000002.30.6 timestamp=2024-04-12T03:39:40.033778+0000 2024-04-12T03:39:44.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:44.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12/obj00000000000000000012 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: entry->timestamp=2024-04-12T03:39:39.804106+0000 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000012[0] 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13/obj00000000000000000002 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: entry->timestamp=2024-04-12T03:39:39.669799+0000 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.79.6 op=0 op_state=1 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000002[0] 2024-04-12T03:39:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.835+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position=00000000002.79.6 2024-04-12T03:39:44.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000000 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:39.854273+0000 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.32.6 op=0 op_state=1 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.836+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000002.32.6 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000006 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:39.553820+0000 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-12T03:39:44.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000006[0] 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3/obj00000000000000000021 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: entry->timestamp=2024-04-12T03:39:39.865730+0000 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000001 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:39.542730+0000 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000021[0] 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:39:44.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.837+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-12T03:39:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.839+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000000]: success 2024-04-12T03:39:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.839+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:2 marker=00000000002.32.6 timestamp=2024-04-12T03:39:39.854273+0000 2024-04-12T03:39:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.839+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000021]: success 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.839+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3:2 marker=00000000002.47.6 timestamp=2024-04-12T03:39:39.865730+0000 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: success 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17:2 marker=00000000002.56.6 timestamp=2024-04-12T03:39:39.542730+0000 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000011 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:39.913928+0000 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.54.6 op=0 op_state=1 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000011[0] 2024-04-12T03:39:44.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.840+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000001.54.6 2024-04-12T03:39:44.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.843+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000030]: success 2024-04-12T03:39:44.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.843+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:2 marker=00000000002.51.6 timestamp=2024-04-12T03:39:39.612289+0000 2024-04-12T03:39:44.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.877+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000005]: success 2024-04-12T03:39:44.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.877+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0:2 marker=00000000002.58.6 timestamp=2024-04-12T03:39:39.730195+0000 2024-04-12T03:39:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.881+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000012]: success 2024-04-12T03:39:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.882+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12:2 marker=00000000002.41.6 timestamp=2024-04-12T03:39:39.804106+0000 2024-04-12T03:39:44.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.882+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000002]: success 2024-04-12T03:39:44.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.882+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13:2 marker=00000000002.79.6 timestamp=2024-04-12T03:39:39.669799+0000 2024-04-12T03:39:44.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.887+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000011]: success 2024-04-12T03:39:44.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.887+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:2 marker=00000000001.54.6 timestamp=2024-04-12T03:39:39.913928+0000 2024-04-12T03:39:44.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.888+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000006]: success 2024-04-12T03:39:44.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.888+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15:2 marker=00000000002.45.6 timestamp=2024-04-12T03:39:39.553820+0000 2024-04-12T03:39:44.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.906+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.906+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.906+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.906+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.907+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.908+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.909+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.910+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.910+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.910+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.910+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8/obj00000000000000000039 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: entry->timestamp=2024-04-12T03:39:40.837649+0000 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.68.6 op=0 op_state=1 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000039[0] 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000001.68.6 2024-04-12T03:39:44.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket shard: 0 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket shard: 0 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.911+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12/obj00000000000000000009 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: entry->timestamp=2024-04-12T03:39:40.666144+0000 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000009[0] 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.912+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket shard: 0 2024-04-12T03:39:44.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket shard: 0 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.913+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000033 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:40.714159+0000 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000013 2024-04-12T03:39:44.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:40.293431+0000 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000033[0] 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000013[0] 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000001 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:40.725414+0000 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.914+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:44.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000007 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:40.365450+0000 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000007[0] 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21/obj00000000000000000055 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: entry->timestamp=2024-04-12T03:39:40.414311+0000 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.48.6 op=0 op_state=1 2024-04-12T03:39:44.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000055[0] 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position=00000000001.48.6 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000003 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:40.353878+0000 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000003[0] 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000009]: success 2024-04-12T03:39:44.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.915+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12:3 marker=00000000002.55.6 timestamp=2024-04-12T03:39:40.666144+0000 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.918+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000033]: success 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.918+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:3 marker=00000000002.54.6 timestamp=2024-04-12T03:39:40.714159+0000 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.918+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000013]: success 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.918+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:3 marker=00000000002.74.6 timestamp=2024-04-12T03:39:40.293431+0000 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.919+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000001]: success 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.919+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17:3 marker=00000000002.41.6 timestamp=2024-04-12T03:39:40.725414+0000 2024-04-12T03:39:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.920+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000003]: success 2024-04-12T03:39:44.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.920+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20:3 marker=00000000002.64.6 timestamp=2024-04-12T03:39:40.353878+0000 2024-04-12T03:39:44.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.920+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000007]: success 2024-04-12T03:39:44.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.920+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:3 marker=00000000002.46.6 timestamp=2024-04-12T03:39:40.365450+0000 2024-04-12T03:39:44.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:39:44.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000014 2024-04-12T03:39:44.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:40.787277+0000 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000014[0] 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000026 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:40.424705+0000 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3/obj00000000000000000046 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: entry->timestamp=2024-04-12T03:39:40.898206+0000 2024-04-12T03:39:44.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: parsed entry: id=00000000001.84.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0/obj00000000000000000058 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: entry->timestamp=2024-04-12T03:39:40.484783+0000 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.84.6 op=0 op_state=1 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000011 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:40.848190+0000 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-12T03:39:44.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000032 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:40.534134+0000 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000026[0] 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000046[0] 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000001.71.6 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000058[0] 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000001.84.6 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000011[0] 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-12T03:39:44.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000032[0] 2024-04-12T03:39:44.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.950+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-12T03:39:44.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-12T03:39:44.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000050 2024-04-12T03:39:44.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:40.593619+0000 2024-04-12T03:39:44.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000002 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:40.603652+0000 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000050[0] 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000002[0] 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.951+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.132.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000006 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:40.304810+0000 2024-04-12T03:39:44.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.132.6 op=0 op_state=1 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000006[0] 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.953+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.132.6 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000040 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:40.777238+0000 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.76.6 op=0 op_state=1 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000040[0] 2024-04-12T03:39:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.954+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.76.6 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13/obj00000000000000000045 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: entry->timestamp=2024-04-12T03:39:40.656292+0000 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.50.6 op=0 op_state=1 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000045[0] 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-12T03:39:44.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:3 marker=00000000002.58.6 timestamp=2024-04-12T03:39:40.848190+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000026]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:3 marker=00000000002.68.6 timestamp=2024-04-12T03:39:40.424705+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000014]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4:3 marker=00000000002.35.6 timestamp=2024-04-12T03:39:40.787277+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000002]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.955+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:3 marker=00000000002.77.6 timestamp=2024-04-12T03:39:40.603652+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000032]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:3 marker=00000000002.50.6 timestamp=2024-04-12T03:39:40.534134+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000006]: success 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:3 marker=00000000002.132.6 timestamp=2024-04-12T03:39:40.304810+0000 2024-04-12T03:39:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18/obj00000000000000000000 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: entry->timestamp=2024-04-12T03:39:40.545705+0000 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.956+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position=00000000002.52.6 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22/obj00000000000000000008 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: entry->timestamp=2024-04-12T03:39:40.473833+0000 2024-04-12T03:39:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:39:44.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000008[0] 2024-04-12T03:39:44.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.957+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:39:44.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.960+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000000]: success 2024-04-12T03:39:44.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.960+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18:3 marker=00000000002.52.6 timestamp=2024-04-12T03:39:40.545705+0000 2024-04-12T03:39:44.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.963+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000050]: success 2024-04-12T03:39:44.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.963+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:3 marker=00000000001.56.6 timestamp=2024-04-12T03:39:40.593619+0000 2024-04-12T03:39:44.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.964+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000058]: success 2024-04-12T03:39:44.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.964+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0:3 marker=00000000001.84.6 timestamp=2024-04-12T03:39:40.484783+0000 2024-04-12T03:39:44.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.964+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000055]: success 2024-04-12T03:39:44.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.964+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21:3 marker=00000000001.48.6 timestamp=2024-04-12T03:39:40.414311+0000 2024-04-12T03:39:44.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.966+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000039]: success 2024-04-12T03:39:44.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:44.966+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8:3 marker=00000000001.68.6 timestamp=2024-04-12T03:39:40.837649+0000 2024-04-12T03:39:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.000+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000046]: success 2024-04-12T03:39:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.000+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3:3 marker=00000000001.71.6 timestamp=2024-04-12T03:39:40.898206+0000 2024-04-12T03:39:45.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.005+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000008]: success 2024-04-12T03:39:45.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.006+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22:3 marker=00000000002.37.6 timestamp=2024-04-12T03:39:40.473833+0000 2024-04-12T03:39:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.006+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000040]: success 2024-04-12T03:39:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.006+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11:3 marker=00000000001.76.6 timestamp=2024-04-12T03:39:40.777238+0000 2024-04-12T03:39:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.006+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000045]: success 2024-04-12T03:39:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.006+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13:3 marker=00000000001.50.6 timestamp=2024-04-12T03:39:40.656292+0000 2024-04-12T03:39:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.029+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:45.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.030+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:45.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:45.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.031+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:45.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.032+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.033+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket shard: 0 2024-04-12T03:39:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket shard: 0 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.034+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.035+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket shard: 0 2024-04-12T03:39:45.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.035+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:39:45.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000005 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:41.285629+0000 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.036+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.037+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000005[0] 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.037+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.038+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket shard: 0 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.038+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.038+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket shard: 0 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.038+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000012 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:41.468039+0000 2024-04-12T03:39:45.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18/obj00000000000000000003 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: entry->timestamp=2024-04-12T03:39:41.398595+0000 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19/obj00000000000000000042 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: entry->timestamp=2024-04-12T03:39:41.528174+0000 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.53.6 op=0 op_state=1 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000010 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:41.334991+0000 2024-04-12T03:39:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22/obj00000000000000000009 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: entry->timestamp=2024-04-12T03:39:41.877729+0000 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000003[0] 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000042[0] 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position=00000000001.53.6 2024-04-12T03:39:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000009[0] 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000012 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.039+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000010 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000027 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:41.517996+0000 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.62.6 op=0 op_state=1 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: : injecting data sync error on key=obj00000000000000000027 2024-04-12T03:39:45.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000001.62.6 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000005]: success 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:4 marker=00000000002.62.6 timestamp=2024-04-12T03:39:41.285629+0000 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000027 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000001.75.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000059 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.041+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:41.578039+0000 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.042+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.75.6 op=0 op_state=1 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.042+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000059[0] 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.042+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000001.75.6 2024-04-12T03:39:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: a sync operation returned error 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: backing out with sync_status=-5 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28/obj00000000000000000021 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: entry->timestamp=2024-04-12T03:39:41.828647+0000 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000021[0] 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-12T03:39:45.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: a sync operation returned error 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.043+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: backing out with sync_status=-5 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26/obj00000000000000000014 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: entry->timestamp=2024-04-12T03:39:41.758148+0000 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-12T03:39:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000014[0] 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.044+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.045+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000009]: success 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.045+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22:4 marker=00000000002.46.6 timestamp=2024-04-12T03:39:41.877729+0000 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.045+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.045+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000003]: success 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.045+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18:4 marker=00000000002.84.6 timestamp=2024-04-12T03:39:41.398595+0000 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.046+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.046+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.048+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000014]: success 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.048+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26:4 marker=00000000002.59.6 timestamp=2024-04-12T03:39:41.758148+0000 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.048+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000021]: success 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.048+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28:4 marker=00000000002.74.6 timestamp=2024-04-12T03:39:41.828647+0000 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.050+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000059]: success 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.050+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:4 marker=00000000001.75.6 timestamp=2024-04-12T03:39:41.578039+0000 2024-04-12T03:39:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.051+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000042]: success 2024-04-12T03:39:45.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.051+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19:4 marker=00000000001.53.6 timestamp=2024-04-12T03:39:41.528174+0000 2024-04-12T03:39:45.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:39:45.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000011 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:41.410257+0000 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000011[0] 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3/obj00000000000000000004 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: entry->timestamp=2024-04-12T03:39:41.889145+0000 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000004[0] 2024-04-12T03:39:45.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.072+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000002.86.6 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000002 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:41.588099+0000 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000002[0] 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000019 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:41.937266+0000 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.21.6 op=0 op_state=1 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000008 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:41.273623+0000 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.21.6 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000008[0] 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-12T03:39:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.073+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000019 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000031 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:41.214019+0000 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.39.6 op=0 op_state=1 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000031[0] 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000015 2024-04-12T03:39:45.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:41.165858+0000 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000015[0] 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000011]: success 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4:4 marker=00000000002.69.6 timestamp=2024-04-12T03:39:41.410257+0000 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000004]: success 2024-04-12T03:39:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.074+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3:4 marker=00000000002.86.6 timestamp=2024-04-12T03:39:41.889145+0000 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000049 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:41.817901+0000 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: : injecting data sync error on key=obj00000000000000000049 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-12T03:39:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12/obj00000000000000000022 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: entry->timestamp=2024-04-12T03:39:41.224838+0000 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000020 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:41.769840+0000 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.075+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000049 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000022[0] 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-12T03:39:45.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000020[0] 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.076+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8/obj00000000000000000017 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: entry->timestamp=2024-04-12T03:39:41.347233+0000 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000017[0] 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-04-12T03:39:45.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000037 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:41.458188+0000 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.73.6 op=0 op_state=1 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000037[0] 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.077+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000001.73.6 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000002]: success 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:4 marker=00000000002.53.6 timestamp=2024-04-12T03:39:41.588099+0000 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000015]: success 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:4 marker=00000000002.66.6 timestamp=2024-04-12T03:39:41.165858+0000 2024-04-12T03:39:45.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000008]: success 2024-04-12T03:39:45.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.078+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:4 marker=00000000002.60.6 timestamp=2024-04-12T03:39:41.273623+0000 2024-04-12T03:39:45.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:45.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21/obj00000000000000000001 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: entry->timestamp=2024-04-12T03:39:41.637573+0000 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.080+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24/obj00000000000000000016 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: entry->timestamp=2024-04-12T03:39:41.649168+0000 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-12T03:39:45.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23/obj00000000000000000025 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: entry->timestamp=2024-04-12T03:39:41.949191+0000 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000020]: success 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:4 marker=00000000002.57.6 timestamp=2024-04-12T03:39:41.769840+0000 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27/obj00000000000000000030 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: entry->timestamp=2024-04-12T03:39:41.709719+0000 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.89.6 op=0 op_state=1 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:45.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25/obj00000000000000000000 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: entry->timestamp=2024-04-12T03:39:41.697714+0000 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000001]: success 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21:4 marker=00000000002.84.6 timestamp=2024-04-12T03:39:41.637573+0000 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000016[0] 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000025[0] 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000030[0] 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position=00000000002.89.6 2024-04-12T03:39:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:45.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.083+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-12T03:39:45.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000022]: success 2024-04-12T03:39:45.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12:4 marker=00000000002.57.6 timestamp=2024-04-12T03:39:41.224838+0000 2024-04-12T03:39:45.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000025]: success 2024-04-12T03:39:45.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23:4 marker=00000000002.84.6 timestamp=2024-04-12T03:39:41.949191+0000 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000016]: success 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.086+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24:4 marker=00000000002.55.6 timestamp=2024-04-12T03:39:41.649168+0000 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000000]: success 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25:4 marker=00000000002.55.6 timestamp=2024-04-12T03:39:41.697714+0000 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000012 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:41.468039+0000 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: a sync operation returned error 2024-04-12T03:39:45.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000030]: success 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27:4 marker=00000000002.89.6 timestamp=2024-04-12T03:39:41.709719+0000 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000012[0] 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: backing out with sync_status=-5 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.087+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000010 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:41.334991+0000 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:45.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000010[0] 2024-04-12T03:39:45.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:45.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.089+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000012]: success 2024-04-12T03:39:45.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17:4 marker=00000000002.83.6 timestamp=2024-04-12T03:39:41.468039+0000 2024-04-12T03:39:45.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000010]: success 2024-04-12T03:39:45.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20:4 marker=00000000002.68.6 timestamp=2024-04-12T03:39:41.334991+0000 2024-04-12T03:39:45.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:45.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.091+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.120+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: a sync operation returned error 2024-04-12T03:39:45.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.120+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: backing out with sync_status=-5 2024-04-12T03:39:45.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.120+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.120+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.121+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.121+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:45.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.121+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.122+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: a sync operation returned error 2024-04-12T03:39:45.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.122+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: backing out with sync_status=-5 2024-04-12T03:39:45.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.122+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.122+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.123+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000031]: success 2024-04-12T03:39:45.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.123+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11:4 marker=00000000001.39.6 timestamp=2024-04-12T03:39:41.214019+0000 2024-04-12T03:39:45.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.124+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.124+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:45.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.124+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.128+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000037]: success 2024-04-12T03:39:45.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.128+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15:4 marker=00000000001.73.6 timestamp=2024-04-12T03:39:41.458188+0000 2024-04-12T03:39:45.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.130+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000017]: success 2024-04-12T03:39:45.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.130+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8:4 marker=00000000001.41.6 timestamp=2024-04-12T03:39:41.347233+0000 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000027 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:41.517996+0000 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.62.6 op=0 op_state=1 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000027[0] 2024-04-12T03:39:45.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.133+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000001.62.6 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000019 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:41.937266+0000 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.21.6 op=0 op_state=1 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000019[0] 2024-04-12T03:39:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.164+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.21.6 2024-04-12T03:39:45.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.165+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000019]: success 2024-04-12T03:39:45.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.165+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:4 marker=00000000002.21.6 timestamp=2024-04-12T03:39:41.937266+0000 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000049 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:41.817901+0000 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000049[0] 2024-04-12T03:39:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.168+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:45.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.179+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000027]: success 2024-04-12T03:39:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.179+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:4 marker=00000000001.62.6 timestamp=2024-04-12T03:39:41.517996+0000 2024-04-12T03:39:45.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.212+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000049]: success 2024-04-12T03:39:45.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.212+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:4 marker=00000000001.41.6 timestamp=2024-04-12T03:39:41.817901+0000 2024-04-12T03:39:45.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.217+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.217+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.217+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.218+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.219+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:45.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.220+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000004 2024-04-12T03:39:45.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:42.465954+0000 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.79.6 op=0 op_state=1 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000317 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:42.903162+0000 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.67.6 op=0 op_state=1 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000004[0] 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000002.79.6 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000317[0] 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000001.67.6 2024-04-12T03:39:45.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000467 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:42.586205+0000 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.37.6 op=0 op_state=1 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000467[0] 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.221+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000056 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:42.537801+0000 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000001.88.6 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.223+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000056 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12/obj00000000000000000271 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: entry->timestamp=2024-04-12T03:39:42.777368+0000 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000271[0] 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.224+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-12T03:39:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket shard: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket shard: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket shard: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000023 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:42.402445+0000 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000023[0] 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.225+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-12T03:39:45.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.226+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000004]: success 2024-04-12T03:39:45.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.226+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:5 marker=00000000002.79.6 timestamp=2024-04-12T03:39:42.465954+0000 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket shard: 0 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket shard: 0 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket shard: 0 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket shard: 0 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.227+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-12T03:39:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000018 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:42.705503+0000 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000018[0] 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18/obj00000000000000000016 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: entry->timestamp=2024-04-12T03:39:43.013365+0000 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.89.6 op=0 op_state=1 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000016[0] 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position=00000000002.89.6 2024-04-12T03:39:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: sync status for source bucket shard: 0 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: sync status for source bucket shard: 0 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket shard: 0 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket shard: 0 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.228+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23/obj00000000000000000001 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: entry->timestamp=2024-04-12T03:39:42.827263+0000 2024-04-12T03:39:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.42.6 op=0 op_state=1 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22/obj00000000000000000012 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: entry->timestamp=2024-04-12T03:39:42.230351+0000 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21/obj00000000000000000002 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: entry->timestamp=2024-04-12T03:39:42.765991+0000 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position=00000000002.42.6 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2024-04-12T03:39:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000002[0] 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23/obj00000000000000000001 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.230+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22/obj00000000000000000012 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.231+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: a sync operation returned error 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.231+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: backing out with sync_status=-5 2024-04-12T03:39:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.231+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.231+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000017 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:42.337899+0000 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000006 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:42.289806+0000 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-12T03:39:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000006[0] 2024-04-12T03:39:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-12T03:39:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000017 2024-04-12T03:39:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000016]: success 2024-04-12T03:39:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.232+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18:5 marker=00000000002.89.6 timestamp=2024-04-12T03:39:43.013365+0000 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30/obj00000000000000000133 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: entry->timestamp=2024-04-12T03:39:42.415930+0000 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: parsed entry: id=00000000001.92.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27/obj00000000000000000061 2024-04-12T03:39:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: entry->timestamp=2024-04-12T03:39:42.715792+0000 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.92.6 op=0 op_state=1 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000133[0] 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: : injecting data sync error on key=obj00000000000000000061 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position=00000000001.92.6 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.233+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27/obj00000000000000000061 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.236+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000006]: success 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.236+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:5 marker=00000000002.66.6 timestamp=2024-04-12T03:39:42.289806+0000 2024-04-12T03:39:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.237+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.237+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000467]: success 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.237+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:5 marker=00000000001.37.6 timestamp=2024-04-12T03:39:42.586205+0000 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.238+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.238+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.240+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:entry[obj00000000000000000002]: success 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.240+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21:5 marker=00000000002.83.6 timestamp=2024-04-12T03:39:42.765991+0000 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.240+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000317]: success 2024-04-12T03:39:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.240+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:5 marker=00000000001.67.6 timestamp=2024-04-12T03:39:42.903162+0000 2024-04-12T03:39:45.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.243+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000018]: success 2024-04-12T03:39:45.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.243+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:5 marker=00000000002.68.6 timestamp=2024-04-12T03:39:42.705503+0000 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0/obj00000000000000000019 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: entry->timestamp=2024-04-12T03:39:42.596893+0000 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000019[0] 2024-04-12T03:39:45.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.260+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-12T03:39:45.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.261+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000019]: success 2024-04-12T03:39:45.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.261+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0:5 marker=00000000002.77.6 timestamp=2024-04-12T03:39:42.596893+0000 2024-04-12T03:39:45.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:45.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000000 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:42.525883+0000 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000021 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:42.838808+0000 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.263+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000021 2024-04-12T03:39:45.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.265+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000000]: success 2024-04-12T03:39:45.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.265+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:5 marker=00000000002.43.6 timestamp=2024-04-12T03:39:42.525883+0000 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11/obj00000000000000000009 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: entry->timestamp=2024-04-12T03:39:42.656752+0000 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000009[0] 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:39:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000007 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:43.024745+0000 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.267+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000007 2024-04-12T03:39:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.269+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:entry[obj00000000000000000009]: success 2024-04-12T03:39:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.269+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11:5 marker=00000000002.37.6 timestamp=2024-04-12T03:39:42.656752+0000 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: parsed entry: id=00000000002.27.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13/obj00000000000000000008 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: entry->timestamp=2024-04-12T03:39:42.477694+0000 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.27.6 op=0 op_state=1 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000008[0] 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position=00000000002.27.6 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:39:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:42.349885+0000 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.25.6 op=0 op_state=1 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000001.25.6 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.270+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000271]: success 2024-04-12T03:39:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12:5 marker=00000000001.45.6 timestamp=2024-04-12T03:39:42.777368+0000 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: parsed entry: id=00000000002.137.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24/obj00000000000000000011 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: entry->timestamp=2024-04-12T03:39:42.965338+0000 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.137.6 op=0 op_state=1 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000011[0] 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position=00000000002.137.6 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000008]: success 2024-04-12T03:39:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.272+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13:5 marker=00000000002.27.6 timestamp=2024-04-12T03:39:42.477694+0000 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28/obj00000000000000000050 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: entry->timestamp=2024-04-12T03:39:42.889863+0000 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000050[0] 2024-04-12T03:39:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.273+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000162 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:42.646091+0000 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.73.6 op=0 op_state=1 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000162[0] 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000001.73.6 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26/obj00000000000000000010 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: entry->timestamp=2024-04-12T03:39:42.277937+0000 2024-04-12T03:39:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19/obj00000000000000000003 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: entry->timestamp=2024-04-12T03:39:42.954071+0000 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.79.6 op=0 op_state=1 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000010[0] 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000003[0] 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position=00000000002.79.6 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000011]: success 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.274+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24:5 marker=00000000002.137.6 timestamp=2024-04-12T03:39:42.965338+0000 2024-04-12T03:39:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.275+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: a sync operation returned error 2024-04-12T03:39:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.275+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: backing out with sync_status=-5 2024-04-12T03:39:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.275+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.275+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.276+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket shard: 0 2024-04-12T03:39:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000050]: success 2024-04-12T03:39:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28:5 marker=00000000002.39.6 timestamp=2024-04-12T03:39:42.889863+0000 2024-04-12T03:39:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000010]: success 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.277+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26:5 marker=00000000002.81.6 timestamp=2024-04-12T03:39:42.277937+0000 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25/obj00000000000000000283 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: entry->timestamp=2024-04-12T03:39:43.073811+0000 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000283[0] 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position=00000000001.94.6 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: a sync operation returned error 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: a sync operation returned error 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: a sync operation returned error 2024-04-12T03:39:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: backing out with sync_status=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: backing out with sync_status=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: backing out with sync_status=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000003]: success 2024-04-12T03:39:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.278+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19:5 marker=00000000002.79.6 timestamp=2024-04-12T03:39:42.954071+0000 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.281+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.281+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.281+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000056 2024-04-12T03:39:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:42.537801+0000 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000056[0] 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000001.88.6 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000023]: success 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:5 marker=00000000002.46.6 timestamp=2024-04-12T03:39:42.402445+0000 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket shard: 0 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.282+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.284+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket shard: 0 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.284+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.284+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000133]: success 2024-04-12T03:39:45.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.284+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30:5 marker=00000000001.41.6 timestamp=2024-04-12T03:39:42.415930+0000 2024-04-12T03:39:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: a sync operation returned error 2024-04-12T03:39:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: backing out with sync_status=-5 2024-04-12T03:39:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.307+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: a sync operation returned error 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: backing out with sync_status=-5 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:45.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.311+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.312+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: a sync operation returned error 2024-04-12T03:39:45.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.312+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: backing out with sync_status=-5 2024-04-12T03:39:45.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.312+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.312+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.312+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.313+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.313+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22/obj00000000000000000012 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: entry->timestamp=2024-04-12T03:39:42.230351+0000 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000012[0] 2024-04-12T03:39:45.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.319+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-12T03:39:45.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.321+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:entry[obj00000000000000000012]: success 2024-04-12T03:39:45.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.321+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22:5 marker=00000000002.64.6 timestamp=2024-04-12T03:39:42.230351+0000 2024-04-12T03:39:45.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.323+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000162]: success 2024-04-12T03:39:45.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.323+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17:5 marker=00000000001.73.6 timestamp=2024-04-12T03:39:42.646091+0000 2024-04-12T03:39:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:39:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000017 2024-04-12T03:39:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:42.337899+0000 2024-04-12T03:39:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000017[0] 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23/obj00000000000000000001 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: entry->timestamp=2024-04-12T03:39:42.827263+0000 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.42.6 op=0 op_state=1 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.325+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position=00000000002.42.6 2024-04-12T03:39:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: parsed entry: id=00000000001.92.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-04-12T03:39:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27/obj00000000000000000061 2024-04-12T03:39:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: entry->timestamp=2024-04-12T03:39:42.715792+0000 2024-04-12T03:39:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.92.6 op=0 op_state=1 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000061[0] 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position=00000000001.92.6 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000283]: success 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.327+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25:5 marker=00000000001.94.6 timestamp=2024-04-12T03:39:43.073811+0000 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.328+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000017]: success 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.328+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:5 marker=00000000002.49.6 timestamp=2024-04-12T03:39:42.337899+0000 2024-04-12T03:39:45.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.329+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000001]: success 2024-04-12T03:39:45.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.329+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23:5 marker=00000000002.42.6 timestamp=2024-04-12T03:39:42.827263+0000 2024-04-12T03:39:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.332+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000056]: success 2024-04-12T03:39:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.332+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4:5 marker=00000000001.88.6 timestamp=2024-04-12T03:39:42.537801+0000 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000021 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:42.838808+0000 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000021[0] 2024-04-12T03:39:45.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.349+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-12T03:39:45.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.350+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000021]: success 2024-04-12T03:39:45.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.350+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:5 marker=00000000002.39.6 timestamp=2024-04-12T03:39:42.838808+0000 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000007 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:43.024745+0000 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000007[0] 2024-04-12T03:39:45.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.353+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:39:45.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.354+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000007]: success 2024-04-12T03:39:45.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.354+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15:5 marker=00000000002.68.6 timestamp=2024-04-12T03:39:43.024745+0000 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:42.349885+0000 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.25.6 op=0 op_state=1 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000001.25.6 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.355+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.373+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000061]: success 2024-04-12T03:39:45.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.373+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27:5 marker=00000000001.92.6 timestamp=2024-04-12T03:39:42.715792+0000 2024-04-12T03:39:45.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: a sync operation returned error 2024-04-12T03:39:45.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: backing out with sync_status=-5 2024-04-12T03:39:45.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.397+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:42.349885+0000 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.25.6 op=0 op_state=1 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000001.25.6 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.440+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.482+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: a sync operation returned error 2024-04-12T03:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.482+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: backing out with sync_status=-5 2024-04-12T03:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.482+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.482+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.483+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.483+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.483+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:39:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000423 2024-04-12T03:39:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:42.349885+0000 2024-04-12T03:39:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.25.6 op=0 op_state=1 2024-04-12T03:39:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000423[0] 2024-04-12T03:39:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.525+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000001.25.6 2024-04-12T03:39:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.530+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000423]: success 2024-04-12T03:39:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.530+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20:5 marker=00000000001.25.6 timestamp=2024-04-12T03:39:42.349885+0000 2024-04-12T03:39:45.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.571+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.571+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: sync status for source bucket shard: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: sync status for source bucket shard: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: sync status for source bucket shard: 0 2024-04-12T03:39:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.572+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: sync status for source bucket shard: 0 2024-04-12T03:39:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: sync status for source bucket shard: 0 2024-04-12T03:39:45.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: sync status for source bucket shard: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.573+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: sync status for source bucket shard: 0 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: sync status for source bucket shard: 0 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: sync status for source bucket shard: 0 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5/obj00000000000000000065 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: entry->timestamp=2024-04-12T03:39:43.412402+0000 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.35.6 op=0 op_state=1 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000065[0] 2024-04-12T03:39:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.574+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: listing bilog for incremental sync; position=00000000001.35.6 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: sync status for source bucket shard: 0 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3/obj00000000000000000001 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: entry->timestamp=2024-04-12T03:39:43.653960+0000 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000001[0] 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-12T03:39:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4/obj00000000000000000033 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: entry->timestamp=2024-04-12T03:39:43.642231+0000 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.90.6 op=0 op_state=1 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000033[0] 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: listing bilog for incremental sync; position=00000000002.90.6 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: sync status for source bucket shard: 0 2024-04-12T03:39:45.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.575+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: sync status for source bucket shard: 0 2024-04-12T03:39:45.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: parsed entry: id=00000000002.148.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7/obj00000000000000000016 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: entry->timestamp=2024-04-12T03:39:43.867458+0000 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.148.6 op=0 op_state=1 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000016[0] 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: listing bilog for incremental sync; position=00000000002.148.6 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: sync status for source bucket shard: 0 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.577+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: sync status for source bucket shard: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6/obj00000000000000000005 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: entry->timestamp=2024-04-12T03:39:43.702263+0000 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000005[0] 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.578+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: sync status for source bucket shard: 0 2024-04-12T03:39:45.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: sync status for source bucket shard: 0 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.579+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.580+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.580+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:entry[obj00000000000000000001]: success 2024-04-12T03:39:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.580+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:3:6 marker=00000000002.81.6 timestamp=2024-04-12T03:39:43.653960+0000 2024-04-12T03:39:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: sync status for source bucket shard: 0 2024-04-12T03:39:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: sync status for source bucket shard: 0 2024-04-12T03:39:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: sync status for source bucket shard: 0 2024-04-12T03:39:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: sync status for source bucket shard: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: sync status for source bucket shard: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: sync status for source bucket shard: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: sync status for source bucket shard: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: sync status for source bucket shard: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.581+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12/obj00000000000000000062 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: entry->timestamp=2024-04-12T03:39:43.401681+0000 2024-04-12T03:39:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.87.6 op=0 op_state=1 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13/obj00000000000000000009 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: entry->timestamp=2024-04-12T03:39:44.365579+0000 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16/obj00000000000000000012 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: entry->timestamp=2024-04-12T03:39:43.713870+0000 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000062[0] 2024-04-12T03:39:45.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: listing bilog for incremental sync; position=00000000001.87.6 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000009[0] 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000012[0] 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.582+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: listing bilog for incremental sync; position=00000000002.92.6 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.583+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:entry[obj00000000000000000033]: success 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.583+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:4:6 marker=00000000002.90.6 timestamp=2024-04-12T03:39:43.642231+0000 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.583+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:entry[obj00000000000000000005]: success 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.583+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:6:6 marker=00000000002.45.6 timestamp=2024-04-12T03:39:43.702263+0000 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.584+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: sync status for source bucket shard: 0 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.584+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.584+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: sync status for source bucket shard: 0 2024-04-12T03:39:45.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.584+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: sync status for source bucket shard: 0 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: sync status for source bucket shard: 0 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: sync status for source bucket shard: 0 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27/obj00000000000000000053 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: entry->timestamp=2024-04-12T03:39:43.594276+0000 2024-04-12T03:39:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000053[0] 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: listing bilog for incremental sync; position=00000000002.94.6 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14/obj00000000000000000040 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: entry->timestamp=2024-04-12T03:39:44.046701+0000 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31/obj00000000000000000003 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: entry->timestamp=2024-04-12T03:39:43.582269+0000 2024-04-12T03:39:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2/obj00000000000000000011 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: entry->timestamp=2024-04-12T03:39:43.762337+0000 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.78.6 op=0 op_state=1 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000040[0] 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: listing bilog for incremental sync; position=00000000002.98.6 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000003[0] 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000011[0] 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.587+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: listing bilog for incremental sync; position=00000000002.78.6 2024-04-12T03:39:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28/obj00000000000000000020 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: entry->timestamp=2024-04-12T03:39:44.306035+0000 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29/obj00000000000000000004 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: entry->timestamp=2024-04-12T03:39:43.534085+0000 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000020[0] 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000004[0] 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-12T03:39:45.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:entry[obj00000000000000000016]: success 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:7:6 marker=00000000002.148.6 timestamp=2024-04-12T03:39:43.867458+0000 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:entry[obj00000000000000000012]: success 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:16:6 marker=00000000002.92.6 timestamp=2024-04-12T03:39:43.713870+0000 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.588+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30/obj00000000000000000036 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: entry->timestamp=2024-04-12T03:39:43.354674+0000 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000036[0] 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: listing bilog for incremental sync; position=00000000001.86.6 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:39:45.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000015 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:44.062376+0000 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: : injecting data sync error on key=obj00000000000000000015 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.589+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000015 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20/obj00000000000000000006 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: entry->timestamp=2024-04-12T03:39:43.929551+0000 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:39:45.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000006[0] 2024-04-12T03:39:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:39:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:entry[obj00000000000000000009]: success 2024-04-12T03:39:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.592+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:13:6 marker=00000000002.53.6 timestamp=2024-04-12T03:39:44.365579+0000 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:entry[obj00000000000000000040]: success 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:14:6 marker=00000000002.98.6 timestamp=2024-04-12T03:39:44.046701+0000 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:entry[obj00000000000000000003]: success 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:31:6 marker=00000000002.96.6 timestamp=2024-04-12T03:39:43.582269+0000 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:entry[obj00000000000000000011]: success 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:2:6 marker=00000000002.78.6 timestamp=2024-04-12T03:39:43.762337+0000 2024-04-12T03:39:45.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:entry[obj00000000000000000004]: success 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:29:6 marker=00000000002.59.6 timestamp=2024-04-12T03:39:43.534085+0000 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:entry[obj00000000000000000020]: success 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.593+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:28:6 marker=00000000002.53.6 timestamp=2024-04-12T03:39:44.306035+0000 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: a sync operation returned error 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: backing out with sync_status=-5 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26/obj00000000000000000021 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: entry->timestamp=2024-04-12T03:39:43.462369+0000 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-12T03:39:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000021[0] 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.595+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.596+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:entry[obj00000000000000000006]: success 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.596+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:20:6 marker=00000000002.37.6 timestamp=2024-04-12T03:39:43.929551+0000 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.598+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:entry[obj00000000000000000065]: success 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:5:6 marker=00000000001.35.6 timestamp=2024-04-12T03:39:43.412402+0000 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: sync status for source bucket shard: 0 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.598+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.599+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:entry[obj00000000000000000021]: success 2024-04-12T03:39:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.599+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:26:6 marker=00000000002.55.6 timestamp=2024-04-12T03:39:43.462369+0000 2024-04-12T03:39:45.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.601+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:entry[obj00000000000000000053]: success 2024-04-12T03:39:45.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.601+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:27:6 marker=00000000002.94.6 timestamp=2024-04-12T03:39:43.594276+0000 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: parsed entry: id=00000000002.150.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1/obj00000000000000000000 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: entry->timestamp=2024-04-12T03:39:44.186035+0000 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.150.6 op=0 op_state=1 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000000[0] 2024-04-12T03:39:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.614+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: listing bilog for incremental sync; position=00000000002.150.6 2024-04-12T03:39:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.615+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:entry[obj00000000000000000000]: success 2024-04-12T03:39:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.615+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:1:6 marker=00000000002.150.6 timestamp=2024-04-12T03:39:44.186035+0000 2024-04-12T03:39:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: parsed entry: id=00000000001.79.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-04-12T03:39:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0/obj00000000000000000060 2024-04-12T03:39:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: entry->timestamp=2024-04-12T03:39:43.774037+0000 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.79.6 op=0 op_state=1 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000060[0] 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: listing bilog for incremental sync; position=00000000001.79.6 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8/obj00000000000000000048 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: entry->timestamp=2024-04-12T03:39:43.992271+0000 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000048[0] 2024-04-12T03:39:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.617+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: listing bilog for incremental sync; position=00000000001.88.6 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10/obj00000000000000000026 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: entry->timestamp=2024-04-12T03:39:44.234315+0000 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000026[0] 2024-04-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.618+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: listing bilog for incremental sync; position=00000000002.94.6 2024-04-12T03:39:45.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.620+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:entry[obj00000000000000000026]: success 2024-04-12T03:39:45.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.620+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:10:6 marker=00000000002.94.6 timestamp=2024-04-12T03:39:44.234315+0000 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15/obj00000000000000000024 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: entry->timestamp=2024-04-12T03:39:43.917457+0000 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000024[0] 2024-04-12T03:39:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.622+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:39:45.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.623+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:entry[obj00000000000000000024]: success 2024-04-12T03:39:45.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.623+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:15:6 marker=00000000002.83.6 timestamp=2024-04-12T03:39:43.917457+0000 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19/obj00000000000000000045 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: entry->timestamp=2024-04-12T03:39:43.979475+0000 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: : injecting data sync error on key=obj00000000000000000045 2024-04-12T03:39:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-12T03:39:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:39:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.626+0000 7f3eb95d7780 0 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: ERROR: failed to sync object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19/obj00000000000000000045 2024-04-12T03:39:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-12T03:39:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23/obj00000000000000000039 2024-04-12T03:39:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: entry->timestamp=2024-04-12T03:39:44.124442+0000 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000039[0] 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.627+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25/obj00000000000000000010 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: entry->timestamp=2024-04-12T03:39:44.174175+0000 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-12T03:39:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000010[0] 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24/obj00000000000000000027 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: entry->timestamp=2024-04-12T03:39:44.354090+0000 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: parsed entry: id=00000000002.146.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32/obj00000000000000000014 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: entry->timestamp=2024-04-12T03:39:43.855435+0000 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.146.6 op=0 op_state=1 2024-04-12T03:39:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000027[0] 2024-04-12T03:39:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:39:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000014[0] 2024-04-12T03:39:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.628+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: listing bilog for incremental sync; position=00000000002.146.6 2024-04-12T03:39:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.629+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:entry[obj00000000000000000039]: success 2024-04-12T03:39:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.629+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:23:6 marker=00000000002.51.6 timestamp=2024-04-12T03:39:44.124442+0000 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18/obj00000000000000000013 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: entry->timestamp=2024-04-12T03:39:44.414393+0000 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000013[0] 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17/obj00000000000000000002 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: entry->timestamp=2024-04-12T03:39:44.110430+0000 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-12T03:39:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36/obj00000000000000000008 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: entry->timestamp=2024-04-12T03:39:43.522192+0000 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000002[0] 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000008[0] 2024-04-12T03:39:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.630+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: listing bilog for incremental sync; position=00000000002.92.6 2024-04-12T03:39:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.632+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:entry[obj00000000000000000010]: success 2024-04-12T03:39:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.632+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:25:6 marker=00000000002.39.6 timestamp=2024-04-12T03:39:44.174175+0000 2024-04-12T03:39:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.632+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:entry[obj00000000000000000027]: success 2024-04-12T03:39:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.632+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:24:6 marker=00000000002.69.6 timestamp=2024-04-12T03:39:44.354090+0000 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.633+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:entry[obj00000000000000000014]: success 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.633+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:32:6 marker=00000000002.146.6 timestamp=2024-04-12T03:39:43.855435+0000 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.633+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:entry[obj00000000000000000002]: success 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.633+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:17:6 marker=00000000002.85.6 timestamp=2024-04-12T03:39:44.110430+0000 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:entry[obj00000000000000000008]: success 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:36:6 marker=00000000002.92.6 timestamp=2024-04-12T03:39:43.522192+0000 2024-04-12T03:39:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: parsed entry: id=00000000001.79.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34/obj00000000000000000057 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: entry->timestamp=2024-04-12T03:39:43.473958+0000 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000001.79.6 op=0 op_state=1 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33/obj00000000000000000019 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: entry->timestamp=2024-04-12T03:39:44.245783+0000 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35/obj00000000000000000018 2024-04-12T03:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: entry->timestamp=2024-04-12T03:39:44.294181+0000 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000057[0] 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: listing bilog for incremental sync; position=00000000001.79.6 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:entry[obj00000000000000000013]: success 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:18:6 marker=00000000002.83.6 timestamp=2024-04-12T03:39:44.414393+0000 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000019[0] 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000018[0] 2024-04-12T03:39:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.634+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:entry[obj00000000000000000062]: success 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:12:6 marker=00000000001.87.6 timestamp=2024-04-12T03:39:43.401681+0000 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:entry[obj00000000000000000018]: success 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:35:6 marker=00000000002.81.6 timestamp=2024-04-12T03:39:44.294181+0000 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:entry[obj00000000000000000019]: success 2024-04-12T03:39:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.638+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:33:6 marker=00000000002.51.6 timestamp=2024-04-12T03:39:44.245783+0000 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.640+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:entry[obj00000000000000000036]: success 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:30:6 marker=00000000001.86.6 timestamp=2024-04-12T03:39:43.354674+0000 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9/obj00000000000000000015 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: entry->timestamp=2024-04-12T03:39:44.062376+0000 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000015[0] 2024-04-12T03:39:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.641+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-12T03:39:45.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.642+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:entry[obj00000000000000000015]: success 2024-04-12T03:39:45.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.642+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:9:6 marker=00000000002.50.6 timestamp=2024-04-12T03:39:44.062376+0000 2024-04-12T03:39:45.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.662+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:entry[obj00000000000000000048]: success 2024-04-12T03:39:45.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.662+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:8:6 marker=00000000001.88.6 timestamp=2024-04-12T03:39:43.992271+0000 2024-04-12T03:39:45.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.665+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:entry[obj00000000000000000060]: success 2024-04-12T03:39:45.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.665+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:0:6 marker=00000000001.79.6 timestamp=2024-04-12T03:39:43.774037+0000 2024-04-12T03:39:45.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: a sync operation returned error 2024-04-12T03:39:45.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: backing out with sync_status=-5 2024-04-12T03:39:45.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:39:45.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: ERROR: incremental sync failed. error: -5 2024-04-12T03:39:45.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:39:45.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: sync status for source bucket shard: 0 2024-04-12T03:39:45.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.668+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:39:45.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.680+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:entry[obj00000000000000000057]: success 2024-04-12T03:39:45.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.680+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:34:6 marker=00000000001.79.6 timestamp=2024-04-12T03:39:43.473958+0000 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: syncing object: dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19/obj00000000000000000045 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: entry->timestamp=2024-04-12T03:39:43.979475+0000 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 5 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-60[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23])/obj00000000000000000045[0] 2024-04-12T03:39:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.710+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-12T03:39:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.711+0000 7f3eb95d7780 10 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:entry[obj00000000000000000045]: success 2024-04-12T03:39:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:45.711+0000 7f3eb95d7780 20 RGW-SYNC:bucket[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23<-dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]:inc_sync[dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19]: updating marker marker_oid=bucket.sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23:19:6 marker=00000000002.96.6 timestamp=2024-04-12T03:39:43.979475+0000 2024-04-12T03:39:45.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:45.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:45.773 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000000", 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.898 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.186035Z", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.899 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3946708074969333297", 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000001", 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 17 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.653960Z", 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.901 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.827690747788700621", 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000002", 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 16 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.110430Z", 2024-04-12T03:39:45.903 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.8403089733981922343", 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.904 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000003", 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.905 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.582269Z", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.906 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10895103945894806887", 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000004", 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:39:45.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.534085Z", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.6960678194822082227", 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000005", 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.702263Z", 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.910 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15082604080464774192", 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.911 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000006", 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 23 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.929551Z", 2024-04-12T03:39:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.5859411952607831270", 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.913 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000007", 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 20 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.914 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.024745Z", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.915 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2450669678484010543", 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000008", 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.916 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.522192Z", 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.917 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9311159570438446184", 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.918 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000009", 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 29 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.365579Z", 2024-04-12T03:39:45.919 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.5643858935400240524", 2024-04-12T03:39:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000010", 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 27 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.921 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.174175Z", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.922 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4796995599921058803", 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000011", 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.923 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.762337Z", 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.924 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1399558375925147321", 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000012", 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.926 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.713870Z", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.927 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17919680853489785484", 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000013", 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.928 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.414393Z", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.929 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.6497237980529268359", 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000014", 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.930 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 24 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.855435Z", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.931 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15176248872141710127", 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000015", 2024-04-12T03:39:45.932 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.062376Z", 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.933 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2239304555597595318", 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.934 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000016", 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.935 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.867458Z", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.7218637510485069198", 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000017", 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.337899Z", 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.938 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17517285536470190967", 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000018", 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.940 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.294181Z", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10643967782489035675", 2024-04-12T03:39:45.941 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000019", 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 25 2024-04-12T03:39:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.245783Z", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.943 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17641226961742352495", 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000020", 2024-04-12T03:39:45.944 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 26 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.306035Z", 2024-04-12T03:39:45.945 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15618260379865479354", 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000021", 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.947 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.462369Z", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.948 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17151340612231783660", 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000022", 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.949 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.224838Z", 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.950 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15052356596712433216", 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000023", 2024-04-12T03:39:45.951 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.402445Z", 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.952 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2079632987721228668", 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.953 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000024", 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.954 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.917457Z", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.955 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4990243897879749018", 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000025", 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.956 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.949191Z", 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.957 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10327069294679634786", 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.958 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000026", 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.234315Z", 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.959 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15339114915662130445", 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000027", 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.961 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.354090Z", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15474185858670157098", 2024-04-12T03:39:45.962 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000028", 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:39.106025Z", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14732230948302083999", 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000030", 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.968 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.709719Z", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.969 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17588045337095805341", 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000031", 2024-04-12T03:39:45.970 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.214019Z", 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.971 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9991974805748307230", 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.972 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000032", 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.534134Z", 2024-04-12T03:39:45.973 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3044741877185715313", 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000033", 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 21 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.975 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.642231Z", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.976 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14439793263754011924", 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000036", 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.977 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.354674Z", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.978 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16321522382989188263", 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000037", 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.458188Z", 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.980 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1992680984286492321", 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000039", 2024-04-12T03:39:45.981 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.982 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.124442Z", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10624219110621983283", 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.983 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000040", 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 17 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.984 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.046701Z", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.985 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1907243793311378956", 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000042", 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.986 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.528174Z", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.987 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.11497704395033506628", 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000045", 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.979475Z", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.989 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.8307670376780536746", 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000046", 2024-04-12T03:39:45.990 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.898206Z", 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15969274190870866626", 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000048", 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:45.993 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.992271Z", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.11111466699623256231", 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000049", 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.995 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.817901Z", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.996 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.13145367207042218904", 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000050", 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:45.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.889863Z", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:45.998 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.8698456089452944852", 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:45.999 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000053", 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.594276Z", 2024-04-12T03:39:46.000 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.13720245967971527591", 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000055", 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.002 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.414311Z", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.003 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14149643646448742671", 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000056", 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.004 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.537801Z", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.005 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9638764617179475230", 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000057", 2024-04-12T03:39:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.473958Z", 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.007 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9199164569927020945", 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000058", 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.009 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.484783Z", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.010 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16959557596782285241", 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000059", 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.011 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.578039Z", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.012 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2681221712888945184", 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000060", 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.013 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.774037Z", 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.014 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9610298706245562055", 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000061", 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.016 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.715792Z", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.7196696174082855315", 2024-04-12T03:39:46.017 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000062", 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.401681Z", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4061274198315769928", 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000065", 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.020 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.412402Z", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.021 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14093133945859959813", 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000076", 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.022 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:39.017566Z", 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.023 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17195434672095818834", 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.024 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000133", 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 17 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.025 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.415930Z", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15073922562920431805", 2024-04-12T03:39:46.026 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000162", 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.027 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.646091Z", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.18263044404585917276", 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000271", 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.029 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.777368Z", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.030 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.12673586577734549422", 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000283", 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.031 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 38 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.073811Z", 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.032 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16072274737924404252", 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000317", 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.903162Z", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2463625514357742177", 2024-04-12T03:39:46.035 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.036 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000423", 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.037 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.349885Z", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.038 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.5222302644372614731", 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000467", 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.039 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.586205Z", 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.040 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16397979616134488572", 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:46.041 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:39:46.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:46.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:39:46.042 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:39:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000000", 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.128 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.186035Z", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.129 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o53QlJ0V2TlWT3_0_akaXTGfgWnxAn5", 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000001", 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.130 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.653960Z", 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.131 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uInrwXXGKkMqopXvcSlGiiXNqkGTOZZ", 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000002", 2024-04-12T03:39:46.132 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.110430Z", 2024-04-12T03:39:46.133 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_x41xEaYwVk9yZOiS9nSoQdLxd-Wb7m5", 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.134 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000003", 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.135 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.582269Z", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_I2pX9idowJ7dMgA53z3ak4RJhO1iCgq", 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000004", 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.137 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.534085Z", 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.138 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vpHz11l54OHQRQMPN_H0rdkugPCGq0k", 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.139 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000005", 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.702263Z", 2024-04-12T03:39:46.140 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fwr4IZhG1oa5Jk6HiQOdGQ1OEeMQhtt", 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000006", 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.142 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.929551Z", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.143 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PJ0xuX6g4ZNqWogsmkSM-ZkLPaB4Su5", 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000007", 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.144 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.024745Z", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.145 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vQ74I_HAkoRUEOAryIwDBO6Fg6aujIQ", 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.146 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000008", 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.147 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.522192Z", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NoU9MGeamSh46TbeiJ3iJ6ZTF0XxORv", 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.148 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000009", 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.149 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.365579Z", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.150 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_iZYyTCatv73q-vndHSpAfJ847x6VaQi", 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000010", 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.151 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.174175Z", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.152 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6IbCADHQcJ3VLv1KW2VrjXOMCFLyTtK", 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000011", 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.762337Z", 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.154 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_69f5nMjaP0HdDb25WDT9WqxAI9_G4dE", 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000012", 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.156 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.713870Z", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_sHzEEvbTZpvdP4Yg4LktixCjRGLUITJ", 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000013", 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.414393Z", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.159 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UgncilYKx8QX_5SA6t_jlwh7NtIvO1F", 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000014", 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.160 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.855435Z", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.161 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_mXfvn7y8dj6b-rqNVO376IZcQhG3OxF", 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000015", 2024-04-12T03:39:46.162 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.062376Z", 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.163 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Ok6pq7l8tSpxg0blob9AVy7NoJUPIcm", 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000016", 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.165 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.867458Z", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.166 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ffEXTcWN8lCl1BKH3rlsqQiukfzR-xy", 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000017", 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.167 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.337899Z", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.168 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Xhgn4mTqsZ4simL_sUkM4a7co4S3J1n", 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000018", 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.169 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.294181Z", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.170 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CgKWAVHQre3umksMs4LBLfYV_RjdPnu", 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.171 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000019", 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.245783Z", 2024-04-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MDw2PxDpe6MlePQmHkMuPDgan9ze3KA", 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.173 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000020", 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.306035Z", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0nPIb5-7Uq_35rZo1aH0kmjwc4O9zeS", 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000021", 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.462369Z", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8gEJfTrGpoBcZcQ0cFajVnq3MrUNK9Y", 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000022", 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.224838Z", 2024-04-12T03:39:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ukm8xDXL5LZ_oI8NF0j_IlSr7WSkhWb", 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000023", 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.402445Z", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CItBsDS7dHrvrSrDrRLUPUjiURQiD9Q", 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000024", 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.917457Z", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.184 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0IsicUniMhDYRDPpVHwTGH_7GmpBCxg", 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000025", 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.185 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.949191Z", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MvjubAlJIj7HFQpNayq2HFjMmfagAy_", 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000026", 2024-04-12T03:39:46.187 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.234315Z", 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.188 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_eHzBovMXKUhHfuw_97L7oQYoce4c7KG", 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000027", 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.354090Z", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LP3PylTQXOQia3GY_QM_tGGT-suAPC9", 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000028", 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:39.106025Z", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RzhZEM7r8i7IeP2DJd6SUKRGoTCFSE8", 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000030", 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.709719Z", 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_oWyCcFqKnxWKAAVbWKlK5v2lKeGnvQb", 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000031", 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.197 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.214019Z", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.198 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zvi5phu01WL9kLHWQdQYb7W2IkrPAta", 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000032", 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.199 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.534134Z", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.200 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hLShA9sAz1TeFvhI8tVb-bwUto8reFN", 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000033", 2024-04-12T03:39:46.201 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.642231Z", 2024-04-12T03:39:46.202 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_HtBV6V-CKvn4YEDY7cLj5QH6OB_urr1", 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.203 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000036", 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.204 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.354674Z", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uCLx2tcqH7HXguAZ9tYtwm-zwQbid5V", 2024-04-12T03:39:46.205 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000037", 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.206 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.458188Z", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.207 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lnHz-bJZqejKPCgc_vXsjxCaL-3Wnok", 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000039", 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.124442Z", 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.209 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_5H7TDoLUkHBhbGuN8wGsbuGcc5i7hAd", 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.210 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000040", 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:44.046701Z", 2024-04-12T03:39:46.211 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_epmF2gpiaYe50aW2__O2ABB9QsnegLZ", 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.212 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000042", 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.213 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.528174Z", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_aad-Pt4NSa0qhPqiWCZbLrYv6zRKEka", 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000045", 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.979475Z", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.216 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BPHPT_Y2CTF63NM2j3moX2WF0_GhlN2", 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000046", 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.898206Z", 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.218 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Ld8DYuEib3Xo_tk0D6gaNhmot3LY3HM", 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.219 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000048", 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.992271Z", 2024-04-12T03:39:46.220 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PQf40C6MPpNAgAyCBAy9o6vxNeq2l8b", 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.221 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000049", 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.222 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.817901Z", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.223 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1zi6G_TXeV46eowUCcin-FXKcf8Zs9s", 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000050", 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.224 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.889863Z", 2024-04-12T03:39:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.226 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Mxgaaz-RfFjq59XOefplhD6drTGYgek", 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000053", 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.228 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.594276Z", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_23UIzck2qsnoQ47SuL_Q9xLmcAgFITa", 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000055", 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.230 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.414311Z", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.231 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_htSvnduCO5h3hm1cjtpetOxhSS93NsD", 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000056", 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.232 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.537801Z", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.233 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_yxygSHtiOr9bG8VXhUHbUib7qE4UNRV", 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000057", 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.473958Z", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.235 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rmtv38vYys94CleUCfIBeHulYZ3ehzv", 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.236 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000058", 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:40.484783Z", 2024-04-12T03:39:46.237 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_g0Xu1PsGx2VWHMAYq0g5hcY_iyw_VmV", 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.238 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000059", 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.239 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:41.578039Z", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.240 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_deaCxdI1woWUH2SdIavo6hFAkRcQaZN", 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000060", 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.241 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.774037Z", 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.242 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_v9ECDkUnG0_SjeOJ5M7Mt36Rm0ow7Nw", 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000061", 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.244 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.715792Z", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JnuexVwcJwrqdJSwkyEOwROtM-yT3Kp", 2024-04-12T03:39:46.245 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000062", 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.246 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.401681Z", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.247 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qdIWc7mD--jxYn39p6gcCsAutOHV9qr", 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000065", 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.412402Z", 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnSzx9OPPs-qRytdHDLOi30gbUo1Jwa", 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000076", 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:39.017566Z", 2024-04-12T03:39:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_u0qvUIMXQGRKf7s485ilVrM-2qV7cji", 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000133", 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.415930Z", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4u0Tq0TgcDD20S5_3CfTnmlOtgjPY7n", 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000162", 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.255 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.646091Z", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.257 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_I9QU-lDF75aiIBCPCiE_yZFIzrZ0WrX", 2024-04-12T03:39:46.257 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.257 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.257 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.258 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000271", 2024-04-12T03:39:46.258 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.258 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.258 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.777368Z", 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.259 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_cb1cCfwjL-LbM-JHTJI7Csfss5dVDAH", 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000283", 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 46 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:43.073811Z", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xKcdtFyHiq4Mih75rY-Hikw0ueY8A3V", 2024-04-12T03:39:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000317", 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.263 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.903162Z", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.264 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_H_UIgr_pw5u7uK56YDE2a0gTyGDFRhD", 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000423", 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 16 2024-04-12T03:39:46.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.349885Z", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.266 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BkJ2wCcdY7YczImuIVurBwf6Q5xm5tl", 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000467", 2024-04-12T03:39:46.267 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:39:42.586205Z", 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:39:46.268 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uIOP0Gmdr7BqrIc2C_3CiSacVf1PbcZ", 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:39:46.269 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:39:46.270 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:39:46.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:39:46.273 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:39:46.273 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:39:46.294 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:39:46.294 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:39:46.294 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:39:47.295 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:47.316 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:39:47.316 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:39:47.316 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:39:47.317 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T03:39:47.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-60 2024-04-12T03:39:47.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:47.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:47.317 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:39:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:47.452+0000 7f4214893780 1 bucket sync caught up with source: 2024-04-12T03:39:47.453 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.79.6, 00000000002.150.6, 00000000002.78.6, 00000000002.81.6, 00000000002.90.6, 00000000001.35.6, 00000000002.45.6, 00000000002.148.6, 00000000001.88.6, 00000000002.50.6, 00000000002.94.6, , 00000000001.87.6, 00000000002.53.6, 00000000002.98.6, 00000000002.83.6, 00000000002.92.6, 00000000002.85.6, 00000000002.83.6, 00000000002.96.6, 00000000002.37.6, , , 00000000002.51.6, 00000000002.69.6, 00000000002.39.6, 00000000002.55.6, 00000000002.94.6, 00000000002.53.6, 00000000002.59.6, 00000000001.86.6, 00000000002.96.6, 00000000002.146.6, 00000000002.51.6, 00000000001.79.6, 00000000002.81.6, 00000000002.92.6] 2024-04-12T03:39:47.453 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.79.6, 00000000002.150.6, 00000000002.78.6, 00000000002.81.6, 00000000002.90.6, 00000000001.35.6, 00000000002.45.6, 00000000002.148.6, 00000000001.88.6, 00000000002.50.6, 00000000002.94.6, , 00000000001.87.6, 00000000002.53.6, 00000000002.98.6, 00000000002.83.6, 00000000002.92.6, 00000000002.85.6, 00000000002.83.6, 00000000002.96.6, 00000000002.37.6, , , 00000000002.51.6, 00000000002.69.6, 00000000002.39.6, 00000000002.55.6, 00000000002.94.6, 00000000002.53.6, 00000000002.59.6, 00000000001.86.6, 00000000002.96.6, 00000000002.146.6, 00000000002.51.6, 00000000001.79.6, 00000000002.81.6, 00000000002.92.6] 2024-04-12T03:39:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:47.452+0000 7f4214893780 0 bucket checkpoint complete 2024-04-12T03:39:47.467 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-60 2024-04-12T03:39:47.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:47.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:39:47.467 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:39:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:47.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:47.556+0000 7f82ad801780 1 waiting for incremental sync to catch up: 2024-04-12T03:39:47.556 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.1440.6, 00000000001.1190.6, 00000000006.664.6, 00000000010.1438.6, 00000000004.1371.6, 00000000001.2191.6, 00000000007.1368.6, 00000000006.1374.6, 00000000007.122.6, 00000000001.3053.6, 00000000004.1188.6] 2024-04-12T03:39:47.557 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.1440.6, 00000000007.1202.6, 00000000006.664.6, 00000000010.1438.6, 00000000004.1371.6, 00000000008.2205.6, 00000000007.1368.6, 00000000006.1374.6, 00000000007.122.6, 00000000007.3065.6, 00000000004.1188.6] 2024-04-12T03:39:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:52.557+0000 7f82ad801780 1 bucket sync caught up with source: 2024-04-12T03:39:52.558 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.1440.6, 00000000007.1202.6, 00000000006.664.6, 00000000010.1438.6, 00000000004.1371.6, 00000000008.2205.6, 00000000007.1368.6, 00000000006.1374.6, 00000000007.122.6, 00000000007.3065.6, 00000000004.1188.6] 2024-04-12T03:39:52.558 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.1440.6, 00000000007.1202.6, 00000000006.664.6, 00000000010.1438.6, 00000000004.1371.6, 00000000008.2205.6, 00000000007.1368.6, 00000000006.1374.6, 00000000007.122.6, 00000000007.3065.6, 00000000004.1188.6] 2024-04-12T03:39:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:39:52.557+0000 7f82ad801780 0 bucket checkpoint complete 2024-04-12T03:39:52.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-60 zones={a2, a1} 2024-04-12T03:39:52.575 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.575 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.575 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.575 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:52.575 DEBUG:boto:Data: 2024-04-12T03:39:52.575 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.575 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.575 DEBUG:boto:Params: {} 2024-04-12T03:39:52.576 DEBUG:boto:Token: None 2024-04-12T03:39:52.576 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw= 2024-04-12T03:39:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw=', 'Content-Length': '0'} 2024-04-12T03:39:52.577 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-12T03:39:52.577 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:52.792 DEBUG:boto:Token: None 2024-04-12T03:39:52.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw= 2024-04-12T03:39:52.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw=', 'Content-Length': '0'} 2024-04-12T03:39:52.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efbf184e16b0b899-006618ad08-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.809 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.809 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.809 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.809 DEBUG:boto:Path: /dgbpxg-60/ 2024-04-12T03:39:52.809 DEBUG:boto:Data: 2024-04-12T03:39:52.809 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.810 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.810 DEBUG:boto:Params: {} 2024-04-12T03:39:52.810 DEBUG:boto:Token: None 2024-04-12T03:39:52.810 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw= 2024-04-12T03:39:52.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:74Gu5xYL1i2tynfWx0hj2L7TSfw=', 'Content-Length': '0'} 2024-04-12T03:39:52.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e0c2d34caa98d91-006618ad08-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:39:52.815 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.816 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.816 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.816 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.816 DEBUG:boto:Method: GET 2024-04-12T03:39:52.816 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.816 DEBUG:boto:Data: 2024-04-12T03:39:52.816 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.816 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.816 DEBUG:boto:Params: {} 2024-04-12T03:39:52.816 DEBUG:boto:Token: None 2024-04-12T03:39:52.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/?versions 2024-04-12T03:39:52.817 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU= 2024-04-12T03:39:52.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU=', 'Content-Length': '0'} 2024-04-12T03:39:52.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab0bbe9ef3f33399-006618ad08-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.820 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:39:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:39:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:39:52.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-12T03:39:52.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-12T03:39:52.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-12T03:39:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-12T03:39:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-12T03:39:52.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-12T03:39:52.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:39:52.829 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.829 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.829 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.829 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.829 DEBUG:boto:Method: GET 2024-04-12T03:39:52.829 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.829 DEBUG:boto:Data: 2024-04-12T03:39:52.829 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.830 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.830 DEBUG:boto:Params: {} 2024-04-12T03:39:52.830 DEBUG:boto:Token: None 2024-04-12T03:39:52.830 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/?versions 2024-04-12T03:39:52.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU= 2024-04-12T03:39:52.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU=', 'Content-Length': '0'} 2024-04-12T03:39:52.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000084336bdc8711566-006618ad08-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.835 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-12T03:39:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-12T03:39:52.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-12T03:39:52.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-12T03:39:52.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-12T03:39:52.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-12T03:39:52.843 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.843 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.843 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.843 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.843 DEBUG:boto:Method: GET 2024-04-12T03:39:52.843 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.843 DEBUG:boto:Data: 2024-04-12T03:39:52.843 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.843 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.843 DEBUG:boto:Params: {} 2024-04-12T03:39:52.844 DEBUG:boto:Token: None 2024-04-12T03:39:52.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/?versions 2024-04-12T03:39:52.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU= 2024-04-12T03:39:52.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU=', 'Content-Length': '0'} 2024-04-12T03:39:52.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010326ed4fd4b5ad5-006618ad08-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.847 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.850 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.850 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.850 DEBUG:boto:path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.851 DEBUG:boto:auth_path=/dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.851 DEBUG:boto:Method: GET 2024-04-12T03:39:52.851 DEBUG:boto:Path: /dgbpxg-60/?versions&max-keys=999 2024-04-12T03:39:52.851 DEBUG:boto:Data: 2024-04-12T03:39:52.851 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.851 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.851 DEBUG:boto:Params: {} 2024-04-12T03:39:52.851 DEBUG:boto:Token: None 2024-04-12T03:39:52.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/?versions 2024-04-12T03:39:52.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU= 2024-04-12T03:39:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkLnir1YeT8JJ90PCFbCD7K1DaU=', 'Content-Length': '0'} 2024-04-12T03:39:52.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037162b104472a012-006618ad08-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.856 DEBUG:boto:b'dgbpxg-60999falseobj00000000000000000000nulltrue2024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:52.860 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.861 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.861 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.861 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.861 DEBUG:boto:Method: GET 2024-04-12T03:39:52.861 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.861 DEBUG:boto:Data: 2024-04-12T03:39:52.861 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.861 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.861 DEBUG:boto:Params: {} 2024-04-12T03:39:52.861 DEBUG:boto:Token: None 2024-04-12T03:39:52.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464= 2024-04-12T03:39:52.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464=', 'Content-Length': '0'} 2024-04-12T03:39:52.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024da8d2bfd6afbd1-006618ad08-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.906 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.906 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.907 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.907 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.907 DEBUG:boto:Method: GET 2024-04-12T03:39:52.907 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.907 DEBUG:boto:Data: 2024-04-12T03:39:52.907 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.907 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.907 DEBUG:boto:Params: {} 2024-04-12T03:39:52.907 DEBUG:boto:Token: None 2024-04-12T03:39:52.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464= 2024-04-12T03:39:52.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464=', 'Content-Length': '0'} 2024-04-12T03:39:52.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18c28daab6ef5ad-006618ad08-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.910 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.910 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.910 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.910 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.910 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.910 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.910 DEBUG:boto:Data: 2024-04-12T03:39:52.911 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.911 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.911 DEBUG:boto:Params: {} 2024-04-12T03:39:52.911 DEBUG:boto:Token: None 2024-04-12T03:39:52.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ydm9jbThrY0ocH5L7O3uHQBfpng= 2024-04-12T03:39:52.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ydm9jbThrY0ocH5L7O3uHQBfpng=', 'Content-Length': '0'} 2024-04-12T03:39:52.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adfdc51f273c979e-006618ad08-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.913 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.913 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.914 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.914 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.914 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.914 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.914 DEBUG:boto:Data: 2024-04-12T03:39:52.914 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.914 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.914 DEBUG:boto:Params: {} 2024-04-12T03:39:52.914 DEBUG:boto:Token: None 2024-04-12T03:39:52.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ydm9jbThrY0ocH5L7O3uHQBfpng= 2024-04-12T03:39:52.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ydm9jbThrY0ocH5L7O3uHQBfpng=', 'Content-Length': '0'} 2024-04-12T03:39:52.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c2ca91cdf730785-006618ad08-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:52.917 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.917 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.917 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.917 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.917 DEBUG:boto:Method: GET 2024-04-12T03:39:52.917 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.917 DEBUG:boto:Data: 2024-04-12T03:39:52.917 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.918 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.918 DEBUG:boto:Params: {} 2024-04-12T03:39:52.918 DEBUG:boto:Token: None 2024-04-12T03:39:52.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.918 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464= 2024-04-12T03:39:52.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464=', 'Content-Length': '0'} 2024-04-12T03:39:52.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f568c62b90f9da-006618ad08-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.920 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.920 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.921 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.921 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.921 DEBUG:boto:Method: GET 2024-04-12T03:39:52.921 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.921 DEBUG:boto:Data: 2024-04-12T03:39:52.921 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.921 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.921 DEBUG:boto:Params: {} 2024-04-12T03:39:52.921 DEBUG:boto:Token: None 2024-04-12T03:39:52.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000?versionId=null 2024-04-12T03:39:52.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464= 2024-04-12T03:39:52.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iSNndsi+0eDCDtLUD97b/0s464=', 'Content-Length': '0'} 2024-04-12T03:39:52.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a439d7691207a86-006618ad08-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.923 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.924 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.924 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.924 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.924 DEBUG:boto:Method: GET 2024-04-12T03:39:52.924 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.924 DEBUG:boto:Data: 2024-04-12T03:39:52.924 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.924 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.924 DEBUG:boto:Params: {} 2024-04-12T03:39:52.924 DEBUG:boto:Token: None 2024-04-12T03:39:52.925 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc= 2024-04-12T03:39:52.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc=', 'Content-Length': '0'} 2024-04-12T03:39:52.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c595a527b0c84f13-006618ad08-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.927 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.928 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.928 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.928 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.928 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.928 DEBUG:boto:Method: GET 2024-04-12T03:39:52.928 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.928 DEBUG:boto:Data: 2024-04-12T03:39:52.928 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.929 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.929 DEBUG:boto:Params: {} 2024-04-12T03:39:52.929 DEBUG:boto:Token: None 2024-04-12T03:39:52.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.929 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc= 2024-04-12T03:39:52.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc=', 'Content-Length': '0'} 2024-04-12T03:39:52.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000badae36005a2a69a-006618ad08-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.933 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.933 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.933 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.933 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.934 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.934 DEBUG:boto:Data: 2024-04-12T03:39:52.934 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.934 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.934 DEBUG:boto:Params: {} 2024-04-12T03:39:52.934 DEBUG:boto:Token: None 2024-04-12T03:39:52.934 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.934 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gTN5OyVlS/8TMW1DJ03bJ2Ajfv8= 2024-04-12T03:39:52.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gTN5OyVlS/8TMW1DJ03bJ2Ajfv8=', 'Content-Length': '0'} 2024-04-12T03:39:52.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071242b9bb6c6e9be-006618ad08-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.936 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.937 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.937 DEBUG:boto:Method: HEAD 2024-04-12T03:39:52.937 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.937 DEBUG:boto:Data: 2024-04-12T03:39:52.937 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.937 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.937 DEBUG:boto:Params: {} 2024-04-12T03:39:52.937 DEBUG:boto:Token: None 2024-04-12T03:39:52.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gTN5OyVlS/8TMW1DJ03bJ2Ajfv8= 2024-04-12T03:39:52.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gTN5OyVlS/8TMW1DJ03bJ2Ajfv8=', 'Content-Length': '0'} 2024-04-12T03:39:52.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad62484f6985346-006618ad08-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:39:52.940 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.940 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.940 DEBUG:boto:Method: GET 2024-04-12T03:39:52.940 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.940 DEBUG:boto:Data: 2024-04-12T03:39:52.941 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.941 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.941 DEBUG:boto:Params: {} 2024-04-12T03:39:52.941 DEBUG:boto:Token: None 2024-04-12T03:39:52.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K5iKXZ0DcyY3IvWkMstIPZPzVHM= 2024-04-12T03:39:52.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K5iKXZ0DcyY3IvWkMstIPZPzVHM=', 'Content-Length': '0'} 2024-04-12T03:39:52.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd34bf6daf376f10-006618ad08-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.943 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.944 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.944 DEBUG:boto:Method: GET 2024-04-12T03:39:52.944 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.944 DEBUG:boto:Data: 2024-04-12T03:39:52.944 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.944 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.944 DEBUG:boto:Params: {} 2024-04-12T03:39:52.944 DEBUG:boto:Token: None 2024-04-12T03:39:52.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/obj00000000000000000000 2024-04-12T03:39:52.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K5iKXZ0DcyY3IvWkMstIPZPzVHM= 2024-04-12T03:39:52.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K5iKXZ0DcyY3IvWkMstIPZPzVHM=', 'Content-Length': '0'} 2024-04-12T03:39:52.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d523d9e674e15b3e-006618ad08-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.947 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.947 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.947 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.947 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.947 DEBUG:boto:Method: GET 2024-04-12T03:39:52.947 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.948 DEBUG:boto:Data: 2024-04-12T03:39:52.948 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:52.948 DEBUG:boto:Port: 8001 2024-04-12T03:39:52.948 DEBUG:boto:Params: {} 2024-04-12T03:39:52.948 DEBUG:boto:Token: None 2024-04-12T03:39:52.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc= 2024-04-12T03:39:52.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc=', 'Content-Length': '0'} 2024-04-12T03:39:52.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a997e54dc041ac0c-006618ad08-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:52.994 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:52.995 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:52.995 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:52.995 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.995 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.995 DEBUG:boto:Method: GET 2024-04-12T03:39:52.995 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000000 2024-04-12T03:39:52.995 DEBUG:boto:Data: 2024-04-12T03:39:52.995 DEBUG:boto:Headers: {} 2024-04-12T03:39:52.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:52.995 DEBUG:boto:Port: 8000 2024-04-12T03:39:52.996 DEBUG:boto:Params: {} 2024-04-12T03:39:52.996 DEBUG:boto:Token: None 2024-04-12T03:39:52.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:52 GMT /dgbpxg-60/ 2024-04-12T03:39:52.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc= 2024-04-12T03:39:52.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3j3TAS2eZGCCSQNootRCoIyx+zc=', 'Content-Length': '0'} 2024-04-12T03:39:53.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000843ef35070b6d9f6-006618ad08-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.000 DEBUG:boto:b'dgbpxg-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:39:44.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:53.001 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.001 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.001 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.001 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.001 DEBUG:boto:Method: GET 2024-04-12T03:39:53.001 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.001 DEBUG:boto:Data: 2024-04-12T03:39:53.001 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.001 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.001 DEBUG:boto:Params: {} 2024-04-12T03:39:53.001 DEBUG:boto:Token: None 2024-04-12T03:39:53.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.002 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo= 2024-04-12T03:39:53.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo=', 'Content-Length': '0'} 2024-04-12T03:39:53.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051f608e097ad9f3-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.004 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.004 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.004 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.004 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.004 DEBUG:boto:Method: GET 2024-04-12T03:39:53.004 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.005 DEBUG:boto:Data: 2024-04-12T03:39:53.005 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.005 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.005 DEBUG:boto:Params: {} 2024-04-12T03:39:53.005 DEBUG:boto:Token: None 2024-04-12T03:39:53.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.005 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo= 2024-04-12T03:39:53.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo=', 'Content-Length': '0'} 2024-04-12T03:39:53.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8ea217f42d80da-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.007 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.008 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.008 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.008 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.008 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.008 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.008 DEBUG:boto:Data: 2024-04-12T03:39:53.008 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.008 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.008 DEBUG:boto:Params: {} 2024-04-12T03:39:53.008 DEBUG:boto:Token: None 2024-04-12T03:39:53.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OlOdglbQ6f6IPas5uhhIOXB6Xyg= 2024-04-12T03:39:53.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OlOdglbQ6f6IPas5uhhIOXB6Xyg=', 'Content-Length': '0'} 2024-04-12T03:39:53.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fc6fd370335301-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.010 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.011 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.011 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.011 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.011 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.011 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.011 DEBUG:boto:Data: 2024-04-12T03:39:53.011 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.011 DEBUG:boto:Params: {} 2024-04-12T03:39:53.012 DEBUG:boto:Token: None 2024-04-12T03:39:53.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OlOdglbQ6f6IPas5uhhIOXB6Xyg= 2024-04-12T03:39:53.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OlOdglbQ6f6IPas5uhhIOXB6Xyg=', 'Content-Length': '0'} 2024-04-12T03:39:53.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6d6d6a749f71c2-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:53.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.015 DEBUG:boto:Method: GET 2024-04-12T03:39:53.015 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.015 DEBUG:boto:Data: 2024-04-12T03:39:53.015 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.015 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.015 DEBUG:boto:Params: {} 2024-04-12T03:39:53.015 DEBUG:boto:Token: None 2024-04-12T03:39:53.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo= 2024-04-12T03:39:53.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo=', 'Content-Length': '0'} 2024-04-12T03:39:53.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008425fe60a4293c7c-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.017 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.018 DEBUG:boto:Method: GET 2024-04-12T03:39:53.018 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.018 DEBUG:boto:Data: 2024-04-12T03:39:53.018 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.018 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.018 DEBUG:boto:Params: {} 2024-04-12T03:39:53.019 DEBUG:boto:Token: None 2024-04-12T03:39:53.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001?versionId=null 2024-04-12T03:39:53.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo= 2024-04-12T03:39:53.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NCac4Nhk7ICIz0XpQAPnf9TEhAo=', 'Content-Length': '0'} 2024-04-12T03:39:53.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003941b24aab3aa8b5-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.062 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.062 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.063 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.063 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.063 DEBUG:boto:Method: GET 2024-04-12T03:39:53.063 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.063 DEBUG:boto:Data: 2024-04-12T03:39:53.063 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.063 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.063 DEBUG:boto:Params: {} 2024-04-12T03:39:53.064 DEBUG:boto:Token: None 2024-04-12T03:39:53.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c27cad6e84cd056c-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.066 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.067 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.067 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.067 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.067 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.067 DEBUG:boto:Method: GET 2024-04-12T03:39:53.067 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.067 DEBUG:boto:Data: 2024-04-12T03:39:53.068 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.068 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.068 DEBUG:boto:Params: {} 2024-04-12T03:39:53.068 DEBUG:boto:Token: None 2024-04-12T03:39:53.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dbc859730db94db-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.072 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.073 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.073 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.073 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.073 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.073 DEBUG:boto:Data: 2024-04-12T03:39:53.073 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.073 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.073 DEBUG:boto:Params: {} 2024-04-12T03:39:53.074 DEBUG:boto:Token: None 2024-04-12T03:39:53.074 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5aiS1Bs2A4EzDekQsVKmWAaNKyw= 2024-04-12T03:39:53.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5aiS1Bs2A4EzDekQsVKmWAaNKyw=', 'Content-Length': '0'} 2024-04-12T03:39:53.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9d229a5a21cf94-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.076 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.076 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.076 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.076 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.076 DEBUG:boto:Data: 2024-04-12T03:39:53.076 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.077 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.077 DEBUG:boto:Params: {} 2024-04-12T03:39:53.077 DEBUG:boto:Token: None 2024-04-12T03:39:53.077 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.077 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5aiS1Bs2A4EzDekQsVKmWAaNKyw= 2024-04-12T03:39:53.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5aiS1Bs2A4EzDekQsVKmWAaNKyw=', 'Content-Length': '0'} 2024-04-12T03:39:53.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cd1818609eef99-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:39:53.079 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.080 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.080 DEBUG:boto:Method: GET 2024-04-12T03:39:53.080 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.080 DEBUG:boto:Data: 2024-04-12T03:39:53.080 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.080 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.080 DEBUG:boto:Params: {} 2024-04-12T03:39:53.080 DEBUG:boto:Token: None 2024-04-12T03:39:53.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qM2NDBiXmJx6eP3bcMVbMTR1Rmo= 2024-04-12T03:39:53.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qM2NDBiXmJx6eP3bcMVbMTR1Rmo=', 'Content-Length': '0'} 2024-04-12T03:39:53.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7904717244d0f8-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.126 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.126 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.127 DEBUG:boto:Method: GET 2024-04-12T03:39:53.127 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.127 DEBUG:boto:Data: 2024-04-12T03:39:53.127 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.127 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.127 DEBUG:boto:Params: {} 2024-04-12T03:39:53.127 DEBUG:boto:Token: None 2024-04-12T03:39:53.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000001 2024-04-12T03:39:53.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qM2NDBiXmJx6eP3bcMVbMTR1Rmo= 2024-04-12T03:39:53.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qM2NDBiXmJx6eP3bcMVbMTR1Rmo=', 'Content-Length': '0'} 2024-04-12T03:39:53.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4b95ea499064f9d-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.130 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.130 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.130 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.130 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.130 DEBUG:boto:Method: GET 2024-04-12T03:39:53.130 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.130 DEBUG:boto:Data: 2024-04-12T03:39:53.130 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.130 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.131 DEBUG:boto:Params: {} 2024-04-12T03:39:53.131 DEBUG:boto:Token: None 2024-04-12T03:39:53.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071fed04cd8415a6e-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.134 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.134 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.134 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.134 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.134 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.134 DEBUG:boto:Method: GET 2024-04-12T03:39:53.135 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000001 2024-04-12T03:39:53.135 DEBUG:boto:Data: 2024-04-12T03:39:53.135 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.135 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.135 DEBUG:boto:Params: {} 2024-04-12T03:39:53.135 DEBUG:boto:Token: None 2024-04-12T03:39:53.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007287185a3c851761-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.139 DEBUG:boto:b'dgbpxg-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:39:43.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:53.139 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.139 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.140 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.140 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.140 DEBUG:boto:Method: GET 2024-04-12T03:39:53.140 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.140 DEBUG:boto:Data: 2024-04-12T03:39:53.140 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.140 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.140 DEBUG:boto:Params: {} 2024-04-12T03:39:53.140 DEBUG:boto:Token: None 2024-04-12T03:39:53.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA= 2024-04-12T03:39:53.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA=', 'Content-Length': '0'} 2024-04-12T03:39:53.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182a6843af9d4a18-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.143 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.143 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.143 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.143 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.143 DEBUG:boto:Method: GET 2024-04-12T03:39:53.143 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.143 DEBUG:boto:Data: 2024-04-12T03:39:53.143 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.143 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.144 DEBUG:boto:Params: {} 2024-04-12T03:39:53.144 DEBUG:boto:Token: None 2024-04-12T03:39:53.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA= 2024-04-12T03:39:53.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA=', 'Content-Length': '0'} 2024-04-12T03:39:53.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c9dd9ab0c1db1e-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.186 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.186 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.186 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.186 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.187 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.187 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.187 DEBUG:boto:Data: 2024-04-12T03:39:53.187 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.187 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.187 DEBUG:boto:Params: {} 2024-04-12T03:39:53.187 DEBUG:boto:Token: None 2024-04-12T03:39:53.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Vx5Shm7KzyVNWVFkHk20srEAig= 2024-04-12T03:39:53.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Vx5Shm7KzyVNWVFkHk20srEAig=', 'Content-Length': '0'} 2024-04-12T03:39:53.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dfbc4b0206d9d56-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.189 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.190 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.190 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.190 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.190 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.190 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.190 DEBUG:boto:Data: 2024-04-12T03:39:53.190 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.190 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.191 DEBUG:boto:Params: {} 2024-04-12T03:39:53.191 DEBUG:boto:Token: None 2024-04-12T03:39:53.191 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.191 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Vx5Shm7KzyVNWVFkHk20srEAig= 2024-04-12T03:39:53.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Vx5Shm7KzyVNWVFkHk20srEAig=', 'Content-Length': '0'} 2024-04-12T03:39:53.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cae89585abdc537-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:53.193 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.193 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.193 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.193 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.193 DEBUG:boto:Method: GET 2024-04-12T03:39:53.193 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.194 DEBUG:boto:Data: 2024-04-12T03:39:53.194 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.194 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.194 DEBUG:boto:Params: {} 2024-04-12T03:39:53.194 DEBUG:boto:Token: None 2024-04-12T03:39:53.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA= 2024-04-12T03:39:53.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA=', 'Content-Length': '0'} 2024-04-12T03:39:53.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016056c6a240bece0-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.196 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.197 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.197 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.197 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.197 DEBUG:boto:Method: GET 2024-04-12T03:39:53.197 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.197 DEBUG:boto:Data: 2024-04-12T03:39:53.197 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.197 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.197 DEBUG:boto:Params: {} 2024-04-12T03:39:53.197 DEBUG:boto:Token: None 2024-04-12T03:39:53.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002?versionId=null 2024-04-12T03:39:53.198 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA= 2024-04-12T03:39:53.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ujsJLOFqRpBwC4SPsqQpbf4drKA=', 'Content-Length': '0'} 2024-04-12T03:39:53.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009174af47c16ee38e-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.200 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.200 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.200 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.200 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.200 DEBUG:boto:Method: GET 2024-04-12T03:39:53.200 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.201 DEBUG:boto:Data: 2024-04-12T03:39:53.201 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.201 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.201 DEBUG:boto:Params: {} 2024-04-12T03:39:53.201 DEBUG:boto:Token: None 2024-04-12T03:39:53.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361f02d7036b91f5-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.204 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.204 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.204 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.205 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.205 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.205 DEBUG:boto:Method: GET 2024-04-12T03:39:53.205 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.205 DEBUG:boto:Data: 2024-04-12T03:39:53.205 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.205 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.205 DEBUG:boto:Params: {} 2024-04-12T03:39:53.205 DEBUG:boto:Token: None 2024-04-12T03:39:53.205 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.206 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2f14b0533257fe-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.210 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.210 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.210 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.210 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.210 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.210 DEBUG:boto:Data: 2024-04-12T03:39:53.210 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.211 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.211 DEBUG:boto:Params: {} 2024-04-12T03:39:53.211 DEBUG:boto:Token: None 2024-04-12T03:39:53.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5CMjjeOt7gg6mrvTiZf23VzpIn4= 2024-04-12T03:39:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5CMjjeOt7gg6mrvTiZf23VzpIn4=', 'Content-Length': '0'} 2024-04-12T03:39:53.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0461bc9d47f2864-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.213 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.213 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.213 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.213 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.214 DEBUG:boto:Data: 2024-04-12T03:39:53.214 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.214 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.214 DEBUG:boto:Params: {} 2024-04-12T03:39:53.214 DEBUG:boto:Token: None 2024-04-12T03:39:53.214 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.214 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5CMjjeOt7gg6mrvTiZf23VzpIn4= 2024-04-12T03:39:53.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5CMjjeOt7gg6mrvTiZf23VzpIn4=', 'Content-Length': '0'} 2024-04-12T03:39:53.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c01c1bc7ea7e18d-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:39:53.216 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.217 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.217 DEBUG:boto:Method: GET 2024-04-12T03:39:53.217 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.217 DEBUG:boto:Data: 2024-04-12T03:39:53.217 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.217 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.217 DEBUG:boto:Params: {} 2024-04-12T03:39:53.217 DEBUG:boto:Token: None 2024-04-12T03:39:53.218 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n/4QWdIPzemM+w89tXKsIGHkBqQ= 2024-04-12T03:39:53.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n/4QWdIPzemM+w89tXKsIGHkBqQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d158d24f774ee0b4-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.220 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.220 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.220 DEBUG:boto:Method: GET 2024-04-12T03:39:53.220 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.220 DEBUG:boto:Data: 2024-04-12T03:39:53.220 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.221 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.221 DEBUG:boto:Params: {} 2024-04-12T03:39:53.221 DEBUG:boto:Token: None 2024-04-12T03:39:53.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000002 2024-04-12T03:39:53.221 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n/4QWdIPzemM+w89tXKsIGHkBqQ= 2024-04-12T03:39:53.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n/4QWdIPzemM+w89tXKsIGHkBqQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586f1727e8f78ade-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.223 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.223 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.223 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.224 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.224 DEBUG:boto:Method: GET 2024-04-12T03:39:53.224 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.224 DEBUG:boto:Data: 2024-04-12T03:39:53.224 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.224 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.224 DEBUG:boto:Params: {} 2024-04-12T03:39:53.224 DEBUG:boto:Token: None 2024-04-12T03:39:53.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb3de45acbd874e-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.227 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.228 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.228 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.228 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.228 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.228 DEBUG:boto:Method: GET 2024-04-12T03:39:53.228 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000002 2024-04-12T03:39:53.228 DEBUG:boto:Data: 2024-04-12T03:39:53.228 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.228 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.228 DEBUG:boto:Params: {} 2024-04-12T03:39:53.229 DEBUG:boto:Token: None 2024-04-12T03:39:53.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002bcb47adae71f5d-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.233 DEBUG:boto:b'dgbpxg-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:39:44.110Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:53.233 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.233 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.234 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.234 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.234 DEBUG:boto:Method: GET 2024-04-12T03:39:53.234 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.234 DEBUG:boto:Data: 2024-04-12T03:39:53.234 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.234 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.234 DEBUG:boto:Params: {} 2024-04-12T03:39:53.234 DEBUG:boto:Token: None 2024-04-12T03:39:53.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I= 2024-04-12T03:39:53.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I=', 'Content-Length': '0'} 2024-04-12T03:39:53.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda6a231372d5898-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.278 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.278 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.279 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.279 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.279 DEBUG:boto:Method: GET 2024-04-12T03:39:53.279 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.279 DEBUG:boto:Data: 2024-04-12T03:39:53.279 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.279 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.279 DEBUG:boto:Params: {} 2024-04-12T03:39:53.279 DEBUG:boto:Token: None 2024-04-12T03:39:53.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I= 2024-04-12T03:39:53.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I=', 'Content-Length': '0'} 2024-04-12T03:39:53.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007069c946a094fd6d-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.282 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.282 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.282 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.282 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.283 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.283 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.283 DEBUG:boto:Data: 2024-04-12T03:39:53.283 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.283 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.283 DEBUG:boto:Params: {} 2024-04-12T03:39:53.283 DEBUG:boto:Token: None 2024-04-12T03:39:53.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9nXzLCFkvY2EFr0SzgRVP5ZsrzU= 2024-04-12T03:39:53.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9nXzLCFkvY2EFr0SzgRVP5ZsrzU=', 'Content-Length': '0'} 2024-04-12T03:39:53.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882cda5f50b813ae-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.286 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.286 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.286 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.286 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.286 DEBUG:boto:Data: 2024-04-12T03:39:53.286 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.286 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.287 DEBUG:boto:Params: {} 2024-04-12T03:39:53.287 DEBUG:boto:Token: None 2024-04-12T03:39:53.287 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.287 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9nXzLCFkvY2EFr0SzgRVP5ZsrzU= 2024-04-12T03:39:53.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9nXzLCFkvY2EFr0SzgRVP5ZsrzU=', 'Content-Length': '0'} 2024-04-12T03:39:53.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000596af2a813177ee0-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:53.289 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.289 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.289 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.289 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.289 DEBUG:boto:Method: GET 2024-04-12T03:39:53.290 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.290 DEBUG:boto:Data: 2024-04-12T03:39:53.290 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.290 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.290 DEBUG:boto:Params: {} 2024-04-12T03:39:53.290 DEBUG:boto:Token: None 2024-04-12T03:39:53.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.290 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I= 2024-04-12T03:39:53.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I=', 'Content-Length': '0'} 2024-04-12T03:39:53.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974270b138e1ce81-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.293 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.293 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.293 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.293 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.293 DEBUG:boto:Method: GET 2024-04-12T03:39:53.293 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.293 DEBUG:boto:Data: 2024-04-12T03:39:53.293 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.293 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.293 DEBUG:boto:Params: {} 2024-04-12T03:39:53.294 DEBUG:boto:Token: None 2024-04-12T03:39:53.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003?versionId=null 2024-04-12T03:39:53.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I= 2024-04-12T03:39:53.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lFwEIdA5V6Nq7uP3v0hZu8O1V3I=', 'Content-Length': '0'} 2024-04-12T03:39:53.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded54d849056ec81-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.296 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.296 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.297 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.297 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.297 DEBUG:boto:Method: GET 2024-04-12T03:39:53.297 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.297 DEBUG:boto:Data: 2024-04-12T03:39:53.297 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.297 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.297 DEBUG:boto:Params: {} 2024-04-12T03:39:53.297 DEBUG:boto:Token: None 2024-04-12T03:39:53.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.298 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000827b2b42bebfda1a-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.300 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.301 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.301 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.301 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.301 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.301 DEBUG:boto:Method: GET 2024-04-12T03:39:53.301 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.301 DEBUG:boto:Data: 2024-04-12T03:39:53.301 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.301 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.302 DEBUG:boto:Params: {} 2024-04-12T03:39:53.302 DEBUG:boto:Token: None 2024-04-12T03:39:53.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2e0b55df1c9807-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.306 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.307 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.307 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.307 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.307 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.307 DEBUG:boto:Data: 2024-04-12T03:39:53.307 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.307 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.307 DEBUG:boto:Params: {} 2024-04-12T03:39:53.308 DEBUG:boto:Token: None 2024-04-12T03:39:53.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qBWn3kRKPC04+2LPKr5risUm664= 2024-04-12T03:39:53.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qBWn3kRKPC04+2LPKr5risUm664=', 'Content-Length': '0'} 2024-04-12T03:39:53.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965dab90e5e130b1-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.310 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.310 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.310 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.310 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.310 DEBUG:boto:Data: 2024-04-12T03:39:53.310 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.310 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.310 DEBUG:boto:Params: {} 2024-04-12T03:39:53.311 DEBUG:boto:Token: None 2024-04-12T03:39:53.311 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qBWn3kRKPC04+2LPKr5risUm664= 2024-04-12T03:39:53.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qBWn3kRKPC04+2LPKr5risUm664=', 'Content-Length': '0'} 2024-04-12T03:39:53.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce19a6f0b3f396c-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:39:53.313 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.313 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.313 DEBUG:boto:Method: GET 2024-04-12T03:39:53.313 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.313 DEBUG:boto:Data: 2024-04-12T03:39:53.314 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.314 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.314 DEBUG:boto:Params: {} 2024-04-12T03:39:53.314 DEBUG:boto:Token: None 2024-04-12T03:39:53.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.314 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OKuGlN6mwmSNkOsIE9CDGplVPGI= 2024-04-12T03:39:53.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OKuGlN6mwmSNkOsIE9CDGplVPGI=', 'Content-Length': '0'} 2024-04-12T03:39:53.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab4efea6f130863-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.316 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.316 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.316 DEBUG:boto:Method: GET 2024-04-12T03:39:53.316 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.316 DEBUG:boto:Data: 2024-04-12T03:39:53.316 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.317 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.317 DEBUG:boto:Params: {} 2024-04-12T03:39:53.317 DEBUG:boto:Token: None 2024-04-12T03:39:53.317 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000003 2024-04-12T03:39:53.317 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OKuGlN6mwmSNkOsIE9CDGplVPGI= 2024-04-12T03:39:53.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OKuGlN6mwmSNkOsIE9CDGplVPGI=', 'Content-Length': '0'} 2024-04-12T03:39:53.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f248f8fbe084b780-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.362 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.363 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.363 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.363 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.363 DEBUG:boto:Method: GET 2024-04-12T03:39:53.363 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.363 DEBUG:boto:Data: 2024-04-12T03:39:53.363 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.363 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.363 DEBUG:boto:Params: {} 2024-04-12T03:39:53.364 DEBUG:boto:Token: None 2024-04-12T03:39:53.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c56f7e7320084bb-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.367 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.367 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.367 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.367 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.367 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.367 DEBUG:boto:Method: GET 2024-04-12T03:39:53.368 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000003 2024-04-12T03:39:53.368 DEBUG:boto:Data: 2024-04-12T03:39:53.368 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.368 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.368 DEBUG:boto:Params: {} 2024-04-12T03:39:53.368 DEBUG:boto:Token: None 2024-04-12T03:39:53.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d61c92da837217f-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.372 DEBUG:boto:b'dgbpxg-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:39:43.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:53.373 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.373 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.373 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.373 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.373 DEBUG:boto:Method: GET 2024-04-12T03:39:53.373 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.373 DEBUG:boto:Data: 2024-04-12T03:39:53.373 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.374 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.374 DEBUG:boto:Params: {} 2024-04-12T03:39:53.374 DEBUG:boto:Token: None 2024-04-12T03:39:53.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.374 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA= 2024-04-12T03:39:53.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA=', 'Content-Length': '0'} 2024-04-12T03:39:53.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008316feeaa91a014a-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.376 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.376 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.376 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.377 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.377 DEBUG:boto:Method: GET 2024-04-12T03:39:53.377 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.377 DEBUG:boto:Data: 2024-04-12T03:39:53.377 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.377 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.377 DEBUG:boto:Params: {} 2024-04-12T03:39:53.377 DEBUG:boto:Token: None 2024-04-12T03:39:53.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA= 2024-04-12T03:39:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA=', 'Content-Length': '0'} 2024-04-12T03:39:53.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b4e3c38ad99a39-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.379 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.380 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.380 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.380 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.380 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.380 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.380 DEBUG:boto:Data: 2024-04-12T03:39:53.380 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.380 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.380 DEBUG:boto:Params: {} 2024-04-12T03:39:53.381 DEBUG:boto:Token: None 2024-04-12T03:39:53.381 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh0ShyAmxLIgz5pinMkEWTD+KgY= 2024-04-12T03:39:53.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh0ShyAmxLIgz5pinMkEWTD+KgY=', 'Content-Length': '0'} 2024-04-12T03:39:53.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9202790578de0e2-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.383 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.383 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.383 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.383 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.383 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.383 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.383 DEBUG:boto:Data: 2024-04-12T03:39:53.383 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.383 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.384 DEBUG:boto:Params: {} 2024-04-12T03:39:53.384 DEBUG:boto:Token: None 2024-04-12T03:39:53.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wh0ShyAmxLIgz5pinMkEWTD+KgY= 2024-04-12T03:39:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wh0ShyAmxLIgz5pinMkEWTD+KgY=', 'Content-Length': '0'} 2024-04-12T03:39:53.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce2c37ec399a0b6-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:53.386 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.386 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.386 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.386 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.386 DEBUG:boto:Method: GET 2024-04-12T03:39:53.387 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.387 DEBUG:boto:Data: 2024-04-12T03:39:53.387 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.387 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.387 DEBUG:boto:Params: {} 2024-04-12T03:39:53.387 DEBUG:boto:Token: None 2024-04-12T03:39:53.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA= 2024-04-12T03:39:53.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA=', 'Content-Length': '0'} 2024-04-12T03:39:53.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fbd04a92606da2c-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.430 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.430 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.431 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.431 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.431 DEBUG:boto:Method: GET 2024-04-12T03:39:53.431 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.431 DEBUG:boto:Data: 2024-04-12T03:39:53.431 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.431 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.431 DEBUG:boto:Params: {} 2024-04-12T03:39:53.432 DEBUG:boto:Token: None 2024-04-12T03:39:53.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004?versionId=null 2024-04-12T03:39:53.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA= 2024-04-12T03:39:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qLBhN04UWXVIqrf39B10rLy0kbA=', 'Content-Length': '0'} 2024-04-12T03:39:53.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000946f4277d731515-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.434 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.434 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.435 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.435 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.435 DEBUG:boto:Method: GET 2024-04-12T03:39:53.435 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.435 DEBUG:boto:Data: 2024-04-12T03:39:53.435 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.435 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.435 DEBUG:boto:Params: {} 2024-04-12T03:39:53.435 DEBUG:boto:Token: None 2024-04-12T03:39:53.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.436 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000effa53c5f73e5186-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.438 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.439 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.439 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.439 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.439 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.439 DEBUG:boto:Method: GET 2024-04-12T03:39:53.439 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.440 DEBUG:boto:Data: 2024-04-12T03:39:53.440 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.440 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.440 DEBUG:boto:Params: {} 2024-04-12T03:39:53.440 DEBUG:boto:Token: None 2024-04-12T03:39:53.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f34be73e2b495d9d-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.446 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.446 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.446 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.447 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.447 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.447 DEBUG:boto:Data: 2024-04-12T03:39:53.447 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.447 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.447 DEBUG:boto:Params: {} 2024-04-12T03:39:53.447 DEBUG:boto:Token: None 2024-04-12T03:39:53.447 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LaOGi0+MNNkD1rVa73hW2QY2g6M= 2024-04-12T03:39:53.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LaOGi0+MNNkD1rVa73hW2QY2g6M=', 'Content-Length': '0'} 2024-04-12T03:39:53.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fedf4017ea58a681-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.450 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.450 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.450 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.450 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.450 DEBUG:boto:Data: 2024-04-12T03:39:53.450 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.451 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.451 DEBUG:boto:Params: {} 2024-04-12T03:39:53.451 DEBUG:boto:Token: None 2024-04-12T03:39:53.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LaOGi0+MNNkD1rVa73hW2QY2g6M= 2024-04-12T03:39:53.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LaOGi0+MNNkD1rVa73hW2QY2g6M=', 'Content-Length': '0'} 2024-04-12T03:39:53.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a17005defe7d07-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:39:53.453 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.453 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.454 DEBUG:boto:Method: GET 2024-04-12T03:39:53.454 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.454 DEBUG:boto:Data: 2024-04-12T03:39:53.454 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.454 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.454 DEBUG:boto:Params: {} 2024-04-12T03:39:53.454 DEBUG:boto:Token: None 2024-04-12T03:39:53.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.455 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rUTIKWSkNy5yG6xDtxJH9ZDuHHY= 2024-04-12T03:39:53.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rUTIKWSkNy5yG6xDtxJH9ZDuHHY=', 'Content-Length': '0'} 2024-04-12T03:39:53.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000190a4891801bc9e4-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.456 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.456 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.457 DEBUG:boto:Method: GET 2024-04-12T03:39:53.457 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.457 DEBUG:boto:Data: 2024-04-12T03:39:53.457 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.457 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.457 DEBUG:boto:Params: {} 2024-04-12T03:39:53.457 DEBUG:boto:Token: None 2024-04-12T03:39:53.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000004 2024-04-12T03:39:53.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rUTIKWSkNy5yG6xDtxJH9ZDuHHY= 2024-04-12T03:39:53.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rUTIKWSkNy5yG6xDtxJH9ZDuHHY=', 'Content-Length': '0'} 2024-04-12T03:39:53.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4f7e8c28be91e1-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.460 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.460 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.460 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.460 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.460 DEBUG:boto:Method: GET 2024-04-12T03:39:53.461 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.461 DEBUG:boto:Data: 2024-04-12T03:39:53.461 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.461 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.461 DEBUG:boto:Params: {} 2024-04-12T03:39:53.461 DEBUG:boto:Token: None 2024-04-12T03:39:53.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a1e94f27328c46-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.464 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.464 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.464 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.465 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.465 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.465 DEBUG:boto:Method: GET 2024-04-12T03:39:53.465 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000004 2024-04-12T03:39:53.465 DEBUG:boto:Data: 2024-04-12T03:39:53.465 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.465 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.465 DEBUG:boto:Params: {} 2024-04-12T03:39:53.465 DEBUG:boto:Token: None 2024-04-12T03:39:53.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c81b624fdf234b-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.470 DEBUG:boto:b'dgbpxg-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:39:43.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:53.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.471 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.471 DEBUG:boto:Method: GET 2024-04-12T03:39:53.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.471 DEBUG:boto:Data: 2024-04-12T03:39:53.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.471 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.471 DEBUG:boto:Params: {} 2024-04-12T03:39:53.471 DEBUG:boto:Token: None 2024-04-12T03:39:53.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk= 2024-04-12T03:39:53.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk=', 'Content-Length': '0'} 2024-04-12T03:39:53.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efc2fa7a948cdefa-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.514 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.515 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.515 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.515 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.515 DEBUG:boto:Method: GET 2024-04-12T03:39:53.515 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.515 DEBUG:boto:Data: 2024-04-12T03:39:53.517 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.517 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.517 DEBUG:boto:Params: {} 2024-04-12T03:39:53.518 DEBUG:boto:Token: None 2024-04-12T03:39:53.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.518 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk= 2024-04-12T03:39:53.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk=', 'Content-Length': '0'} 2024-04-12T03:39:53.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e701d2d8b57081c-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.520 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.520 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.521 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.521 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.521 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.521 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.521 DEBUG:boto:Data: 2024-04-12T03:39:53.521 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.521 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.521 DEBUG:boto:Params: {} 2024-04-12T03:39:53.522 DEBUG:boto:Token: None 2024-04-12T03:39:53.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:su8W59+ckiyG9SXvnVuO4QqtdXE= 2024-04-12T03:39:53.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:su8W59+ckiyG9SXvnVuO4QqtdXE=', 'Content-Length': '0'} 2024-04-12T03:39:53.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c6ed6bcaff4fab-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.524 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.525 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.525 DEBUG:boto:Data: 2024-04-12T03:39:53.525 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.525 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.525 DEBUG:boto:Params: {} 2024-04-12T03:39:53.525 DEBUG:boto:Token: None 2024-04-12T03:39:53.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:su8W59+ckiyG9SXvnVuO4QqtdXE= 2024-04-12T03:39:53.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:su8W59+ckiyG9SXvnVuO4QqtdXE=', 'Content-Length': '0'} 2024-04-12T03:39:53.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d60348418b0d51-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:53.528 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.528 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.528 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.528 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.528 DEBUG:boto:Method: GET 2024-04-12T03:39:53.528 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.528 DEBUG:boto:Data: 2024-04-12T03:39:53.528 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.528 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.529 DEBUG:boto:Params: {} 2024-04-12T03:39:53.529 DEBUG:boto:Token: None 2024-04-12T03:39:53.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk= 2024-04-12T03:39:53.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk=', 'Content-Length': '0'} 2024-04-12T03:39:53.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005de74d588174de2-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.531 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.531 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.531 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.531 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.531 DEBUG:boto:Method: GET 2024-04-12T03:39:53.531 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.532 DEBUG:boto:Data: 2024-04-12T03:39:53.532 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.532 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.532 DEBUG:boto:Params: {} 2024-04-12T03:39:53.532 DEBUG:boto:Token: None 2024-04-12T03:39:53.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005?versionId=null 2024-04-12T03:39:53.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk= 2024-04-12T03:39:53.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zhs/q0C0evy9KZzvKHZ8ksb9VJk=', 'Content-Length': '0'} 2024-04-12T03:39:53.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c39d0677e81e97-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.535 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.535 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.535 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.535 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.535 DEBUG:boto:Method: GET 2024-04-12T03:39:53.535 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.535 DEBUG:boto:Data: 2024-04-12T03:39:53.535 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.536 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.536 DEBUG:boto:Params: {} 2024-04-12T03:39:53.536 DEBUG:boto:Token: None 2024-04-12T03:39:53.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad5701efa12a322-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.539 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.539 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.539 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.540 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.540 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.540 DEBUG:boto:Method: GET 2024-04-12T03:39:53.540 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.540 DEBUG:boto:Data: 2024-04-12T03:39:53.540 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.540 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.540 DEBUG:boto:Params: {} 2024-04-12T03:39:53.540 DEBUG:boto:Token: None 2024-04-12T03:39:53.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5931ed6a57651e-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.545 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.545 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.545 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.545 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.545 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.545 DEBUG:boto:Data: 2024-04-12T03:39:53.545 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.546 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.546 DEBUG:boto:Params: {} 2024-04-12T03:39:53.546 DEBUG:boto:Token: None 2024-04-12T03:39:53.546 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.546 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rKnjeP9+/OCMlSB7N6lfE6KZcI4= 2024-04-12T03:39:53.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rKnjeP9+/OCMlSB7N6lfE6KZcI4=', 'Content-Length': '0'} 2024-04-12T03:39:53.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0eeb9234dedce9-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.548 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.548 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.548 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.548 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.549 DEBUG:boto:Data: 2024-04-12T03:39:53.549 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.549 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.549 DEBUG:boto:Params: {} 2024-04-12T03:39:53.549 DEBUG:boto:Token: None 2024-04-12T03:39:53.549 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rKnjeP9+/OCMlSB7N6lfE6KZcI4= 2024-04-12T03:39:53.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rKnjeP9+/OCMlSB7N6lfE6KZcI4=', 'Content-Length': '0'} 2024-04-12T03:39:53.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072970795e88bb124-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:39:53.552 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.552 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.552 DEBUG:boto:Method: GET 2024-04-12T03:39:53.552 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.552 DEBUG:boto:Data: 2024-04-12T03:39:53.552 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.552 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.552 DEBUG:boto:Params: {} 2024-04-12T03:39:53.553 DEBUG:boto:Token: None 2024-04-12T03:39:53.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:za7N91Ksspwz/NCPct3MxR3ZWkQ= 2024-04-12T03:39:53.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:za7N91Ksspwz/NCPct3MxR3ZWkQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62b91f5144beb94-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.599 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.599 DEBUG:boto:Method: GET 2024-04-12T03:39:53.599 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.599 DEBUG:boto:Data: 2024-04-12T03:39:53.599 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.599 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.599 DEBUG:boto:Params: {} 2024-04-12T03:39:53.599 DEBUG:boto:Token: None 2024-04-12T03:39:53.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000005 2024-04-12T03:39:53.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:za7N91Ksspwz/NCPct3MxR3ZWkQ= 2024-04-12T03:39:53.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:za7N91Ksspwz/NCPct3MxR3ZWkQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4709c07289c387-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.602 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.602 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.602 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.602 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.603 DEBUG:boto:Method: GET 2024-04-12T03:39:53.603 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.603 DEBUG:boto:Data: 2024-04-12T03:39:53.603 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.603 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.603 DEBUG:boto:Params: {} 2024-04-12T03:39:53.603 DEBUG:boto:Token: None 2024-04-12T03:39:53.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5038bcbc3a82eb-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.607 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.607 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.607 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.607 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.607 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.608 DEBUG:boto:Method: GET 2024-04-12T03:39:53.608 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000005 2024-04-12T03:39:53.608 DEBUG:boto:Data: 2024-04-12T03:39:53.608 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.608 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.608 DEBUG:boto:Params: {} 2024-04-12T03:39:53.608 DEBUG:boto:Token: None 2024-04-12T03:39:53.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa214b7fb744cd0d-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.613 DEBUG:boto:b'dgbpxg-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:39:43.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:53.613 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.613 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.613 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.613 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.614 DEBUG:boto:Method: GET 2024-04-12T03:39:53.614 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.614 DEBUG:boto:Data: 2024-04-12T03:39:53.614 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.614 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.614 DEBUG:boto:Params: {} 2024-04-12T03:39:53.614 DEBUG:boto:Token: None 2024-04-12T03:39:53.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE= 2024-04-12T03:39:53.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE=', 'Content-Length': '0'} 2024-04-12T03:39:53.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c14d495aece3da6c-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.617 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.617 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.617 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.617 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.617 DEBUG:boto:Method: GET 2024-04-12T03:39:53.617 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.618 DEBUG:boto:Data: 2024-04-12T03:39:53.618 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.618 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.618 DEBUG:boto:Params: {} 2024-04-12T03:39:53.618 DEBUG:boto:Token: None 2024-04-12T03:39:53.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE= 2024-04-12T03:39:53.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE=', 'Content-Length': '0'} 2024-04-12T03:39:53.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc2fd752e138450-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.621 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.621 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.621 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.621 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.621 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.621 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.621 DEBUG:boto:Data: 2024-04-12T03:39:53.622 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.622 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.622 DEBUG:boto:Params: {} 2024-04-12T03:39:53.622 DEBUG:boto:Token: None 2024-04-12T03:39:53.622 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.622 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1sCmrZDyOjus9GS35f1RRYHwo9E= 2024-04-12T03:39:53.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1sCmrZDyOjus9GS35f1RRYHwo9E=', 'Content-Length': '0'} 2024-04-12T03:39:53.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1fb33d4fe40ed0-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.624 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.624 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.624 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.624 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.625 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.625 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.625 DEBUG:boto:Data: 2024-04-12T03:39:53.625 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.625 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.625 DEBUG:boto:Params: {} 2024-04-12T03:39:53.625 DEBUG:boto:Token: None 2024-04-12T03:39:53.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1sCmrZDyOjus9GS35f1RRYHwo9E= 2024-04-12T03:39:53.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1sCmrZDyOjus9GS35f1RRYHwo9E=', 'Content-Length': '0'} 2024-04-12T03:39:53.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000760a8a8c83f3b36f-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:53.628 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.628 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.628 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.628 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.628 DEBUG:boto:Method: GET 2024-04-12T03:39:53.629 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.629 DEBUG:boto:Data: 2024-04-12T03:39:53.629 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.629 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.629 DEBUG:boto:Params: {} 2024-04-12T03:39:53.629 DEBUG:boto:Token: None 2024-04-12T03:39:53.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.630 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE= 2024-04-12T03:39:53.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE=', 'Content-Length': '0'} 2024-04-12T03:39:53.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51a46e42be2e70a-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.631 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.631 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.632 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.632 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.632 DEBUG:boto:Method: GET 2024-04-12T03:39:53.632 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.632 DEBUG:boto:Data: 2024-04-12T03:39:53.632 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.632 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.632 DEBUG:boto:Params: {} 2024-04-12T03:39:53.632 DEBUG:boto:Token: None 2024-04-12T03:39:53.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006?versionId=null 2024-04-12T03:39:53.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE= 2024-04-12T03:39:53.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gjM0MrlqiKJ2zd7fIAls1ld6tRE=', 'Content-Length': '0'} 2024-04-12T03:39:53.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776c9f2a1ada42e1-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.635 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.635 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.635 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.635 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.635 DEBUG:boto:Method: GET 2024-04-12T03:39:53.636 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.636 DEBUG:boto:Data: 2024-04-12T03:39:53.636 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.636 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.636 DEBUG:boto:Params: {} 2024-04-12T03:39:53.636 DEBUG:boto:Token: None 2024-04-12T03:39:53.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070cc06448230993e-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.682 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.683 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.683 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.683 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.683 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.683 DEBUG:boto:Method: GET 2024-04-12T03:39:53.683 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.683 DEBUG:boto:Data: 2024-04-12T03:39:53.683 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.684 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.684 DEBUG:boto:Params: {} 2024-04-12T03:39:53.684 DEBUG:boto:Token: None 2024-04-12T03:39:53.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c67c8dfb1d6894-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.688 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.689 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.689 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.689 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.689 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.689 DEBUG:boto:Data: 2024-04-12T03:39:53.689 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.689 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.689 DEBUG:boto:Params: {} 2024-04-12T03:39:53.689 DEBUG:boto:Token: None 2024-04-12T03:39:53.690 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.690 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpUYAySeVAmjOQvwkVjemiAxDTE= 2024-04-12T03:39:53.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpUYAySeVAmjOQvwkVjemiAxDTE=', 'Content-Length': '0'} 2024-04-12T03:39:53.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008718137e5ef17288-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.692 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.692 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.692 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.692 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.692 DEBUG:boto:Data: 2024-04-12T03:39:53.692 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.692 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.692 DEBUG:boto:Params: {} 2024-04-12T03:39:53.692 DEBUG:boto:Token: None 2024-04-12T03:39:53.693 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpUYAySeVAmjOQvwkVjemiAxDTE= 2024-04-12T03:39:53.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpUYAySeVAmjOQvwkVjemiAxDTE=', 'Content-Length': '0'} 2024-04-12T03:39:53.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ada7888ced9cca-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:39:53.695 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.695 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.695 DEBUG:boto:Method: GET 2024-04-12T03:39:53.695 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.695 DEBUG:boto:Data: 2024-04-12T03:39:53.695 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.695 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.695 DEBUG:boto:Params: {} 2024-04-12T03:39:53.696 DEBUG:boto:Token: None 2024-04-12T03:39:53.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j00kztaLnBFqDijawQNKz9v1BhA= 2024-04-12T03:39:53.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j00kztaLnBFqDijawQNKz9v1BhA=', 'Content-Length': '0'} 2024-04-12T03:39:53.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047650763d5fd5d16-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.698 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.699 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.699 DEBUG:boto:Method: GET 2024-04-12T03:39:53.699 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.699 DEBUG:boto:Data: 2024-04-12T03:39:53.699 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.699 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.699 DEBUG:boto:Params: {} 2024-04-12T03:39:53.699 DEBUG:boto:Token: None 2024-04-12T03:39:53.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000006 2024-04-12T03:39:53.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j00kztaLnBFqDijawQNKz9v1BhA= 2024-04-12T03:39:53.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j00kztaLnBFqDijawQNKz9v1BhA=', 'Content-Length': '0'} 2024-04-12T03:39:53.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feee7ba7ba3a8e31-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.746 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.747 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.747 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.747 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.747 DEBUG:boto:Method: GET 2024-04-12T03:39:53.747 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.747 DEBUG:boto:Data: 2024-04-12T03:39:53.748 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.748 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.748 DEBUG:boto:Params: {} 2024-04-12T03:39:53.748 DEBUG:boto:Token: None 2024-04-12T03:39:53.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eee2b5851093fa5-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.751 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.751 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.751 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.751 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.752 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.752 DEBUG:boto:Method: GET 2024-04-12T03:39:53.752 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000006 2024-04-12T03:39:53.752 DEBUG:boto:Data: 2024-04-12T03:39:53.752 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.752 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.752 DEBUG:boto:Params: {} 2024-04-12T03:39:53.752 DEBUG:boto:Token: None 2024-04-12T03:39:53.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e283b8bae19e8fe3-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.756 DEBUG:boto:b'dgbpxg-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:39:43.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:39:53.757 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.757 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.757 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.757 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.757 DEBUG:boto:Method: GET 2024-04-12T03:39:53.757 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.757 DEBUG:boto:Data: 2024-04-12T03:39:53.758 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.758 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.758 DEBUG:boto:Params: {} 2024-04-12T03:39:53.758 DEBUG:boto:Token: None 2024-04-12T03:39:53.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE= 2024-04-12T03:39:53.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE=', 'Content-Length': '0'} 2024-04-12T03:39:53.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087954a87164cf6a3-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.760 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.760 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.761 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.761 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.761 DEBUG:boto:Method: GET 2024-04-12T03:39:53.761 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.761 DEBUG:boto:Data: 2024-04-12T03:39:53.761 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.761 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.761 DEBUG:boto:Params: {} 2024-04-12T03:39:53.761 DEBUG:boto:Token: None 2024-04-12T03:39:53.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.762 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE= 2024-04-12T03:39:53.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE=', 'Content-Length': '0'} 2024-04-12T03:39:53.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c338be7e780846a-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.764 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.764 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.764 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.764 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.764 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.764 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.764 DEBUG:boto:Data: 2024-04-12T03:39:53.764 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.765 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.765 DEBUG:boto:Params: {} 2024-04-12T03:39:53.765 DEBUG:boto:Token: None 2024-04-12T03:39:53.765 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.765 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VV0WF30Fo7Y815bY25Hzd4NWwdU= 2024-04-12T03:39:53.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VV0WF30Fo7Y815bY25Hzd4NWwdU=', 'Content-Length': '0'} 2024-04-12T03:39:53.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6121113402a1fd-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.767 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.767 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.767 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.767 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.767 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.767 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.767 DEBUG:boto:Data: 2024-04-12T03:39:53.767 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.768 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.768 DEBUG:boto:Params: {} 2024-04-12T03:39:53.768 DEBUG:boto:Token: None 2024-04-12T03:39:53.768 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VV0WF30Fo7Y815bY25Hzd4NWwdU= 2024-04-12T03:39:53.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VV0WF30Fo7Y815bY25Hzd4NWwdU=', 'Content-Length': '0'} 2024-04-12T03:39:53.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db09c1914efa2fb-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:39:53.770 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.770 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.770 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.770 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.770 DEBUG:boto:Method: GET 2024-04-12T03:39:53.770 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.770 DEBUG:boto:Data: 2024-04-12T03:39:53.770 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.770 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.771 DEBUG:boto:Params: {} 2024-04-12T03:39:53.771 DEBUG:boto:Token: None 2024-04-12T03:39:53.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE= 2024-04-12T03:39:53.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE=', 'Content-Length': '0'} 2024-04-12T03:39:53.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bd70345730175c-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.814 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.814 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.815 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.815 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.815 DEBUG:boto:Method: GET 2024-04-12T03:39:53.815 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.815 DEBUG:boto:Data: 2024-04-12T03:39:53.815 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.815 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.815 DEBUG:boto:Params: {} 2024-04-12T03:39:53.816 DEBUG:boto:Token: None 2024-04-12T03:39:53.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007?versionId=null 2024-04-12T03:39:53.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE= 2024-04-12T03:39:53.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CWnJDRFmplyuJmCIT5H5vL+kvwE=', 'Content-Length': '0'} 2024-04-12T03:39:53.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630c623477d6ee84-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.818 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.818 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.818 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.818 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.818 DEBUG:boto:Method: GET 2024-04-12T03:39:53.818 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.819 DEBUG:boto:Data: 2024-04-12T03:39:53.819 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.819 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.819 DEBUG:boto:Params: {} 2024-04-12T03:39:53.819 DEBUG:boto:Token: None 2024-04-12T03:39:53.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc87a0e926a0a12-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.822 DEBUG:boto:b'dgbpxg-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.822 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.822 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.822 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.822 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.822 DEBUG:boto:Method: GET 2024-04-12T03:39:53.822 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.823 DEBUG:boto:Data: 2024-04-12T03:39:53.823 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.823 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.823 DEBUG:boto:Params: {} 2024-04-12T03:39:53.823 DEBUG:boto:Token: None 2024-04-12T03:39:53.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c50d70f8da1d52-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.827 DEBUG:boto:b'dgbpxg-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.827 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.828 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.828 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.828 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.828 DEBUG:boto:Data: 2024-04-12T03:39:53.828 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.829 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.829 DEBUG:boto:Params: {} 2024-04-12T03:39:53.829 DEBUG:boto:Token: None 2024-04-12T03:39:53.830 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:izH50JZVgS5CHIGSQQT2aQ2MdbI= 2024-04-12T03:39:53.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:izH50JZVgS5CHIGSQQT2aQ2MdbI=', 'Content-Length': '0'} 2024-04-12T03:39:53.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dd21dc7843603b-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.832 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.833 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.833 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.833 DEBUG:boto:Data: 2024-04-12T03:39:53.833 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.833 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.833 DEBUG:boto:Params: {} 2024-04-12T03:39:53.833 DEBUG:boto:Token: None 2024-04-12T03:39:53.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.834 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:izH50JZVgS5CHIGSQQT2aQ2MdbI= 2024-04-12T03:39:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:izH50JZVgS5CHIGSQQT2aQ2MdbI=', 'Content-Length': '0'} 2024-04-12T03:39:53.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e41555ad59060718-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:39:53.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.836 DEBUG:boto:Method: GET 2024-04-12T03:39:53.836 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.837 DEBUG:boto:Data: 2024-04-12T03:39:53.837 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.837 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.837 DEBUG:boto:Params: {} 2024-04-12T03:39:53.837 DEBUG:boto:Token: None 2024-04-12T03:39:53.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kQQWy/TLIihvVB6VlvpzoJ9XBdA= 2024-04-12T03:39:53.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kQQWy/TLIihvVB6VlvpzoJ9XBdA=', 'Content-Length': '0'} 2024-04-12T03:39:53.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bdb7da4bef0934-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.839 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.840 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.840 DEBUG:boto:Method: GET 2024-04-12T03:39:53.840 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.840 DEBUG:boto:Data: 2024-04-12T03:39:53.840 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.840 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.840 DEBUG:boto:Params: {} 2024-04-12T03:39:53.840 DEBUG:boto:Token: None 2024-04-12T03:39:53.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000007 2024-04-12T03:39:53.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kQQWy/TLIihvVB6VlvpzoJ9XBdA= 2024-04-12T03:39:53.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kQQWy/TLIihvVB6VlvpzoJ9XBdA=', 'Content-Length': '0'} 2024-04-12T03:39:53.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f4106530f1bc98-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.882 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.882 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.882 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.883 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.883 DEBUG:boto:Method: GET 2024-04-12T03:39:53.883 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.883 DEBUG:boto:Data: 2024-04-12T03:39:53.883 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.883 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.883 DEBUG:boto:Params: {} 2024-04-12T03:39:53.883 DEBUG:boto:Token: None 2024-04-12T03:39:53.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e401cfced8b8c2-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.886 DEBUG:boto:b'dgbpxg-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.887 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.887 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.887 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.889 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.889 DEBUG:boto:Method: GET 2024-04-12T03:39:53.889 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000007 2024-04-12T03:39:53.890 DEBUG:boto:Data: 2024-04-12T03:39:53.890 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.890 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.890 DEBUG:boto:Params: {} 2024-04-12T03:39:53.890 DEBUG:boto:Token: None 2024-04-12T03:39:53.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da98ebc370dc601-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.895 DEBUG:boto:b'dgbpxg-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:39:43.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:39:53.895 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.895 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.895 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.895 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.895 DEBUG:boto:Method: GET 2024-04-12T03:39:53.896 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.896 DEBUG:boto:Data: 2024-04-12T03:39:53.896 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.896 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.896 DEBUG:boto:Params: {} 2024-04-12T03:39:53.896 DEBUG:boto:Token: None 2024-04-12T03:39:53.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU= 2024-04-12T03:39:53.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU=', 'Content-Length': '0'} 2024-04-12T03:39:53.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb55535b5054b84b-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.899 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.899 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.899 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.899 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.900 DEBUG:boto:Method: GET 2024-04-12T03:39:53.900 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.900 DEBUG:boto:Data: 2024-04-12T03:39:53.900 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.900 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.900 DEBUG:boto:Params: {} 2024-04-12T03:39:53.900 DEBUG:boto:Token: None 2024-04-12T03:39:53.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU= 2024-04-12T03:39:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU=', 'Content-Length': '0'} 2024-04-12T03:39:53.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb07990a73867029-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.903 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.903 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.903 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.903 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.903 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.903 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.903 DEBUG:boto:Data: 2024-04-12T03:39:53.903 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.903 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.903 DEBUG:boto:Params: {} 2024-04-12T03:39:53.904 DEBUG:boto:Token: None 2024-04-12T03:39:53.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lMlg3jTGxg9TU9RnEj+IOOvtsSQ= 2024-04-12T03:39:53.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lMlg3jTGxg9TU9RnEj+IOOvtsSQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019cb5185ff7d7ed7-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.906 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.906 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.906 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.906 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.906 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.906 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.907 DEBUG:boto:Data: 2024-04-12T03:39:53.907 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.907 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.907 DEBUG:boto:Params: {} 2024-04-12T03:39:53.907 DEBUG:boto:Token: None 2024-04-12T03:39:53.907 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lMlg3jTGxg9TU9RnEj+IOOvtsSQ= 2024-04-12T03:39:53.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lMlg3jTGxg9TU9RnEj+IOOvtsSQ=', 'Content-Length': '0'} 2024-04-12T03:39:53.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e62602d4fa738ec7-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:39:53.909 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.910 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.910 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.910 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.910 DEBUG:boto:Method: GET 2024-04-12T03:39:53.910 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.910 DEBUG:boto:Data: 2024-04-12T03:39:53.910 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.910 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.910 DEBUG:boto:Params: {} 2024-04-12T03:39:53.911 DEBUG:boto:Token: None 2024-04-12T03:39:53.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU= 2024-04-12T03:39:53.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU=', 'Content-Length': '0'} 2024-04-12T03:39:53.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39cee397947b2ab-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.913 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.913 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.913 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.913 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.913 DEBUG:boto:Method: GET 2024-04-12T03:39:53.913 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.913 DEBUG:boto:Data: 2024-04-12T03:39:53.914 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.914 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.914 DEBUG:boto:Params: {} 2024-04-12T03:39:53.914 DEBUG:boto:Token: None 2024-04-12T03:39:53.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008?versionId=null 2024-04-12T03:39:53.914 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU= 2024-04-12T03:39:53.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MCnju4sx4c4JWVoFYsVnI+NXIvU=', 'Content-Length': '0'} 2024-04-12T03:39:53.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a75218eb614195-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.958 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.958 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.959 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.959 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.959 DEBUG:boto:Method: GET 2024-04-12T03:39:53.959 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.959 DEBUG:boto:Data: 2024-04-12T03:39:53.959 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.959 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.959 DEBUG:boto:Params: {} 2024-04-12T03:39:53.959 DEBUG:boto:Token: None 2024-04-12T03:39:53.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d35eb34c930aba-006618ad09-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.962 DEBUG:boto:b'dgbpxg-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.962 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:53.963 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:53.963 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.963 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.963 DEBUG:boto:Method: GET 2024-04-12T03:39:53.963 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:53.963 DEBUG:boto:Data: 2024-04-12T03:39:53.963 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.963 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.963 DEBUG:boto:Params: {} 2024-04-12T03:39:53.963 DEBUG:boto:Token: None 2024-04-12T03:39:53.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/ 2024-04-12T03:39:53.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g= 2024-04-12T03:39:53.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2tVc63z2FK0GKHpQ6WyAJ6DUN8g=', 'Content-Length': '0'} 2024-04-12T03:39:53.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84a732dde789f85-006618ad09-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.967 DEBUG:boto:b'dgbpxg-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:53.968 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.968 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.968 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.968 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.968 DEBUG:boto:Data: 2024-04-12T03:39:53.968 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.968 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.968 DEBUG:boto:Params: {} 2024-04-12T03:39:53.969 DEBUG:boto:Token: None 2024-04-12T03:39:53.969 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qdElmfXdiiwFKLtPIJge1AKYy3s= 2024-04-12T03:39:53.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qdElmfXdiiwFKLtPIJge1AKYy3s=', 'Content-Length': '0'} 2024-04-12T03:39:53.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d0f669be8188b8-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.971 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.971 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.971 DEBUG:boto:Method: HEAD 2024-04-12T03:39:53.971 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.971 DEBUG:boto:Data: 2024-04-12T03:39:53.971 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:53.971 DEBUG:boto:Port: 8000 2024-04-12T03:39:53.971 DEBUG:boto:Params: {} 2024-04-12T03:39:53.972 DEBUG:boto:Token: None 2024-04-12T03:39:53.972 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qdElmfXdiiwFKLtPIJge1AKYy3s= 2024-04-12T03:39:53.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qdElmfXdiiwFKLtPIJge1AKYy3s=', 'Content-Length': '0'} 2024-04-12T03:39:53.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9234cc7b481a6f8-006618ad09-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:53.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:39:53.974 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.974 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.974 DEBUG:boto:Method: GET 2024-04-12T03:39:53.974 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.974 DEBUG:boto:Data: 2024-04-12T03:39:53.974 DEBUG:boto:Headers: {} 2024-04-12T03:39:53.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:53.974 DEBUG:boto:Port: 8001 2024-04-12T03:39:53.975 DEBUG:boto:Params: {} 2024-04-12T03:39:53.975 DEBUG:boto:Token: None 2024-04-12T03:39:53.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:53 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:53.975 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rmmYBa+syH4VADhiHyGdENqkVsY= 2024-04-12T03:39:53.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rmmYBa+syH4VADhiHyGdENqkVsY=', 'Content-Length': '0'} 2024-04-12T03:39:53.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf75e3ee0beec4d-006618ad09-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.022 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:54.023 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:54.023 DEBUG:boto:Method: GET 2024-04-12T03:39:54.023 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:54.023 DEBUG:boto:Data: 2024-04-12T03:39:54.023 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.023 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.023 DEBUG:boto:Params: {} 2024-04-12T03:39:54.024 DEBUG:boto:Token: None 2024-04-12T03:39:54.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000008 2024-04-12T03:39:54.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K9w5005OAOMk2XpNMQdyIh5JfNo= 2024-04-12T03:39:54.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K9w5005OAOMk2XpNMQdyIh5JfNo=', 'Content-Length': '0'} 2024-04-12T03:39:54.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0ad34af9444b65-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.026 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.026 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.026 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.026 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.026 DEBUG:boto:Method: GET 2024-04-12T03:39:54.026 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.027 DEBUG:boto:Data: 2024-04-12T03:39:54.027 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.027 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.027 DEBUG:boto:Params: {} 2024-04-12T03:39:54.027 DEBUG:boto:Token: None 2024-04-12T03:39:54.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012792911d70a69cd-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.030 DEBUG:boto:b'dgbpxg-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.030 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.030 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.030 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.030 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.031 DEBUG:boto:Method: GET 2024-04-12T03:39:54.031 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000008 2024-04-12T03:39:54.031 DEBUG:boto:Data: 2024-04-12T03:39:54.031 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.031 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.031 DEBUG:boto:Params: {} 2024-04-12T03:39:54.031 DEBUG:boto:Token: None 2024-04-12T03:39:54.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0ec32f9e260a4c-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.035 DEBUG:boto:b'dgbpxg-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:39:43.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:39:54.035 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.036 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.036 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.036 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.036 DEBUG:boto:Method: GET 2024-04-12T03:39:54.036 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.036 DEBUG:boto:Data: 2024-04-12T03:39:54.036 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.036 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.037 DEBUG:boto:Params: {} 2024-04-12T03:39:54.037 DEBUG:boto:Token: None 2024-04-12T03:39:54.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528= 2024-04-12T03:39:54.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528=', 'Content-Length': '0'} 2024-04-12T03:39:54.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063578cb584513b1b-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.039 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.039 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.039 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.039 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.040 DEBUG:boto:Method: GET 2024-04-12T03:39:54.040 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.040 DEBUG:boto:Data: 2024-04-12T03:39:54.040 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.040 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.040 DEBUG:boto:Params: {} 2024-04-12T03:39:54.040 DEBUG:boto:Token: None 2024-04-12T03:39:54.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528= 2024-04-12T03:39:54.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528=', 'Content-Length': '0'} 2024-04-12T03:39:54.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e0a08c391969fa-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.086 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.086 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.087 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.087 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.087 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.087 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.087 DEBUG:boto:Data: 2024-04-12T03:39:54.087 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.087 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.087 DEBUG:boto:Params: {} 2024-04-12T03:39:54.088 DEBUG:boto:Token: None 2024-04-12T03:39:54.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t8TAAZMcdGZj/15jU3wdbg+xyUA= 2024-04-12T03:39:54.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t8TAAZMcdGZj/15jU3wdbg+xyUA=', 'Content-Length': '0'} 2024-04-12T03:39:54.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe3ad25d96aac015-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.090 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.090 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.090 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.090 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.090 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.091 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.091 DEBUG:boto:Data: 2024-04-12T03:39:54.091 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.091 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.091 DEBUG:boto:Params: {} 2024-04-12T03:39:54.091 DEBUG:boto:Token: None 2024-04-12T03:39:54.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t8TAAZMcdGZj/15jU3wdbg+xyUA= 2024-04-12T03:39:54.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t8TAAZMcdGZj/15jU3wdbg+xyUA=', 'Content-Length': '0'} 2024-04-12T03:39:54.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e10df48b327923-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:39:54.093 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.093 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.094 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.094 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.094 DEBUG:boto:Method: GET 2024-04-12T03:39:54.094 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.094 DEBUG:boto:Data: 2024-04-12T03:39:54.094 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.094 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.094 DEBUG:boto:Params: {} 2024-04-12T03:39:54.095 DEBUG:boto:Token: None 2024-04-12T03:39:54.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528= 2024-04-12T03:39:54.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528=', 'Content-Length': '0'} 2024-04-12T03:39:54.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003558d1135eaa2427-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.097 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.097 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.097 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.097 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.098 DEBUG:boto:Method: GET 2024-04-12T03:39:54.098 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.098 DEBUG:boto:Data: 2024-04-12T03:39:54.098 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.098 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.098 DEBUG:boto:Params: {} 2024-04-12T03:39:54.098 DEBUG:boto:Token: None 2024-04-12T03:39:54.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009?versionId=null 2024-04-12T03:39:54.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528= 2024-04-12T03:39:54.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6OuLS9kIPpL3xvLHvoJjanEl528=', 'Content-Length': '0'} 2024-04-12T03:39:54.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a44f4aae124a436-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.100 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.101 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.101 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.101 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.101 DEBUG:boto:Method: GET 2024-04-12T03:39:54.101 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.101 DEBUG:boto:Data: 2024-04-12T03:39:54.101 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.101 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.101 DEBUG:boto:Params: {} 2024-04-12T03:39:54.102 DEBUG:boto:Token: None 2024-04-12T03:39:54.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b885ce05cf26a58-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.104 DEBUG:boto:b'dgbpxg-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.104 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.105 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.105 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.105 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.105 DEBUG:boto:Method: GET 2024-04-12T03:39:54.105 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.105 DEBUG:boto:Data: 2024-04-12T03:39:54.105 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.105 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.106 DEBUG:boto:Params: {} 2024-04-12T03:39:54.106 DEBUG:boto:Token: None 2024-04-12T03:39:54.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d036192bde9e3e8-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.110 DEBUG:boto:b'dgbpxg-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.110 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.111 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.111 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.111 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.111 DEBUG:boto:Data: 2024-04-12T03:39:54.112 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.112 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.112 DEBUG:boto:Params: {} 2024-04-12T03:39:54.112 DEBUG:boto:Token: None 2024-04-12T03:39:54.113 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5MR9Mb1d4LzLAbN9qrlOhhv+Aes= 2024-04-12T03:39:54.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5MR9Mb1d4LzLAbN9qrlOhhv+Aes=', 'Content-Length': '0'} 2024-04-12T03:39:54.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8483f635d55598a-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.115 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.115 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.115 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.115 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.115 DEBUG:boto:Data: 2024-04-12T03:39:54.115 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.115 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.116 DEBUG:boto:Params: {} 2024-04-12T03:39:54.116 DEBUG:boto:Token: None 2024-04-12T03:39:54.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5MR9Mb1d4LzLAbN9qrlOhhv+Aes= 2024-04-12T03:39:54.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5MR9Mb1d4LzLAbN9qrlOhhv+Aes=', 'Content-Length': '0'} 2024-04-12T03:39:54.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b82fdcfbbf4a3b-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:39:54.118 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.118 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.118 DEBUG:boto:Method: GET 2024-04-12T03:39:54.118 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.118 DEBUG:boto:Data: 2024-04-12T03:39:54.118 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.119 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.119 DEBUG:boto:Params: {} 2024-04-12T03:39:54.119 DEBUG:boto:Token: None 2024-04-12T03:39:54.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TfsizozXjWCHVBL1KWtD/goN3jo= 2024-04-12T03:39:54.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TfsizozXjWCHVBL1KWtD/goN3jo=', 'Content-Length': '0'} 2024-04-12T03:39:54.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f909e78a0e61c01-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.121 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.121 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.121 DEBUG:boto:Method: GET 2024-04-12T03:39:54.121 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.121 DEBUG:boto:Data: 2024-04-12T03:39:54.121 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.121 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.122 DEBUG:boto:Params: {} 2024-04-12T03:39:54.122 DEBUG:boto:Token: None 2024-04-12T03:39:54.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000009 2024-04-12T03:39:54.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TfsizozXjWCHVBL1KWtD/goN3jo= 2024-04-12T03:39:54.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TfsizozXjWCHVBL1KWtD/goN3jo=', 'Content-Length': '0'} 2024-04-12T03:39:54.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076273bb23ae55a89-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.124 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.124 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.125 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.125 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.125 DEBUG:boto:Method: GET 2024-04-12T03:39:54.125 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.125 DEBUG:boto:Data: 2024-04-12T03:39:54.125 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.125 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.126 DEBUG:boto:Params: {} 2024-04-12T03:39:54.126 DEBUG:boto:Token: None 2024-04-12T03:39:54.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001091686e3278cc70-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.128 DEBUG:boto:b'dgbpxg-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.129 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.129 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.129 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.129 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.129 DEBUG:boto:Method: GET 2024-04-12T03:39:54.129 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000009 2024-04-12T03:39:54.129 DEBUG:boto:Data: 2024-04-12T03:39:54.129 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.130 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.130 DEBUG:boto:Params: {} 2024-04-12T03:39:54.130 DEBUG:boto:Token: None 2024-04-12T03:39:54.130 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be306f7ec31107fc-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.134 DEBUG:boto:b'dgbpxg-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:39:44.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:54.135 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.135 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.135 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.135 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.135 DEBUG:boto:Method: GET 2024-04-12T03:39:54.135 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.135 DEBUG:boto:Data: 2024-04-12T03:39:54.135 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.135 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.136 DEBUG:boto:Params: {} 2024-04-12T03:39:54.136 DEBUG:boto:Token: None 2024-04-12T03:39:54.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8= 2024-04-12T03:39:54.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8=', 'Content-Length': '0'} 2024-04-12T03:39:54.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0934473e69be7d7-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.139 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.139 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.139 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.139 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.140 DEBUG:boto:Method: GET 2024-04-12T03:39:54.140 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.140 DEBUG:boto:Data: 2024-04-12T03:39:54.140 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.140 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.140 DEBUG:boto:Params: {} 2024-04-12T03:39:54.140 DEBUG:boto:Token: None 2024-04-12T03:39:54.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8= 2024-04-12T03:39:54.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8=', 'Content-Length': '0'} 2024-04-12T03:39:54.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fc2c8e83b59e38-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.142 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.143 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.143 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.143 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.143 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.143 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.143 DEBUG:boto:Data: 2024-04-12T03:39:54.143 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.143 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.143 DEBUG:boto:Params: {} 2024-04-12T03:39:54.143 DEBUG:boto:Token: None 2024-04-12T03:39:54.144 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4qir1riTIrfUWV31NOBNcdjdw28= 2024-04-12T03:39:54.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4qir1riTIrfUWV31NOBNcdjdw28=', 'Content-Length': '0'} 2024-04-12T03:39:54.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004548dfef51a955bf-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.146 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.146 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.146 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.146 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.146 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.146 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.146 DEBUG:boto:Data: 2024-04-12T03:39:54.146 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.147 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.147 DEBUG:boto:Params: {} 2024-04-12T03:39:54.147 DEBUG:boto:Token: None 2024-04-12T03:39:54.147 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4qir1riTIrfUWV31NOBNcdjdw28= 2024-04-12T03:39:54.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4qir1riTIrfUWV31NOBNcdjdw28=', 'Content-Length': '0'} 2024-04-12T03:39:54.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b69097e6c54690-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:54.149 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.149 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.149 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.149 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.149 DEBUG:boto:Method: GET 2024-04-12T03:39:54.149 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.149 DEBUG:boto:Data: 2024-04-12T03:39:54.150 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.150 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.150 DEBUG:boto:Params: {} 2024-04-12T03:39:54.150 DEBUG:boto:Token: None 2024-04-12T03:39:54.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8= 2024-04-12T03:39:54.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8=', 'Content-Length': '0'} 2024-04-12T03:39:54.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda1c89b1a6e5f27-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.194 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.194 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.195 DEBUG:boto:Method: GET 2024-04-12T03:39:54.195 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.195 DEBUG:boto:Data: 2024-04-12T03:39:54.195 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.195 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.195 DEBUG:boto:Params: {} 2024-04-12T03:39:54.195 DEBUG:boto:Token: None 2024-04-12T03:39:54.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010?versionId=null 2024-04-12T03:39:54.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8= 2024-04-12T03:39:54.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wyTWhu85c1CRwmIUYSZLBgcLpO8=', 'Content-Length': '0'} 2024-04-12T03:39:54.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d56a95f33d9f67-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.198 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.198 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.198 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.198 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.198 DEBUG:boto:Method: GET 2024-04-12T03:39:54.198 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.198 DEBUG:boto:Data: 2024-04-12T03:39:54.198 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.199 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.199 DEBUG:boto:Params: {} 2024-04-12T03:39:54.199 DEBUG:boto:Token: None 2024-04-12T03:39:54.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d282fca361281a2e-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.202 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.203 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.203 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.203 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.203 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.203 DEBUG:boto:Method: GET 2024-04-12T03:39:54.203 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.203 DEBUG:boto:Data: 2024-04-12T03:39:54.203 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.203 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.203 DEBUG:boto:Params: {} 2024-04-12T03:39:54.204 DEBUG:boto:Token: None 2024-04-12T03:39:54.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985692be56204e5e-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.207 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.208 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.208 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.208 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.208 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.208 DEBUG:boto:Data: 2024-04-12T03:39:54.208 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.208 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.208 DEBUG:boto:Params: {} 2024-04-12T03:39:54.208 DEBUG:boto:Token: None 2024-04-12T03:39:54.209 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.209 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uZe3aNUgCsvnohcalXs1KNxHNI= 2024-04-12T03:39:54.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uZe3aNUgCsvnohcalXs1KNxHNI=', 'Content-Length': '0'} 2024-04-12T03:39:54.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf27376bf2b5ae31-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.211 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.211 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.211 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.211 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.211 DEBUG:boto:Data: 2024-04-12T03:39:54.211 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.211 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.212 DEBUG:boto:Params: {} 2024-04-12T03:39:54.212 DEBUG:boto:Token: None 2024-04-12T03:39:54.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uZe3aNUgCsvnohcalXs1KNxHNI= 2024-04-12T03:39:54.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uZe3aNUgCsvnohcalXs1KNxHNI=', 'Content-Length': '0'} 2024-04-12T03:39:54.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a965809ae82938c0-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:39:54.215 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.215 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.215 DEBUG:boto:Method: GET 2024-04-12T03:39:54.215 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.215 DEBUG:boto:Data: 2024-04-12T03:39:54.216 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.216 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.216 DEBUG:boto:Params: {} 2024-04-12T03:39:54.217 DEBUG:boto:Token: None 2024-04-12T03:39:54.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.217 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vlN1g24DZWbVbV1B/YrnmuYvYko= 2024-04-12T03:39:54.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vlN1g24DZWbVbV1B/YrnmuYvYko=', 'Content-Length': '0'} 2024-04-12T03:39:54.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a885d0073028082-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.266 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.266 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.267 DEBUG:boto:Method: GET 2024-04-12T03:39:54.267 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.267 DEBUG:boto:Data: 2024-04-12T03:39:54.267 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.267 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.267 DEBUG:boto:Params: {} 2024-04-12T03:39:54.267 DEBUG:boto:Token: None 2024-04-12T03:39:54.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000010 2024-04-12T03:39:54.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vlN1g24DZWbVbV1B/YrnmuYvYko= 2024-04-12T03:39:54.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vlN1g24DZWbVbV1B/YrnmuYvYko=', 'Content-Length': '0'} 2024-04-12T03:39:54.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741170d0f9c74eb6-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.270 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.270 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.270 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.270 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.270 DEBUG:boto:Method: GET 2024-04-12T03:39:54.270 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.270 DEBUG:boto:Data: 2024-04-12T03:39:54.270 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.271 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.271 DEBUG:boto:Params: {} 2024-04-12T03:39:54.271 DEBUG:boto:Token: None 2024-04-12T03:39:54.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc2a73da9da28af-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.274 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.274 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.274 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.274 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.274 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.274 DEBUG:boto:Method: GET 2024-04-12T03:39:54.274 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000010 2024-04-12T03:39:54.275 DEBUG:boto:Data: 2024-04-12T03:39:54.275 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.275 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.275 DEBUG:boto:Params: {} 2024-04-12T03:39:54.275 DEBUG:boto:Token: None 2024-04-12T03:39:54.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0ab6b89a997243d-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.279 DEBUG:boto:b'dgbpxg-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:39:44.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:39:54.279 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.279 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.279 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.280 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.280 DEBUG:boto:Method: GET 2024-04-12T03:39:54.280 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.280 DEBUG:boto:Data: 2024-04-12T03:39:54.280 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.280 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.280 DEBUG:boto:Params: {} 2024-04-12T03:39:54.280 DEBUG:boto:Token: None 2024-04-12T03:39:54.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0= 2024-04-12T03:39:54.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0=', 'Content-Length': '0'} 2024-04-12T03:39:54.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ffcd66f40d2c5d8-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.283 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.283 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.283 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.283 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.283 DEBUG:boto:Method: GET 2024-04-12T03:39:54.283 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.283 DEBUG:boto:Data: 2024-04-12T03:39:54.283 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.283 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.283 DEBUG:boto:Params: {} 2024-04-12T03:39:54.284 DEBUG:boto:Token: None 2024-04-12T03:39:54.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0= 2024-04-12T03:39:54.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0=', 'Content-Length': '0'} 2024-04-12T03:39:54.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e494e85845fc646-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.286 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.286 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.286 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.286 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.286 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.286 DEBUG:boto:Data: 2024-04-12T03:39:54.286 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.287 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.287 DEBUG:boto:Params: {} 2024-04-12T03:39:54.287 DEBUG:boto:Token: None 2024-04-12T03:39:54.287 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.287 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/vhbaO8e+vvIEExP7CEEk/9ZYlo= 2024-04-12T03:39:54.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/vhbaO8e+vvIEExP7CEEk/9ZYlo=', 'Content-Length': '0'} 2024-04-12T03:39:54.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e2a7b88f5bb32e-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.289 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.289 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.289 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.289 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.290 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.290 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.290 DEBUG:boto:Data: 2024-04-12T03:39:54.290 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.290 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.290 DEBUG:boto:Params: {} 2024-04-12T03:39:54.290 DEBUG:boto:Token: None 2024-04-12T03:39:54.290 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.290 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/vhbaO8e+vvIEExP7CEEk/9ZYlo= 2024-04-12T03:39:54.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/vhbaO8e+vvIEExP7CEEk/9ZYlo=', 'Content-Length': '0'} 2024-04-12T03:39:54.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb8bebb81047834-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:39:54.292 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.292 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.292 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.293 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.293 DEBUG:boto:Method: GET 2024-04-12T03:39:54.293 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.293 DEBUG:boto:Data: 2024-04-12T03:39:54.293 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.293 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.293 DEBUG:boto:Params: {} 2024-04-12T03:39:54.293 DEBUG:boto:Token: None 2024-04-12T03:39:54.293 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.293 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0= 2024-04-12T03:39:54.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0=', 'Content-Length': '0'} 2024-04-12T03:39:54.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c7fa6528f04f910-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.295 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.295 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.295 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.296 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.296 DEBUG:boto:Method: GET 2024-04-12T03:39:54.296 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.296 DEBUG:boto:Data: 2024-04-12T03:39:54.296 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.296 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.296 DEBUG:boto:Params: {} 2024-04-12T03:39:54.296 DEBUG:boto:Token: None 2024-04-12T03:39:54.297 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011?versionId=null 2024-04-12T03:39:54.297 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0= 2024-04-12T03:39:54.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qd68HajXwBgj7wAQ+FIrxJ3zHe0=', 'Content-Length': '0'} 2024-04-12T03:39:54.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b46629966f2998-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.299 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.299 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.299 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.299 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.299 DEBUG:boto:Method: GET 2024-04-12T03:39:54.299 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.299 DEBUG:boto:Data: 2024-04-12T03:39:54.299 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.300 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.300 DEBUG:boto:Params: {} 2024-04-12T03:39:54.300 DEBUG:boto:Token: None 2024-04-12T03:39:54.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce361b5f4bd2f8c-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.346 DEBUG:boto:b'dgbpxg-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.347 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.347 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.347 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.347 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.347 DEBUG:boto:Method: GET 2024-04-12T03:39:54.347 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.347 DEBUG:boto:Data: 2024-04-12T03:39:54.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.348 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.348 DEBUG:boto:Params: {} 2024-04-12T03:39:54.348 DEBUG:boto:Token: None 2024-04-12T03:39:54.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce93c74f55b8afdd-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.352 DEBUG:boto:b'dgbpxg-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.352 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.352 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.352 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.352 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.352 DEBUG:boto:Data: 2024-04-12T03:39:54.352 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.353 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.353 DEBUG:boto:Params: {} 2024-04-12T03:39:54.353 DEBUG:boto:Token: None 2024-04-12T03:39:54.353 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.353 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wkFabFWSLNJlEQJQEKxCJtTwyU= 2024-04-12T03:39:54.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wkFabFWSLNJlEQJQEKxCJtTwyU=', 'Content-Length': '0'} 2024-04-12T03:39:54.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f2ee1ea0141d07-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.355 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.355 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.355 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.355 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.355 DEBUG:boto:Data: 2024-04-12T03:39:54.355 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.355 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.355 DEBUG:boto:Params: {} 2024-04-12T03:39:54.356 DEBUG:boto:Token: None 2024-04-12T03:39:54.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wkFabFWSLNJlEQJQEKxCJtTwyU= 2024-04-12T03:39:54.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wkFabFWSLNJlEQJQEKxCJtTwyU=', 'Content-Length': '0'} 2024-04-12T03:39:54.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8138f2f37ef7c3c-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:39:54.358 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.358 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.358 DEBUG:boto:Method: GET 2024-04-12T03:39:54.358 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.358 DEBUG:boto:Data: 2024-04-12T03:39:54.358 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.359 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.359 DEBUG:boto:Params: {} 2024-04-12T03:39:54.359 DEBUG:boto:Token: None 2024-04-12T03:39:54.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aq1JEjCpEKeapadbjDYWWPdblt0= 2024-04-12T03:39:54.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aq1JEjCpEKeapadbjDYWWPdblt0=', 'Content-Length': '0'} 2024-04-12T03:39:54.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a53050bf7b2a4e-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.361 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.361 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.362 DEBUG:boto:Method: GET 2024-04-12T03:39:54.362 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.362 DEBUG:boto:Data: 2024-04-12T03:39:54.362 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.362 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.362 DEBUG:boto:Params: {} 2024-04-12T03:39:54.362 DEBUG:boto:Token: None 2024-04-12T03:39:54.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000011 2024-04-12T03:39:54.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aq1JEjCpEKeapadbjDYWWPdblt0= 2024-04-12T03:39:54.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aq1JEjCpEKeapadbjDYWWPdblt0=', 'Content-Length': '0'} 2024-04-12T03:39:54.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004887b7626a13baf7-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.364 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.365 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.365 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.365 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.365 DEBUG:boto:Method: GET 2024-04-12T03:39:54.365 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.365 DEBUG:boto:Data: 2024-04-12T03:39:54.365 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.365 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.365 DEBUG:boto:Params: {} 2024-04-12T03:39:54.366 DEBUG:boto:Token: None 2024-04-12T03:39:54.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4776374c26a651f-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.368 DEBUG:boto:b'dgbpxg-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.368 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.369 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.369 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.369 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.369 DEBUG:boto:Method: GET 2024-04-12T03:39:54.369 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000011 2024-04-12T03:39:54.369 DEBUG:boto:Data: 2024-04-12T03:39:54.369 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.369 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.369 DEBUG:boto:Params: {} 2024-04-12T03:39:54.370 DEBUG:boto:Token: None 2024-04-12T03:39:54.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b9c4a5ff5739cd-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.373 DEBUG:boto:b'dgbpxg-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:39:43.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:54.374 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.374 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.374 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.374 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.374 DEBUG:boto:Method: GET 2024-04-12T03:39:54.375 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.375 DEBUG:boto:Data: 2024-04-12T03:39:54.375 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.375 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.375 DEBUG:boto:Params: {} 2024-04-12T03:39:54.375 DEBUG:boto:Token: None 2024-04-12T03:39:54.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4= 2024-04-12T03:39:54.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4=', 'Content-Length': '0'} 2024-04-12T03:39:54.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724fc354dc04b311-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.377 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.378 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.378 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.378 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.378 DEBUG:boto:Method: GET 2024-04-12T03:39:54.378 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.378 DEBUG:boto:Data: 2024-04-12T03:39:54.378 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.378 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.378 DEBUG:boto:Params: {} 2024-04-12T03:39:54.379 DEBUG:boto:Token: None 2024-04-12T03:39:54.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4= 2024-04-12T03:39:54.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4=', 'Content-Length': '0'} 2024-04-12T03:39:54.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65c199097b7c1a2-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.381 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.381 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.381 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.381 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.382 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.382 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.382 DEBUG:boto:Data: 2024-04-12T03:39:54.382 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.382 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.382 DEBUG:boto:Params: {} 2024-04-12T03:39:54.382 DEBUG:boto:Token: None 2024-04-12T03:39:54.382 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MaDomNTFJhlUV/KvMkm+z9os3iU= 2024-04-12T03:39:54.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MaDomNTFJhlUV/KvMkm+z9os3iU=', 'Content-Length': '0'} 2024-04-12T03:39:54.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbdbed2ad7f363e8-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.385 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.385 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.385 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.385 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.385 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.385 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.385 DEBUG:boto:Data: 2024-04-12T03:39:54.385 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.386 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.386 DEBUG:boto:Params: {} 2024-04-12T03:39:54.386 DEBUG:boto:Token: None 2024-04-12T03:39:54.386 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MaDomNTFJhlUV/KvMkm+z9os3iU= 2024-04-12T03:39:54.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MaDomNTFJhlUV/KvMkm+z9os3iU=', 'Content-Length': '0'} 2024-04-12T03:39:54.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000994ad9fb9d315fa0-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:54.388 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.388 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.389 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.389 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.389 DEBUG:boto:Method: GET 2024-04-12T03:39:54.389 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.389 DEBUG:boto:Data: 2024-04-12T03:39:54.389 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.389 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.389 DEBUG:boto:Params: {} 2024-04-12T03:39:54.389 DEBUG:boto:Token: None 2024-04-12T03:39:54.389 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4= 2024-04-12T03:39:54.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4=', 'Content-Length': '0'} 2024-04-12T03:39:54.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b995917caeb1d0af-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.391 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.391 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.392 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.392 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.392 DEBUG:boto:Method: GET 2024-04-12T03:39:54.392 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.392 DEBUG:boto:Data: 2024-04-12T03:39:54.392 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.392 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.392 DEBUG:boto:Params: {} 2024-04-12T03:39:54.392 DEBUG:boto:Token: None 2024-04-12T03:39:54.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012?versionId=null 2024-04-12T03:39:54.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4= 2024-04-12T03:39:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+q1S1GJe5px0LJ7MG3uDdOUHk4=', 'Content-Length': '0'} 2024-04-12T03:39:54.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acaf26628b483e4a-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.395 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.395 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.395 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.395 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.395 DEBUG:boto:Method: GET 2024-04-12T03:39:54.395 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.395 DEBUG:boto:Data: 2024-04-12T03:39:54.395 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.396 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.396 DEBUG:boto:Params: {} 2024-04-12T03:39:54.396 DEBUG:boto:Token: None 2024-04-12T03:39:54.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0754675e4f836d5-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.398 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.399 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.399 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.399 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.399 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.399 DEBUG:boto:Method: GET 2024-04-12T03:39:54.399 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.399 DEBUG:boto:Data: 2024-04-12T03:39:54.399 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.400 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.400 DEBUG:boto:Params: {} 2024-04-12T03:39:54.400 DEBUG:boto:Token: None 2024-04-12T03:39:54.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050bff0060f4fcd28-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.446 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.447 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.447 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.447 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.447 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.447 DEBUG:boto:Data: 2024-04-12T03:39:54.447 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.447 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.447 DEBUG:boto:Params: {} 2024-04-12T03:39:54.448 DEBUG:boto:Token: None 2024-04-12T03:39:54.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HQSaR5V08e1n3kZnBS2m20P/9bk= 2024-04-12T03:39:54.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HQSaR5V08e1n3kZnBS2m20P/9bk=', 'Content-Length': '0'} 2024-04-12T03:39:54.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b527434d74e7d0f-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.450 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.450 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.450 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.450 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.450 DEBUG:boto:Data: 2024-04-12T03:39:54.451 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.451 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.451 DEBUG:boto:Params: {} 2024-04-12T03:39:54.451 DEBUG:boto:Token: None 2024-04-12T03:39:54.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HQSaR5V08e1n3kZnBS2m20P/9bk= 2024-04-12T03:39:54.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HQSaR5V08e1n3kZnBS2m20P/9bk=', 'Content-Length': '0'} 2024-04-12T03:39:54.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b964ff4449c19e-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:39:54.453 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.454 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.454 DEBUG:boto:Method: GET 2024-04-12T03:39:54.454 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.454 DEBUG:boto:Data: 2024-04-12T03:39:54.454 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.454 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.454 DEBUG:boto:Params: {} 2024-04-12T03:39:54.454 DEBUG:boto:Token: None 2024-04-12T03:39:54.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sSe/dOOMYAw8X6aGa2lmYa9PWpM= 2024-04-12T03:39:54.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sSe/dOOMYAw8X6aGa2lmYa9PWpM=', 'Content-Length': '0'} 2024-04-12T03:39:54.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db63a4dfe82fba46-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.457 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.457 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.457 DEBUG:boto:Method: GET 2024-04-12T03:39:54.457 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.457 DEBUG:boto:Data: 2024-04-12T03:39:54.457 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.457 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.457 DEBUG:boto:Params: {} 2024-04-12T03:39:54.458 DEBUG:boto:Token: None 2024-04-12T03:39:54.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000012 2024-04-12T03:39:54.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sSe/dOOMYAw8X6aGa2lmYa9PWpM= 2024-04-12T03:39:54.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sSe/dOOMYAw8X6aGa2lmYa9PWpM=', 'Content-Length': '0'} 2024-04-12T03:39:54.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d564e68551245e77-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.460 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.460 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.460 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.460 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.461 DEBUG:boto:Method: GET 2024-04-12T03:39:54.461 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.461 DEBUG:boto:Data: 2024-04-12T03:39:54.461 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.461 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.461 DEBUG:boto:Params: {} 2024-04-12T03:39:54.461 DEBUG:boto:Token: None 2024-04-12T03:39:54.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15aa1f90517efac-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.464 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.464 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.464 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.465 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.465 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.465 DEBUG:boto:Method: GET 2024-04-12T03:39:54.465 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000012 2024-04-12T03:39:54.465 DEBUG:boto:Data: 2024-04-12T03:39:54.465 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.465 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.465 DEBUG:boto:Params: {} 2024-04-12T03:39:54.466 DEBUG:boto:Token: None 2024-04-12T03:39:54.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91128a3e6c50a23-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.470 DEBUG:boto:b'dgbpxg-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:39:43.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:54.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.471 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.471 DEBUG:boto:Method: GET 2024-04-12T03:39:54.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.471 DEBUG:boto:Data: 2024-04-12T03:39:54.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.471 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.471 DEBUG:boto:Params: {} 2024-04-12T03:39:54.471 DEBUG:boto:Token: None 2024-04-12T03:39:54.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y= 2024-04-12T03:39:54.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y=', 'Content-Length': '0'} 2024-04-12T03:39:54.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009658292e1290d70d-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.514 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.514 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.515 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.515 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.515 DEBUG:boto:Method: GET 2024-04-12T03:39:54.515 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.515 DEBUG:boto:Data: 2024-04-12T03:39:54.515 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.515 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.515 DEBUG:boto:Params: {} 2024-04-12T03:39:54.515 DEBUG:boto:Token: None 2024-04-12T03:39:54.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y= 2024-04-12T03:39:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y=', 'Content-Length': '0'} 2024-04-12T03:39:54.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401fa56d3acde0e0-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.518 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.518 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.518 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.518 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.518 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.518 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.518 DEBUG:boto:Data: 2024-04-12T03:39:54.518 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.519 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.519 DEBUG:boto:Params: {} 2024-04-12T03:39:54.519 DEBUG:boto:Token: None 2024-04-12T03:39:54.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vu9N20XOl7O/IVf1fg8GAwu3O+g= 2024-04-12T03:39:54.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vu9N20XOl7O/IVf1fg8GAwu3O+g=', 'Content-Length': '0'} 2024-04-12T03:39:54.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ad5c16b03cba82-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.521 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.521 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.521 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.521 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.521 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.521 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.521 DEBUG:boto:Data: 2024-04-12T03:39:54.521 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.521 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.522 DEBUG:boto:Params: {} 2024-04-12T03:39:54.522 DEBUG:boto:Token: None 2024-04-12T03:39:54.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vu9N20XOl7O/IVf1fg8GAwu3O+g= 2024-04-12T03:39:54.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vu9N20XOl7O/IVf1fg8GAwu3O+g=', 'Content-Length': '0'} 2024-04-12T03:39:54.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8e785e08c733b3-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:54.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.525 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.525 DEBUG:boto:Method: GET 2024-04-12T03:39:54.525 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.525 DEBUG:boto:Data: 2024-04-12T03:39:54.525 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.525 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.525 DEBUG:boto:Params: {} 2024-04-12T03:39:54.525 DEBUG:boto:Token: None 2024-04-12T03:39:54.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y= 2024-04-12T03:39:54.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y=', 'Content-Length': '0'} 2024-04-12T03:39:54.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687c3450c2ae042f-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.528 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.528 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.528 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.528 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.528 DEBUG:boto:Method: GET 2024-04-12T03:39:54.529 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.529 DEBUG:boto:Data: 2024-04-12T03:39:54.529 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.529 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.529 DEBUG:boto:Params: {} 2024-04-12T03:39:54.529 DEBUG:boto:Token: None 2024-04-12T03:39:54.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013?versionId=null 2024-04-12T03:39:54.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y= 2024-04-12T03:39:54.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KIc9oF4n6f3aB6/fQQBrttePK6Y=', 'Content-Length': '0'} 2024-04-12T03:39:54.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002713bf79e6b172a5-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.531 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.531 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.532 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.532 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.532 DEBUG:boto:Method: GET 2024-04-12T03:39:54.532 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.532 DEBUG:boto:Data: 2024-04-12T03:39:54.532 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.532 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.532 DEBUG:boto:Params: {} 2024-04-12T03:39:54.532 DEBUG:boto:Token: None 2024-04-12T03:39:54.533 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f03aa15f71a2679-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.535 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.535 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.535 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.536 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.536 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.536 DEBUG:boto:Method: GET 2024-04-12T03:39:54.536 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.536 DEBUG:boto:Data: 2024-04-12T03:39:54.536 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.536 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.536 DEBUG:boto:Params: {} 2024-04-12T03:39:54.536 DEBUG:boto:Token: None 2024-04-12T03:39:54.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064166cf3683d8625-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.540 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.541 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.541 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.541 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.541 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.541 DEBUG:boto:Data: 2024-04-12T03:39:54.541 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.542 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.542 DEBUG:boto:Params: {} 2024-04-12T03:39:54.542 DEBUG:boto:Token: None 2024-04-12T03:39:54.542 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ctlLypOzaXuoTogUx0EGvnZ4GsE= 2024-04-12T03:39:54.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ctlLypOzaXuoTogUx0EGvnZ4GsE=', 'Content-Length': '0'} 2024-04-12T03:39:54.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066acca601986237-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.544 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.544 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.544 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.544 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.545 DEBUG:boto:Data: 2024-04-12T03:39:54.545 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.545 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.545 DEBUG:boto:Params: {} 2024-04-12T03:39:54.545 DEBUG:boto:Token: None 2024-04-12T03:39:54.545 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ctlLypOzaXuoTogUx0EGvnZ4GsE= 2024-04-12T03:39:54.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ctlLypOzaXuoTogUx0EGvnZ4GsE=', 'Content-Length': '0'} 2024-04-12T03:39:54.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034534b91bfc0472c-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:39:54.548 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.548 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.548 DEBUG:boto:Method: GET 2024-04-12T03:39:54.548 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.548 DEBUG:boto:Data: 2024-04-12T03:39:54.548 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.548 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.548 DEBUG:boto:Params: {} 2024-04-12T03:39:54.548 DEBUG:boto:Token: None 2024-04-12T03:39:54.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a/VEg+JcMAJFOhL45WcNY5eCwNg= 2024-04-12T03:39:54.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a/VEg+JcMAJFOhL45WcNY5eCwNg=', 'Content-Length': '0'} 2024-04-12T03:39:54.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a955a13cc4454bde-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.550 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.550 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.551 DEBUG:boto:Method: GET 2024-04-12T03:39:54.551 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.551 DEBUG:boto:Data: 2024-04-12T03:39:54.551 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.551 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.551 DEBUG:boto:Params: {} 2024-04-12T03:39:54.551 DEBUG:boto:Token: None 2024-04-12T03:39:54.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000013 2024-04-12T03:39:54.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a/VEg+JcMAJFOhL45WcNY5eCwNg= 2024-04-12T03:39:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a/VEg+JcMAJFOhL45WcNY5eCwNg=', 'Content-Length': '0'} 2024-04-12T03:39:54.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c80fefa3fccfad-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.594 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.595 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.595 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.595 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.595 DEBUG:boto:Method: GET 2024-04-12T03:39:54.595 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.595 DEBUG:boto:Data: 2024-04-12T03:39:54.595 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.596 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.596 DEBUG:boto:Params: {} 2024-04-12T03:39:54.596 DEBUG:boto:Token: None 2024-04-12T03:39:54.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa243338d4e2cda-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.599 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.599 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.599 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.599 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.599 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.599 DEBUG:boto:Method: GET 2024-04-12T03:39:54.600 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000013 2024-04-12T03:39:54.600 DEBUG:boto:Data: 2024-04-12T03:39:54.600 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.600 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.600 DEBUG:boto:Params: {} 2024-04-12T03:39:54.600 DEBUG:boto:Token: None 2024-04-12T03:39:54.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003472b663dc7e9e00-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.604 DEBUG:boto:b'dgbpxg-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:39:44.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:39:54.604 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.605 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.605 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.605 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.605 DEBUG:boto:Method: GET 2024-04-12T03:39:54.605 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.605 DEBUG:boto:Data: 2024-04-12T03:39:54.605 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.605 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.605 DEBUG:boto:Params: {} 2024-04-12T03:39:54.605 DEBUG:boto:Token: None 2024-04-12T03:39:54.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4= 2024-04-12T03:39:54.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4=', 'Content-Length': '0'} 2024-04-12T03:39:54.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dff49c4ce8d8f8d-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.650 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.651 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.651 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.651 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.651 DEBUG:boto:Method: GET 2024-04-12T03:39:54.651 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.651 DEBUG:boto:Data: 2024-04-12T03:39:54.651 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.651 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.651 DEBUG:boto:Params: {} 2024-04-12T03:39:54.652 DEBUG:boto:Token: None 2024-04-12T03:39:54.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4= 2024-04-12T03:39:54.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4=', 'Content-Length': '0'} 2024-04-12T03:39:54.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae3e62ee4ae0799-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.654 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.654 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.654 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.654 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.654 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.654 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.654 DEBUG:boto:Data: 2024-04-12T03:39:54.655 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.655 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.655 DEBUG:boto:Params: {} 2024-04-12T03:39:54.655 DEBUG:boto:Token: None 2024-04-12T03:39:54.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N5/xTQEk7Jw1fwJ3GlEEWa8R/8A= 2024-04-12T03:39:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N5/xTQEk7Jw1fwJ3GlEEWa8R/8A=', 'Content-Length': '0'} 2024-04-12T03:39:54.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000820f8a608c4fb258-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.657 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.657 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.657 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.657 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.657 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.657 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.658 DEBUG:boto:Data: 2024-04-12T03:39:54.658 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.658 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.658 DEBUG:boto:Params: {} 2024-04-12T03:39:54.658 DEBUG:boto:Token: None 2024-04-12T03:39:54.658 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N5/xTQEk7Jw1fwJ3GlEEWa8R/8A= 2024-04-12T03:39:54.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N5/xTQEk7Jw1fwJ3GlEEWa8R/8A=', 'Content-Length': '0'} 2024-04-12T03:39:54.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014dc2b3ab1c4071a-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:39:54.660 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.660 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.660 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.660 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.660 DEBUG:boto:Method: GET 2024-04-12T03:39:54.660 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.661 DEBUG:boto:Data: 2024-04-12T03:39:54.661 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.661 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.661 DEBUG:boto:Params: {} 2024-04-12T03:39:54.661 DEBUG:boto:Token: None 2024-04-12T03:39:54.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4= 2024-04-12T03:39:54.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4=', 'Content-Length': '0'} 2024-04-12T03:39:54.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c236c8e42ccb864-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.663 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.663 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.664 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.664 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.664 DEBUG:boto:Method: GET 2024-04-12T03:39:54.664 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.664 DEBUG:boto:Data: 2024-04-12T03:39:54.664 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.664 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.664 DEBUG:boto:Params: {} 2024-04-12T03:39:54.665 DEBUG:boto:Token: None 2024-04-12T03:39:54.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014?versionId=null 2024-04-12T03:39:54.665 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4= 2024-04-12T03:39:54.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBSKLN9V/tSzDhI89xZ7oLxvku4=', 'Content-Length': '0'} 2024-04-12T03:39:54.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543efadc1494749a-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.667 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.667 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.667 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.667 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.667 DEBUG:boto:Method: GET 2024-04-12T03:39:54.667 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.668 DEBUG:boto:Data: 2024-04-12T03:39:54.668 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.668 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.668 DEBUG:boto:Params: {} 2024-04-12T03:39:54.668 DEBUG:boto:Token: None 2024-04-12T03:39:54.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae6b9005f5cd73d5-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.671 DEBUG:boto:b'dgbpxg-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.671 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.671 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.671 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.671 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.671 DEBUG:boto:Method: GET 2024-04-12T03:39:54.671 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.672 DEBUG:boto:Data: 2024-04-12T03:39:54.672 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.672 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.672 DEBUG:boto:Params: {} 2024-04-12T03:39:54.672 DEBUG:boto:Token: None 2024-04-12T03:39:54.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e2129022d9282c7-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.677 DEBUG:boto:b'dgbpxg-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.677 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.677 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.677 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.677 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.677 DEBUG:boto:Data: 2024-04-12T03:39:54.677 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.678 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.678 DEBUG:boto:Params: {} 2024-04-12T03:39:54.678 DEBUG:boto:Token: None 2024-04-12T03:39:54.678 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.678 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EW36PPNJsZEOZWgs8pT1h6/6K2E= 2024-04-12T03:39:54.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EW36PPNJsZEOZWgs8pT1h6/6K2E=', 'Content-Length': '0'} 2024-04-12T03:39:54.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453a6384e3cedf67-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.680 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.680 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.680 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.680 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.680 DEBUG:boto:Data: 2024-04-12T03:39:54.680 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.681 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.681 DEBUG:boto:Params: {} 2024-04-12T03:39:54.681 DEBUG:boto:Token: None 2024-04-12T03:39:54.681 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EW36PPNJsZEOZWgs8pT1h6/6K2E= 2024-04-12T03:39:54.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EW36PPNJsZEOZWgs8pT1h6/6K2E=', 'Content-Length': '0'} 2024-04-12T03:39:54.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a3a39e7e0ad2f1-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:39:54.683 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.683 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.683 DEBUG:boto:Method: GET 2024-04-12T03:39:54.684 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.684 DEBUG:boto:Data: 2024-04-12T03:39:54.684 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.684 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.684 DEBUG:boto:Params: {} 2024-04-12T03:39:54.684 DEBUG:boto:Token: None 2024-04-12T03:39:54.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Mnjz8cclUHOJE1qLIVRJkrIm94E= 2024-04-12T03:39:54.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Mnjz8cclUHOJE1qLIVRJkrIm94E=', 'Content-Length': '0'} 2024-04-12T03:39:54.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c5077b45c8b848-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.726 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.726 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.727 DEBUG:boto:Method: GET 2024-04-12T03:39:54.727 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.727 DEBUG:boto:Data: 2024-04-12T03:39:54.727 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.727 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.727 DEBUG:boto:Params: {} 2024-04-12T03:39:54.727 DEBUG:boto:Token: None 2024-04-12T03:39:54.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000014 2024-04-12T03:39:54.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Mnjz8cclUHOJE1qLIVRJkrIm94E= 2024-04-12T03:39:54.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Mnjz8cclUHOJE1qLIVRJkrIm94E=', 'Content-Length': '0'} 2024-04-12T03:39:54.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab2646ca0f6428c-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.730 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.730 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.730 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.730 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.730 DEBUG:boto:Method: GET 2024-04-12T03:39:54.731 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.731 DEBUG:boto:Data: 2024-04-12T03:39:54.731 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.731 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.731 DEBUG:boto:Params: {} 2024-04-12T03:39:54.732 DEBUG:boto:Token: None 2024-04-12T03:39:54.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e815f4b452af854-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.734 DEBUG:boto:b'dgbpxg-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.735 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.735 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.735 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.735 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.735 DEBUG:boto:Method: GET 2024-04-12T03:39:54.735 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000014 2024-04-12T03:39:54.735 DEBUG:boto:Data: 2024-04-12T03:39:54.735 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.735 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.735 DEBUG:boto:Params: {} 2024-04-12T03:39:54.735 DEBUG:boto:Token: None 2024-04-12T03:39:54.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e840b7ce40caa15-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.739 DEBUG:boto:b'dgbpxg-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:39:43.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:39:54.740 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.740 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.740 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.740 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.740 DEBUG:boto:Method: GET 2024-04-12T03:39:54.740 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.741 DEBUG:boto:Data: 2024-04-12T03:39:54.741 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.741 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.741 DEBUG:boto:Params: {} 2024-04-12T03:39:54.741 DEBUG:boto:Token: None 2024-04-12T03:39:54.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM= 2024-04-12T03:39:54.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM=', 'Content-Length': '0'} 2024-04-12T03:39:54.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813d459136519f05-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.744 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.744 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.744 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.744 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.744 DEBUG:boto:Method: GET 2024-04-12T03:39:54.744 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.744 DEBUG:boto:Data: 2024-04-12T03:39:54.744 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.744 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.744 DEBUG:boto:Params: {} 2024-04-12T03:39:54.745 DEBUG:boto:Token: None 2024-04-12T03:39:54.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM= 2024-04-12T03:39:54.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM=', 'Content-Length': '0'} 2024-04-12T03:39:54.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d47943017a02f5-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.747 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.747 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.747 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.747 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.747 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.747 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.747 DEBUG:boto:Data: 2024-04-12T03:39:54.747 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.748 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.748 DEBUG:boto:Params: {} 2024-04-12T03:39:54.748 DEBUG:boto:Token: None 2024-04-12T03:39:54.748 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M9cH/iLlooa09NPB0F3JRYPhbMU= 2024-04-12T03:39:54.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M9cH/iLlooa09NPB0F3JRYPhbMU=', 'Content-Length': '0'} 2024-04-12T03:39:54.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70184644d32cd85-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.750 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.750 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.750 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.750 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.750 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.750 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.750 DEBUG:boto:Data: 2024-04-12T03:39:54.750 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.751 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.751 DEBUG:boto:Params: {} 2024-04-12T03:39:54.751 DEBUG:boto:Token: None 2024-04-12T03:39:54.751 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M9cH/iLlooa09NPB0F3JRYPhbMU= 2024-04-12T03:39:54.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M9cH/iLlooa09NPB0F3JRYPhbMU=', 'Content-Length': '0'} 2024-04-12T03:39:54.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c4851142ea3de4-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:39:54.753 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.753 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.753 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.753 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.753 DEBUG:boto:Method: GET 2024-04-12T03:39:54.753 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.753 DEBUG:boto:Data: 2024-04-12T03:39:54.754 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.754 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.754 DEBUG:boto:Params: {} 2024-04-12T03:39:54.754 DEBUG:boto:Token: None 2024-04-12T03:39:54.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM= 2024-04-12T03:39:54.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM=', 'Content-Length': '0'} 2024-04-12T03:39:54.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954c9db4dcdb9a01-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.756 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.756 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.756 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.756 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.756 DEBUG:boto:Method: GET 2024-04-12T03:39:54.757 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.757 DEBUG:boto:Data: 2024-04-12T03:39:54.757 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.757 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.757 DEBUG:boto:Params: {} 2024-04-12T03:39:54.757 DEBUG:boto:Token: None 2024-04-12T03:39:54.757 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015?versionId=null 2024-04-12T03:39:54.757 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM= 2024-04-12T03:39:54.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SsGechYrr5FciE3UicWQdOuO+WM=', 'Content-Length': '0'} 2024-04-12T03:39:54.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000392282952ccdc29-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.759 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.759 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.760 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.760 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.760 DEBUG:boto:Method: GET 2024-04-12T03:39:54.760 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.760 DEBUG:boto:Data: 2024-04-12T03:39:54.760 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.760 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.760 DEBUG:boto:Params: {} 2024-04-12T03:39:54.760 DEBUG:boto:Token: None 2024-04-12T03:39:54.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d739db5951fba1a0-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.763 DEBUG:boto:b'dgbpxg-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.763 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.764 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.764 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.764 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.764 DEBUG:boto:Method: GET 2024-04-12T03:39:54.764 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.764 DEBUG:boto:Data: 2024-04-12T03:39:54.764 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.764 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.764 DEBUG:boto:Params: {} 2024-04-12T03:39:54.764 DEBUG:boto:Token: None 2024-04-12T03:39:54.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.765 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053421892e3afaf7d-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.810 DEBUG:boto:b'dgbpxg-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.811 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.811 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.811 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.811 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.811 DEBUG:boto:Data: 2024-04-12T03:39:54.811 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.811 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.811 DEBUG:boto:Params: {} 2024-04-12T03:39:54.811 DEBUG:boto:Token: None 2024-04-12T03:39:54.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UB/PnhUNI2Duiuy8byh68ruRgq8= 2024-04-12T03:39:54.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UB/PnhUNI2Duiuy8byh68ruRgq8=', 'Content-Length': '0'} 2024-04-12T03:39:54.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0caf7c43798a10-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.814 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.814 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.814 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.814 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.814 DEBUG:boto:Data: 2024-04-12T03:39:54.814 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.814 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.814 DEBUG:boto:Params: {} 2024-04-12T03:39:54.815 DEBUG:boto:Token: None 2024-04-12T03:39:54.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UB/PnhUNI2Duiuy8byh68ruRgq8= 2024-04-12T03:39:54.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UB/PnhUNI2Duiuy8byh68ruRgq8=', 'Content-Length': '0'} 2024-04-12T03:39:54.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b43cd02938436ac-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:39:54.817 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.817 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.817 DEBUG:boto:Method: GET 2024-04-12T03:39:54.817 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.817 DEBUG:boto:Data: 2024-04-12T03:39:54.817 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.818 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.818 DEBUG:boto:Params: {} 2024-04-12T03:39:54.818 DEBUG:boto:Token: None 2024-04-12T03:39:54.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/hF63qKc+Xmi4iao+vPkEL3+jL4= 2024-04-12T03:39:54.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/hF63qKc+Xmi4iao+vPkEL3+jL4=', 'Content-Length': '0'} 2024-04-12T03:39:54.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf149e0d690f128-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.820 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.820 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.820 DEBUG:boto:Method: GET 2024-04-12T03:39:54.820 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.820 DEBUG:boto:Data: 2024-04-12T03:39:54.821 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.821 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.821 DEBUG:boto:Params: {} 2024-04-12T03:39:54.821 DEBUG:boto:Token: None 2024-04-12T03:39:54.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000015 2024-04-12T03:39:54.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/hF63qKc+Xmi4iao+vPkEL3+jL4= 2024-04-12T03:39:54.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/hF63qKc+Xmi4iao+vPkEL3+jL4=', 'Content-Length': '0'} 2024-04-12T03:39:54.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002efae762f8ef3f1a-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.823 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.823 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.824 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.824 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.824 DEBUG:boto:Method: GET 2024-04-12T03:39:54.824 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.824 DEBUG:boto:Data: 2024-04-12T03:39:54.824 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.824 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.824 DEBUG:boto:Params: {} 2024-04-12T03:39:54.824 DEBUG:boto:Token: None 2024-04-12T03:39:54.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.825 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000479153340d97ffa-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.827 DEBUG:boto:b'dgbpxg-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.827 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.827 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.828 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.828 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.828 DEBUG:boto:Method: GET 2024-04-12T03:39:54.828 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000015 2024-04-12T03:39:54.828 DEBUG:boto:Data: 2024-04-12T03:39:54.828 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.828 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.828 DEBUG:boto:Params: {} 2024-04-12T03:39:54.828 DEBUG:boto:Token: None 2024-04-12T03:39:54.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ad553d16ba8b2b-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.832 DEBUG:boto:b'dgbpxg-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:39:44.062Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:39:54.833 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.833 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.833 DEBUG:boto:Method: GET 2024-04-12T03:39:54.833 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.833 DEBUG:boto:Data: 2024-04-12T03:39:54.833 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.834 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.834 DEBUG:boto:Params: {} 2024-04-12T03:39:54.834 DEBUG:boto:Token: None 2024-04-12T03:39:54.834 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.834 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk= 2024-04-12T03:39:54.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk=', 'Content-Length': '0'} 2024-04-12T03:39:54.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084e7c4848490efab-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.878 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.878 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.879 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.879 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.879 DEBUG:boto:Method: GET 2024-04-12T03:39:54.879 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.879 DEBUG:boto:Data: 2024-04-12T03:39:54.879 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.879 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.879 DEBUG:boto:Params: {} 2024-04-12T03:39:54.879 DEBUG:boto:Token: None 2024-04-12T03:39:54.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk= 2024-04-12T03:39:54.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk=', 'Content-Length': '0'} 2024-04-12T03:39:54.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23b728cb0d06a3e-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.882 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.882 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.882 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.882 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.882 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.882 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.882 DEBUG:boto:Data: 2024-04-12T03:39:54.883 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.883 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.883 DEBUG:boto:Params: {} 2024-04-12T03:39:54.883 DEBUG:boto:Token: None 2024-04-12T03:39:54.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jfccedLKbX5eS13+8GSt6dz5+lY= 2024-04-12T03:39:54.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jfccedLKbX5eS13+8GSt6dz5+lY=', 'Content-Length': '0'} 2024-04-12T03:39:54.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ffb2f9a882c641-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.885 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.885 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.885 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.885 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.886 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.886 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.886 DEBUG:boto:Data: 2024-04-12T03:39:54.886 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.886 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.886 DEBUG:boto:Params: {} 2024-04-12T03:39:54.886 DEBUG:boto:Token: None 2024-04-12T03:39:54.886 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jfccedLKbX5eS13+8GSt6dz5+lY= 2024-04-12T03:39:54.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jfccedLKbX5eS13+8GSt6dz5+lY=', 'Content-Length': '0'} 2024-04-12T03:39:54.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28c1b17ce5fe145-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:39:54.888 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.889 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.889 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.889 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.889 DEBUG:boto:Method: GET 2024-04-12T03:39:54.889 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.889 DEBUG:boto:Data: 2024-04-12T03:39:54.889 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.889 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.889 DEBUG:boto:Params: {} 2024-04-12T03:39:54.889 DEBUG:boto:Token: None 2024-04-12T03:39:54.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk= 2024-04-12T03:39:54.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk=', 'Content-Length': '0'} 2024-04-12T03:39:54.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be285169995ac50-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.892 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.892 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.892 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.892 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.892 DEBUG:boto:Method: GET 2024-04-12T03:39:54.892 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.892 DEBUG:boto:Data: 2024-04-12T03:39:54.892 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.893 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.893 DEBUG:boto:Params: {} 2024-04-12T03:39:54.893 DEBUG:boto:Token: None 2024-04-12T03:39:54.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016?versionId=null 2024-04-12T03:39:54.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk= 2024-04-12T03:39:54.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+fGV/cCJe7sXMyEnMyr9zdEYMk=', 'Content-Length': '0'} 2024-04-12T03:39:54.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac539d1b99933a66-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.895 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.895 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.895 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.895 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.895 DEBUG:boto:Method: GET 2024-04-12T03:39:54.895 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.895 DEBUG:boto:Data: 2024-04-12T03:39:54.896 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.896 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.896 DEBUG:boto:Params: {} 2024-04-12T03:39:54.896 DEBUG:boto:Token: None 2024-04-12T03:39:54.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f55acfb46837c6-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.898 DEBUG:boto:b'dgbpxg-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.899 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.899 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.899 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.899 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.899 DEBUG:boto:Method: GET 2024-04-12T03:39:54.899 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.899 DEBUG:boto:Data: 2024-04-12T03:39:54.900 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.900 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.900 DEBUG:boto:Params: {} 2024-04-12T03:39:54.900 DEBUG:boto:Token: None 2024-04-12T03:39:54.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7b5a8210fc2132-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.946 DEBUG:boto:b'dgbpxg-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.947 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.947 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.947 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.947 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.947 DEBUG:boto:Data: 2024-04-12T03:39:54.947 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.947 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.947 DEBUG:boto:Params: {} 2024-04-12T03:39:54.948 DEBUG:boto:Token: None 2024-04-12T03:39:54.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MltS+9bgdw62BQKRavzyAxkXFuU= 2024-04-12T03:39:54.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MltS+9bgdw62BQKRavzyAxkXFuU=', 'Content-Length': '0'} 2024-04-12T03:39:54.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099693cf6b2a06d4d-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.950 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.950 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.950 DEBUG:boto:Method: HEAD 2024-04-12T03:39:54.950 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.950 DEBUG:boto:Data: 2024-04-12T03:39:54.950 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.950 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.950 DEBUG:boto:Params: {} 2024-04-12T03:39:54.950 DEBUG:boto:Token: None 2024-04-12T03:39:54.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MltS+9bgdw62BQKRavzyAxkXFuU= 2024-04-12T03:39:54.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MltS+9bgdw62BQKRavzyAxkXFuU=', 'Content-Length': '0'} 2024-04-12T03:39:54.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b2aa088ca79f71f-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:39:54.953 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.953 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.953 DEBUG:boto:Method: GET 2024-04-12T03:39:54.953 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.953 DEBUG:boto:Data: 2024-04-12T03:39:54.953 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.953 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.953 DEBUG:boto:Params: {} 2024-04-12T03:39:54.954 DEBUG:boto:Token: None 2024-04-12T03:39:54.954 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.954 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lIrAzxwMxThIxz697TpLP9/yah8= 2024-04-12T03:39:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lIrAzxwMxThIxz697TpLP9/yah8=', 'Content-Length': '0'} 2024-04-12T03:39:54.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd211b07bc6a0990-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.956 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.956 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.956 DEBUG:boto:Method: GET 2024-04-12T03:39:54.956 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.956 DEBUG:boto:Data: 2024-04-12T03:39:54.956 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.956 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.956 DEBUG:boto:Params: {} 2024-04-12T03:39:54.957 DEBUG:boto:Token: None 2024-04-12T03:39:54.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000016 2024-04-12T03:39:54.957 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lIrAzxwMxThIxz697TpLP9/yah8= 2024-04-12T03:39:54.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lIrAzxwMxThIxz697TpLP9/yah8=', 'Content-Length': '0'} 2024-04-12T03:39:54.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e779defc4703b0-006618ad0a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.959 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.959 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.959 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.959 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.959 DEBUG:boto:Method: GET 2024-04-12T03:39:54.959 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.959 DEBUG:boto:Data: 2024-04-12T03:39:54.960 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.960 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.960 DEBUG:boto:Params: {} 2024-04-12T03:39:54.960 DEBUG:boto:Token: None 2024-04-12T03:39:54.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042df242f84060f6b-006618ad0a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.962 DEBUG:boto:b'dgbpxg-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.963 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:54.963 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:54.963 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.963 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.963 DEBUG:boto:Method: GET 2024-04-12T03:39:54.963 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000016 2024-04-12T03:39:54.963 DEBUG:boto:Data: 2024-04-12T03:39:54.963 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:54.964 DEBUG:boto:Port: 8000 2024-04-12T03:39:54.964 DEBUG:boto:Params: {} 2024-04-12T03:39:54.964 DEBUG:boto:Token: None 2024-04-12T03:39:54.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/ 2024-04-12T03:39:54.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk= 2024-04-12T03:39:54.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NA7EnRo8wUpYxcxypXtaRGAPmKk=', 'Content-Length': '0'} 2024-04-12T03:39:54.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680c5fa0e884ba82-006618ad0a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:54.967 DEBUG:boto:b'dgbpxg-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:39:43.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:54.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:39:54.968 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:54.968 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:54.968 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:54.968 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:54.968 DEBUG:boto:Method: GET 2024-04-12T03:39:54.968 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:54.968 DEBUG:boto:Data: 2024-04-12T03:39:54.969 DEBUG:boto:Headers: {} 2024-04-12T03:39:54.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:54.969 DEBUG:boto:Port: 8001 2024-04-12T03:39:54.969 DEBUG:boto:Params: {} 2024-04-12T03:39:54.969 DEBUG:boto:Token: None 2024-04-12T03:39:54.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:54 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:54.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6Q7IGXYea+XbUMXBqExh5mwg6oQ= 2024-04-12T03:39:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6Q7IGXYea+XbUMXBqExh5mwg6oQ=', 'Content-Length': '0'} 2024-04-12T03:39:54.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1b3dd12e84e446-006618ad0a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.015 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.015 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.015 DEBUG:boto:Method: GET 2024-04-12T03:39:55.015 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.015 DEBUG:boto:Data: 2024-04-12T03:39:55.015 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.015 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.015 DEBUG:boto:Params: {} 2024-04-12T03:39:55.015 DEBUG:boto:Token: None 2024-04-12T03:39:55.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs= 2024-04-12T03:39:55.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs=', 'Content-Length': '0'} 2024-04-12T03:39:55.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da5e382c55fe72d7-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.018 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.018 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.018 DEBUG:boto:Data: 2024-04-12T03:39:55.018 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.018 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.019 DEBUG:boto:Params: {} 2024-04-12T03:39:55.019 DEBUG:boto:Token: None 2024-04-12T03:39:55.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6z9zoaUn55fGQkUepladyVT63yA= 2024-04-12T03:39:55.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6z9zoaUn55fGQkUepladyVT63yA=', 'Content-Length': '0'} 2024-04-12T03:39:55.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a481018791293c8-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.021 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.021 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.021 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.021 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.021 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.021 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.021 DEBUG:boto:Data: 2024-04-12T03:39:55.021 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.021 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.021 DEBUG:boto:Params: {} 2024-04-12T03:39:55.022 DEBUG:boto:Token: None 2024-04-12T03:39:55.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6z9zoaUn55fGQkUepladyVT63yA= 2024-04-12T03:39:55.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6z9zoaUn55fGQkUepladyVT63yA=', 'Content-Length': '0'} 2024-04-12T03:39:55.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b242f2cc499b98a7-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:39:55.024 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.024 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.024 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.024 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.024 DEBUG:boto:Method: GET 2024-04-12T03:39:55.024 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.024 DEBUG:boto:Data: 2024-04-12T03:39:55.024 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.025 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.025 DEBUG:boto:Params: {} 2024-04-12T03:39:55.025 DEBUG:boto:Token: None 2024-04-12T03:39:55.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs= 2024-04-12T03:39:55.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs=', 'Content-Length': '0'} 2024-04-12T03:39:55.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb09611d5d6bfea9-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.027 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.027 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.027 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.027 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.027 DEBUG:boto:Method: GET 2024-04-12T03:39:55.027 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.028 DEBUG:boto:Data: 2024-04-12T03:39:55.028 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.028 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.028 DEBUG:boto:Params: {} 2024-04-12T03:39:55.028 DEBUG:boto:Token: None 2024-04-12T03:39:55.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017?versionId=null 2024-04-12T03:39:55.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs= 2024-04-12T03:39:55.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4Ty0mu5azeBzZAKMglrf7tYMWGs=', 'Content-Length': '0'} 2024-04-12T03:39:55.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae39ec1980920bc5-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.030 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.030 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.031 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.031 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.031 DEBUG:boto:Method: GET 2024-04-12T03:39:55.031 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.031 DEBUG:boto:Data: 2024-04-12T03:39:55.031 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.031 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.031 DEBUG:boto:Params: {} 2024-04-12T03:39:55.031 DEBUG:boto:Token: None 2024-04-12T03:39:55.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca618b3b6a7511c7-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.034 DEBUG:boto:b'dgbpxg-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.035 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.035 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.035 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.035 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.035 DEBUG:boto:Method: GET 2024-04-12T03:39:55.035 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.035 DEBUG:boto:Data: 2024-04-12T03:39:55.035 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.035 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.035 DEBUG:boto:Params: {} 2024-04-12T03:39:55.036 DEBUG:boto:Token: None 2024-04-12T03:39:55.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004912b40231866020-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.040 DEBUG:boto:b'dgbpxg-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.040 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.040 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.041 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.041 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.041 DEBUG:boto:Data: 2024-04-12T03:39:55.041 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.041 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.041 DEBUG:boto:Params: {} 2024-04-12T03:39:55.041 DEBUG:boto:Token: None 2024-04-12T03:39:55.041 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XTJiiFemOvErCQ6YCtNNv2TUuAM= 2024-04-12T03:39:55.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XTJiiFemOvErCQ6YCtNNv2TUuAM=', 'Content-Length': '0'} 2024-04-12T03:39:55.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005870abf0388f2655-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.043 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.043 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.043 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.044 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.044 DEBUG:boto:Data: 2024-04-12T03:39:55.044 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.044 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.044 DEBUG:boto:Params: {} 2024-04-12T03:39:55.044 DEBUG:boto:Token: None 2024-04-12T03:39:55.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XTJiiFemOvErCQ6YCtNNv2TUuAM= 2024-04-12T03:39:55.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XTJiiFemOvErCQ6YCtNNv2TUuAM=', 'Content-Length': '0'} 2024-04-12T03:39:55.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243abbec87250052-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:39:55.046 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.046 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.046 DEBUG:boto:Method: GET 2024-04-12T03:39:55.047 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.047 DEBUG:boto:Data: 2024-04-12T03:39:55.047 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.047 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.047 DEBUG:boto:Params: {} 2024-04-12T03:39:55.047 DEBUG:boto:Token: None 2024-04-12T03:39:55.047 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.047 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WuZ4N/xC6HK+4ovO/WcGSIq61QE= 2024-04-12T03:39:55.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WuZ4N/xC6HK+4ovO/WcGSIq61QE=', 'Content-Length': '0'} 2024-04-12T03:39:55.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f5f7400ca2f57d-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.090 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.090 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.091 DEBUG:boto:Method: GET 2024-04-12T03:39:55.091 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.091 DEBUG:boto:Data: 2024-04-12T03:39:55.091 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.091 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.091 DEBUG:boto:Params: {} 2024-04-12T03:39:55.091 DEBUG:boto:Token: None 2024-04-12T03:39:55.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000017 2024-04-12T03:39:55.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WuZ4N/xC6HK+4ovO/WcGSIq61QE= 2024-04-12T03:39:55.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WuZ4N/xC6HK+4ovO/WcGSIq61QE=', 'Content-Length': '0'} 2024-04-12T03:39:55.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1bc68faaa4df39-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.094 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.094 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.094 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.094 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.094 DEBUG:boto:Method: GET 2024-04-12T03:39:55.094 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.094 DEBUG:boto:Data: 2024-04-12T03:39:55.094 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.095 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.095 DEBUG:boto:Params: {} 2024-04-12T03:39:55.095 DEBUG:boto:Token: None 2024-04-12T03:39:55.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d9bd84d2ebe4a0d-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.097 DEBUG:boto:b'dgbpxg-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.098 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.098 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.098 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.098 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.098 DEBUG:boto:Method: GET 2024-04-12T03:39:55.098 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000017 2024-04-12T03:39:55.098 DEBUG:boto:Data: 2024-04-12T03:39:55.098 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.098 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.099 DEBUG:boto:Params: {} 2024-04-12T03:39:55.099 DEBUG:boto:Token: None 2024-04-12T03:39:55.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014630f17955bb9da-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.103 DEBUG:boto:b'dgbpxg-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:39:42.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:39:55.103 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.103 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.103 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.103 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.104 DEBUG:boto:Method: GET 2024-04-12T03:39:55.104 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.104 DEBUG:boto:Data: 2024-04-12T03:39:55.104 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.104 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.104 DEBUG:boto:Params: {} 2024-04-12T03:39:55.104 DEBUG:boto:Token: None 2024-04-12T03:39:55.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00= 2024-04-12T03:39:55.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00=', 'Content-Length': '0'} 2024-04-12T03:39:55.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a44ac6c5c1573b4c-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.107 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.107 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.107 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.107 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.107 DEBUG:boto:Method: GET 2024-04-12T03:39:55.108 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.108 DEBUG:boto:Data: 2024-04-12T03:39:55.108 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.108 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.108 DEBUG:boto:Params: {} 2024-04-12T03:39:55.108 DEBUG:boto:Token: None 2024-04-12T03:39:55.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00= 2024-04-12T03:39:55.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00=', 'Content-Length': '0'} 2024-04-12T03:39:55.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7743e6f63988bb-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.110 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.111 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.111 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.111 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.111 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.111 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.111 DEBUG:boto:Data: 2024-04-12T03:39:55.111 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.111 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.111 DEBUG:boto:Params: {} 2024-04-12T03:39:55.111 DEBUG:boto:Token: None 2024-04-12T03:39:55.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0foce2UsJNxZTQj58j+2LcZG61k= 2024-04-12T03:39:55.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0foce2UsJNxZTQj58j+2LcZG61k=', 'Content-Length': '0'} 2024-04-12T03:39:55.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc72135978e3c691-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.114 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.114 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.114 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.114 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.114 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.114 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.114 DEBUG:boto:Data: 2024-04-12T03:39:55.114 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.115 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.115 DEBUG:boto:Params: {} 2024-04-12T03:39:55.115 DEBUG:boto:Token: None 2024-04-12T03:39:55.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0foce2UsJNxZTQj58j+2LcZG61k= 2024-04-12T03:39:55.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0foce2UsJNxZTQj58j+2LcZG61k=', 'Content-Length': '0'} 2024-04-12T03:39:55.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ac7ad1c6b5b10e-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:39:55.117 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.117 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.117 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.117 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.118 DEBUG:boto:Method: GET 2024-04-12T03:39:55.118 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.118 DEBUG:boto:Data: 2024-04-12T03:39:55.118 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.118 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.118 DEBUG:boto:Params: {} 2024-04-12T03:39:55.118 DEBUG:boto:Token: None 2024-04-12T03:39:55.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00= 2024-04-12T03:39:55.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00=', 'Content-Length': '0'} 2024-04-12T03:39:55.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378e8a660006efc4-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.120 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.121 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.121 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.121 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.121 DEBUG:boto:Method: GET 2024-04-12T03:39:55.121 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.121 DEBUG:boto:Data: 2024-04-12T03:39:55.121 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.121 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.121 DEBUG:boto:Params: {} 2024-04-12T03:39:55.122 DEBUG:boto:Token: None 2024-04-12T03:39:55.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018?versionId=null 2024-04-12T03:39:55.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00= 2024-04-12T03:39:55.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XG5tKf0Xiz3F06Pp4lVRofD4l00=', 'Content-Length': '0'} 2024-04-12T03:39:55.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf16bb87deae7eb-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.124 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.124 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.124 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.124 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.124 DEBUG:boto:Method: GET 2024-04-12T03:39:55.124 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.124 DEBUG:boto:Data: 2024-04-12T03:39:55.124 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.125 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.125 DEBUG:boto:Params: {} 2024-04-12T03:39:55.125 DEBUG:boto:Token: None 2024-04-12T03:39:55.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cc2d8c985443315-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.128 DEBUG:boto:b'dgbpxg-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.128 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.128 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.128 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.128 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.128 DEBUG:boto:Method: GET 2024-04-12T03:39:55.128 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.128 DEBUG:boto:Data: 2024-04-12T03:39:55.128 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.129 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.129 DEBUG:boto:Params: {} 2024-04-12T03:39:55.129 DEBUG:boto:Token: None 2024-04-12T03:39:55.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7cec3694fc5f0aa-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.132 DEBUG:boto:b'dgbpxg-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.133 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.133 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.133 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.133 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.133 DEBUG:boto:Data: 2024-04-12T03:39:55.133 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.133 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.133 DEBUG:boto:Params: {} 2024-04-12T03:39:55.134 DEBUG:boto:Token: None 2024-04-12T03:39:55.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sh4NrYPZDl1J4Z9wdNVP+GYuEwQ= 2024-04-12T03:39:55.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sh4NrYPZDl1J4Z9wdNVP+GYuEwQ=', 'Content-Length': '0'} 2024-04-12T03:39:55.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0e19d6a94ffebe-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.136 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.136 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.136 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.136 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.136 DEBUG:boto:Data: 2024-04-12T03:39:55.136 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.136 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.136 DEBUG:boto:Params: {} 2024-04-12T03:39:55.137 DEBUG:boto:Token: None 2024-04-12T03:39:55.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Sh4NrYPZDl1J4Z9wdNVP+GYuEwQ= 2024-04-12T03:39:55.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sh4NrYPZDl1J4Z9wdNVP+GYuEwQ=', 'Content-Length': '0'} 2024-04-12T03:39:55.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c1d48637c951f5-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:39:55.139 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.139 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.139 DEBUG:boto:Method: GET 2024-04-12T03:39:55.139 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.139 DEBUG:boto:Data: 2024-04-12T03:39:55.139 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.140 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.140 DEBUG:boto:Params: {} 2024-04-12T03:39:55.140 DEBUG:boto:Token: None 2024-04-12T03:39:55.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsuSCpmue5PAyzoGrfGhqPQWtjA= 2024-04-12T03:39:55.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsuSCpmue5PAyzoGrfGhqPQWtjA=', 'Content-Length': '0'} 2024-04-12T03:39:55.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658a9b75479f7f37-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.182 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.182 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.182 DEBUG:boto:Method: GET 2024-04-12T03:39:55.183 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.183 DEBUG:boto:Data: 2024-04-12T03:39:55.183 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.183 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.183 DEBUG:boto:Params: {} 2024-04-12T03:39:55.183 DEBUG:boto:Token: None 2024-04-12T03:39:55.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000018 2024-04-12T03:39:55.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KsuSCpmue5PAyzoGrfGhqPQWtjA= 2024-04-12T03:39:55.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KsuSCpmue5PAyzoGrfGhqPQWtjA=', 'Content-Length': '0'} 2024-04-12T03:39:55.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9967c96275e7df-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.185 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.185 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.185 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.185 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.186 DEBUG:boto:Method: GET 2024-04-12T03:39:55.186 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.186 DEBUG:boto:Data: 2024-04-12T03:39:55.186 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.186 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.186 DEBUG:boto:Params: {} 2024-04-12T03:39:55.186 DEBUG:boto:Token: None 2024-04-12T03:39:55.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fb05057c21e937-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.189 DEBUG:boto:b'dgbpxg-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.189 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.189 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.189 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.189 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.190 DEBUG:boto:Method: GET 2024-04-12T03:39:55.190 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000018 2024-04-12T03:39:55.190 DEBUG:boto:Data: 2024-04-12T03:39:55.190 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.190 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.190 DEBUG:boto:Params: {} 2024-04-12T03:39:55.190 DEBUG:boto:Token: None 2024-04-12T03:39:55.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b526ec704840b106-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.194 DEBUG:boto:b'dgbpxg-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:39:44.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:39:55.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.195 DEBUG:boto:Method: GET 2024-04-12T03:39:55.195 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.195 DEBUG:boto:Data: 2024-04-12T03:39:55.195 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.195 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.196 DEBUG:boto:Params: {} 2024-04-12T03:39:55.196 DEBUG:boto:Token: None 2024-04-12T03:39:55.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8= 2024-04-12T03:39:55.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8=', 'Content-Length': '0'} 2024-04-12T03:39:55.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0c9e4570a5abc9-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.198 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.198 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.198 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.198 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.198 DEBUG:boto:Method: GET 2024-04-12T03:39:55.199 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.199 DEBUG:boto:Data: 2024-04-12T03:39:55.199 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.199 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.199 DEBUG:boto:Params: {} 2024-04-12T03:39:55.199 DEBUG:boto:Token: None 2024-04-12T03:39:55.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8= 2024-04-12T03:39:55.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8=', 'Content-Length': '0'} 2024-04-12T03:39:55.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed1bdb05cb24e56f-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.201 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.201 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.202 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.202 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.202 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.202 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.202 DEBUG:boto:Data: 2024-04-12T03:39:55.202 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.202 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.202 DEBUG:boto:Params: {} 2024-04-12T03:39:55.202 DEBUG:boto:Token: None 2024-04-12T03:39:55.202 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EIbIbjRfhMI2F5V388l25yXiQtc= 2024-04-12T03:39:55.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EIbIbjRfhMI2F5V388l25yXiQtc=', 'Content-Length': '0'} 2024-04-12T03:39:55.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083efc324542c2747-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.204 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.204 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.205 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.205 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.205 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.205 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.205 DEBUG:boto:Data: 2024-04-12T03:39:55.205 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.205 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.205 DEBUG:boto:Params: {} 2024-04-12T03:39:55.205 DEBUG:boto:Token: None 2024-04-12T03:39:55.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.206 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EIbIbjRfhMI2F5V388l25yXiQtc= 2024-04-12T03:39:55.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EIbIbjRfhMI2F5V388l25yXiQtc=', 'Content-Length': '0'} 2024-04-12T03:39:55.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005243caa1f84ad3ce-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:39:55.208 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.208 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.208 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.208 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.208 DEBUG:boto:Method: GET 2024-04-12T03:39:55.208 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.208 DEBUG:boto:Data: 2024-04-12T03:39:55.208 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.209 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.209 DEBUG:boto:Params: {} 2024-04-12T03:39:55.209 DEBUG:boto:Token: None 2024-04-12T03:39:55.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.209 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8= 2024-04-12T03:39:55.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8=', 'Content-Length': '0'} 2024-04-12T03:39:55.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bcbf01e24d4cb8b-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.211 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.211 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.211 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.211 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.211 DEBUG:boto:Method: GET 2024-04-12T03:39:55.211 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.212 DEBUG:boto:Data: 2024-04-12T03:39:55.212 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.212 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.212 DEBUG:boto:Params: {} 2024-04-12T03:39:55.212 DEBUG:boto:Token: None 2024-04-12T03:39:55.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019?versionId=null 2024-04-12T03:39:55.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8= 2024-04-12T03:39:55.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrqybyeCSvObTOnvaPq5BBaLyP8=', 'Content-Length': '0'} 2024-04-12T03:39:55.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027563f0881303d9a-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.215 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.215 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.215 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.215 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.215 DEBUG:boto:Method: GET 2024-04-12T03:39:55.215 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.215 DEBUG:boto:Data: 2024-04-12T03:39:55.215 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.215 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.215 DEBUG:boto:Params: {} 2024-04-12T03:39:55.216 DEBUG:boto:Token: None 2024-04-12T03:39:55.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d69d73a09c4b12-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.218 DEBUG:boto:b'dgbpxg-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.219 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.219 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.219 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.219 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.219 DEBUG:boto:Method: GET 2024-04-12T03:39:55.219 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.219 DEBUG:boto:Data: 2024-04-12T03:39:55.219 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.220 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.220 DEBUG:boto:Params: {} 2024-04-12T03:39:55.220 DEBUG:boto:Token: None 2024-04-12T03:39:55.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ea5a52abfa1257-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.224 DEBUG:boto:b'dgbpxg-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.224 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.224 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.225 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.225 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.225 DEBUG:boto:Data: 2024-04-12T03:39:55.225 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.225 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.225 DEBUG:boto:Params: {} 2024-04-12T03:39:55.225 DEBUG:boto:Token: None 2024-04-12T03:39:55.225 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JQZrtFzG2NJw871WEaxwRuvjB9g= 2024-04-12T03:39:55.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JQZrtFzG2NJw871WEaxwRuvjB9g=', 'Content-Length': '0'} 2024-04-12T03:39:55.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c690b3d871df223f-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.228 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.228 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.228 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.228 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.228 DEBUG:boto:Data: 2024-04-12T03:39:55.228 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.229 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.229 DEBUG:boto:Params: {} 2024-04-12T03:39:55.229 DEBUG:boto:Token: None 2024-04-12T03:39:55.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JQZrtFzG2NJw871WEaxwRuvjB9g= 2024-04-12T03:39:55.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JQZrtFzG2NJw871WEaxwRuvjB9g=', 'Content-Length': '0'} 2024-04-12T03:39:55.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca35e33e06dfc9d3-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:39:55.231 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.231 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.231 DEBUG:boto:Method: GET 2024-04-12T03:39:55.232 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.232 DEBUG:boto:Data: 2024-04-12T03:39:55.232 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.232 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.232 DEBUG:boto:Params: {} 2024-04-12T03:39:55.232 DEBUG:boto:Token: None 2024-04-12T03:39:55.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WiGJrGTcrJOf8q1Ycpp6uigVRVo= 2024-04-12T03:39:55.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WiGJrGTcrJOf8q1Ycpp6uigVRVo=', 'Content-Length': '0'} 2024-04-12T03:39:55.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba004b5b6aa866ed-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.274 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.274 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.275 DEBUG:boto:Method: GET 2024-04-12T03:39:55.275 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.275 DEBUG:boto:Data: 2024-04-12T03:39:55.275 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.275 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.275 DEBUG:boto:Params: {} 2024-04-12T03:39:55.275 DEBUG:boto:Token: None 2024-04-12T03:39:55.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000019 2024-04-12T03:39:55.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WiGJrGTcrJOf8q1Ycpp6uigVRVo= 2024-04-12T03:39:55.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WiGJrGTcrJOf8q1Ycpp6uigVRVo=', 'Content-Length': '0'} 2024-04-12T03:39:55.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67f22c8cf90b814-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.277 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.277 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.278 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.278 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.278 DEBUG:boto:Method: GET 2024-04-12T03:39:55.278 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.278 DEBUG:boto:Data: 2024-04-12T03:39:55.278 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.278 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.278 DEBUG:boto:Params: {} 2024-04-12T03:39:55.278 DEBUG:boto:Token: None 2024-04-12T03:39:55.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006534edc956fc4727-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.281 DEBUG:boto:b'dgbpxg-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.281 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.281 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.281 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.282 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.282 DEBUG:boto:Method: GET 2024-04-12T03:39:55.282 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000019 2024-04-12T03:39:55.282 DEBUG:boto:Data: 2024-04-12T03:39:55.282 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.282 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.282 DEBUG:boto:Params: {} 2024-04-12T03:39:55.282 DEBUG:boto:Token: None 2024-04-12T03:39:55.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae581ae2fb3caea3-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.287 DEBUG:boto:b'dgbpxg-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:39:44.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:39:55.287 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.287 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.287 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.287 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.287 DEBUG:boto:Method: GET 2024-04-12T03:39:55.288 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.288 DEBUG:boto:Data: 2024-04-12T03:39:55.288 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.288 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.288 DEBUG:boto:Params: {} 2024-04-12T03:39:55.288 DEBUG:boto:Token: None 2024-04-12T03:39:55.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w= 2024-04-12T03:39:55.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w=', 'Content-Length': '0'} 2024-04-12T03:39:55.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054beac789160e1da-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.290 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.291 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.291 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.291 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.291 DEBUG:boto:Method: GET 2024-04-12T03:39:55.291 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.291 DEBUG:boto:Data: 2024-04-12T03:39:55.291 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.291 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.291 DEBUG:boto:Params: {} 2024-04-12T03:39:55.291 DEBUG:boto:Token: None 2024-04-12T03:39:55.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w= 2024-04-12T03:39:55.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w=', 'Content-Length': '0'} 2024-04-12T03:39:55.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1d0f1bb5220f8e-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.335 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.335 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.335 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.335 DEBUG:boto:Data: 2024-04-12T03:39:55.335 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.335 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.335 DEBUG:boto:Params: {} 2024-04-12T03:39:55.335 DEBUG:boto:Token: None 2024-04-12T03:39:55.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HK1fdc7plZG9HuQORypocDnFCvA= 2024-04-12T03:39:55.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HK1fdc7plZG9HuQORypocDnFCvA=', 'Content-Length': '0'} 2024-04-12T03:39:55.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa04bdedb2938363-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.337 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.338 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.338 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.338 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.338 DEBUG:boto:Data: 2024-04-12T03:39:55.338 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.338 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.338 DEBUG:boto:Params: {} 2024-04-12T03:39:55.338 DEBUG:boto:Token: None 2024-04-12T03:39:55.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HK1fdc7plZG9HuQORypocDnFCvA= 2024-04-12T03:39:55.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HK1fdc7plZG9HuQORypocDnFCvA=', 'Content-Length': '0'} 2024-04-12T03:39:55.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30b9bc7b215e99a-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:39:55.340 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.341 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.341 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.341 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.341 DEBUG:boto:Method: GET 2024-04-12T03:39:55.341 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.341 DEBUG:boto:Data: 2024-04-12T03:39:55.341 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.341 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.341 DEBUG:boto:Params: {} 2024-04-12T03:39:55.341 DEBUG:boto:Token: None 2024-04-12T03:39:55.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.342 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w= 2024-04-12T03:39:55.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w=', 'Content-Length': '0'} 2024-04-12T03:39:55.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000961c46da5c70b02d-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.343 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.343 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.344 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.344 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.344 DEBUG:boto:Method: GET 2024-04-12T03:39:55.344 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.344 DEBUG:boto:Data: 2024-04-12T03:39:55.344 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.344 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.344 DEBUG:boto:Params: {} 2024-04-12T03:39:55.344 DEBUG:boto:Token: None 2024-04-12T03:39:55.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020?versionId=null 2024-04-12T03:39:55.345 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w= 2024-04-12T03:39:55.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrVt8AFdBLJPeRMz4I0tmiAuY3w=', 'Content-Length': '0'} 2024-04-12T03:39:55.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffa5769991c5b58-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.346 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.346 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.346 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.347 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.347 DEBUG:boto:Method: GET 2024-04-12T03:39:55.347 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.347 DEBUG:boto:Data: 2024-04-12T03:39:55.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.347 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.347 DEBUG:boto:Params: {} 2024-04-12T03:39:55.347 DEBUG:boto:Token: None 2024-04-12T03:39:55.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098adf09d246a8e2e-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.350 DEBUG:boto:b'dgbpxg-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.350 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.350 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.350 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.351 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.351 DEBUG:boto:Method: GET 2024-04-12T03:39:55.351 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.351 DEBUG:boto:Data: 2024-04-12T03:39:55.351 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.351 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.351 DEBUG:boto:Params: {} 2024-04-12T03:39:55.351 DEBUG:boto:Token: None 2024-04-12T03:39:55.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.351 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2636a249010b3ee-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.355 DEBUG:boto:b'dgbpxg-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.355 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.355 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.356 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.356 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.356 DEBUG:boto:Data: 2024-04-12T03:39:55.356 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.356 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.356 DEBUG:boto:Params: {} 2024-04-12T03:39:55.356 DEBUG:boto:Token: None 2024-04-12T03:39:55.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRo2KQKgQa9krc6gtSCoW19kG14= 2024-04-12T03:39:55.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRo2KQKgQa9krc6gtSCoW19kG14=', 'Content-Length': '0'} 2024-04-12T03:39:55.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7d6b13360bf8e6-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.358 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.358 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.358 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.359 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.359 DEBUG:boto:Data: 2024-04-12T03:39:55.359 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.359 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.359 DEBUG:boto:Params: {} 2024-04-12T03:39:55.359 DEBUG:boto:Token: None 2024-04-12T03:39:55.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRo2KQKgQa9krc6gtSCoW19kG14= 2024-04-12T03:39:55.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRo2KQKgQa9krc6gtSCoW19kG14=', 'Content-Length': '0'} 2024-04-12T03:39:55.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a082f66d69f7b456-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:39:55.362 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.362 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.362 DEBUG:boto:Method: GET 2024-04-12T03:39:55.362 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.362 DEBUG:boto:Data: 2024-04-12T03:39:55.362 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.363 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.363 DEBUG:boto:Params: {} 2024-04-12T03:39:55.363 DEBUG:boto:Token: None 2024-04-12T03:39:55.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AAm1DT5TiyEAOHpmZ+3+7BjjCjY= 2024-04-12T03:39:55.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AAm1DT5TiyEAOHpmZ+3+7BjjCjY=', 'Content-Length': '0'} 2024-04-12T03:39:55.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009724e0fb0fcbe021-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.365 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.365 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.365 DEBUG:boto:Method: GET 2024-04-12T03:39:55.366 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.366 DEBUG:boto:Data: 2024-04-12T03:39:55.366 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.366 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.366 DEBUG:boto:Params: {} 2024-04-12T03:39:55.366 DEBUG:boto:Token: None 2024-04-12T03:39:55.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000020 2024-04-12T03:39:55.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AAm1DT5TiyEAOHpmZ+3+7BjjCjY= 2024-04-12T03:39:55.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AAm1DT5TiyEAOHpmZ+3+7BjjCjY=', 'Content-Length': '0'} 2024-04-12T03:39:55.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a92d0570f0d7d86-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.368 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.368 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.368 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.368 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.368 DEBUG:boto:Method: GET 2024-04-12T03:39:55.369 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.369 DEBUG:boto:Data: 2024-04-12T03:39:55.369 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.369 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.369 DEBUG:boto:Params: {} 2024-04-12T03:39:55.369 DEBUG:boto:Token: None 2024-04-12T03:39:55.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010359f139fe74869-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.371 DEBUG:boto:b'dgbpxg-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.372 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.372 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.372 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.372 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.372 DEBUG:boto:Method: GET 2024-04-12T03:39:55.372 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000020 2024-04-12T03:39:55.372 DEBUG:boto:Data: 2024-04-12T03:39:55.372 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.372 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.373 DEBUG:boto:Params: {} 2024-04-12T03:39:55.373 DEBUG:boto:Token: None 2024-04-12T03:39:55.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8157e90160054a-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.377 DEBUG:boto:b'dgbpxg-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:39:44.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:39:55.377 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.377 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.377 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.377 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.378 DEBUG:boto:Method: GET 2024-04-12T03:39:55.378 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.378 DEBUG:boto:Data: 2024-04-12T03:39:55.378 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.378 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.378 DEBUG:boto:Params: {} 2024-04-12T03:39:55.378 DEBUG:boto:Token: None 2024-04-12T03:39:55.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA= 2024-04-12T03:39:55.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA=', 'Content-Length': '0'} 2024-04-12T03:39:55.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d15c9c52e99229-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.381 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.381 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.381 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.381 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.381 DEBUG:boto:Method: GET 2024-04-12T03:39:55.381 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.381 DEBUG:boto:Data: 2024-04-12T03:39:55.381 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.381 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.381 DEBUG:boto:Params: {} 2024-04-12T03:39:55.381 DEBUG:boto:Token: None 2024-04-12T03:39:55.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA= 2024-04-12T03:39:55.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA=', 'Content-Length': '0'} 2024-04-12T03:39:55.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614fca8575b44580-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.426 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.426 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.426 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.427 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.427 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.427 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.427 DEBUG:boto:Data: 2024-04-12T03:39:55.427 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.427 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.427 DEBUG:boto:Params: {} 2024-04-12T03:39:55.427 DEBUG:boto:Token: None 2024-04-12T03:39:55.427 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8fC8U0E53I6VYk0fKHN4VIhAqm4= 2024-04-12T03:39:55.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8fC8U0E53I6VYk0fKHN4VIhAqm4=', 'Content-Length': '0'} 2024-04-12T03:39:55.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabdea4829cca5ca-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.430 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.430 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.430 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.430 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.430 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.430 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.430 DEBUG:boto:Data: 2024-04-12T03:39:55.430 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.430 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.431 DEBUG:boto:Params: {} 2024-04-12T03:39:55.431 DEBUG:boto:Token: None 2024-04-12T03:39:55.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8fC8U0E53I6VYk0fKHN4VIhAqm4= 2024-04-12T03:39:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8fC8U0E53I6VYk0fKHN4VIhAqm4=', 'Content-Length': '0'} 2024-04-12T03:39:55.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc33773ffce9aadb-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:39:55.433 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.433 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.433 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.433 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.433 DEBUG:boto:Method: GET 2024-04-12T03:39:55.433 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.433 DEBUG:boto:Data: 2024-04-12T03:39:55.433 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.434 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.434 DEBUG:boto:Params: {} 2024-04-12T03:39:55.434 DEBUG:boto:Token: None 2024-04-12T03:39:55.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.434 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA= 2024-04-12T03:39:55.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA=', 'Content-Length': '0'} 2024-04-12T03:39:55.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f089a3bdf6ab16b-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.436 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.436 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.436 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.436 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.436 DEBUG:boto:Method: GET 2024-04-12T03:39:55.437 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.437 DEBUG:boto:Data: 2024-04-12T03:39:55.437 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.437 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.437 DEBUG:boto:Params: {} 2024-04-12T03:39:55.437 DEBUG:boto:Token: None 2024-04-12T03:39:55.437 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021?versionId=null 2024-04-12T03:39:55.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA= 2024-04-12T03:39:55.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IglF9NM7oELSg3Bk4+loSTfV1mA=', 'Content-Length': '0'} 2024-04-12T03:39:55.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da9f3bceadb81e4-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.439 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.439 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.440 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.440 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.440 DEBUG:boto:Method: GET 2024-04-12T03:39:55.440 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.440 DEBUG:boto:Data: 2024-04-12T03:39:55.440 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.440 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.440 DEBUG:boto:Params: {} 2024-04-12T03:39:55.440 DEBUG:boto:Token: None 2024-04-12T03:39:55.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000467ea92d31303ce9-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.443 DEBUG:boto:b'dgbpxg-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.443 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.443 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.443 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.443 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.443 DEBUG:boto:Method: GET 2024-04-12T03:39:55.443 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.444 DEBUG:boto:Data: 2024-04-12T03:39:55.444 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.444 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.444 DEBUG:boto:Params: {} 2024-04-12T03:39:55.444 DEBUG:boto:Token: None 2024-04-12T03:39:55.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb7a077423343e1-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.448 DEBUG:boto:b'dgbpxg-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.448 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.448 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.448 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.449 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.449 DEBUG:boto:Data: 2024-04-12T03:39:55.449 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.449 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.449 DEBUG:boto:Params: {} 2024-04-12T03:39:55.449 DEBUG:boto:Token: None 2024-04-12T03:39:55.449 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.450 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zpaauB4YecbfNaqd2CaIBJ2W1w8= 2024-04-12T03:39:55.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zpaauB4YecbfNaqd2CaIBJ2W1w8=', 'Content-Length': '0'} 2024-04-12T03:39:55.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c959a579939a60a4-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.451 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.451 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.451 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.452 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.452 DEBUG:boto:Data: 2024-04-12T03:39:55.452 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.452 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.452 DEBUG:boto:Params: {} 2024-04-12T03:39:55.452 DEBUG:boto:Token: None 2024-04-12T03:39:55.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zpaauB4YecbfNaqd2CaIBJ2W1w8= 2024-04-12T03:39:55.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zpaauB4YecbfNaqd2CaIBJ2W1w8=', 'Content-Length': '0'} 2024-04-12T03:39:55.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c4a3b45e9cb7a5-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:39:55.454 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.455 DEBUG:boto:Method: GET 2024-04-12T03:39:55.455 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.455 DEBUG:boto:Data: 2024-04-12T03:39:55.455 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.455 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.455 DEBUG:boto:Params: {} 2024-04-12T03:39:55.455 DEBUG:boto:Token: None 2024-04-12T03:39:55.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DxMD2UXU05HuRf06dWkaiFQTGfk= 2024-04-12T03:39:55.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DxMD2UXU05HuRf06dWkaiFQTGfk=', 'Content-Length': '0'} 2024-04-12T03:39:55.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f31efb6125238e-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.498 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.498 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.498 DEBUG:boto:Method: GET 2024-04-12T03:39:55.499 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.499 DEBUG:boto:Data: 2024-04-12T03:39:55.499 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.499 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.499 DEBUG:boto:Params: {} 2024-04-12T03:39:55.499 DEBUG:boto:Token: None 2024-04-12T03:39:55.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000021 2024-04-12T03:39:55.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DxMD2UXU05HuRf06dWkaiFQTGfk= 2024-04-12T03:39:55.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DxMD2UXU05HuRf06dWkaiFQTGfk=', 'Content-Length': '0'} 2024-04-12T03:39:55.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db955940e5e2afa-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.501 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.501 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.501 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.501 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.502 DEBUG:boto:Method: GET 2024-04-12T03:39:55.502 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.502 DEBUG:boto:Data: 2024-04-12T03:39:55.502 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.502 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.502 DEBUG:boto:Params: {} 2024-04-12T03:39:55.502 DEBUG:boto:Token: None 2024-04-12T03:39:55.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba709d85052b63df-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.505 DEBUG:boto:b'dgbpxg-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.505 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.505 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.505 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.506 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.506 DEBUG:boto:Method: GET 2024-04-12T03:39:55.506 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000021 2024-04-12T03:39:55.506 DEBUG:boto:Data: 2024-04-12T03:39:55.506 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.506 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.506 DEBUG:boto:Params: {} 2024-04-12T03:39:55.506 DEBUG:boto:Token: None 2024-04-12T03:39:55.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4fe4e09b380e3a-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.510 DEBUG:boto:b'dgbpxg-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:39:43.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:39:55.511 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.511 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.511 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.511 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.511 DEBUG:boto:Method: GET 2024-04-12T03:39:55.511 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.511 DEBUG:boto:Data: 2024-04-12T03:39:55.511 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.512 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.512 DEBUG:boto:Params: {} 2024-04-12T03:39:55.512 DEBUG:boto:Token: None 2024-04-12T03:39:55.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI= 2024-04-12T03:39:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI=', 'Content-Length': '0'} 2024-04-12T03:39:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003469f4215e8e6f3e-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.514 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.514 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.514 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.514 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.514 DEBUG:boto:Method: GET 2024-04-12T03:39:55.514 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.515 DEBUG:boto:Data: 2024-04-12T03:39:55.515 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.515 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.515 DEBUG:boto:Params: {} 2024-04-12T03:39:55.515 DEBUG:boto:Token: None 2024-04-12T03:39:55.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI= 2024-04-12T03:39:55.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI=', 'Content-Length': '0'} 2024-04-12T03:39:55.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6569317a27185c-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.517 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.518 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.518 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.518 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.518 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.518 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.518 DEBUG:boto:Data: 2024-04-12T03:39:55.518 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.518 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.518 DEBUG:boto:Params: {} 2024-04-12T03:39:55.519 DEBUG:boto:Token: None 2024-04-12T03:39:55.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PP2d7UcHmCrXOsRlI5DePjPgAzo= 2024-04-12T03:39:55.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PP2d7UcHmCrXOsRlI5DePjPgAzo=', 'Content-Length': '0'} 2024-04-12T03:39:55.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009675dbdccd960ac2-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.520 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.521 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.521 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.521 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.521 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.521 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.521 DEBUG:boto:Data: 2024-04-12T03:39:55.521 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.521 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.521 DEBUG:boto:Params: {} 2024-04-12T03:39:55.521 DEBUG:boto:Token: None 2024-04-12T03:39:55.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PP2d7UcHmCrXOsRlI5DePjPgAzo= 2024-04-12T03:39:55.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PP2d7UcHmCrXOsRlI5DePjPgAzo=', 'Content-Length': '0'} 2024-04-12T03:39:55.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e6bd135576b6d8-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:39:55.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.524 DEBUG:boto:Method: GET 2024-04-12T03:39:55.524 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.524 DEBUG:boto:Data: 2024-04-12T03:39:55.524 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.524 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.525 DEBUG:boto:Params: {} 2024-04-12T03:39:55.525 DEBUG:boto:Token: None 2024-04-12T03:39:55.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI= 2024-04-12T03:39:55.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI=', 'Content-Length': '0'} 2024-04-12T03:39:55.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f005c4b4d55743a8-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.527 DEBUG:boto:Method: GET 2024-04-12T03:39:55.527 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.527 DEBUG:boto:Data: 2024-04-12T03:39:55.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.528 DEBUG:boto:Params: {} 2024-04-12T03:39:55.528 DEBUG:boto:Token: None 2024-04-12T03:39:55.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022?versionId=null 2024-04-12T03:39:55.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI= 2024-04-12T03:39:55.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t4vs/rH54OuCQA0Ec8sYvK2hrgI=', 'Content-Length': '0'} 2024-04-12T03:39:55.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8876e4784973139-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.530 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.530 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.530 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.530 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.530 DEBUG:boto:Method: GET 2024-04-12T03:39:55.530 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.530 DEBUG:boto:Data: 2024-04-12T03:39:55.531 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.531 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.531 DEBUG:boto:Params: {} 2024-04-12T03:39:55.531 DEBUG:boto:Token: None 2024-04-12T03:39:55.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4ff1002d720b08-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.534 DEBUG:boto:b'dgbpxg-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.534 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.534 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.534 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.534 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.534 DEBUG:boto:Method: GET 2024-04-12T03:39:55.535 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.535 DEBUG:boto:Data: 2024-04-12T03:39:55.535 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.535 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.535 DEBUG:boto:Params: {} 2024-04-12T03:39:55.535 DEBUG:boto:Token: None 2024-04-12T03:39:55.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ed2df973195535-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.539 DEBUG:boto:b'dgbpxg-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.539 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.539 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.540 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.540 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.540 DEBUG:boto:Data: 2024-04-12T03:39:55.540 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.540 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.540 DEBUG:boto:Params: {} 2024-04-12T03:39:55.540 DEBUG:boto:Token: None 2024-04-12T03:39:55.540 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+OhW3jlv5V0LuZcK+JjyittCgcc= 2024-04-12T03:39:55.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+OhW3jlv5V0LuZcK+JjyittCgcc=', 'Content-Length': '0'} 2024-04-12T03:39:55.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794ea2f2ce1f37ea-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.542 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.542 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.543 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.543 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.543 DEBUG:boto:Data: 2024-04-12T03:39:55.543 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.543 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.543 DEBUG:boto:Params: {} 2024-04-12T03:39:55.543 DEBUG:boto:Token: None 2024-04-12T03:39:55.543 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+OhW3jlv5V0LuZcK+JjyittCgcc= 2024-04-12T03:39:55.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+OhW3jlv5V0LuZcK+JjyittCgcc=', 'Content-Length': '0'} 2024-04-12T03:39:55.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b63f6a9721f5d6c-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:39:55.545 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.546 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.546 DEBUG:boto:Method: GET 2024-04-12T03:39:55.546 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.546 DEBUG:boto:Data: 2024-04-12T03:39:55.546 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.546 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.546 DEBUG:boto:Params: {} 2024-04-12T03:39:55.546 DEBUG:boto:Token: None 2024-04-12T03:39:55.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0jVysfZWDIBlH8aRIpIA8Z2gP4= 2024-04-12T03:39:55.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0jVysfZWDIBlH8aRIpIA8Z2gP4=', 'Content-Length': '0'} 2024-04-12T03:39:55.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872813bc13b13e85-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.548 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.549 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.549 DEBUG:boto:Method: GET 2024-04-12T03:39:55.549 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.549 DEBUG:boto:Data: 2024-04-12T03:39:55.549 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.549 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.549 DEBUG:boto:Params: {} 2024-04-12T03:39:55.549 DEBUG:boto:Token: None 2024-04-12T03:39:55.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000022 2024-04-12T03:39:55.550 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P0jVysfZWDIBlH8aRIpIA8Z2gP4= 2024-04-12T03:39:55.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P0jVysfZWDIBlH8aRIpIA8Z2gP4=', 'Content-Length': '0'} 2024-04-12T03:39:55.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b402cdc778393e7-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.552 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.552 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.552 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.552 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.552 DEBUG:boto:Method: GET 2024-04-12T03:39:55.552 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.552 DEBUG:boto:Data: 2024-04-12T03:39:55.552 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.552 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.553 DEBUG:boto:Params: {} 2024-04-12T03:39:55.553 DEBUG:boto:Token: None 2024-04-12T03:39:55.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ba7e806aa81279-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.555 DEBUG:boto:b'dgbpxg-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.556 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.556 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.556 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.556 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.556 DEBUG:boto:Method: GET 2024-04-12T03:39:55.556 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000022 2024-04-12T03:39:55.556 DEBUG:boto:Data: 2024-04-12T03:39:55.557 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.557 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.557 DEBUG:boto:Params: {} 2024-04-12T03:39:55.557 DEBUG:boto:Token: None 2024-04-12T03:39:55.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.557 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038db8160c2abda7-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.561 DEBUG:boto:b'dgbpxg-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:39:41.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:39:55.562 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.562 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.562 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.562 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.562 DEBUG:boto:Method: GET 2024-04-12T03:39:55.562 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.562 DEBUG:boto:Data: 2024-04-12T03:39:55.562 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.562 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.562 DEBUG:boto:Params: {} 2024-04-12T03:39:55.563 DEBUG:boto:Token: None 2024-04-12T03:39:55.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.563 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw= 2024-04-12T03:39:55.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw=', 'Content-Length': '0'} 2024-04-12T03:39:55.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d22327acbb7d7ad-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.606 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.606 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.607 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.607 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.607 DEBUG:boto:Method: GET 2024-04-12T03:39:55.607 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.607 DEBUG:boto:Data: 2024-04-12T03:39:55.607 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.607 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.607 DEBUG:boto:Params: {} 2024-04-12T03:39:55.607 DEBUG:boto:Token: None 2024-04-12T03:39:55.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw= 2024-04-12T03:39:55.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw=', 'Content-Length': '0'} 2024-04-12T03:39:55.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001443d13c8facc76c-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.610 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.610 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.610 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.610 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.610 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.610 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.610 DEBUG:boto:Data: 2024-04-12T03:39:55.611 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.611 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.611 DEBUG:boto:Params: {} 2024-04-12T03:39:55.611 DEBUG:boto:Token: None 2024-04-12T03:39:55.611 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QQkRlbdtIsjI+iPVCM3PMunknxo= 2024-04-12T03:39:55.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QQkRlbdtIsjI+iPVCM3PMunknxo=', 'Content-Length': '0'} 2024-04-12T03:39:55.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000effb37a26c1a04c4-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.613 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.613 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.613 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.613 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.613 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.614 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.614 DEBUG:boto:Data: 2024-04-12T03:39:55.614 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.614 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.614 DEBUG:boto:Params: {} 2024-04-12T03:39:55.614 DEBUG:boto:Token: None 2024-04-12T03:39:55.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QQkRlbdtIsjI+iPVCM3PMunknxo= 2024-04-12T03:39:55.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QQkRlbdtIsjI+iPVCM3PMunknxo=', 'Content-Length': '0'} 2024-04-12T03:39:55.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbab1f45907b0c66-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:39:55.616 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.616 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.617 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.617 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.617 DEBUG:boto:Method: GET 2024-04-12T03:39:55.617 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.617 DEBUG:boto:Data: 2024-04-12T03:39:55.617 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.617 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.617 DEBUG:boto:Params: {} 2024-04-12T03:39:55.617 DEBUG:boto:Token: None 2024-04-12T03:39:55.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw= 2024-04-12T03:39:55.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw=', 'Content-Length': '0'} 2024-04-12T03:39:55.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef035e81d096658-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.619 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.620 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.620 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.620 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.620 DEBUG:boto:Method: GET 2024-04-12T03:39:55.620 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.620 DEBUG:boto:Data: 2024-04-12T03:39:55.620 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.620 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.620 DEBUG:boto:Params: {} 2024-04-12T03:39:55.620 DEBUG:boto:Token: None 2024-04-12T03:39:55.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023?versionId=null 2024-04-12T03:39:55.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw= 2024-04-12T03:39:55.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EAKxpjjdMhojTIfbmtrbQOWi4Gw=', 'Content-Length': '0'} 2024-04-12T03:39:55.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c4383f95848c01-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.623 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.623 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.623 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.623 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.623 DEBUG:boto:Method: GET 2024-04-12T03:39:55.623 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.624 DEBUG:boto:Data: 2024-04-12T03:39:55.624 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.624 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.624 DEBUG:boto:Params: {} 2024-04-12T03:39:55.624 DEBUG:boto:Token: None 2024-04-12T03:39:55.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000941c1e093381eeb6-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.627 DEBUG:boto:b'dgbpxg-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.627 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.627 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.627 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.627 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.628 DEBUG:boto:Method: GET 2024-04-12T03:39:55.628 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.628 DEBUG:boto:Data: 2024-04-12T03:39:55.628 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.628 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.628 DEBUG:boto:Params: {} 2024-04-12T03:39:55.628 DEBUG:boto:Token: None 2024-04-12T03:39:55.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085901a07d8fbb135-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.674 DEBUG:boto:b'dgbpxg-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.675 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.675 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.675 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.675 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.675 DEBUG:boto:Data: 2024-04-12T03:39:55.675 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.675 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.675 DEBUG:boto:Params: {} 2024-04-12T03:39:55.675 DEBUG:boto:Token: None 2024-04-12T03:39:55.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E3pCwjO1nbjtSpkTVrO/eloeoPs= 2024-04-12T03:39:55.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E3pCwjO1nbjtSpkTVrO/eloeoPs=', 'Content-Length': '0'} 2024-04-12T03:39:55.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702ed468643c9b05-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.677 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.677 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.678 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.678 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.678 DEBUG:boto:Data: 2024-04-12T03:39:55.678 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.678 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.678 DEBUG:boto:Params: {} 2024-04-12T03:39:55.678 DEBUG:boto:Token: None 2024-04-12T03:39:55.678 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.678 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E3pCwjO1nbjtSpkTVrO/eloeoPs= 2024-04-12T03:39:55.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E3pCwjO1nbjtSpkTVrO/eloeoPs=', 'Content-Length': '0'} 2024-04-12T03:39:55.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf024dfea4cac7a-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:39:55.681 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.681 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.681 DEBUG:boto:Method: GET 2024-04-12T03:39:55.681 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.681 DEBUG:boto:Data: 2024-04-12T03:39:55.681 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.681 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.681 DEBUG:boto:Params: {} 2024-04-12T03:39:55.682 DEBUG:boto:Token: None 2024-04-12T03:39:55.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.682 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lgAPhd04KAgkLQ4ErzuR9yjgSbk= 2024-04-12T03:39:55.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lgAPhd04KAgkLQ4ErzuR9yjgSbk=', 'Content-Length': '0'} 2024-04-12T03:39:55.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936e5ef318802e02-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.684 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.684 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.684 DEBUG:boto:Method: GET 2024-04-12T03:39:55.684 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.684 DEBUG:boto:Data: 2024-04-12T03:39:55.684 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.684 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.684 DEBUG:boto:Params: {} 2024-04-12T03:39:55.684 DEBUG:boto:Token: None 2024-04-12T03:39:55.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000023 2024-04-12T03:39:55.685 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lgAPhd04KAgkLQ4ErzuR9yjgSbk= 2024-04-12T03:39:55.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lgAPhd04KAgkLQ4ErzuR9yjgSbk=', 'Content-Length': '0'} 2024-04-12T03:39:55.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028bae7116a456ca2-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.687 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.687 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.687 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.687 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.687 DEBUG:boto:Method: GET 2024-04-12T03:39:55.687 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.687 DEBUG:boto:Data: 2024-04-12T03:39:55.687 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.687 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.688 DEBUG:boto:Params: {} 2024-04-12T03:39:55.688 DEBUG:boto:Token: None 2024-04-12T03:39:55.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007623895655924f0-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.690 DEBUG:boto:b'dgbpxg-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.690 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.691 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.691 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.691 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.691 DEBUG:boto:Method: GET 2024-04-12T03:39:55.691 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000023 2024-04-12T03:39:55.691 DEBUG:boto:Data: 2024-04-12T03:39:55.691 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.691 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.691 DEBUG:boto:Params: {} 2024-04-12T03:39:55.691 DEBUG:boto:Token: None 2024-04-12T03:39:55.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092bf0592ab13f5e9-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.695 DEBUG:boto:b'dgbpxg-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:39:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:39:55.695 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.695 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.696 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.696 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.696 DEBUG:boto:Method: GET 2024-04-12T03:39:55.696 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.696 DEBUG:boto:Data: 2024-04-12T03:39:55.696 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.696 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.696 DEBUG:boto:Params: {} 2024-04-12T03:39:55.696 DEBUG:boto:Token: None 2024-04-12T03:39:55.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM= 2024-04-12T03:39:55.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM=', 'Content-Length': '0'} 2024-04-12T03:39:55.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c1ed27266a2c4c-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.699 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.699 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.699 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.699 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.699 DEBUG:boto:Method: GET 2024-04-12T03:39:55.699 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.699 DEBUG:boto:Data: 2024-04-12T03:39:55.699 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.700 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.700 DEBUG:boto:Params: {} 2024-04-12T03:39:55.700 DEBUG:boto:Token: None 2024-04-12T03:39:55.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM= 2024-04-12T03:39:55.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM=', 'Content-Length': '0'} 2024-04-12T03:39:55.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ccde243ae8ae36-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.742 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.743 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.743 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.743 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.743 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.743 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.743 DEBUG:boto:Data: 2024-04-12T03:39:55.743 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.743 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.743 DEBUG:boto:Params: {} 2024-04-12T03:39:55.744 DEBUG:boto:Token: None 2024-04-12T03:39:55.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gTUAZ7NJVfyLH3wbjangHsLJnL8= 2024-04-12T03:39:55.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gTUAZ7NJVfyLH3wbjangHsLJnL8=', 'Content-Length': '0'} 2024-04-12T03:39:55.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001917093e0a64864-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.746 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.746 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.746 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.746 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.746 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.746 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.746 DEBUG:boto:Data: 2024-04-12T03:39:55.746 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.747 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.747 DEBUG:boto:Params: {} 2024-04-12T03:39:55.747 DEBUG:boto:Token: None 2024-04-12T03:39:55.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gTUAZ7NJVfyLH3wbjangHsLJnL8= 2024-04-12T03:39:55.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gTUAZ7NJVfyLH3wbjangHsLJnL8=', 'Content-Length': '0'} 2024-04-12T03:39:55.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969fc7884d4ced00-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:39:55.749 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.749 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.749 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.749 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.750 DEBUG:boto:Method: GET 2024-04-12T03:39:55.750 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.750 DEBUG:boto:Data: 2024-04-12T03:39:55.750 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.751 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.751 DEBUG:boto:Params: {} 2024-04-12T03:39:55.751 DEBUG:boto:Token: None 2024-04-12T03:39:55.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM= 2024-04-12T03:39:55.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM=', 'Content-Length': '0'} 2024-04-12T03:39:55.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005756dd0b323a2802-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.754 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.754 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.754 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.755 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.755 DEBUG:boto:Method: GET 2024-04-12T03:39:55.755 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.755 DEBUG:boto:Data: 2024-04-12T03:39:55.755 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.755 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.755 DEBUG:boto:Params: {} 2024-04-12T03:39:55.755 DEBUG:boto:Token: None 2024-04-12T03:39:55.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024?versionId=null 2024-04-12T03:39:55.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM= 2024-04-12T03:39:55.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynBT3CNDmnKKTl83DG4OnCtYAcM=', 'Content-Length': '0'} 2024-04-12T03:39:55.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a75cf4eef25c3d0-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.758 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.758 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.758 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.758 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.758 DEBUG:boto:Method: GET 2024-04-12T03:39:55.758 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.759 DEBUG:boto:Data: 2024-04-12T03:39:55.759 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.759 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.759 DEBUG:boto:Params: {} 2024-04-12T03:39:55.759 DEBUG:boto:Token: None 2024-04-12T03:39:55.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77af49874b88d91-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.763 DEBUG:boto:b'dgbpxg-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.764 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.764 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.764 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.764 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.764 DEBUG:boto:Method: GET 2024-04-12T03:39:55.764 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.764 DEBUG:boto:Data: 2024-04-12T03:39:55.764 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.764 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.764 DEBUG:boto:Params: {} 2024-04-12T03:39:55.765 DEBUG:boto:Token: None 2024-04-12T03:39:55.765 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.765 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7da03121499d290-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.769 DEBUG:boto:b'dgbpxg-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.769 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.769 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.770 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.770 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.770 DEBUG:boto:Data: 2024-04-12T03:39:55.770 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.770 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.770 DEBUG:boto:Params: {} 2024-04-12T03:39:55.770 DEBUG:boto:Token: None 2024-04-12T03:39:55.770 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CT03Jw8K401lNdNzNix0IEB7dCw= 2024-04-12T03:39:55.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CT03Jw8K401lNdNzNix0IEB7dCw=', 'Content-Length': '0'} 2024-04-12T03:39:55.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0be71db0b53c361-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.772 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.772 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.772 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.772 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.772 DEBUG:boto:Data: 2024-04-12T03:39:55.773 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.773 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.773 DEBUG:boto:Params: {} 2024-04-12T03:39:55.773 DEBUG:boto:Token: None 2024-04-12T03:39:55.773 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CT03Jw8K401lNdNzNix0IEB7dCw= 2024-04-12T03:39:55.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CT03Jw8K401lNdNzNix0IEB7dCw=', 'Content-Length': '0'} 2024-04-12T03:39:55.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e029d0bc3273f7-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:39:55.775 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.775 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.775 DEBUG:boto:Method: GET 2024-04-12T03:39:55.776 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.776 DEBUG:boto:Data: 2024-04-12T03:39:55.776 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.776 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.776 DEBUG:boto:Params: {} 2024-04-12T03:39:55.776 DEBUG:boto:Token: None 2024-04-12T03:39:55.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rFtlu3Kywqw1OEekQ5X4PLKOz40= 2024-04-12T03:39:55.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rFtlu3Kywqw1OEekQ5X4PLKOz40=', 'Content-Length': '0'} 2024-04-12T03:39:55.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048cd5ce821e8e455-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.779 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.779 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.779 DEBUG:boto:Method: GET 2024-04-12T03:39:55.779 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.779 DEBUG:boto:Data: 2024-04-12T03:39:55.779 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.779 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.780 DEBUG:boto:Params: {} 2024-04-12T03:39:55.780 DEBUG:boto:Token: None 2024-04-12T03:39:55.780 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000024 2024-04-12T03:39:55.780 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rFtlu3Kywqw1OEekQ5X4PLKOz40= 2024-04-12T03:39:55.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rFtlu3Kywqw1OEekQ5X4PLKOz40=', 'Content-Length': '0'} 2024-04-12T03:39:55.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65984d3c4dc4da3-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.822 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.823 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.823 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.823 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.823 DEBUG:boto:Method: GET 2024-04-12T03:39:55.823 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.823 DEBUG:boto:Data: 2024-04-12T03:39:55.823 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.823 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.823 DEBUG:boto:Params: {} 2024-04-12T03:39:55.823 DEBUG:boto:Token: None 2024-04-12T03:39:55.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6a0fcf7cc01f46-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.826 DEBUG:boto:b'dgbpxg-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.827 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.827 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.827 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.827 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.827 DEBUG:boto:Method: GET 2024-04-12T03:39:55.827 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000024 2024-04-12T03:39:55.827 DEBUG:boto:Data: 2024-04-12T03:39:55.827 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.828 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.828 DEBUG:boto:Params: {} 2024-04-12T03:39:55.828 DEBUG:boto:Token: None 2024-04-12T03:39:55.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b17d465c6955dfd-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.832 DEBUG:boto:b'dgbpxg-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:39:43.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:39:55.832 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.832 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.832 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.833 DEBUG:boto:Method: GET 2024-04-12T03:39:55.833 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.833 DEBUG:boto:Data: 2024-04-12T03:39:55.833 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.833 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.833 DEBUG:boto:Params: {} 2024-04-12T03:39:55.833 DEBUG:boto:Token: None 2024-04-12T03:39:55.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.833 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs= 2024-04-12T03:39:55.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs=', 'Content-Length': '0'} 2024-04-12T03:39:55.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003917b2eea16ac3f2-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.836 DEBUG:boto:Method: GET 2024-04-12T03:39:55.836 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.836 DEBUG:boto:Data: 2024-04-12T03:39:55.836 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.836 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.836 DEBUG:boto:Params: {} 2024-04-12T03:39:55.837 DEBUG:boto:Token: None 2024-04-12T03:39:55.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs= 2024-04-12T03:39:55.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs=', 'Content-Length': '0'} 2024-04-12T03:39:55.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad5bfa6c2de560c-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.839 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.839 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.839 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.839 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.839 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.839 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.839 DEBUG:boto:Data: 2024-04-12T03:39:55.839 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.839 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.839 DEBUG:boto:Params: {} 2024-04-12T03:39:55.839 DEBUG:boto:Token: None 2024-04-12T03:39:55.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJT9FeyjARRQF6AIpw/UBgpIhKY= 2024-04-12T03:39:55.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJT9FeyjARRQF6AIpw/UBgpIhKY=', 'Content-Length': '0'} 2024-04-12T03:39:55.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f45e4c82e47d1b8-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.841 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.841 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.842 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.842 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.842 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.842 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.842 DEBUG:boto:Data: 2024-04-12T03:39:55.842 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.842 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.842 DEBUG:boto:Params: {} 2024-04-12T03:39:55.842 DEBUG:boto:Token: None 2024-04-12T03:39:55.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UJT9FeyjARRQF6AIpw/UBgpIhKY= 2024-04-12T03:39:55.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UJT9FeyjARRQF6AIpw/UBgpIhKY=', 'Content-Length': '0'} 2024-04-12T03:39:55.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000accb7b90130f260e-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:39:55.844 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.845 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.845 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.845 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.845 DEBUG:boto:Method: GET 2024-04-12T03:39:55.845 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.845 DEBUG:boto:Data: 2024-04-12T03:39:55.845 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.845 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.845 DEBUG:boto:Params: {} 2024-04-12T03:39:55.845 DEBUG:boto:Token: None 2024-04-12T03:39:55.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs= 2024-04-12T03:39:55.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs=', 'Content-Length': '0'} 2024-04-12T03:39:55.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ec84548815c6ad-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.890 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.890 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.891 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.891 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.891 DEBUG:boto:Method: GET 2024-04-12T03:39:55.891 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.891 DEBUG:boto:Data: 2024-04-12T03:39:55.891 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.891 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.891 DEBUG:boto:Params: {} 2024-04-12T03:39:55.891 DEBUG:boto:Token: None 2024-04-12T03:39:55.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025?versionId=null 2024-04-12T03:39:55.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs= 2024-04-12T03:39:55.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NMIlhuOfQ4emJDxkbZtj5rXxKWs=', 'Content-Length': '0'} 2024-04-12T03:39:55.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003541a287fb28e45-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.893 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.893 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.894 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.894 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.894 DEBUG:boto:Method: GET 2024-04-12T03:39:55.894 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.894 DEBUG:boto:Data: 2024-04-12T03:39:55.894 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.894 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.894 DEBUG:boto:Params: {} 2024-04-12T03:39:55.894 DEBUG:boto:Token: None 2024-04-12T03:39:55.894 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb92e7cce53c619-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.897 DEBUG:boto:b'dgbpxg-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.897 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.897 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.897 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.898 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.898 DEBUG:boto:Method: GET 2024-04-12T03:39:55.898 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.898 DEBUG:boto:Data: 2024-04-12T03:39:55.898 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.898 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.898 DEBUG:boto:Params: {} 2024-04-12T03:39:55.898 DEBUG:boto:Token: None 2024-04-12T03:39:55.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fafaa227cd712b23-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.902 DEBUG:boto:b'dgbpxg-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.902 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.902 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.902 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.903 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.903 DEBUG:boto:Data: 2024-04-12T03:39:55.903 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.903 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.903 DEBUG:boto:Params: {} 2024-04-12T03:39:55.905 DEBUG:boto:Token: None 2024-04-12T03:39:55.906 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.906 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GtVsazuykRMck0sVPmUzi1/Z5M= 2024-04-12T03:39:55.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GtVsazuykRMck0sVPmUzi1/Z5M=', 'Content-Length': '0'} 2024-04-12T03:39:55.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25b99666dfc8f44-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.908 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.908 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.908 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.908 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.908 DEBUG:boto:Data: 2024-04-12T03:39:55.908 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.908 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.909 DEBUG:boto:Params: {} 2024-04-12T03:39:55.909 DEBUG:boto:Token: None 2024-04-12T03:39:55.909 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.909 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GtVsazuykRMck0sVPmUzi1/Z5M= 2024-04-12T03:39:55.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GtVsazuykRMck0sVPmUzi1/Z5M=', 'Content-Length': '0'} 2024-04-12T03:39:55.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093719e5dc2e7996-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:39:55.911 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.911 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.911 DEBUG:boto:Method: GET 2024-04-12T03:39:55.911 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.911 DEBUG:boto:Data: 2024-04-12T03:39:55.911 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.911 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.911 DEBUG:boto:Params: {} 2024-04-12T03:39:55.911 DEBUG:boto:Token: None 2024-04-12T03:39:55.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2EGdEAmRU2814JIuWGKPvsedXo= 2024-04-12T03:39:55.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2EGdEAmRU2814JIuWGKPvsedXo=', 'Content-Length': '0'} 2024-04-12T03:39:55.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431c7529762f65ee-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.913 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.914 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.914 DEBUG:boto:Method: GET 2024-04-12T03:39:55.914 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.914 DEBUG:boto:Data: 2024-04-12T03:39:55.914 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.914 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.914 DEBUG:boto:Params: {} 2024-04-12T03:39:55.914 DEBUG:boto:Token: None 2024-04-12T03:39:55.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000025 2024-04-12T03:39:55.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2EGdEAmRU2814JIuWGKPvsedXo= 2024-04-12T03:39:55.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2EGdEAmRU2814JIuWGKPvsedXo=', 'Content-Length': '0'} 2024-04-12T03:39:55.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020acefecf0c404ac-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.916 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.916 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.916 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.917 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.917 DEBUG:boto:Method: GET 2024-04-12T03:39:55.917 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.917 DEBUG:boto:Data: 2024-04-12T03:39:55.917 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.917 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.917 DEBUG:boto:Params: {} 2024-04-12T03:39:55.917 DEBUG:boto:Token: None 2024-04-12T03:39:55.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.918 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da822103c08607c7-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.920 DEBUG:boto:b'dgbpxg-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.920 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.920 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.920 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.921 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.921 DEBUG:boto:Method: GET 2024-04-12T03:39:55.921 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000025 2024-04-12T03:39:55.921 DEBUG:boto:Data: 2024-04-12T03:39:55.921 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.921 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.921 DEBUG:boto:Params: {} 2024-04-12T03:39:55.921 DEBUG:boto:Token: None 2024-04-12T03:39:55.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a40de302873160a-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.925 DEBUG:boto:b'dgbpxg-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:39:41.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:39:55.925 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.925 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.925 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.926 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.926 DEBUG:boto:Method: GET 2024-04-12T03:39:55.926 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.926 DEBUG:boto:Data: 2024-04-12T03:39:55.926 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.926 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.926 DEBUG:boto:Params: {} 2024-04-12T03:39:55.926 DEBUG:boto:Token: None 2024-04-12T03:39:55.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs= 2024-04-12T03:39:55.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs=', 'Content-Length': '0'} 2024-04-12T03:39:55.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0633f67c62654c6-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.929 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.929 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.929 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.929 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.929 DEBUG:boto:Method: GET 2024-04-12T03:39:55.929 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.929 DEBUG:boto:Data: 2024-04-12T03:39:55.929 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.930 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.930 DEBUG:boto:Params: {} 2024-04-12T03:39:55.930 DEBUG:boto:Token: None 2024-04-12T03:39:55.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs= 2024-04-12T03:39:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs=', 'Content-Length': '0'} 2024-04-12T03:39:55.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6d613866617565-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.932 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.932 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.932 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.932 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.932 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.932 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.932 DEBUG:boto:Data: 2024-04-12T03:39:55.932 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.933 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.933 DEBUG:boto:Params: {} 2024-04-12T03:39:55.933 DEBUG:boto:Token: None 2024-04-12T03:39:55.933 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CxWfOYHxVqpxNReaKt+vfQYYwqg= 2024-04-12T03:39:55.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CxWfOYHxVqpxNReaKt+vfQYYwqg=', 'Content-Length': '0'} 2024-04-12T03:39:55.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c67492ff04f063f-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.935 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.935 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.935 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.935 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.935 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.935 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.935 DEBUG:boto:Data: 2024-04-12T03:39:55.935 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.936 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.936 DEBUG:boto:Params: {} 2024-04-12T03:39:55.936 DEBUG:boto:Token: None 2024-04-12T03:39:55.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CxWfOYHxVqpxNReaKt+vfQYYwqg= 2024-04-12T03:39:55.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CxWfOYHxVqpxNReaKt+vfQYYwqg=', 'Content-Length': '0'} 2024-04-12T03:39:55.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330a8e9f28094e9c-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:39:55.938 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.938 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.938 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.938 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.938 DEBUG:boto:Method: GET 2024-04-12T03:39:55.938 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.938 DEBUG:boto:Data: 2024-04-12T03:39:55.938 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.939 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.939 DEBUG:boto:Params: {} 2024-04-12T03:39:55.939 DEBUG:boto:Token: None 2024-04-12T03:39:55.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs= 2024-04-12T03:39:55.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs=', 'Content-Length': '0'} 2024-04-12T03:39:55.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007293b82bb5f37e7d-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.982 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.982 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.982 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.983 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.983 DEBUG:boto:Method: GET 2024-04-12T03:39:55.983 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.983 DEBUG:boto:Data: 2024-04-12T03:39:55.983 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.983 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.983 DEBUG:boto:Params: {} 2024-04-12T03:39:55.983 DEBUG:boto:Token: None 2024-04-12T03:39:55.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026?versionId=null 2024-04-12T03:39:55.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs= 2024-04-12T03:39:55.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7idTnjxiYxrjzmMkCNr4+dL0kCs=', 'Content-Length': '0'} 2024-04-12T03:39:55.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b0269326aab73d-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.986 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.986 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.986 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.986 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.986 DEBUG:boto:Method: GET 2024-04-12T03:39:55.986 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.986 DEBUG:boto:Data: 2024-04-12T03:39:55.986 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.986 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.986 DEBUG:boto:Params: {} 2024-04-12T03:39:55.987 DEBUG:boto:Token: None 2024-04-12T03:39:55.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.987 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0f01a3542ff2e0-006618ad0b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.989 DEBUG:boto:b'dgbpxg-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.990 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:55.990 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:55.990 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.990 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.990 DEBUG:boto:Method: GET 2024-04-12T03:39:55.990 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:55.990 DEBUG:boto:Data: 2024-04-12T03:39:55.990 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.990 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.990 DEBUG:boto:Params: {} 2024-04-12T03:39:55.991 DEBUG:boto:Token: None 2024-04-12T03:39:55.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/ 2024-04-12T03:39:55.991 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4= 2024-04-12T03:39:55.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+S432CD+YSb+mOvBMfWCz6etUG4=', 'Content-Length': '0'} 2024-04-12T03:39:55.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057c8dd01a9294938-006618ad0b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.994 DEBUG:boto:b'dgbpxg-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:55.995 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.995 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.995 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.995 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.995 DEBUG:boto:Data: 2024-04-12T03:39:55.995 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:55.995 DEBUG:boto:Port: 8001 2024-04-12T03:39:55.995 DEBUG:boto:Params: {} 2024-04-12T03:39:55.995 DEBUG:boto:Token: None 2024-04-12T03:39:55.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrBM3/a3p+A4uxm2RO3WngYMnns= 2024-04-12T03:39:55.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrBM3/a3p+A4uxm2RO3WngYMnns=', 'Content-Length': '0'} 2024-04-12T03:39:55.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1c9e382f3f22f1-006618ad0b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:55.998 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.998 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.998 DEBUG:boto:Method: HEAD 2024-04-12T03:39:55.998 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.998 DEBUG:boto:Data: 2024-04-12T03:39:55.998 DEBUG:boto:Headers: {} 2024-04-12T03:39:55.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:55.998 DEBUG:boto:Port: 8000 2024-04-12T03:39:55.998 DEBUG:boto:Params: {} 2024-04-12T03:39:55.999 DEBUG:boto:Token: None 2024-04-12T03:39:55.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:55 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:55.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrBM3/a3p+A4uxm2RO3WngYMnns= 2024-04-12T03:39:55.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrBM3/a3p+A4uxm2RO3WngYMnns=', 'Content-Length': '0'} 2024-04-12T03:39:56.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833a608f4e4b687b-006618ad0b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:39:56.000 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.000 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.001 DEBUG:boto:Method: GET 2024-04-12T03:39:56.001 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.001 DEBUG:boto:Data: 2024-04-12T03:39:56.001 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.001 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.001 DEBUG:boto:Params: {} 2024-04-12T03:39:56.001 DEBUG:boto:Token: None 2024-04-12T03:39:56.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EmTxK8tn48ADL1Fhg+7IwgEigmU= 2024-04-12T03:39:56.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EmTxK8tn48ADL1Fhg+7IwgEigmU=', 'Content-Length': '0'} 2024-04-12T03:39:56.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6714fbc38914a51-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.003 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.003 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.003 DEBUG:boto:Method: GET 2024-04-12T03:39:56.003 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.003 DEBUG:boto:Data: 2024-04-12T03:39:56.004 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.004 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.004 DEBUG:boto:Params: {} 2024-04-12T03:39:56.004 DEBUG:boto:Token: None 2024-04-12T03:39:56.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000026 2024-04-12T03:39:56.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EmTxK8tn48ADL1Fhg+7IwgEigmU= 2024-04-12T03:39:56.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EmTxK8tn48ADL1Fhg+7IwgEigmU=', 'Content-Length': '0'} 2024-04-12T03:39:56.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ee028f561ea3c6-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.050 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.050 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.050 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.051 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.051 DEBUG:boto:Method: GET 2024-04-12T03:39:56.051 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.051 DEBUG:boto:Data: 2024-04-12T03:39:56.051 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.051 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.051 DEBUG:boto:Params: {} 2024-04-12T03:39:56.051 DEBUG:boto:Token: None 2024-04-12T03:39:56.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86cc57d18836b34-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.054 DEBUG:boto:b'dgbpxg-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.054 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.054 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.054 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.054 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.055 DEBUG:boto:Method: GET 2024-04-12T03:39:56.055 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000026 2024-04-12T03:39:56.055 DEBUG:boto:Data: 2024-04-12T03:39:56.055 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.055 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.055 DEBUG:boto:Params: {} 2024-04-12T03:39:56.055 DEBUG:boto:Token: None 2024-04-12T03:39:56.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3e16b79994d3f6-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.059 DEBUG:boto:b'dgbpxg-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:39:44.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:39:56.060 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.060 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.060 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.060 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.060 DEBUG:boto:Method: GET 2024-04-12T03:39:56.060 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.060 DEBUG:boto:Data: 2024-04-12T03:39:56.060 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.060 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.060 DEBUG:boto:Params: {} 2024-04-12T03:39:56.061 DEBUG:boto:Token: None 2024-04-12T03:39:56.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80= 2024-04-12T03:39:56.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80=', 'Content-Length': '0'} 2024-04-12T03:39:56.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f58648aab19627-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.063 DEBUG:boto:Method: GET 2024-04-12T03:39:56.063 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.063 DEBUG:boto:Data: 2024-04-12T03:39:56.063 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.063 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.064 DEBUG:boto:Params: {} 2024-04-12T03:39:56.064 DEBUG:boto:Token: None 2024-04-12T03:39:56.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80= 2024-04-12T03:39:56.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80=', 'Content-Length': '0'} 2024-04-12T03:39:56.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000527565dee911eaa3-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.066 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.066 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.066 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.066 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.066 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.067 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.067 DEBUG:boto:Data: 2024-04-12T03:39:56.067 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.067 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.067 DEBUG:boto:Params: {} 2024-04-12T03:39:56.067 DEBUG:boto:Token: None 2024-04-12T03:39:56.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vj75sUtGbltli2e7TBScqR/qEa0= 2024-04-12T03:39:56.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vj75sUtGbltli2e7TBScqR/qEa0=', 'Content-Length': '0'} 2024-04-12T03:39:56.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002dc193074831c74-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.069 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.069 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.069 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.069 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.069 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.069 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.070 DEBUG:boto:Data: 2024-04-12T03:39:56.070 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.070 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.070 DEBUG:boto:Params: {} 2024-04-12T03:39:56.070 DEBUG:boto:Token: None 2024-04-12T03:39:56.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vj75sUtGbltli2e7TBScqR/qEa0= 2024-04-12T03:39:56.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vj75sUtGbltli2e7TBScqR/qEa0=', 'Content-Length': '0'} 2024-04-12T03:39:56.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d54829d49447402f-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:39:56.072 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.072 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.072 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.072 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.072 DEBUG:boto:Method: GET 2024-04-12T03:39:56.073 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.073 DEBUG:boto:Data: 2024-04-12T03:39:56.073 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.073 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.073 DEBUG:boto:Params: {} 2024-04-12T03:39:56.073 DEBUG:boto:Token: None 2024-04-12T03:39:56.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80= 2024-04-12T03:39:56.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80=', 'Content-Length': '0'} 2024-04-12T03:39:56.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46fd20d40d31b35-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.075 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.075 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.075 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.075 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.076 DEBUG:boto:Method: GET 2024-04-12T03:39:56.076 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.076 DEBUG:boto:Data: 2024-04-12T03:39:56.076 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.076 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.076 DEBUG:boto:Params: {} 2024-04-12T03:39:56.076 DEBUG:boto:Token: None 2024-04-12T03:39:56.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027?versionId=null 2024-04-12T03:39:56.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80= 2024-04-12T03:39:56.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GGxoA3pPSTWgmWQegkGAfQ5p80=', 'Content-Length': '0'} 2024-04-12T03:39:56.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a528a5b18b07c9-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.079 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.079 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.079 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.079 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.079 DEBUG:boto:Method: GET 2024-04-12T03:39:56.079 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.079 DEBUG:boto:Data: 2024-04-12T03:39:56.079 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.079 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.079 DEBUG:boto:Params: {} 2024-04-12T03:39:56.080 DEBUG:boto:Token: None 2024-04-12T03:39:56.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e27322ca603a3b7-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.122 DEBUG:boto:b'dgbpxg-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.123 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.123 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.123 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.123 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.123 DEBUG:boto:Method: GET 2024-04-12T03:39:56.123 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.123 DEBUG:boto:Data: 2024-04-12T03:39:56.123 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.123 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.123 DEBUG:boto:Params: {} 2024-04-12T03:39:56.123 DEBUG:boto:Token: None 2024-04-12T03:39:56.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062e71478d6474205-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.127 DEBUG:boto:b'dgbpxg-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.128 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.128 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.128 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.128 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.128 DEBUG:boto:Data: 2024-04-12T03:39:56.128 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.128 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.128 DEBUG:boto:Params: {} 2024-04-12T03:39:56.129 DEBUG:boto:Token: None 2024-04-12T03:39:56.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E5bmlbecOmOWnvlM/VEh574/dLE= 2024-04-12T03:39:56.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E5bmlbecOmOWnvlM/VEh574/dLE=', 'Content-Length': '0'} 2024-04-12T03:39:56.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0315aa62fae024-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.130 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.131 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.131 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.131 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.131 DEBUG:boto:Data: 2024-04-12T03:39:56.131 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.131 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.131 DEBUG:boto:Params: {} 2024-04-12T03:39:56.131 DEBUG:boto:Token: None 2024-04-12T03:39:56.131 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E5bmlbecOmOWnvlM/VEh574/dLE= 2024-04-12T03:39:56.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E5bmlbecOmOWnvlM/VEh574/dLE=', 'Content-Length': '0'} 2024-04-12T03:39:56.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ceeeb41dfa85c7-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:39:56.133 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.133 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.133 DEBUG:boto:Method: GET 2024-04-12T03:39:56.133 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.133 DEBUG:boto:Data: 2024-04-12T03:39:56.133 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.134 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.134 DEBUG:boto:Params: {} 2024-04-12T03:39:56.134 DEBUG:boto:Token: None 2024-04-12T03:39:56.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MVkT2LjJrDm+wXtkOanVoNdShE= 2024-04-12T03:39:56.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MVkT2LjJrDm+wXtkOanVoNdShE=', 'Content-Length': '0'} 2024-04-12T03:39:56.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280ac062938eb057-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.136 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.136 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.136 DEBUG:boto:Method: GET 2024-04-12T03:39:56.136 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.136 DEBUG:boto:Data: 2024-04-12T03:39:56.136 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.136 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.136 DEBUG:boto:Params: {} 2024-04-12T03:39:56.137 DEBUG:boto:Token: None 2024-04-12T03:39:56.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000027 2024-04-12T03:39:56.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/MVkT2LjJrDm+wXtkOanVoNdShE= 2024-04-12T03:39:56.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/MVkT2LjJrDm+wXtkOanVoNdShE=', 'Content-Length': '0'} 2024-04-12T03:39:56.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004207cbf993a2ecf2-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.182 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.182 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.183 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.183 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.183 DEBUG:boto:Method: GET 2024-04-12T03:39:56.183 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.183 DEBUG:boto:Data: 2024-04-12T03:39:56.183 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.183 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.183 DEBUG:boto:Params: {} 2024-04-12T03:39:56.183 DEBUG:boto:Token: None 2024-04-12T03:39:56.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd12bafd3d5854f-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.186 DEBUG:boto:b'dgbpxg-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.186 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.186 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.187 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.187 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.187 DEBUG:boto:Method: GET 2024-04-12T03:39:56.187 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000027 2024-04-12T03:39:56.187 DEBUG:boto:Data: 2024-04-12T03:39:56.187 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.187 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.187 DEBUG:boto:Params: {} 2024-04-12T03:39:56.187 DEBUG:boto:Token: None 2024-04-12T03:39:56.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9a73c6a3e45c880-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.191 DEBUG:boto:b'dgbpxg-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:39:44.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:39:56.192 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.192 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.192 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.192 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.192 DEBUG:boto:Method: GET 2024-04-12T03:39:56.192 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.192 DEBUG:boto:Data: 2024-04-12T03:39:56.192 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.192 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.192 DEBUG:boto:Params: {} 2024-04-12T03:39:56.193 DEBUG:boto:Token: None 2024-04-12T03:39:56.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8= 2024-04-12T03:39:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8=', 'Content-Length': '0'} 2024-04-12T03:39:56.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79cdad03752693d-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.195 DEBUG:boto:Method: GET 2024-04-12T03:39:56.195 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.195 DEBUG:boto:Data: 2024-04-12T03:39:56.195 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.196 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.196 DEBUG:boto:Params: {} 2024-04-12T03:39:56.196 DEBUG:boto:Token: None 2024-04-12T03:39:56.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8= 2024-04-12T03:39:56.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8=', 'Content-Length': '0'} 2024-04-12T03:39:56.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9aab43c8596e137-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.198 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.198 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.198 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.198 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.198 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.198 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.198 DEBUG:boto:Data: 2024-04-12T03:39:56.199 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.199 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.199 DEBUG:boto:Params: {} 2024-04-12T03:39:56.199 DEBUG:boto:Token: None 2024-04-12T03:39:56.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0LhMYArfVF6PPDG2bMu7k+G0buM= 2024-04-12T03:39:56.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0LhMYArfVF6PPDG2bMu7k+G0buM=', 'Content-Length': '0'} 2024-04-12T03:39:56.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b1d0feab5aabc8-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.201 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.201 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.201 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.201 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.201 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.201 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.201 DEBUG:boto:Data: 2024-04-12T03:39:56.202 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.202 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.202 DEBUG:boto:Params: {} 2024-04-12T03:39:56.202 DEBUG:boto:Token: None 2024-04-12T03:39:56.202 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.202 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0LhMYArfVF6PPDG2bMu7k+G0buM= 2024-04-12T03:39:56.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0LhMYArfVF6PPDG2bMu7k+G0buM=', 'Content-Length': '0'} 2024-04-12T03:39:56.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224533e3518ecf10-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:39:56.204 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.204 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.204 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.204 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.205 DEBUG:boto:Method: GET 2024-04-12T03:39:56.205 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.205 DEBUG:boto:Data: 2024-04-12T03:39:56.205 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.205 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.205 DEBUG:boto:Params: {} 2024-04-12T03:39:56.205 DEBUG:boto:Token: None 2024-04-12T03:39:56.205 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.205 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8= 2024-04-12T03:39:56.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8=', 'Content-Length': '0'} 2024-04-12T03:39:56.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bf44f6df94ab54-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.207 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.207 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.207 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.207 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.207 DEBUG:boto:Method: GET 2024-04-12T03:39:56.208 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.208 DEBUG:boto:Data: 2024-04-12T03:39:56.208 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.208 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.208 DEBUG:boto:Params: {} 2024-04-12T03:39:56.208 DEBUG:boto:Token: None 2024-04-12T03:39:56.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028?versionId=null 2024-04-12T03:39:56.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8= 2024-04-12T03:39:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Exv8rb5eK0V3Inpv09XcA5w+H8=', 'Content-Length': '0'} 2024-04-12T03:39:56.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b24e23fb8abc0c-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.210 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.211 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.211 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.211 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.211 DEBUG:boto:Method: GET 2024-04-12T03:39:56.211 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.211 DEBUG:boto:Data: 2024-04-12T03:39:56.211 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.211 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.211 DEBUG:boto:Params: {} 2024-04-12T03:39:56.211 DEBUG:boto:Token: None 2024-04-12T03:39:56.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cbf5d639c1b37fc-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.258 DEBUG:boto:b'dgbpxg-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.259 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.259 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.259 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.259 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.259 DEBUG:boto:Method: GET 2024-04-12T03:39:56.259 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.259 DEBUG:boto:Data: 2024-04-12T03:39:56.259 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.259 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.259 DEBUG:boto:Params: {} 2024-04-12T03:39:56.260 DEBUG:boto:Token: None 2024-04-12T03:39:56.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e662f2613acc2a6-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.263 DEBUG:boto:b'dgbpxg-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.264 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.264 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.264 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.264 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.264 DEBUG:boto:Data: 2024-04-12T03:39:56.264 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.264 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.265 DEBUG:boto:Params: {} 2024-04-12T03:39:56.265 DEBUG:boto:Token: None 2024-04-12T03:39:56.265 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.265 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tRHzeNvxt/4C+3E3W6vYrkQE6FE= 2024-04-12T03:39:56.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tRHzeNvxt/4C+3E3W6vYrkQE6FE=', 'Content-Length': '0'} 2024-04-12T03:39:56.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4832eb36ed4b472-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.267 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.267 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.267 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.267 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.267 DEBUG:boto:Data: 2024-04-12T03:39:56.267 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.267 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.267 DEBUG:boto:Params: {} 2024-04-12T03:39:56.268 DEBUG:boto:Token: None 2024-04-12T03:39:56.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tRHzeNvxt/4C+3E3W6vYrkQE6FE= 2024-04-12T03:39:56.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tRHzeNvxt/4C+3E3W6vYrkQE6FE=', 'Content-Length': '0'} 2024-04-12T03:39:56.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003508514395fdc1b4-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:39:56.270 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.270 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.270 DEBUG:boto:Method: GET 2024-04-12T03:39:56.270 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.270 DEBUG:boto:Data: 2024-04-12T03:39:56.270 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.270 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.271 DEBUG:boto:Params: {} 2024-04-12T03:39:56.271 DEBUG:boto:Token: None 2024-04-12T03:39:56.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N2Z2htDoIUV29MyaWp0mOIAg9LA= 2024-04-12T03:39:56.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N2Z2htDoIUV29MyaWp0mOIAg9LA=', 'Content-Length': '0'} 2024-04-12T03:39:56.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895e20c6dffda6f8-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.273 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.273 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.273 DEBUG:boto:Method: GET 2024-04-12T03:39:56.273 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.273 DEBUG:boto:Data: 2024-04-12T03:39:56.273 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.273 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.273 DEBUG:boto:Params: {} 2024-04-12T03:39:56.274 DEBUG:boto:Token: None 2024-04-12T03:39:56.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000028 2024-04-12T03:39:56.274 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N2Z2htDoIUV29MyaWp0mOIAg9LA= 2024-04-12T03:39:56.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N2Z2htDoIUV29MyaWp0mOIAg9LA=', 'Content-Length': '0'} 2024-04-12T03:39:56.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007946aa8b37b47377-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.318 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.318 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.319 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.319 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.319 DEBUG:boto:Method: GET 2024-04-12T03:39:56.319 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.319 DEBUG:boto:Data: 2024-04-12T03:39:56.319 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.319 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.319 DEBUG:boto:Params: {} 2024-04-12T03:39:56.319 DEBUG:boto:Token: None 2024-04-12T03:39:56.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111f8563145e8ad4-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.322 DEBUG:boto:b'dgbpxg-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.322 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.322 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.322 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.323 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.323 DEBUG:boto:Method: GET 2024-04-12T03:39:56.323 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000028 2024-04-12T03:39:56.323 DEBUG:boto:Data: 2024-04-12T03:39:56.323 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.323 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.323 DEBUG:boto:Params: {} 2024-04-12T03:39:56.323 DEBUG:boto:Token: None 2024-04-12T03:39:56.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae336823a92904a2-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.327 DEBUG:boto:b'dgbpxg-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:39:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:39:56.328 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.328 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.328 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.328 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.328 DEBUG:boto:Method: GET 2024-04-12T03:39:56.328 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.328 DEBUG:boto:Data: 2024-04-12T03:39:56.328 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.328 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.328 DEBUG:boto:Params: {} 2024-04-12T03:39:56.329 DEBUG:boto:Token: None 2024-04-12T03:39:56.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY= 2024-04-12T03:39:56.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY=', 'Content-Length': '0'} 2024-04-12T03:39:56.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f90bf9a91ebf88-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.331 DEBUG:boto:Method: GET 2024-04-12T03:39:56.331 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.331 DEBUG:boto:Data: 2024-04-12T03:39:56.331 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.332 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.332 DEBUG:boto:Params: {} 2024-04-12T03:39:56.332 DEBUG:boto:Token: None 2024-04-12T03:39:56.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY= 2024-04-12T03:39:56.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY=', 'Content-Length': '0'} 2024-04-12T03:39:56.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a7274fed71af18-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.334 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.334 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.334 DEBUG:boto:Data: 2024-04-12T03:39:56.335 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.335 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.335 DEBUG:boto:Params: {} 2024-04-12T03:39:56.335 DEBUG:boto:Token: None 2024-04-12T03:39:56.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iORkTS1kR6OYmW+Ap9DArJ8NKHA= 2024-04-12T03:39:56.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iORkTS1kR6OYmW+Ap9DArJ8NKHA=', 'Content-Length': '0'} 2024-04-12T03:39:56.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009241e7ee80be7801-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.337 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.337 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.337 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.337 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.337 DEBUG:boto:Data: 2024-04-12T03:39:56.337 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.338 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.338 DEBUG:boto:Params: {} 2024-04-12T03:39:56.338 DEBUG:boto:Token: None 2024-04-12T03:39:56.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iORkTS1kR6OYmW+Ap9DArJ8NKHA= 2024-04-12T03:39:56.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iORkTS1kR6OYmW+Ap9DArJ8NKHA=', 'Content-Length': '0'} 2024-04-12T03:39:56.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2baae111143e10-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:39:56.340 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.340 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.340 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.340 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.340 DEBUG:boto:Method: GET 2024-04-12T03:39:56.340 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.340 DEBUG:boto:Data: 2024-04-12T03:39:56.340 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.341 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.341 DEBUG:boto:Params: {} 2024-04-12T03:39:56.341 DEBUG:boto:Token: None 2024-04-12T03:39:56.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY= 2024-04-12T03:39:56.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY=', 'Content-Length': '0'} 2024-04-12T03:39:56.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e73377c82b382d7-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.386 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.386 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.387 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.387 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.387 DEBUG:boto:Method: GET 2024-04-12T03:39:56.387 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.387 DEBUG:boto:Data: 2024-04-12T03:39:56.387 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.387 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.387 DEBUG:boto:Params: {} 2024-04-12T03:39:56.388 DEBUG:boto:Token: None 2024-04-12T03:39:56.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030?versionId=null 2024-04-12T03:39:56.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY= 2024-04-12T03:39:56.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+7JLxFQfb15iwbUERlinbfUOQlY=', 'Content-Length': '0'} 2024-04-12T03:39:56.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4967dc8f5e1a1a6-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.390 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.390 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.390 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.390 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.390 DEBUG:boto:Method: GET 2024-04-12T03:39:56.391 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.391 DEBUG:boto:Data: 2024-04-12T03:39:56.391 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.391 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.391 DEBUG:boto:Params: {} 2024-04-12T03:39:56.391 DEBUG:boto:Token: None 2024-04-12T03:39:56.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec36757425fc220-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.393 DEBUG:boto:b'dgbpxg-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.394 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.394 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.394 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.394 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.394 DEBUG:boto:Method: GET 2024-04-12T03:39:56.394 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.394 DEBUG:boto:Data: 2024-04-12T03:39:56.394 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.395 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.395 DEBUG:boto:Params: {} 2024-04-12T03:39:56.395 DEBUG:boto:Token: None 2024-04-12T03:39:56.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d1c0459a05de10-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.399 DEBUG:boto:b'dgbpxg-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.399 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.399 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.399 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.399 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.399 DEBUG:boto:Data: 2024-04-12T03:39:56.399 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.400 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.400 DEBUG:boto:Params: {} 2024-04-12T03:39:56.400 DEBUG:boto:Token: None 2024-04-12T03:39:56.400 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ylHPtsrjN5C3bAODe9/LWKFbnPc= 2024-04-12T03:39:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ylHPtsrjN5C3bAODe9/LWKFbnPc=', 'Content-Length': '0'} 2024-04-12T03:39:56.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0612f3df74a58a-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.402 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.402 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.402 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.402 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.402 DEBUG:boto:Data: 2024-04-12T03:39:56.402 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.403 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.403 DEBUG:boto:Params: {} 2024-04-12T03:39:56.403 DEBUG:boto:Token: None 2024-04-12T03:39:56.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ylHPtsrjN5C3bAODe9/LWKFbnPc= 2024-04-12T03:39:56.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ylHPtsrjN5C3bAODe9/LWKFbnPc=', 'Content-Length': '0'} 2024-04-12T03:39:56.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c45998729afcada-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:39:56.405 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.405 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.405 DEBUG:boto:Method: GET 2024-04-12T03:39:56.405 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.405 DEBUG:boto:Data: 2024-04-12T03:39:56.405 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.406 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.406 DEBUG:boto:Params: {} 2024-04-12T03:39:56.406 DEBUG:boto:Token: None 2024-04-12T03:39:56.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZSmz0ua4EmimSg7dLqzrT7soun4= 2024-04-12T03:39:56.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZSmz0ua4EmimSg7dLqzrT7soun4=', 'Content-Length': '0'} 2024-04-12T03:39:56.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e892fea674751f-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.408 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.408 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.408 DEBUG:boto:Method: GET 2024-04-12T03:39:56.408 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.408 DEBUG:boto:Data: 2024-04-12T03:39:56.408 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.408 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.408 DEBUG:boto:Params: {} 2024-04-12T03:39:56.408 DEBUG:boto:Token: None 2024-04-12T03:39:56.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000030 2024-04-12T03:39:56.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZSmz0ua4EmimSg7dLqzrT7soun4= 2024-04-12T03:39:56.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZSmz0ua4EmimSg7dLqzrT7soun4=', 'Content-Length': '0'} 2024-04-12T03:39:56.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000480138b770619bff-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.454 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.454 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.455 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.455 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.455 DEBUG:boto:Method: GET 2024-04-12T03:39:56.455 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.455 DEBUG:boto:Data: 2024-04-12T03:39:56.455 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.455 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.455 DEBUG:boto:Params: {} 2024-04-12T03:39:56.455 DEBUG:boto:Token: None 2024-04-12T03:39:56.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afbc2d3417896e7a-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.458 DEBUG:boto:b'dgbpxg-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.458 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.458 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.458 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.459 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.459 DEBUG:boto:Method: GET 2024-04-12T03:39:56.459 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000030 2024-04-12T03:39:56.459 DEBUG:boto:Data: 2024-04-12T03:39:56.459 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.459 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.459 DEBUG:boto:Params: {} 2024-04-12T03:39:56.459 DEBUG:boto:Token: None 2024-04-12T03:39:56.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022501a02e87772b6-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.463 DEBUG:boto:b'dgbpxg-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:39:41.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:39:56.463 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.463 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.464 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.464 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.464 DEBUG:boto:Method: GET 2024-04-12T03:39:56.464 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.464 DEBUG:boto:Data: 2024-04-12T03:39:56.464 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.464 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.464 DEBUG:boto:Params: {} 2024-04-12T03:39:56.464 DEBUG:boto:Token: None 2024-04-12T03:39:56.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk= 2024-04-12T03:39:56.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk=', 'Content-Length': '0'} 2024-04-12T03:39:56.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343de3e245817fdb-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.467 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.467 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.467 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.467 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.467 DEBUG:boto:Method: GET 2024-04-12T03:39:56.467 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.467 DEBUG:boto:Data: 2024-04-12T03:39:56.467 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.467 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.468 DEBUG:boto:Params: {} 2024-04-12T03:39:56.468 DEBUG:boto:Token: None 2024-04-12T03:39:56.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk= 2024-04-12T03:39:56.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk=', 'Content-Length': '0'} 2024-04-12T03:39:56.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1750986cfcaa3a-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.470 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.470 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.470 DEBUG:boto:Data: 2024-04-12T03:39:56.470 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.471 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.471 DEBUG:boto:Params: {} 2024-04-12T03:39:56.471 DEBUG:boto:Token: None 2024-04-12T03:39:56.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G4BuHdYyUhN2c3j2L26JOkwYB0Y= 2024-04-12T03:39:56.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G4BuHdYyUhN2c3j2L26JOkwYB0Y=', 'Content-Length': '0'} 2024-04-12T03:39:56.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824431b48d280cd6-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.473 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.473 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.473 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.473 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.473 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.473 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.473 DEBUG:boto:Data: 2024-04-12T03:39:56.473 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.473 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.474 DEBUG:boto:Params: {} 2024-04-12T03:39:56.474 DEBUG:boto:Token: None 2024-04-12T03:39:56.474 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.474 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G4BuHdYyUhN2c3j2L26JOkwYB0Y= 2024-04-12T03:39:56.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G4BuHdYyUhN2c3j2L26JOkwYB0Y=', 'Content-Length': '0'} 2024-04-12T03:39:56.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59555fa682cd15b-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:39:56.476 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.476 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.476 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.476 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.477 DEBUG:boto:Method: GET 2024-04-12T03:39:56.477 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.477 DEBUG:boto:Data: 2024-04-12T03:39:56.477 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.477 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.477 DEBUG:boto:Params: {} 2024-04-12T03:39:56.477 DEBUG:boto:Token: None 2024-04-12T03:39:56.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk= 2024-04-12T03:39:56.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk=', 'Content-Length': '0'} 2024-04-12T03:39:56.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318eae67d67e5831-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.479 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.479 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.480 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.480 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.480 DEBUG:boto:Method: GET 2024-04-12T03:39:56.480 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.480 DEBUG:boto:Data: 2024-04-12T03:39:56.480 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.480 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.480 DEBUG:boto:Params: {} 2024-04-12T03:39:56.480 DEBUG:boto:Token: None 2024-04-12T03:39:56.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031?versionId=null 2024-04-12T03:39:56.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk= 2024-04-12T03:39:56.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x4SOTFLBgAneGkTWkwOW2+/iTwk=', 'Content-Length': '0'} 2024-04-12T03:39:56.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e86b62502275fb8-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.482 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.483 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.483 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.483 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.483 DEBUG:boto:Method: GET 2024-04-12T03:39:56.483 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.483 DEBUG:boto:Data: 2024-04-12T03:39:56.483 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.483 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.483 DEBUG:boto:Params: {} 2024-04-12T03:39:56.484 DEBUG:boto:Token: None 2024-04-12T03:39:56.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e7865996af5dd6-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.526 DEBUG:boto:b'dgbpxg-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.527 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.527 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.527 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.527 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.527 DEBUG:boto:Method: GET 2024-04-12T03:39:56.527 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.527 DEBUG:boto:Data: 2024-04-12T03:39:56.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.528 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.528 DEBUG:boto:Params: {} 2024-04-12T03:39:56.528 DEBUG:boto:Token: None 2024-04-12T03:39:56.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5c605fc4060cf9-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.532 DEBUG:boto:b'dgbpxg-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.532 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.532 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.532 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.532 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.532 DEBUG:boto:Data: 2024-04-12T03:39:56.532 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.533 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.533 DEBUG:boto:Params: {} 2024-04-12T03:39:56.533 DEBUG:boto:Token: None 2024-04-12T03:39:56.533 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JJOU3MmbepUSo1oGj43FSAHaxmU= 2024-04-12T03:39:56.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JJOU3MmbepUSo1oGj43FSAHaxmU=', 'Content-Length': '0'} 2024-04-12T03:39:56.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4884e848f97d9e1-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.535 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.535 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.535 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.535 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.535 DEBUG:boto:Data: 2024-04-12T03:39:56.535 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.535 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.535 DEBUG:boto:Params: {} 2024-04-12T03:39:56.535 DEBUG:boto:Token: None 2024-04-12T03:39:56.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JJOU3MmbepUSo1oGj43FSAHaxmU= 2024-04-12T03:39:56.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JJOU3MmbepUSo1oGj43FSAHaxmU=', 'Content-Length': '0'} 2024-04-12T03:39:56.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6696f130e538c60-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:39:56.538 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.538 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.538 DEBUG:boto:Method: GET 2024-04-12T03:39:56.538 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.538 DEBUG:boto:Data: 2024-04-12T03:39:56.538 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.539 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.539 DEBUG:boto:Params: {} 2024-04-12T03:39:56.539 DEBUG:boto:Token: None 2024-04-12T03:39:56.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bJoG7smV7n72uywYjPn5jy7yXtg= 2024-04-12T03:39:56.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bJoG7smV7n72uywYjPn5jy7yXtg=', 'Content-Length': '0'} 2024-04-12T03:39:56.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e16198f2177dc545-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.541 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.541 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.541 DEBUG:boto:Method: GET 2024-04-12T03:39:56.541 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.541 DEBUG:boto:Data: 2024-04-12T03:39:56.541 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.541 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.541 DEBUG:boto:Params: {} 2024-04-12T03:39:56.542 DEBUG:boto:Token: None 2024-04-12T03:39:56.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000031 2024-04-12T03:39:56.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bJoG7smV7n72uywYjPn5jy7yXtg= 2024-04-12T03:39:56.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bJoG7smV7n72uywYjPn5jy7yXtg=', 'Content-Length': '0'} 2024-04-12T03:39:56.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39611d4b484f290-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.587 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.587 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.587 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.587 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.587 DEBUG:boto:Method: GET 2024-04-12T03:39:56.587 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.588 DEBUG:boto:Data: 2024-04-12T03:39:56.588 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.588 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.588 DEBUG:boto:Params: {} 2024-04-12T03:39:56.588 DEBUG:boto:Token: None 2024-04-12T03:39:56.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f3f4c6aec7f549-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.591 DEBUG:boto:b'dgbpxg-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.591 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.592 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.592 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.592 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.592 DEBUG:boto:Method: GET 2024-04-12T03:39:56.592 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000031 2024-04-12T03:39:56.592 DEBUG:boto:Data: 2024-04-12T03:39:56.592 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.592 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.592 DEBUG:boto:Params: {} 2024-04-12T03:39:56.592 DEBUG:boto:Token: None 2024-04-12T03:39:56.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.593 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e840d0e34da44afb-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.596 DEBUG:boto:b'dgbpxg-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:39:41.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:39:56.597 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.597 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.597 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.597 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.597 DEBUG:boto:Method: GET 2024-04-12T03:39:56.597 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.597 DEBUG:boto:Data: 2024-04-12T03:39:56.597 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.597 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.597 DEBUG:boto:Params: {} 2024-04-12T03:39:56.598 DEBUG:boto:Token: None 2024-04-12T03:39:56.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE= 2024-04-12T03:39:56.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE=', 'Content-Length': '0'} 2024-04-12T03:39:56.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f870b8598060b0-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.642 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.642 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.642 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.642 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.643 DEBUG:boto:Method: GET 2024-04-12T03:39:56.643 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.643 DEBUG:boto:Data: 2024-04-12T03:39:56.643 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.643 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.643 DEBUG:boto:Params: {} 2024-04-12T03:39:56.643 DEBUG:boto:Token: None 2024-04-12T03:39:56.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE= 2024-04-12T03:39:56.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE=', 'Content-Length': '0'} 2024-04-12T03:39:56.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2874be8b71fa65-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.645 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.645 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.645 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.645 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.646 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.646 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.646 DEBUG:boto:Data: 2024-04-12T03:39:56.646 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.646 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.646 DEBUG:boto:Params: {} 2024-04-12T03:39:56.646 DEBUG:boto:Token: None 2024-04-12T03:39:56.646 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.646 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNLX/MZdeQeUUJS6fYvyGYBKA4A= 2024-04-12T03:39:56.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNLX/MZdeQeUUJS6fYvyGYBKA4A=', 'Content-Length': '0'} 2024-04-12T03:39:56.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008014705d7b6a60cc-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.648 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.648 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.648 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.648 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.649 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.649 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.649 DEBUG:boto:Data: 2024-04-12T03:39:56.649 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.649 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.649 DEBUG:boto:Params: {} 2024-04-12T03:39:56.649 DEBUG:boto:Token: None 2024-04-12T03:39:56.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNLX/MZdeQeUUJS6fYvyGYBKA4A= 2024-04-12T03:39:56.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNLX/MZdeQeUUJS6fYvyGYBKA4A=', 'Content-Length': '0'} 2024-04-12T03:39:56.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ce374e1beca881-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:39:56.651 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.651 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.651 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.652 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.652 DEBUG:boto:Method: GET 2024-04-12T03:39:56.652 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.652 DEBUG:boto:Data: 2024-04-12T03:39:56.652 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.652 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.652 DEBUG:boto:Params: {} 2024-04-12T03:39:56.652 DEBUG:boto:Token: None 2024-04-12T03:39:56.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE= 2024-04-12T03:39:56.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE=', 'Content-Length': '0'} 2024-04-12T03:39:56.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa7446dbe5e73ea-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.654 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.655 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.655 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.655 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.655 DEBUG:boto:Method: GET 2024-04-12T03:39:56.655 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.655 DEBUG:boto:Data: 2024-04-12T03:39:56.655 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.655 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.655 DEBUG:boto:Params: {} 2024-04-12T03:39:56.655 DEBUG:boto:Token: None 2024-04-12T03:39:56.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032?versionId=null 2024-04-12T03:39:56.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE= 2024-04-12T03:39:56.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gD3ZWLhrPUux/NOipPx6JigVADE=', 'Content-Length': '0'} 2024-04-12T03:39:56.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058440abdd52bb60e-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.658 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.658 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.658 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.658 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.658 DEBUG:boto:Method: GET 2024-04-12T03:39:56.658 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.658 DEBUG:boto:Data: 2024-04-12T03:39:56.659 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.659 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.659 DEBUG:boto:Params: {} 2024-04-12T03:39:56.659 DEBUG:boto:Token: None 2024-04-12T03:39:56.659 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.659 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a058139beb80ef1b-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.661 DEBUG:boto:b'dgbpxg-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.661 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.662 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.662 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.662 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.662 DEBUG:boto:Method: GET 2024-04-12T03:39:56.662 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.662 DEBUG:boto:Data: 2024-04-12T03:39:56.662 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.662 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.662 DEBUG:boto:Params: {} 2024-04-12T03:39:56.662 DEBUG:boto:Token: None 2024-04-12T03:39:56.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.663 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b793fc844b01954b-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.667 DEBUG:boto:b'dgbpxg-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.667 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.667 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.667 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.667 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.667 DEBUG:boto:Data: 2024-04-12T03:39:56.668 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.668 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.668 DEBUG:boto:Params: {} 2024-04-12T03:39:56.668 DEBUG:boto:Token: None 2024-04-12T03:39:56.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dv4weNN+bNWmi0OlrLZphmiQP4o= 2024-04-12T03:39:56.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dv4weNN+bNWmi0OlrLZphmiQP4o=', 'Content-Length': '0'} 2024-04-12T03:39:56.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b042d3c625b24b38-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.670 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.670 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.670 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.670 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.670 DEBUG:boto:Data: 2024-04-12T03:39:56.671 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.671 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.671 DEBUG:boto:Params: {} 2024-04-12T03:39:56.671 DEBUG:boto:Token: None 2024-04-12T03:39:56.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dv4weNN+bNWmi0OlrLZphmiQP4o= 2024-04-12T03:39:56.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dv4weNN+bNWmi0OlrLZphmiQP4o=', 'Content-Length': '0'} 2024-04-12T03:39:56.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da861169c5adb747-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:39:56.673 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.673 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.673 DEBUG:boto:Method: GET 2024-04-12T03:39:56.673 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.673 DEBUG:boto:Data: 2024-04-12T03:39:56.673 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.674 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.674 DEBUG:boto:Params: {} 2024-04-12T03:39:56.674 DEBUG:boto:Token: None 2024-04-12T03:39:56.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IRIeshyeSeNxZ8pY7k3yTibtvm4= 2024-04-12T03:39:56.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IRIeshyeSeNxZ8pY7k3yTibtvm4=', 'Content-Length': '0'} 2024-04-12T03:39:56.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1527be75742bf07-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.676 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.676 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.676 DEBUG:boto:Method: GET 2024-04-12T03:39:56.676 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.676 DEBUG:boto:Data: 2024-04-12T03:39:56.676 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.676 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.676 DEBUG:boto:Params: {} 2024-04-12T03:39:56.677 DEBUG:boto:Token: None 2024-04-12T03:39:56.677 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000032 2024-04-12T03:39:56.677 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IRIeshyeSeNxZ8pY7k3yTibtvm4= 2024-04-12T03:39:56.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IRIeshyeSeNxZ8pY7k3yTibtvm4=', 'Content-Length': '0'} 2024-04-12T03:39:56.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05b01186be04a9b-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.722 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.722 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.723 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.723 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.723 DEBUG:boto:Method: GET 2024-04-12T03:39:56.723 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.723 DEBUG:boto:Data: 2024-04-12T03:39:56.723 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.723 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.723 DEBUG:boto:Params: {} 2024-04-12T03:39:56.723 DEBUG:boto:Token: None 2024-04-12T03:39:56.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8669c1e4f8f86a7-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.726 DEBUG:boto:b'dgbpxg-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.726 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.727 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.727 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.727 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.727 DEBUG:boto:Method: GET 2024-04-12T03:39:56.727 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000032 2024-04-12T03:39:56.727 DEBUG:boto:Data: 2024-04-12T03:39:56.727 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.727 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.727 DEBUG:boto:Params: {} 2024-04-12T03:39:56.728 DEBUG:boto:Token: None 2024-04-12T03:39:56.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f00620728468e1e-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.731 DEBUG:boto:b'dgbpxg-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:39:40.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:39:56.732 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.732 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.732 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.732 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.732 DEBUG:boto:Method: GET 2024-04-12T03:39:56.732 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.732 DEBUG:boto:Data: 2024-04-12T03:39:56.732 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.733 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.733 DEBUG:boto:Params: {} 2024-04-12T03:39:56.733 DEBUG:boto:Token: None 2024-04-12T03:39:56.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc= 2024-04-12T03:39:56.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc=', 'Content-Length': '0'} 2024-04-12T03:39:56.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b623acbce7cd7b6-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.735 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.735 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.735 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.735 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.735 DEBUG:boto:Method: GET 2024-04-12T03:39:56.735 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.736 DEBUG:boto:Data: 2024-04-12T03:39:56.736 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.736 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.736 DEBUG:boto:Params: {} 2024-04-12T03:39:56.736 DEBUG:boto:Token: None 2024-04-12T03:39:56.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc= 2024-04-12T03:39:56.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc=', 'Content-Length': '0'} 2024-04-12T03:39:56.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004309960ec890170c-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.738 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.738 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.739 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.739 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.739 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.739 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.739 DEBUG:boto:Data: 2024-04-12T03:39:56.739 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.739 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.739 DEBUG:boto:Params: {} 2024-04-12T03:39:56.739 DEBUG:boto:Token: None 2024-04-12T03:39:56.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7nw1dXz/0LMESiLbRGfhdRv7jDQ= 2024-04-12T03:39:56.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7nw1dXz/0LMESiLbRGfhdRv7jDQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63efb5b88d7d6c9-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.741 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.741 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.741 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.742 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.742 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.742 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.742 DEBUG:boto:Data: 2024-04-12T03:39:56.742 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.742 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.742 DEBUG:boto:Params: {} 2024-04-12T03:39:56.742 DEBUG:boto:Token: None 2024-04-12T03:39:56.742 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7nw1dXz/0LMESiLbRGfhdRv7jDQ= 2024-04-12T03:39:56.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7nw1dXz/0LMESiLbRGfhdRv7jDQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b169e382795665-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:39:56.744 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.744 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.745 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.745 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.745 DEBUG:boto:Method: GET 2024-04-12T03:39:56.745 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.745 DEBUG:boto:Data: 2024-04-12T03:39:56.745 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.745 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.745 DEBUG:boto:Params: {} 2024-04-12T03:39:56.745 DEBUG:boto:Token: None 2024-04-12T03:39:56.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.746 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc= 2024-04-12T03:39:56.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc=', 'Content-Length': '0'} 2024-04-12T03:39:56.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035291f8369042056-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.790 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.790 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.790 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.791 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.791 DEBUG:boto:Method: GET 2024-04-12T03:39:56.791 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.791 DEBUG:boto:Data: 2024-04-12T03:39:56.791 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.791 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.791 DEBUG:boto:Params: {} 2024-04-12T03:39:56.791 DEBUG:boto:Token: None 2024-04-12T03:39:56.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033?versionId=null 2024-04-12T03:39:56.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc= 2024-04-12T03:39:56.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:32tGwHC82VVuo+VsAg7t+eRN9Zc=', 'Content-Length': '0'} 2024-04-12T03:39:56.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e207cd95572d8d-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.794 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.794 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.794 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.794 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.794 DEBUG:boto:Method: GET 2024-04-12T03:39:56.794 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.794 DEBUG:boto:Data: 2024-04-12T03:39:56.794 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.794 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.794 DEBUG:boto:Params: {} 2024-04-12T03:39:56.795 DEBUG:boto:Token: None 2024-04-12T03:39:56.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc931789c7bb3ba-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.797 DEBUG:boto:b'dgbpxg-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.798 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.798 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.798 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.798 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.798 DEBUG:boto:Method: GET 2024-04-12T03:39:56.798 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.798 DEBUG:boto:Data: 2024-04-12T03:39:56.798 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.798 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.798 DEBUG:boto:Params: {} 2024-04-12T03:39:56.799 DEBUG:boto:Token: None 2024-04-12T03:39:56.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf8a9eb4ad87424-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.802 DEBUG:boto:b'dgbpxg-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.802 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.802 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.803 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.803 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.803 DEBUG:boto:Data: 2024-04-12T03:39:56.803 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.803 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.803 DEBUG:boto:Params: {} 2024-04-12T03:39:56.803 DEBUG:boto:Token: None 2024-04-12T03:39:56.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5tDfEsMNp7ZNp8I1nL/OmxjkPt8= 2024-04-12T03:39:56.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5tDfEsMNp7ZNp8I1nL/OmxjkPt8=', 'Content-Length': '0'} 2024-04-12T03:39:56.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beed93acc341080a-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.805 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.805 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.805 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.805 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.805 DEBUG:boto:Data: 2024-04-12T03:39:56.805 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.805 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.806 DEBUG:boto:Params: {} 2024-04-12T03:39:56.806 DEBUG:boto:Token: None 2024-04-12T03:39:56.806 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5tDfEsMNp7ZNp8I1nL/OmxjkPt8= 2024-04-12T03:39:56.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5tDfEsMNp7ZNp8I1nL/OmxjkPt8=', 'Content-Length': '0'} 2024-04-12T03:39:56.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58b4792bfe5b0e6-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:39:56.808 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.808 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.808 DEBUG:boto:Method: GET 2024-04-12T03:39:56.808 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.808 DEBUG:boto:Data: 2024-04-12T03:39:56.808 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.808 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.808 DEBUG:boto:Params: {} 2024-04-12T03:39:56.809 DEBUG:boto:Token: None 2024-04-12T03:39:56.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.809 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WoI9DlU51XCWEQRaHc15323S0nY= 2024-04-12T03:39:56.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WoI9DlU51XCWEQRaHc15323S0nY=', 'Content-Length': '0'} 2024-04-12T03:39:56.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa668d6a5c063344-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.810 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.811 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.811 DEBUG:boto:Method: GET 2024-04-12T03:39:56.811 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.811 DEBUG:boto:Data: 2024-04-12T03:39:56.811 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.811 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.811 DEBUG:boto:Params: {} 2024-04-12T03:39:56.811 DEBUG:boto:Token: None 2024-04-12T03:39:56.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000033 2024-04-12T03:39:56.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WoI9DlU51XCWEQRaHc15323S0nY= 2024-04-12T03:39:56.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WoI9DlU51XCWEQRaHc15323S0nY=', 'Content-Length': '0'} 2024-04-12T03:39:56.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008315c60d88a67a67-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.813 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.814 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.814 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.814 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.814 DEBUG:boto:Method: GET 2024-04-12T03:39:56.814 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.814 DEBUG:boto:Data: 2024-04-12T03:39:56.814 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.814 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.814 DEBUG:boto:Params: {} 2024-04-12T03:39:56.814 DEBUG:boto:Token: None 2024-04-12T03:39:56.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024867a085f520712-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.817 DEBUG:boto:b'dgbpxg-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.817 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.817 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.817 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.818 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.818 DEBUG:boto:Method: GET 2024-04-12T03:39:56.818 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000033 2024-04-12T03:39:56.818 DEBUG:boto:Data: 2024-04-12T03:39:56.818 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.818 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.818 DEBUG:boto:Params: {} 2024-04-12T03:39:56.818 DEBUG:boto:Token: None 2024-04-12T03:39:56.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f297b207bff75949-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.822 DEBUG:boto:b'dgbpxg-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:39:43.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:39:56.823 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.823 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.823 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.823 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.823 DEBUG:boto:Method: GET 2024-04-12T03:39:56.823 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.823 DEBUG:boto:Data: 2024-04-12T03:39:56.823 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.823 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.823 DEBUG:boto:Params: {} 2024-04-12T03:39:56.823 DEBUG:boto:Token: None 2024-04-12T03:39:56.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ= 2024-04-12T03:39:56.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed6a062de1db8f5-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.826 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.826 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.826 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.826 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.826 DEBUG:boto:Method: GET 2024-04-12T03:39:56.826 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.827 DEBUG:boto:Data: 2024-04-12T03:39:56.827 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.827 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.827 DEBUG:boto:Params: {} 2024-04-12T03:39:56.827 DEBUG:boto:Token: None 2024-04-12T03:39:56.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ= 2024-04-12T03:39:56.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088fa825d49219ded-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.829 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.830 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.830 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.830 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.830 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.830 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.830 DEBUG:boto:Data: 2024-04-12T03:39:56.830 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.830 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.830 DEBUG:boto:Params: {} 2024-04-12T03:39:56.830 DEBUG:boto:Token: None 2024-04-12T03:39:56.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.831 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:edhYtO/xLF68DoWuoikffoz37zk= 2024-04-12T03:39:56.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:edhYtO/xLF68DoWuoikffoz37zk=', 'Content-Length': '0'} 2024-04-12T03:39:56.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a4088b35c9d2407-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.833 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.833 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.833 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.833 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.833 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.833 DEBUG:boto:Data: 2024-04-12T03:39:56.833 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.834 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.834 DEBUG:boto:Params: {} 2024-04-12T03:39:56.834 DEBUG:boto:Token: None 2024-04-12T03:39:56.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.834 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:edhYtO/xLF68DoWuoikffoz37zk= 2024-04-12T03:39:56.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:edhYtO/xLF68DoWuoikffoz37zk=', 'Content-Length': '0'} 2024-04-12T03:39:56.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf24105eb4f38d3-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:39:56.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.836 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.836 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.836 DEBUG:boto:Method: GET 2024-04-12T03:39:56.837 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.837 DEBUG:boto:Data: 2024-04-12T03:39:56.837 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.837 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.837 DEBUG:boto:Params: {} 2024-04-12T03:39:56.837 DEBUG:boto:Token: None 2024-04-12T03:39:56.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ= 2024-04-12T03:39:56.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfbaf735188ea7d2-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.839 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.839 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.839 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.839 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.839 DEBUG:boto:Method: GET 2024-04-12T03:39:56.840 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.840 DEBUG:boto:Data: 2024-04-12T03:39:56.840 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.840 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.840 DEBUG:boto:Params: {} 2024-04-12T03:39:56.840 DEBUG:boto:Token: None 2024-04-12T03:39:56.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036?versionId=null 2024-04-12T03:39:56.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ= 2024-04-12T03:39:56.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:41YX0qSsZXr3Qr90rTGjzXAytlQ=', 'Content-Length': '0'} 2024-04-12T03:39:56.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858646994ef307ba-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.842 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.842 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.842 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.842 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.843 DEBUG:boto:Method: GET 2024-04-12T03:39:56.843 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.843 DEBUG:boto:Data: 2024-04-12T03:39:56.843 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.843 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.843 DEBUG:boto:Params: {} 2024-04-12T03:39:56.843 DEBUG:boto:Token: None 2024-04-12T03:39:56.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeaacbc163a6b161-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.886 DEBUG:boto:b'dgbpxg-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.887 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.887 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.887 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.887 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.887 DEBUG:boto:Method: GET 2024-04-12T03:39:56.887 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.887 DEBUG:boto:Data: 2024-04-12T03:39:56.887 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.887 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.887 DEBUG:boto:Params: {} 2024-04-12T03:39:56.887 DEBUG:boto:Token: None 2024-04-12T03:39:56.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cb6a3bdc2ad440-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.891 DEBUG:boto:b'dgbpxg-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.891 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.891 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.892 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.892 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.892 DEBUG:boto:Data: 2024-04-12T03:39:56.892 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.892 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.892 DEBUG:boto:Params: {} 2024-04-12T03:39:56.892 DEBUG:boto:Token: None 2024-04-12T03:39:56.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/7p4jav0ikyrZdFngdvMWMJILLA= 2024-04-12T03:39:56.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/7p4jav0ikyrZdFngdvMWMJILLA=', 'Content-Length': '0'} 2024-04-12T03:39:56.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011461776dc4500c4-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.894 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.894 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.894 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.894 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.894 DEBUG:boto:Data: 2024-04-12T03:39:56.895 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.895 DEBUG:boto:Params: {} 2024-04-12T03:39:56.895 DEBUG:boto:Token: None 2024-04-12T03:39:56.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/7p4jav0ikyrZdFngdvMWMJILLA= 2024-04-12T03:39:56.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/7p4jav0ikyrZdFngdvMWMJILLA=', 'Content-Length': '0'} 2024-04-12T03:39:56.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aee75a20f21ad2a-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:39:56.897 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.897 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.897 DEBUG:boto:Method: GET 2024-04-12T03:39:56.897 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.897 DEBUG:boto:Data: 2024-04-12T03:39:56.897 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.898 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.898 DEBUG:boto:Params: {} 2024-04-12T03:39:56.898 DEBUG:boto:Token: None 2024-04-12T03:39:56.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gi8HL2Vk1mC3IPKjDVDB8Kb05fk= 2024-04-12T03:39:56.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gi8HL2Vk1mC3IPKjDVDB8Kb05fk=', 'Content-Length': '0'} 2024-04-12T03:39:56.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f391bd15fbff963-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.900 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.900 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.900 DEBUG:boto:Method: GET 2024-04-12T03:39:56.900 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.900 DEBUG:boto:Data: 2024-04-12T03:39:56.901 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.901 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.901 DEBUG:boto:Params: {} 2024-04-12T03:39:56.901 DEBUG:boto:Token: None 2024-04-12T03:39:56.901 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000036 2024-04-12T03:39:56.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gi8HL2Vk1mC3IPKjDVDB8Kb05fk= 2024-04-12T03:39:56.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gi8HL2Vk1mC3IPKjDVDB8Kb05fk=', 'Content-Length': '0'} 2024-04-12T03:39:56.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed97c8fd59e722a-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.903 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.903 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.903 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.903 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.903 DEBUG:boto:Method: GET 2024-04-12T03:39:56.904 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.904 DEBUG:boto:Data: 2024-04-12T03:39:56.904 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.904 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.904 DEBUG:boto:Params: {} 2024-04-12T03:39:56.904 DEBUG:boto:Token: None 2024-04-12T03:39:56.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af89140d62e575a5-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.907 DEBUG:boto:b'dgbpxg-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.907 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.907 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.907 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.907 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.907 DEBUG:boto:Method: GET 2024-04-12T03:39:56.908 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000036 2024-04-12T03:39:56.908 DEBUG:boto:Data: 2024-04-12T03:39:56.908 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.908 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.908 DEBUG:boto:Params: {} 2024-04-12T03:39:56.908 DEBUG:boto:Token: None 2024-04-12T03:39:56.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ec341348f0761b-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.954 DEBUG:boto:b'dgbpxg-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:39:43.354Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:39:56.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.955 DEBUG:boto:Method: GET 2024-04-12T03:39:56.955 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.955 DEBUG:boto:Data: 2024-04-12T03:39:56.955 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.955 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.956 DEBUG:boto:Params: {} 2024-04-12T03:39:56.956 DEBUG:boto:Token: None 2024-04-12T03:39:56.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE= 2024-04-12T03:39:56.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE=', 'Content-Length': '0'} 2024-04-12T03:39:56.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073bf77ff6fc22d53-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.958 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.958 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.958 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.958 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.959 DEBUG:boto:Method: GET 2024-04-12T03:39:56.959 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.959 DEBUG:boto:Data: 2024-04-12T03:39:56.959 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.959 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.959 DEBUG:boto:Params: {} 2024-04-12T03:39:56.959 DEBUG:boto:Token: None 2024-04-12T03:39:56.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE= 2024-04-12T03:39:56.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE=', 'Content-Length': '0'} 2024-04-12T03:39:56.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb1e1f848114fa4-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.962 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.962 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.962 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.962 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.962 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.962 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.962 DEBUG:boto:Data: 2024-04-12T03:39:56.962 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.962 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.963 DEBUG:boto:Params: {} 2024-04-12T03:39:56.963 DEBUG:boto:Token: None 2024-04-12T03:39:56.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Myjh27o0HA8XYfnVKuvbNyKcHcA= 2024-04-12T03:39:56.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Myjh27o0HA8XYfnVKuvbNyKcHcA=', 'Content-Length': '0'} 2024-04-12T03:39:56.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba93e5599d264e2-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.965 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.965 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.965 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.965 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.965 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.965 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.965 DEBUG:boto:Data: 2024-04-12T03:39:56.965 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.966 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.966 DEBUG:boto:Params: {} 2024-04-12T03:39:56.966 DEBUG:boto:Token: None 2024-04-12T03:39:56.966 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.966 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Myjh27o0HA8XYfnVKuvbNyKcHcA= 2024-04-12T03:39:56.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Myjh27o0HA8XYfnVKuvbNyKcHcA=', 'Content-Length': '0'} 2024-04-12T03:39:56.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000209c8cf4896280ec-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:39:56.968 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.968 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.968 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.968 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.969 DEBUG:boto:Method: GET 2024-04-12T03:39:56.969 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.969 DEBUG:boto:Data: 2024-04-12T03:39:56.969 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.969 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.969 DEBUG:boto:Params: {} 2024-04-12T03:39:56.969 DEBUG:boto:Token: None 2024-04-12T03:39:56.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE= 2024-04-12T03:39:56.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE=', 'Content-Length': '0'} 2024-04-12T03:39:56.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1546b28b56963c-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.971 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.971 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.971 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.971 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.972 DEBUG:boto:Method: GET 2024-04-12T03:39:56.972 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.972 DEBUG:boto:Data: 2024-04-12T03:39:56.972 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.972 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.972 DEBUG:boto:Params: {} 2024-04-12T03:39:56.972 DEBUG:boto:Token: None 2024-04-12T03:39:56.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037?versionId=null 2024-04-12T03:39:56.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE= 2024-04-12T03:39:56.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AqK2nGlkPdE4hNbmtXT6YBt5VCE=', 'Content-Length': '0'} 2024-04-12T03:39:56.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012122b2e0559597b-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.974 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.974 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.974 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.974 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.975 DEBUG:boto:Method: GET 2024-04-12T03:39:56.975 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.975 DEBUG:boto:Data: 2024-04-12T03:39:56.975 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.975 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.975 DEBUG:boto:Params: {} 2024-04-12T03:39:56.975 DEBUG:boto:Token: None 2024-04-12T03:39:56.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.975 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000799ea23a54e09967-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.977 DEBUG:boto:b'dgbpxg-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.978 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.978 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.978 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.978 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.978 DEBUG:boto:Method: GET 2024-04-12T03:39:56.978 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.978 DEBUG:boto:Data: 2024-04-12T03:39:56.978 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.978 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.979 DEBUG:boto:Params: {} 2024-04-12T03:39:56.979 DEBUG:boto:Token: None 2024-04-12T03:39:56.979 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.979 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2dbadd79d784b71-006618ad0c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.982 DEBUG:boto:b'dgbpxg-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:56.982 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.982 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.983 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.983 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.983 DEBUG:boto:Data: 2024-04-12T03:39:56.983 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.983 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.983 DEBUG:boto:Params: {} 2024-04-12T03:39:56.983 DEBUG:boto:Token: None 2024-04-12T03:39:56.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ddugC1ZnImtgq9XNlI1A6m01ON4= 2024-04-12T03:39:56.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddugC1ZnImtgq9XNlI1A6m01ON4=', 'Content-Length': '0'} 2024-04-12T03:39:56.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee468922f89453a9-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.985 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.985 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.986 DEBUG:boto:Method: HEAD 2024-04-12T03:39:56.986 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.986 DEBUG:boto:Data: 2024-04-12T03:39:56.986 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.986 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.986 DEBUG:boto:Params: {} 2024-04-12T03:39:56.986 DEBUG:boto:Token: None 2024-04-12T03:39:56.986 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.987 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ddugC1ZnImtgq9XNlI1A6m01ON4= 2024-04-12T03:39:56.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddugC1ZnImtgq9XNlI1A6m01ON4=', 'Content-Length': '0'} 2024-04-12T03:39:56.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e92aeb8a5cfeff5-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:39:56.988 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.988 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.988 DEBUG:boto:Method: GET 2024-04-12T03:39:56.988 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.989 DEBUG:boto:Data: 2024-04-12T03:39:56.989 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.989 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.989 DEBUG:boto:Params: {} 2024-04-12T03:39:56.989 DEBUG:boto:Token: None 2024-04-12T03:39:56.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e7SGXu+CU8gFyb23XSMNYVxp//0= 2024-04-12T03:39:56.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e7SGXu+CU8gFyb23XSMNYVxp//0=', 'Content-Length': '0'} 2024-04-12T03:39:56.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093da68a600b1952f-006618ad0c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.991 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.991 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.991 DEBUG:boto:Method: GET 2024-04-12T03:39:56.991 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.991 DEBUG:boto:Data: 2024-04-12T03:39:56.991 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:56.992 DEBUG:boto:Port: 8000 2024-04-12T03:39:56.992 DEBUG:boto:Params: {} 2024-04-12T03:39:56.992 DEBUG:boto:Token: None 2024-04-12T03:39:56.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/obj00000000000000000037 2024-04-12T03:39:56.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e7SGXu+CU8gFyb23XSMNYVxp//0= 2024-04-12T03:39:56.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e7SGXu+CU8gFyb23XSMNYVxp//0=', 'Content-Length': '0'} 2024-04-12T03:39:56.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2eb32b69317be5-006618ad0c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:56.994 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:56.994 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:56.994 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.994 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.994 DEBUG:boto:Method: GET 2024-04-12T03:39:56.994 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:56.994 DEBUG:boto:Data: 2024-04-12T03:39:56.994 DEBUG:boto:Headers: {} 2024-04-12T03:39:56.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:56.994 DEBUG:boto:Port: 8001 2024-04-12T03:39:56.994 DEBUG:boto:Params: {} 2024-04-12T03:39:56.995 DEBUG:boto:Token: None 2024-04-12T03:39:56.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:56 GMT /dgbpxg-60/ 2024-04-12T03:39:56.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U= 2024-04-12T03:39:56.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:56 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okBtaZC9Be1Fce4O+YhUZYBdE1U=', 'Content-Length': '0'} 2024-04-12T03:39:56.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d808dd12fd0481-006618ad0c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.042 DEBUG:boto:b'dgbpxg-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.043 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.043 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.043 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:57.043 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:57.043 DEBUG:boto:Method: GET 2024-04-12T03:39:57.043 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000037 2024-04-12T03:39:57.043 DEBUG:boto:Data: 2024-04-12T03:39:57.043 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.043 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.044 DEBUG:boto:Params: {} 2024-04-12T03:39:57.044 DEBUG:boto:Token: None 2024-04-12T03:39:57.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7a9fb5db2da1a7-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.047 DEBUG:boto:b'dgbpxg-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:39:41.458Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:39:57.048 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.048 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.048 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.048 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.048 DEBUG:boto:Method: GET 2024-04-12T03:39:57.048 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.048 DEBUG:boto:Data: 2024-04-12T03:39:57.048 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.049 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.049 DEBUG:boto:Params: {} 2024-04-12T03:39:57.049 DEBUG:boto:Token: None 2024-04-12T03:39:57.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.049 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E= 2024-04-12T03:39:57.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E=', 'Content-Length': '0'} 2024-04-12T03:39:57.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273b4a082e0629d9-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.051 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.051 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.051 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.051 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.051 DEBUG:boto:Method: GET 2024-04-12T03:39:57.051 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.052 DEBUG:boto:Data: 2024-04-12T03:39:57.052 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.052 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.052 DEBUG:boto:Params: {} 2024-04-12T03:39:57.052 DEBUG:boto:Token: None 2024-04-12T03:39:57.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E= 2024-04-12T03:39:57.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E=', 'Content-Length': '0'} 2024-04-12T03:39:57.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6b7711c1011b8d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.054 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.054 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.054 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.054 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.054 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.055 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.055 DEBUG:boto:Data: 2024-04-12T03:39:57.055 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.055 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.055 DEBUG:boto:Params: {} 2024-04-12T03:39:57.055 DEBUG:boto:Token: None 2024-04-12T03:39:57.055 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SaoXyFTIXnnIzn7fMSRXGIuENV4= 2024-04-12T03:39:57.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SaoXyFTIXnnIzn7fMSRXGIuENV4=', 'Content-Length': '0'} 2024-04-12T03:39:57.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae5f15c10fa87e4-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.057 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.057 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.057 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.057 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.057 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.057 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.057 DEBUG:boto:Data: 2024-04-12T03:39:57.057 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.058 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.058 DEBUG:boto:Params: {} 2024-04-12T03:39:57.058 DEBUG:boto:Token: None 2024-04-12T03:39:57.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SaoXyFTIXnnIzn7fMSRXGIuENV4= 2024-04-12T03:39:57.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SaoXyFTIXnnIzn7fMSRXGIuENV4=', 'Content-Length': '0'} 2024-04-12T03:39:57.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000230422e5bd0e2e0d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:39:57.060 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.060 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.061 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.061 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.061 DEBUG:boto:Method: GET 2024-04-12T03:39:57.061 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.061 DEBUG:boto:Data: 2024-04-12T03:39:57.061 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.061 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.061 DEBUG:boto:Params: {} 2024-04-12T03:39:57.061 DEBUG:boto:Token: None 2024-04-12T03:39:57.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E= 2024-04-12T03:39:57.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E=', 'Content-Length': '0'} 2024-04-12T03:39:57.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ee183f5b3be448-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.064 DEBUG:boto:Method: GET 2024-04-12T03:39:57.064 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.064 DEBUG:boto:Data: 2024-04-12T03:39:57.064 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.064 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.064 DEBUG:boto:Params: {} 2024-04-12T03:39:57.064 DEBUG:boto:Token: None 2024-04-12T03:39:57.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039?versionId=null 2024-04-12T03:39:57.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E= 2024-04-12T03:39:57.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mRkdDRsVEw/5BgaYo6PxMbGvK8E=', 'Content-Length': '0'} 2024-04-12T03:39:57.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750a51651ca3f67d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.066 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.067 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.067 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.067 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.067 DEBUG:boto:Method: GET 2024-04-12T03:39:57.067 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.067 DEBUG:boto:Data: 2024-04-12T03:39:57.067 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.067 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.067 DEBUG:boto:Params: {} 2024-04-12T03:39:57.067 DEBUG:boto:Token: None 2024-04-12T03:39:57.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001adcd2fcc3ccd156-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.110 DEBUG:boto:b'dgbpxg-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.110 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.111 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.111 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.111 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.111 DEBUG:boto:Method: GET 2024-04-12T03:39:57.111 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.111 DEBUG:boto:Data: 2024-04-12T03:39:57.111 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.111 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.111 DEBUG:boto:Params: {} 2024-04-12T03:39:57.111 DEBUG:boto:Token: None 2024-04-12T03:39:57.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d76f95a25fd6d5f1-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.116 DEBUG:boto:b'dgbpxg-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.116 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.116 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.116 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.116 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.116 DEBUG:boto:Data: 2024-04-12T03:39:57.116 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.117 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.117 DEBUG:boto:Params: {} 2024-04-12T03:39:57.117 DEBUG:boto:Token: None 2024-04-12T03:39:57.117 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:95wy8Xz/oI+Y3HeHxlH8hPiJMXE= 2024-04-12T03:39:57.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:95wy8Xz/oI+Y3HeHxlH8hPiJMXE=', 'Content-Length': '0'} 2024-04-12T03:39:57.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30e7a73aadc4b55-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.119 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.119 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.119 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.119 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.119 DEBUG:boto:Data: 2024-04-12T03:39:57.119 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.120 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.120 DEBUG:boto:Params: {} 2024-04-12T03:39:57.120 DEBUG:boto:Token: None 2024-04-12T03:39:57.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:95wy8Xz/oI+Y3HeHxlH8hPiJMXE= 2024-04-12T03:39:57.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:95wy8Xz/oI+Y3HeHxlH8hPiJMXE=', 'Content-Length': '0'} 2024-04-12T03:39:57.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c8baaf6b919794-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:39:57.122 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.122 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.122 DEBUG:boto:Method: GET 2024-04-12T03:39:57.122 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.122 DEBUG:boto:Data: 2024-04-12T03:39:57.122 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.122 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.123 DEBUG:boto:Params: {} 2024-04-12T03:39:57.123 DEBUG:boto:Token: None 2024-04-12T03:39:57.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wmpLSxoTeEQuzOl5mVoO3FIWjQw= 2024-04-12T03:39:57.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wmpLSxoTeEQuzOl5mVoO3FIWjQw=', 'Content-Length': '0'} 2024-04-12T03:39:57.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008355a1274839dc87-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.125 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.125 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.125 DEBUG:boto:Method: GET 2024-04-12T03:39:57.125 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.125 DEBUG:boto:Data: 2024-04-12T03:39:57.125 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.125 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.125 DEBUG:boto:Params: {} 2024-04-12T03:39:57.125 DEBUG:boto:Token: None 2024-04-12T03:39:57.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000039 2024-04-12T03:39:57.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wmpLSxoTeEQuzOl5mVoO3FIWjQw= 2024-04-12T03:39:57.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wmpLSxoTeEQuzOl5mVoO3FIWjQw=', 'Content-Length': '0'} 2024-04-12T03:39:57.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4fcafe19f1fb44-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.170 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.170 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.171 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.171 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.171 DEBUG:boto:Method: GET 2024-04-12T03:39:57.171 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.171 DEBUG:boto:Data: 2024-04-12T03:39:57.171 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.171 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.171 DEBUG:boto:Params: {} 2024-04-12T03:39:57.171 DEBUG:boto:Token: None 2024-04-12T03:39:57.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4d31d742d5234e-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.174 DEBUG:boto:b'dgbpxg-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.174 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.174 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.174 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.175 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.175 DEBUG:boto:Method: GET 2024-04-12T03:39:57.175 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000039 2024-04-12T03:39:57.175 DEBUG:boto:Data: 2024-04-12T03:39:57.175 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.175 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.175 DEBUG:boto:Params: {} 2024-04-12T03:39:57.175 DEBUG:boto:Token: None 2024-04-12T03:39:57.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cac19a23b11edc-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.179 DEBUG:boto:b'dgbpxg-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:39:44.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:39:57.179 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.179 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.179 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.179 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.180 DEBUG:boto:Method: GET 2024-04-12T03:39:57.180 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.180 DEBUG:boto:Data: 2024-04-12T03:39:57.180 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.180 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.180 DEBUG:boto:Params: {} 2024-04-12T03:39:57.180 DEBUG:boto:Token: None 2024-04-12T03:39:57.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q= 2024-04-12T03:39:57.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e127dd327e47e9-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.182 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.182 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.182 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.183 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.183 DEBUG:boto:Method: GET 2024-04-12T03:39:57.183 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.183 DEBUG:boto:Data: 2024-04-12T03:39:57.183 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.183 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.183 DEBUG:boto:Params: {} 2024-04-12T03:39:57.183 DEBUG:boto:Token: None 2024-04-12T03:39:57.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q= 2024-04-12T03:39:57.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba5b7e22b032eb4-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.185 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.186 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.186 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.186 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.186 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.186 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.186 DEBUG:boto:Data: 2024-04-12T03:39:57.186 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.186 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.186 DEBUG:boto:Params: {} 2024-04-12T03:39:57.187 DEBUG:boto:Token: None 2024-04-12T03:39:57.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsifEGGwjCUPkr4EzspYE5sdE1s= 2024-04-12T03:39:57.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsifEGGwjCUPkr4EzspYE5sdE1s=', 'Content-Length': '0'} 2024-04-12T03:39:57.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258115bc6570261b-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.189 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.189 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.189 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.189 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.189 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.189 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.189 DEBUG:boto:Data: 2024-04-12T03:39:57.189 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.189 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.189 DEBUG:boto:Params: {} 2024-04-12T03:39:57.189 DEBUG:boto:Token: None 2024-04-12T03:39:57.190 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nsifEGGwjCUPkr4EzspYE5sdE1s= 2024-04-12T03:39:57.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nsifEGGwjCUPkr4EzspYE5sdE1s=', 'Content-Length': '0'} 2024-04-12T03:39:57.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83b7c76d1e5fba0-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:39:57.192 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.192 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.192 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.192 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.192 DEBUG:boto:Method: GET 2024-04-12T03:39:57.192 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.192 DEBUG:boto:Data: 2024-04-12T03:39:57.192 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.192 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.192 DEBUG:boto:Params: {} 2024-04-12T03:39:57.193 DEBUG:boto:Token: None 2024-04-12T03:39:57.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q= 2024-04-12T03:39:57.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3ad3bfb95e3c25-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.195 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.195 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.195 DEBUG:boto:Method: GET 2024-04-12T03:39:57.195 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.195 DEBUG:boto:Data: 2024-04-12T03:39:57.195 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.195 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.195 DEBUG:boto:Params: {} 2024-04-12T03:39:57.196 DEBUG:boto:Token: None 2024-04-12T03:39:57.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040?versionId=null 2024-04-12T03:39:57.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q= 2024-04-12T03:39:57.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aNWYfPBYvOhXA4FJUeudaPUWq6Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a0171af41a2ccc-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.198 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.198 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.198 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.198 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.198 DEBUG:boto:Method: GET 2024-04-12T03:39:57.198 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.198 DEBUG:boto:Data: 2024-04-12T03:39:57.198 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.198 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.198 DEBUG:boto:Params: {} 2024-04-12T03:39:57.199 DEBUG:boto:Token: None 2024-04-12T03:39:57.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a180423550fb92d-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.242 DEBUG:boto:b'dgbpxg-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.243 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.243 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.243 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.243 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.243 DEBUG:boto:Method: GET 2024-04-12T03:39:57.243 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.243 DEBUG:boto:Data: 2024-04-12T03:39:57.243 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.243 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.244 DEBUG:boto:Params: {} 2024-04-12T03:39:57.244 DEBUG:boto:Token: None 2024-04-12T03:39:57.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f5d4e1a70038f6-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.248 DEBUG:boto:b'dgbpxg-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.248 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.248 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.248 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.248 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.248 DEBUG:boto:Data: 2024-04-12T03:39:57.248 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.248 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.248 DEBUG:boto:Params: {} 2024-04-12T03:39:57.249 DEBUG:boto:Token: None 2024-04-12T03:39:57.249 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ps/UkK4a6xGf5+w1bsO9CxhBuS4= 2024-04-12T03:39:57.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ps/UkK4a6xGf5+w1bsO9CxhBuS4=', 'Content-Length': '0'} 2024-04-12T03:39:57.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4570d640d3e24b-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.251 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.251 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.251 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.251 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.251 DEBUG:boto:Data: 2024-04-12T03:39:57.251 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.251 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.251 DEBUG:boto:Params: {} 2024-04-12T03:39:57.251 DEBUG:boto:Token: None 2024-04-12T03:39:57.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ps/UkK4a6xGf5+w1bsO9CxhBuS4= 2024-04-12T03:39:57.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ps/UkK4a6xGf5+w1bsO9CxhBuS4=', 'Content-Length': '0'} 2024-04-12T03:39:57.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd016cfa5d307ef8-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:39:57.253 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.253 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.253 DEBUG:boto:Method: GET 2024-04-12T03:39:57.254 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.254 DEBUG:boto:Data: 2024-04-12T03:39:57.254 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.254 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.254 DEBUG:boto:Params: {} 2024-04-12T03:39:57.254 DEBUG:boto:Token: None 2024-04-12T03:39:57.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bDCJS1vbRjz/uZKutbldYAAZq1M= 2024-04-12T03:39:57.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bDCJS1vbRjz/uZKutbldYAAZq1M=', 'Content-Length': '0'} 2024-04-12T03:39:57.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919635ce72f69a11-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.256 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.256 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.257 DEBUG:boto:Method: GET 2024-04-12T03:39:57.257 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.257 DEBUG:boto:Data: 2024-04-12T03:39:57.257 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.257 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.257 DEBUG:boto:Params: {} 2024-04-12T03:39:57.257 DEBUG:boto:Token: None 2024-04-12T03:39:57.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000040 2024-04-12T03:39:57.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bDCJS1vbRjz/uZKutbldYAAZq1M= 2024-04-12T03:39:57.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bDCJS1vbRjz/uZKutbldYAAZq1M=', 'Content-Length': '0'} 2024-04-12T03:39:57.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afdbad14bf1446ff-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.302 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.302 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.302 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.303 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.303 DEBUG:boto:Method: GET 2024-04-12T03:39:57.303 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.303 DEBUG:boto:Data: 2024-04-12T03:39:57.303 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.303 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.303 DEBUG:boto:Params: {} 2024-04-12T03:39:57.303 DEBUG:boto:Token: None 2024-04-12T03:39:57.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5736bd3c3c4f37-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.306 DEBUG:boto:b'dgbpxg-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.306 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.306 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.306 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.307 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.307 DEBUG:boto:Method: GET 2024-04-12T03:39:57.307 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000040 2024-04-12T03:39:57.307 DEBUG:boto:Data: 2024-04-12T03:39:57.307 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.307 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.307 DEBUG:boto:Params: {} 2024-04-12T03:39:57.307 DEBUG:boto:Token: None 2024-04-12T03:39:57.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4afa184baaa981-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.311 DEBUG:boto:b'dgbpxg-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:39:44.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:39:57.311 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.311 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.312 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.312 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.312 DEBUG:boto:Method: GET 2024-04-12T03:39:57.312 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.312 DEBUG:boto:Data: 2024-04-12T03:39:57.312 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.312 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.312 DEBUG:boto:Params: {} 2024-04-12T03:39:57.312 DEBUG:boto:Token: None 2024-04-12T03:39:57.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes= 2024-04-12T03:39:57.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes=', 'Content-Length': '0'} 2024-04-12T03:39:57.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86556ad08a95585-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.315 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.315 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.315 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.315 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.315 DEBUG:boto:Method: GET 2024-04-12T03:39:57.315 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.315 DEBUG:boto:Data: 2024-04-12T03:39:57.315 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.316 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.316 DEBUG:boto:Params: {} 2024-04-12T03:39:57.316 DEBUG:boto:Token: None 2024-04-12T03:39:57.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes= 2024-04-12T03:39:57.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes=', 'Content-Length': '0'} 2024-04-12T03:39:57.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cdab654c288a001-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.318 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.318 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.318 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.318 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.318 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.318 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.318 DEBUG:boto:Data: 2024-04-12T03:39:57.319 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.319 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.319 DEBUG:boto:Params: {} 2024-04-12T03:39:57.319 DEBUG:boto:Token: None 2024-04-12T03:39:57.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eKHJzQX1ekEAMgOQWf+M8VKRWKA= 2024-04-12T03:39:57.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eKHJzQX1ekEAMgOQWf+M8VKRWKA=', 'Content-Length': '0'} 2024-04-12T03:39:57.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197c7b47911194f5-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.321 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.321 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.321 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.321 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.321 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.321 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.321 DEBUG:boto:Data: 2024-04-12T03:39:57.321 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.321 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.322 DEBUG:boto:Params: {} 2024-04-12T03:39:57.322 DEBUG:boto:Token: None 2024-04-12T03:39:57.322 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eKHJzQX1ekEAMgOQWf+M8VKRWKA= 2024-04-12T03:39:57.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eKHJzQX1ekEAMgOQWf+M8VKRWKA=', 'Content-Length': '0'} 2024-04-12T03:39:57.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a9ae18d5263ae5-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:39:57.324 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.324 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.324 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.324 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.324 DEBUG:boto:Method: GET 2024-04-12T03:39:57.324 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.325 DEBUG:boto:Data: 2024-04-12T03:39:57.325 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.325 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.325 DEBUG:boto:Params: {} 2024-04-12T03:39:57.325 DEBUG:boto:Token: None 2024-04-12T03:39:57.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes= 2024-04-12T03:39:57.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes=', 'Content-Length': '0'} 2024-04-12T03:39:57.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000166c25eb52aad139-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.370 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.370 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.371 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.371 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.371 DEBUG:boto:Method: GET 2024-04-12T03:39:57.371 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.371 DEBUG:boto:Data: 2024-04-12T03:39:57.371 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.371 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.371 DEBUG:boto:Params: {} 2024-04-12T03:39:57.371 DEBUG:boto:Token: None 2024-04-12T03:39:57.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042?versionId=null 2024-04-12T03:39:57.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes= 2024-04-12T03:39:57.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WwK7EN4qzcbWqSBUFhf/dWtgWes=', 'Content-Length': '0'} 2024-04-12T03:39:57.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4d270d9f4a342c-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.373 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.374 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.374 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.374 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.374 DEBUG:boto:Method: GET 2024-04-12T03:39:57.374 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.374 DEBUG:boto:Data: 2024-04-12T03:39:57.374 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.374 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.374 DEBUG:boto:Params: {} 2024-04-12T03:39:57.374 DEBUG:boto:Token: None 2024-04-12T03:39:57.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d23347df00c1325-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.377 DEBUG:boto:b'dgbpxg-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.377 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.377 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.377 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.378 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.378 DEBUG:boto:Method: GET 2024-04-12T03:39:57.378 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.378 DEBUG:boto:Data: 2024-04-12T03:39:57.378 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.378 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.378 DEBUG:boto:Params: {} 2024-04-12T03:39:57.378 DEBUG:boto:Token: None 2024-04-12T03:39:57.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.378 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbecc440bae16fdd-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.382 DEBUG:boto:b'dgbpxg-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.382 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.382 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.382 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.383 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.383 DEBUG:boto:Data: 2024-04-12T03:39:57.383 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.383 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.383 DEBUG:boto:Params: {} 2024-04-12T03:39:57.383 DEBUG:boto:Token: None 2024-04-12T03:39:57.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P5Q1dg/P4XQjxjqoQ35z4clzRpg= 2024-04-12T03:39:57.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P5Q1dg/P4XQjxjqoQ35z4clzRpg=', 'Content-Length': '0'} 2024-04-12T03:39:57.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002523adde7e49c735-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.385 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.385 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.385 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.385 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.386 DEBUG:boto:Data: 2024-04-12T03:39:57.386 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.386 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.386 DEBUG:boto:Params: {} 2024-04-12T03:39:57.386 DEBUG:boto:Token: None 2024-04-12T03:39:57.386 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P5Q1dg/P4XQjxjqoQ35z4clzRpg= 2024-04-12T03:39:57.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P5Q1dg/P4XQjxjqoQ35z4clzRpg=', 'Content-Length': '0'} 2024-04-12T03:39:57.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a601f86b011f59-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:39:57.388 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.388 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.388 DEBUG:boto:Method: GET 2024-04-12T03:39:57.388 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.389 DEBUG:boto:Data: 2024-04-12T03:39:57.389 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.389 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.389 DEBUG:boto:Params: {} 2024-04-12T03:39:57.389 DEBUG:boto:Token: None 2024-04-12T03:39:57.389 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.389 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FZSmaanYrEfBRbi5yfCKpOmAIxA= 2024-04-12T03:39:57.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FZSmaanYrEfBRbi5yfCKpOmAIxA=', 'Content-Length': '0'} 2024-04-12T03:39:57.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b7df266e5742b37-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.393 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.393 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.394 DEBUG:boto:Method: GET 2024-04-12T03:39:57.394 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.394 DEBUG:boto:Data: 2024-04-12T03:39:57.394 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.394 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.394 DEBUG:boto:Params: {} 2024-04-12T03:39:57.394 DEBUG:boto:Token: None 2024-04-12T03:39:57.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000042 2024-04-12T03:39:57.394 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FZSmaanYrEfBRbi5yfCKpOmAIxA= 2024-04-12T03:39:57.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FZSmaanYrEfBRbi5yfCKpOmAIxA=', 'Content-Length': '0'} 2024-04-12T03:39:57.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fdd6cf284e0bdb-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.396 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.396 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.397 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.397 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.397 DEBUG:boto:Method: GET 2024-04-12T03:39:57.397 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.397 DEBUG:boto:Data: 2024-04-12T03:39:57.397 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.397 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.397 DEBUG:boto:Params: {} 2024-04-12T03:39:57.397 DEBUG:boto:Token: None 2024-04-12T03:39:57.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009829f43bf41db4f7-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.400 DEBUG:boto:b'dgbpxg-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.400 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.400 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.400 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.400 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.401 DEBUG:boto:Method: GET 2024-04-12T03:39:57.401 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000042 2024-04-12T03:39:57.401 DEBUG:boto:Data: 2024-04-12T03:39:57.401 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.401 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.401 DEBUG:boto:Params: {} 2024-04-12T03:39:57.401 DEBUG:boto:Token: None 2024-04-12T03:39:57.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fff6525d096010b-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.405 DEBUG:boto:b'dgbpxg-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:39:41.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:39:57.405 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.405 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.406 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.406 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.406 DEBUG:boto:Method: GET 2024-04-12T03:39:57.406 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.406 DEBUG:boto:Data: 2024-04-12T03:39:57.406 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.406 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.406 DEBUG:boto:Params: {} 2024-04-12T03:39:57.406 DEBUG:boto:Token: None 2024-04-12T03:39:57.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk= 2024-04-12T03:39:57.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk=', 'Content-Length': '0'} 2024-04-12T03:39:57.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1733633a00dbec6-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.409 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.409 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.409 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.409 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.409 DEBUG:boto:Method: GET 2024-04-12T03:39:57.410 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.410 DEBUG:boto:Data: 2024-04-12T03:39:57.410 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.410 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.410 DEBUG:boto:Params: {} 2024-04-12T03:39:57.410 DEBUG:boto:Token: None 2024-04-12T03:39:57.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk= 2024-04-12T03:39:57.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk=', 'Content-Length': '0'} 2024-04-12T03:39:57.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7957917ee48a6d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.454 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.454 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.455 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.455 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.455 DEBUG:boto:Data: 2024-04-12T03:39:57.455 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.455 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.455 DEBUG:boto:Params: {} 2024-04-12T03:39:57.455 DEBUG:boto:Token: None 2024-04-12T03:39:57.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zDfmySF1ss+Z4UCqs8J8b8dx4EE= 2024-04-12T03:39:57.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zDfmySF1ss+Z4UCqs8J8b8dx4EE=', 'Content-Length': '0'} 2024-04-12T03:39:57.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008535ef8a1b2d3012-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.457 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.458 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.458 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.458 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.458 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.458 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.458 DEBUG:boto:Data: 2024-04-12T03:39:57.458 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.458 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.458 DEBUG:boto:Params: {} 2024-04-12T03:39:57.459 DEBUG:boto:Token: None 2024-04-12T03:39:57.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zDfmySF1ss+Z4UCqs8J8b8dx4EE= 2024-04-12T03:39:57.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zDfmySF1ss+Z4UCqs8J8b8dx4EE=', 'Content-Length': '0'} 2024-04-12T03:39:57.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e35cccb635b0e9d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:39:57.461 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.461 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.461 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.461 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.461 DEBUG:boto:Method: GET 2024-04-12T03:39:57.461 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.461 DEBUG:boto:Data: 2024-04-12T03:39:57.461 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.462 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.462 DEBUG:boto:Params: {} 2024-04-12T03:39:57.462 DEBUG:boto:Token: None 2024-04-12T03:39:57.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk= 2024-04-12T03:39:57.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk=', 'Content-Length': '0'} 2024-04-12T03:39:57.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a06488e6cf2037d6-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.464 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.464 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.465 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.465 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.465 DEBUG:boto:Method: GET 2024-04-12T03:39:57.465 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.465 DEBUG:boto:Data: 2024-04-12T03:39:57.465 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.465 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.465 DEBUG:boto:Params: {} 2024-04-12T03:39:57.465 DEBUG:boto:Token: None 2024-04-12T03:39:57.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045?versionId=null 2024-04-12T03:39:57.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk= 2024-04-12T03:39:57.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ynTn5nIwWDLq0ctsuag7QE3SQk=', 'Content-Length': '0'} 2024-04-12T03:39:57.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0746e4df656f62-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.468 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.468 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.468 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.468 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.468 DEBUG:boto:Method: GET 2024-04-12T03:39:57.468 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.468 DEBUG:boto:Data: 2024-04-12T03:39:57.469 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.469 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.469 DEBUG:boto:Params: {} 2024-04-12T03:39:57.469 DEBUG:boto:Token: None 2024-04-12T03:39:57.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e0980a54bbb9dd-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.472 DEBUG:boto:b'dgbpxg-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.472 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.472 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.472 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.472 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.472 DEBUG:boto:Method: GET 2024-04-12T03:39:57.473 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.473 DEBUG:boto:Data: 2024-04-12T03:39:57.473 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.473 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.473 DEBUG:boto:Params: {} 2024-04-12T03:39:57.473 DEBUG:boto:Token: None 2024-04-12T03:39:57.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d634da78f8c54f-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.477 DEBUG:boto:b'dgbpxg-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.477 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.477 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.478 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.478 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.478 DEBUG:boto:Data: 2024-04-12T03:39:57.478 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.478 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.478 DEBUG:boto:Params: {} 2024-04-12T03:39:57.478 DEBUG:boto:Token: None 2024-04-12T03:39:57.478 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.478 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jZihzgGGGSdBX8EDO4rIW6Ff1t0= 2024-04-12T03:39:57.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jZihzgGGGSdBX8EDO4rIW6Ff1t0=', 'Content-Length': '0'} 2024-04-12T03:39:57.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf010729fd31ecb-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.480 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.480 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.480 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.481 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.481 DEBUG:boto:Data: 2024-04-12T03:39:57.481 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.481 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.481 DEBUG:boto:Params: {} 2024-04-12T03:39:57.481 DEBUG:boto:Token: None 2024-04-12T03:39:57.481 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jZihzgGGGSdBX8EDO4rIW6Ff1t0= 2024-04-12T03:39:57.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jZihzgGGGSdBX8EDO4rIW6Ff1t0=', 'Content-Length': '0'} 2024-04-12T03:39:57.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6dc18ef7c648c7c-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:39:57.483 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.483 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.483 DEBUG:boto:Method: GET 2024-04-12T03:39:57.484 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.484 DEBUG:boto:Data: 2024-04-12T03:39:57.484 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.484 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.484 DEBUG:boto:Params: {} 2024-04-12T03:39:57.484 DEBUG:boto:Token: None 2024-04-12T03:39:57.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dP5DugGL8hvJplm3SJ1+qi4iRkU= 2024-04-12T03:39:57.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dP5DugGL8hvJplm3SJ1+qi4iRkU=', 'Content-Length': '0'} 2024-04-12T03:39:57.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a022f13c2f7d6b-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.526 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.527 DEBUG:boto:Method: GET 2024-04-12T03:39:57.527 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.527 DEBUG:boto:Data: 2024-04-12T03:39:57.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.527 DEBUG:boto:Params: {} 2024-04-12T03:39:57.527 DEBUG:boto:Token: None 2024-04-12T03:39:57.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000045 2024-04-12T03:39:57.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dP5DugGL8hvJplm3SJ1+qi4iRkU= 2024-04-12T03:39:57.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dP5DugGL8hvJplm3SJ1+qi4iRkU=', 'Content-Length': '0'} 2024-04-12T03:39:57.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000296e8b370b3210b1-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.530 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.530 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.530 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.530 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.530 DEBUG:boto:Method: GET 2024-04-12T03:39:57.530 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.531 DEBUG:boto:Data: 2024-04-12T03:39:57.531 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.531 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.531 DEBUG:boto:Params: {} 2024-04-12T03:39:57.531 DEBUG:boto:Token: None 2024-04-12T03:39:57.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542ecdc848d67267-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.533 DEBUG:boto:b'dgbpxg-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.534 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.534 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.534 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.534 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.534 DEBUG:boto:Method: GET 2024-04-12T03:39:57.534 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000045 2024-04-12T03:39:57.534 DEBUG:boto:Data: 2024-04-12T03:39:57.534 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.535 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.535 DEBUG:boto:Params: {} 2024-04-12T03:39:57.535 DEBUG:boto:Token: None 2024-04-12T03:39:57.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003daadd174287b1d8-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.538 DEBUG:boto:b'dgbpxg-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:39:43.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:39:57.539 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.539 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.539 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.539 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.539 DEBUG:boto:Method: GET 2024-04-12T03:39:57.539 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.539 DEBUG:boto:Data: 2024-04-12T03:39:57.539 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.540 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.540 DEBUG:boto:Params: {} 2024-04-12T03:39:57.540 DEBUG:boto:Token: None 2024-04-12T03:39:57.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc= 2024-04-12T03:39:57.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc=', 'Content-Length': '0'} 2024-04-12T03:39:57.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127384b3724c8af6-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.542 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.542 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.542 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.542 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.543 DEBUG:boto:Method: GET 2024-04-12T03:39:57.543 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.543 DEBUG:boto:Data: 2024-04-12T03:39:57.543 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.543 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.543 DEBUG:boto:Params: {} 2024-04-12T03:39:57.543 DEBUG:boto:Token: None 2024-04-12T03:39:57.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.543 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc= 2024-04-12T03:39:57.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc=', 'Content-Length': '0'} 2024-04-12T03:39:57.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c4dca41dd30dba-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.586 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.587 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.587 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.587 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.587 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.587 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.587 DEBUG:boto:Data: 2024-04-12T03:39:57.587 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.587 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.587 DEBUG:boto:Params: {} 2024-04-12T03:39:57.587 DEBUG:boto:Token: None 2024-04-12T03:39:57.588 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wvBNN0aqC9IieRZxwpPuUNtwF8Q= 2024-04-12T03:39:57.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wvBNN0aqC9IieRZxwpPuUNtwF8Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb79092500d3031-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.589 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.590 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.590 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.590 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.590 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.590 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.590 DEBUG:boto:Data: 2024-04-12T03:39:57.590 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.590 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.590 DEBUG:boto:Params: {} 2024-04-12T03:39:57.590 DEBUG:boto:Token: None 2024-04-12T03:39:57.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wvBNN0aqC9IieRZxwpPuUNtwF8Q= 2024-04-12T03:39:57.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wvBNN0aqC9IieRZxwpPuUNtwF8Q=', 'Content-Length': '0'} 2024-04-12T03:39:57.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd77f95da94d054d-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:39:57.592 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.592 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.593 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.593 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.593 DEBUG:boto:Method: GET 2024-04-12T03:39:57.593 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.593 DEBUG:boto:Data: 2024-04-12T03:39:57.593 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.593 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.593 DEBUG:boto:Params: {} 2024-04-12T03:39:57.593 DEBUG:boto:Token: None 2024-04-12T03:39:57.593 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc= 2024-04-12T03:39:57.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc=', 'Content-Length': '0'} 2024-04-12T03:39:57.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffc15ab3df66f20-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.598 DEBUG:boto:Method: GET 2024-04-12T03:39:57.598 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.598 DEBUG:boto:Data: 2024-04-12T03:39:57.598 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.598 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.598 DEBUG:boto:Params: {} 2024-04-12T03:39:57.599 DEBUG:boto:Token: None 2024-04-12T03:39:57.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046?versionId=null 2024-04-12T03:39:57.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc= 2024-04-12T03:39:57.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cByc6pPRhKK0ey3eL+C64VWrXTc=', 'Content-Length': '0'} 2024-04-12T03:39:57.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008bf95e79117160a-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.601 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.601 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.601 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.601 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.601 DEBUG:boto:Method: GET 2024-04-12T03:39:57.601 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.601 DEBUG:boto:Data: 2024-04-12T03:39:57.601 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.601 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.602 DEBUG:boto:Params: {} 2024-04-12T03:39:57.602 DEBUG:boto:Token: None 2024-04-12T03:39:57.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002289c327fab08849-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.604 DEBUG:boto:b'dgbpxg-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.605 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.605 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.605 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.605 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.605 DEBUG:boto:Method: GET 2024-04-12T03:39:57.605 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.605 DEBUG:boto:Data: 2024-04-12T03:39:57.605 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.605 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.605 DEBUG:boto:Params: {} 2024-04-12T03:39:57.606 DEBUG:boto:Token: None 2024-04-12T03:39:57.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe7efc5c8bdf333-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.610 DEBUG:boto:b'dgbpxg-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.610 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.610 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.610 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.610 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.610 DEBUG:boto:Data: 2024-04-12T03:39:57.610 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.611 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.611 DEBUG:boto:Params: {} 2024-04-12T03:39:57.611 DEBUG:boto:Token: None 2024-04-12T03:39:57.611 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/DFosQpfjELeHxPJtXeW5Mt8EGQ= 2024-04-12T03:39:57.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/DFosQpfjELeHxPJtXeW5Mt8EGQ=', 'Content-Length': '0'} 2024-04-12T03:39:57.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019412b07a5dbe3de-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.613 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.613 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.613 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.613 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.613 DEBUG:boto:Data: 2024-04-12T03:39:57.613 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.614 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.614 DEBUG:boto:Params: {} 2024-04-12T03:39:57.614 DEBUG:boto:Token: None 2024-04-12T03:39:57.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/DFosQpfjELeHxPJtXeW5Mt8EGQ= 2024-04-12T03:39:57.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/DFosQpfjELeHxPJtXeW5Mt8EGQ=', 'Content-Length': '0'} 2024-04-12T03:39:57.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea15310d023d9700-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:39:57.616 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.616 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.616 DEBUG:boto:Method: GET 2024-04-12T03:39:57.616 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.616 DEBUG:boto:Data: 2024-04-12T03:39:57.617 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.617 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.617 DEBUG:boto:Params: {} 2024-04-12T03:39:57.617 DEBUG:boto:Token: None 2024-04-12T03:39:57.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:396j31OSOwC+6hiP/u/Y8hx8iic= 2024-04-12T03:39:57.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:396j31OSOwC+6hiP/u/Y8hx8iic=', 'Content-Length': '0'} 2024-04-12T03:39:57.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d767a5194d75334f-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.619 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.619 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.619 DEBUG:boto:Method: GET 2024-04-12T03:39:57.619 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.619 DEBUG:boto:Data: 2024-04-12T03:39:57.619 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.620 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.620 DEBUG:boto:Params: {} 2024-04-12T03:39:57.620 DEBUG:boto:Token: None 2024-04-12T03:39:57.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000046 2024-04-12T03:39:57.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:396j31OSOwC+6hiP/u/Y8hx8iic= 2024-04-12T03:39:57.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:396j31OSOwC+6hiP/u/Y8hx8iic=', 'Content-Length': '0'} 2024-04-12T03:39:57.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a65e947280baef-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.622 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.622 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.622 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.622 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.623 DEBUG:boto:Method: GET 2024-04-12T03:39:57.623 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.623 DEBUG:boto:Data: 2024-04-12T03:39:57.623 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.623 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.623 DEBUG:boto:Params: {} 2024-04-12T03:39:57.623 DEBUG:boto:Token: None 2024-04-12T03:39:57.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0eef32d4bbb201-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.666 DEBUG:boto:b'dgbpxg-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.667 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.667 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.667 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.667 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.667 DEBUG:boto:Method: GET 2024-04-12T03:39:57.667 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000046 2024-04-12T03:39:57.667 DEBUG:boto:Data: 2024-04-12T03:39:57.667 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.667 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.667 DEBUG:boto:Params: {} 2024-04-12T03:39:57.668 DEBUG:boto:Token: None 2024-04-12T03:39:57.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7025fe05deb483-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.671 DEBUG:boto:b'dgbpxg-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:39:40.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:39:57.672 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.672 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.672 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.672 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.672 DEBUG:boto:Method: GET 2024-04-12T03:39:57.672 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.672 DEBUG:boto:Data: 2024-04-12T03:39:57.672 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.672 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.673 DEBUG:boto:Params: {} 2024-04-12T03:39:57.673 DEBUG:boto:Token: None 2024-04-12T03:39:57.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.673 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI= 2024-04-12T03:39:57.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI=', 'Content-Length': '0'} 2024-04-12T03:39:57.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3149908ae4f441-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.675 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.675 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.675 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.675 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.675 DEBUG:boto:Method: GET 2024-04-12T03:39:57.675 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.676 DEBUG:boto:Data: 2024-04-12T03:39:57.676 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.676 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.676 DEBUG:boto:Params: {} 2024-04-12T03:39:57.676 DEBUG:boto:Token: None 2024-04-12T03:39:57.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI= 2024-04-12T03:39:57.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI=', 'Content-Length': '0'} 2024-04-12T03:39:57.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ffd47323aa68318-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.679 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.679 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.679 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.679 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.679 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.679 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.679 DEBUG:boto:Data: 2024-04-12T03:39:57.679 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.679 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.680 DEBUG:boto:Params: {} 2024-04-12T03:39:57.680 DEBUG:boto:Token: None 2024-04-12T03:39:57.680 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l1z4tDhJadAaEDaZu/rgCT+hWiE= 2024-04-12T03:39:57.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l1z4tDhJadAaEDaZu/rgCT+hWiE=', 'Content-Length': '0'} 2024-04-12T03:39:57.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d0bcefb90563f4b-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.682 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.682 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.682 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.682 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.682 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.682 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.682 DEBUG:boto:Data: 2024-04-12T03:39:57.683 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.683 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.683 DEBUG:boto:Params: {} 2024-04-12T03:39:57.683 DEBUG:boto:Token: None 2024-04-12T03:39:57.683 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.683 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l1z4tDhJadAaEDaZu/rgCT+hWiE= 2024-04-12T03:39:57.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l1z4tDhJadAaEDaZu/rgCT+hWiE=', 'Content-Length': '0'} 2024-04-12T03:39:57.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074622679de114ce3-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:39:57.685 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.685 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.685 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.685 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.685 DEBUG:boto:Method: GET 2024-04-12T03:39:57.685 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.685 DEBUG:boto:Data: 2024-04-12T03:39:57.686 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.686 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.686 DEBUG:boto:Params: {} 2024-04-12T03:39:57.686 DEBUG:boto:Token: None 2024-04-12T03:39:57.686 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.686 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI= 2024-04-12T03:39:57.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI=', 'Content-Length': '0'} 2024-04-12T03:39:57.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e1815b6063a055-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.688 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.688 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.688 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.688 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.688 DEBUG:boto:Method: GET 2024-04-12T03:39:57.689 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.689 DEBUG:boto:Data: 2024-04-12T03:39:57.689 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.689 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.689 DEBUG:boto:Params: {} 2024-04-12T03:39:57.689 DEBUG:boto:Token: None 2024-04-12T03:39:57.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048?versionId=null 2024-04-12T03:39:57.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI= 2024-04-12T03:39:57.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gdn7yvUXz9UiwyUNIGT+p/P/LmI=', 'Content-Length': '0'} 2024-04-12T03:39:57.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027099aa932ebdb30-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.734 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.734 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.734 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.735 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.735 DEBUG:boto:Method: GET 2024-04-12T03:39:57.735 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.735 DEBUG:boto:Data: 2024-04-12T03:39:57.735 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.735 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.735 DEBUG:boto:Params: {} 2024-04-12T03:39:57.735 DEBUG:boto:Token: None 2024-04-12T03:39:57.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738d3b482c2ab994-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.737 DEBUG:boto:b'dgbpxg-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.738 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.738 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.738 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.738 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.738 DEBUG:boto:Method: GET 2024-04-12T03:39:57.738 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.738 DEBUG:boto:Data: 2024-04-12T03:39:57.738 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.739 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.739 DEBUG:boto:Params: {} 2024-04-12T03:39:57.739 DEBUG:boto:Token: None 2024-04-12T03:39:57.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244a97cba93dabd8-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.743 DEBUG:boto:b'dgbpxg-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.743 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.743 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.743 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.743 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.744 DEBUG:boto:Data: 2024-04-12T03:39:57.744 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.744 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.744 DEBUG:boto:Params: {} 2024-04-12T03:39:57.744 DEBUG:boto:Token: None 2024-04-12T03:39:57.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j4r9RF9q9jIuXmp+M0Ti0OPvqe4= 2024-04-12T03:39:57.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j4r9RF9q9jIuXmp+M0Ti0OPvqe4=', 'Content-Length': '0'} 2024-04-12T03:39:57.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be28bd0d50875ea5-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.746 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.746 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.746 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.746 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.746 DEBUG:boto:Data: 2024-04-12T03:39:57.746 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.746 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.747 DEBUG:boto:Params: {} 2024-04-12T03:39:57.747 DEBUG:boto:Token: None 2024-04-12T03:39:57.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j4r9RF9q9jIuXmp+M0Ti0OPvqe4= 2024-04-12T03:39:57.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j4r9RF9q9jIuXmp+M0Ti0OPvqe4=', 'Content-Length': '0'} 2024-04-12T03:39:57.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004650d3d52fc4894a-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:39:57.749 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.749 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.749 DEBUG:boto:Method: GET 2024-04-12T03:39:57.749 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.749 DEBUG:boto:Data: 2024-04-12T03:39:57.749 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.749 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.749 DEBUG:boto:Params: {} 2024-04-12T03:39:57.750 DEBUG:boto:Token: None 2024-04-12T03:39:57.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.750 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BEDYZ/NHhknMb5aAHLh/7GFum64= 2024-04-12T03:39:57.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BEDYZ/NHhknMb5aAHLh/7GFum64=', 'Content-Length': '0'} 2024-04-12T03:39:57.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e4e67c041e7402-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.751 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.752 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.752 DEBUG:boto:Method: GET 2024-04-12T03:39:57.752 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.752 DEBUG:boto:Data: 2024-04-12T03:39:57.752 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.752 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.752 DEBUG:boto:Params: {} 2024-04-12T03:39:57.752 DEBUG:boto:Token: None 2024-04-12T03:39:57.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000048 2024-04-12T03:39:57.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BEDYZ/NHhknMb5aAHLh/7GFum64= 2024-04-12T03:39:57.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BEDYZ/NHhknMb5aAHLh/7GFum64=', 'Content-Length': '0'} 2024-04-12T03:39:57.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f174d31e960ed2-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.754 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.754 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.755 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.755 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.755 DEBUG:boto:Method: GET 2024-04-12T03:39:57.755 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.755 DEBUG:boto:Data: 2024-04-12T03:39:57.755 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.755 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.755 DEBUG:boto:Params: {} 2024-04-12T03:39:57.755 DEBUG:boto:Token: None 2024-04-12T03:39:57.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d5c85e975604d9-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.758 DEBUG:boto:b'dgbpxg-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.758 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.758 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.758 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.758 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.758 DEBUG:boto:Method: GET 2024-04-12T03:39:57.758 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000048 2024-04-12T03:39:57.758 DEBUG:boto:Data: 2024-04-12T03:39:57.758 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.759 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.759 DEBUG:boto:Params: {} 2024-04-12T03:39:57.759 DEBUG:boto:Token: None 2024-04-12T03:39:57.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806f5b9b1c131b16-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.806 DEBUG:boto:b'dgbpxg-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:39:43.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-12T03:39:57.807 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.807 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.807 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.807 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.807 DEBUG:boto:Method: GET 2024-04-12T03:39:57.807 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.807 DEBUG:boto:Data: 2024-04-12T03:39:57.807 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.808 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.808 DEBUG:boto:Params: {} 2024-04-12T03:39:57.808 DEBUG:boto:Token: None 2024-04-12T03:39:57.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78= 2024-04-12T03:39:57.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78=', 'Content-Length': '0'} 2024-04-12T03:39:57.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed132a95d658d6ac-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.811 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.811 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.811 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.811 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.811 DEBUG:boto:Method: GET 2024-04-12T03:39:57.811 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.811 DEBUG:boto:Data: 2024-04-12T03:39:57.811 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.811 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.811 DEBUG:boto:Params: {} 2024-04-12T03:39:57.812 DEBUG:boto:Token: None 2024-04-12T03:39:57.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78= 2024-04-12T03:39:57.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78=', 'Content-Length': '0'} 2024-04-12T03:39:57.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012516db676ae6eb1-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.813 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.814 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.814 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.814 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.814 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.814 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.814 DEBUG:boto:Data: 2024-04-12T03:39:57.814 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.814 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.814 DEBUG:boto:Params: {} 2024-04-12T03:39:57.815 DEBUG:boto:Token: None 2024-04-12T03:39:57.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IQos98DbtoOvI/SoQbwqaoWgSTk= 2024-04-12T03:39:57.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IQos98DbtoOvI/SoQbwqaoWgSTk=', 'Content-Length': '0'} 2024-04-12T03:39:57.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058376407583421c0-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.816 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.817 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.817 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.817 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.817 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.817 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.817 DEBUG:boto:Data: 2024-04-12T03:39:57.817 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.817 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.817 DEBUG:boto:Params: {} 2024-04-12T03:39:57.817 DEBUG:boto:Token: None 2024-04-12T03:39:57.818 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IQos98DbtoOvI/SoQbwqaoWgSTk= 2024-04-12T03:39:57.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IQos98DbtoOvI/SoQbwqaoWgSTk=', 'Content-Length': '0'} 2024-04-12T03:39:57.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4440d444e795dee-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-12T03:39:57.820 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.820 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.820 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.820 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.820 DEBUG:boto:Method: GET 2024-04-12T03:39:57.820 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.820 DEBUG:boto:Data: 2024-04-12T03:39:57.820 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.820 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.820 DEBUG:boto:Params: {} 2024-04-12T03:39:57.821 DEBUG:boto:Token: None 2024-04-12T03:39:57.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78= 2024-04-12T03:39:57.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78=', 'Content-Length': '0'} 2024-04-12T03:39:57.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ae0137b6edc73e-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.866 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.866 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.867 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.867 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.867 DEBUG:boto:Method: GET 2024-04-12T03:39:57.867 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.867 DEBUG:boto:Data: 2024-04-12T03:39:57.867 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.867 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.867 DEBUG:boto:Params: {} 2024-04-12T03:39:57.867 DEBUG:boto:Token: None 2024-04-12T03:39:57.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049?versionId=null 2024-04-12T03:39:57.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78= 2024-04-12T03:39:57.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JtffQqlrORf55DoojFAk2ZLrH78=', 'Content-Length': '0'} 2024-04-12T03:39:57.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d02f912c851cc6f-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.869 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.870 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.870 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.870 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.870 DEBUG:boto:Method: GET 2024-04-12T03:39:57.870 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.870 DEBUG:boto:Data: 2024-04-12T03:39:57.870 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.870 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.870 DEBUG:boto:Params: {} 2024-04-12T03:39:57.870 DEBUG:boto:Token: None 2024-04-12T03:39:57.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375f104fa6d46338-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.873 DEBUG:boto:b'dgbpxg-60obj000000000000000000491000falseobj000000000000000000492024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.873 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.874 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.874 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.874 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.874 DEBUG:boto:Method: GET 2024-04-12T03:39:57.874 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.874 DEBUG:boto:Data: 2024-04-12T03:39:57.874 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.874 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.874 DEBUG:boto:Params: {} 2024-04-12T03:39:57.875 DEBUG:boto:Token: None 2024-04-12T03:39:57.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0b5ab1c4b16724f-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.878 DEBUG:boto:b'dgbpxg-60obj000000000000000000491000falseobj000000000000000000492024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.878 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.879 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.879 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.879 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.879 DEBUG:boto:Data: 2024-04-12T03:39:57.879 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.879 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.879 DEBUG:boto:Params: {} 2024-04-12T03:39:57.879 DEBUG:boto:Token: None 2024-04-12T03:39:57.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xPbuypOaPiPZ3i4hyjg/2BOnGG4= 2024-04-12T03:39:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xPbuypOaPiPZ3i4hyjg/2BOnGG4=', 'Content-Length': '0'} 2024-04-12T03:39:57.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fd079aca0a9930-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.881 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.881 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.882 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.882 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.882 DEBUG:boto:Data: 2024-04-12T03:39:57.882 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.882 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.882 DEBUG:boto:Params: {} 2024-04-12T03:39:57.882 DEBUG:boto:Token: None 2024-04-12T03:39:57.882 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xPbuypOaPiPZ3i4hyjg/2BOnGG4= 2024-04-12T03:39:57.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xPbuypOaPiPZ3i4hyjg/2BOnGG4=', 'Content-Length': '0'} 2024-04-12T03:39:57.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60c17b44fa40631-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-12T03:39:57.884 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.884 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.885 DEBUG:boto:Method: GET 2024-04-12T03:39:57.885 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.885 DEBUG:boto:Data: 2024-04-12T03:39:57.885 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.885 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.885 DEBUG:boto:Params: {} 2024-04-12T03:39:57.885 DEBUG:boto:Token: None 2024-04-12T03:39:57.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.886 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPM39Q+leq2HN8CgftWvqbjvJP4= 2024-04-12T03:39:57.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPM39Q+leq2HN8CgftWvqbjvJP4=', 'Content-Length': '0'} 2024-04-12T03:39:57.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3c2d042072239d-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.887 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.887 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.888 DEBUG:boto:Method: GET 2024-04-12T03:39:57.888 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.888 DEBUG:boto:Data: 2024-04-12T03:39:57.888 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.888 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.888 DEBUG:boto:Params: {} 2024-04-12T03:39:57.888 DEBUG:boto:Token: None 2024-04-12T03:39:57.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000049 2024-04-12T03:39:57.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPM39Q+leq2HN8CgftWvqbjvJP4= 2024-04-12T03:39:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPM39Q+leq2HN8CgftWvqbjvJP4=', 'Content-Length': '0'} 2024-04-12T03:39:57.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb64b5bd34205547-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.890 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.890 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.890 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.890 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.891 DEBUG:boto:Method: GET 2024-04-12T03:39:57.891 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.891 DEBUG:boto:Data: 2024-04-12T03:39:57.891 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.891 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.891 DEBUG:boto:Params: {} 2024-04-12T03:39:57.891 DEBUG:boto:Token: None 2024-04-12T03:39:57.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc663c4dc66d2d94-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.894 DEBUG:boto:b'dgbpxg-60obj000000000000000000491000falseobj000000000000000000492024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.894 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.894 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.894 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.894 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.894 DEBUG:boto:Method: GET 2024-04-12T03:39:57.894 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000049 2024-04-12T03:39:57.894 DEBUG:boto:Data: 2024-04-12T03:39:57.895 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.895 DEBUG:boto:Params: {} 2024-04-12T03:39:57.895 DEBUG:boto:Token: None 2024-04-12T03:39:57.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2e7112494a39f0-006618ad0d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.898 DEBUG:boto:b'dgbpxg-60obj000000000000000000491000falseobj000000000000000000492024-04-12T03:39:41.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:57.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:39:57.899 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.899 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.899 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.899 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.899 DEBUG:boto:Method: GET 2024-04-12T03:39:57.899 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.899 DEBUG:boto:Data: 2024-04-12T03:39:57.899 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.900 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.900 DEBUG:boto:Params: {} 2024-04-12T03:39:57.900 DEBUG:boto:Token: None 2024-04-12T03:39:57.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4= 2024-04-12T03:39:57.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4=', 'Content-Length': '0'} 2024-04-12T03:39:57.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aa453612a2b099a-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.902 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.902 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.902 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.902 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.902 DEBUG:boto:Method: GET 2024-04-12T03:39:57.903 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.903 DEBUG:boto:Data: 2024-04-12T03:39:57.903 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.903 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.903 DEBUG:boto:Params: {} 2024-04-12T03:39:57.903 DEBUG:boto:Token: None 2024-04-12T03:39:57.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4= 2024-04-12T03:39:57.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4=', 'Content-Length': '0'} 2024-04-12T03:39:57.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f2aaeee39e1153-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.946 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.946 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.947 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.947 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.947 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.947 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.947 DEBUG:boto:Data: 2024-04-12T03:39:57.947 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.947 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.947 DEBUG:boto:Params: {} 2024-04-12T03:39:57.947 DEBUG:boto:Token: None 2024-04-12T03:39:57.947 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n0jwauRdf4qtGlyoGXTWvvDcjpI= 2024-04-12T03:39:57.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n0jwauRdf4qtGlyoGXTWvvDcjpI=', 'Content-Length': '0'} 2024-04-12T03:39:57.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089321c0accd7d2cd-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.949 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.950 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.950 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.950 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.950 DEBUG:boto:Method: HEAD 2024-04-12T03:39:57.950 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.950 DEBUG:boto:Data: 2024-04-12T03:39:57.950 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.950 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.950 DEBUG:boto:Params: {} 2024-04-12T03:39:57.950 DEBUG:boto:Token: None 2024-04-12T03:39:57.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n0jwauRdf4qtGlyoGXTWvvDcjpI= 2024-04-12T03:39:57.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n0jwauRdf4qtGlyoGXTWvvDcjpI=', 'Content-Length': '0'} 2024-04-12T03:39:57.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57afd2f251fa80c-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:39:57.952 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.953 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.953 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.953 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.953 DEBUG:boto:Method: GET 2024-04-12T03:39:57.953 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.953 DEBUG:boto:Data: 2024-04-12T03:39:57.953 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.953 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.953 DEBUG:boto:Params: {} 2024-04-12T03:39:57.953 DEBUG:boto:Token: None 2024-04-12T03:39:57.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.954 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4= 2024-04-12T03:39:57.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4=', 'Content-Length': '0'} 2024-04-12T03:39:57.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085caaf46e44a12a7-006618ad0d-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:57.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.956 DEBUG:boto:Method: GET 2024-04-12T03:39:57.956 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.956 DEBUG:boto:Data: 2024-04-12T03:39:57.956 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:57.956 DEBUG:boto:Port: 8000 2024-04-12T03:39:57.956 DEBUG:boto:Params: {} 2024-04-12T03:39:57.956 DEBUG:boto:Token: None 2024-04-12T03:39:57.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/obj00000000000000000050?versionId=null 2024-04-12T03:39:57.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4= 2024-04-12T03:39:57.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LwSW1Lyn0qUm6OkQyRWQXfYTSY4=', 'Content-Length': '0'} 2024-04-12T03:39:57.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca969e9c22b70d62-006618ad0d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:57.958 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:57.958 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:57.958 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:57.959 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:57.959 DEBUG:boto:Method: GET 2024-04-12T03:39:57.959 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:57.959 DEBUG:boto:Data: 2024-04-12T03:39:57.959 DEBUG:boto:Headers: {} 2024-04-12T03:39:57.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:57.959 DEBUG:boto:Port: 8001 2024-04-12T03:39:57.959 DEBUG:boto:Params: {} 2024-04-12T03:39:57.959 DEBUG:boto:Token: None 2024-04-12T03:39:57.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:57 GMT /dgbpxg-60/ 2024-04-12T03:39:57.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY= 2024-04-12T03:39:57.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:57 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vUkEyRcZQ0xzsWPvB9Tx7l5KnMY=', 'Content-Length': '0'} 2024-04-12T03:39:57.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085392a3fc327090c-006618ad0d-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.002 DEBUG:boto:b'dgbpxg-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.003 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.003 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.003 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.003 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.003 DEBUG:boto:Method: GET 2024-04-12T03:39:58.003 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.003 DEBUG:boto:Data: 2024-04-12T03:39:58.003 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.003 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.003 DEBUG:boto:Params: {} 2024-04-12T03:39:58.004 DEBUG:boto:Token: None 2024-04-12T03:39:58.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447809deb7c102a4-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.007 DEBUG:boto:b'dgbpxg-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.007 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.008 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.008 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.008 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.008 DEBUG:boto:Data: 2024-04-12T03:39:58.008 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.008 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.008 DEBUG:boto:Params: {} 2024-04-12T03:39:58.008 DEBUG:boto:Token: None 2024-04-12T03:39:58.008 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ljJ9+yFyxCJVmT3ZTC2rkLB2LEQ= 2024-04-12T03:39:58.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ljJ9+yFyxCJVmT3ZTC2rkLB2LEQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d869d77cb927e7-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.010 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.010 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.011 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.011 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.011 DEBUG:boto:Data: 2024-04-12T03:39:58.011 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.011 DEBUG:boto:Params: {} 2024-04-12T03:39:58.011 DEBUG:boto:Token: None 2024-04-12T03:39:58.011 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ljJ9+yFyxCJVmT3ZTC2rkLB2LEQ= 2024-04-12T03:39:58.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ljJ9+yFyxCJVmT3ZTC2rkLB2LEQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5416c297f77dcc-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:39:58.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.014 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.014 DEBUG:boto:Method: GET 2024-04-12T03:39:58.014 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.014 DEBUG:boto:Data: 2024-04-12T03:39:58.014 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.014 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.014 DEBUG:boto:Params: {} 2024-04-12T03:39:58.014 DEBUG:boto:Token: None 2024-04-12T03:39:58.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CU34J4nFM8Y+I3wB1lS7L2r+W+Y= 2024-04-12T03:39:58.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CU34J4nFM8Y+I3wB1lS7L2r+W+Y=', 'Content-Length': '0'} 2024-04-12T03:39:58.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91501f640167fca-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.016 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.016 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.016 DEBUG:boto:Method: GET 2024-04-12T03:39:58.017 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.017 DEBUG:boto:Data: 2024-04-12T03:39:58.017 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.017 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.017 DEBUG:boto:Params: {} 2024-04-12T03:39:58.017 DEBUG:boto:Token: None 2024-04-12T03:39:58.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000050 2024-04-12T03:39:58.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CU34J4nFM8Y+I3wB1lS7L2r+W+Y= 2024-04-12T03:39:58.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CU34J4nFM8Y+I3wB1lS7L2r+W+Y=', 'Content-Length': '0'} 2024-04-12T03:39:58.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0bd98c75c234be-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.020 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.020 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.020 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.020 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.020 DEBUG:boto:Method: GET 2024-04-12T03:39:58.020 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.020 DEBUG:boto:Data: 2024-04-12T03:39:58.020 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.020 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.020 DEBUG:boto:Params: {} 2024-04-12T03:39:58.021 DEBUG:boto:Token: None 2024-04-12T03:39:58.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5433085a7687e7-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.023 DEBUG:boto:b'dgbpxg-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.023 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.023 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.024 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.024 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.024 DEBUG:boto:Method: GET 2024-04-12T03:39:58.024 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000050 2024-04-12T03:39:58.024 DEBUG:boto:Data: 2024-04-12T03:39:58.024 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.024 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.024 DEBUG:boto:Params: {} 2024-04-12T03:39:58.024 DEBUG:boto:Token: None 2024-04-12T03:39:58.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb7d9b9019158d4-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.074 DEBUG:boto:b'dgbpxg-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:39:42.889Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:39:58.075 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.075 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.075 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.075 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.075 DEBUG:boto:Method: GET 2024-04-12T03:39:58.075 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.075 DEBUG:boto:Data: 2024-04-12T03:39:58.075 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.075 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.075 DEBUG:boto:Params: {} 2024-04-12T03:39:58.076 DEBUG:boto:Token: None 2024-04-12T03:39:58.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws= 2024-04-12T03:39:58.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws=', 'Content-Length': '0'} 2024-04-12T03:39:58.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b42fd966937dd3-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.078 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.078 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.078 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.078 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.078 DEBUG:boto:Method: GET 2024-04-12T03:39:58.079 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.079 DEBUG:boto:Data: 2024-04-12T03:39:58.079 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.079 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.079 DEBUG:boto:Params: {} 2024-04-12T03:39:58.079 DEBUG:boto:Token: None 2024-04-12T03:39:58.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws= 2024-04-12T03:39:58.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws=', 'Content-Length': '0'} 2024-04-12T03:39:58.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a800acc7689ec1af-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.081 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.081 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.082 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.082 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.082 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.082 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.082 DEBUG:boto:Data: 2024-04-12T03:39:58.082 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.082 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.082 DEBUG:boto:Params: {} 2024-04-12T03:39:58.082 DEBUG:boto:Token: None 2024-04-12T03:39:58.082 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:efPu5qTFlrEkF34oe1VWX8+IXfg= 2024-04-12T03:39:58.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:efPu5qTFlrEkF34oe1VWX8+IXfg=', 'Content-Length': '0'} 2024-04-12T03:39:58.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5233d7d64cdee3-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.084 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.084 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.085 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.085 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.085 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.085 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.085 DEBUG:boto:Data: 2024-04-12T03:39:58.085 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.085 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.085 DEBUG:boto:Params: {} 2024-04-12T03:39:58.085 DEBUG:boto:Token: None 2024-04-12T03:39:58.085 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:efPu5qTFlrEkF34oe1VWX8+IXfg= 2024-04-12T03:39:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:efPu5qTFlrEkF34oe1VWX8+IXfg=', 'Content-Length': '0'} 2024-04-12T03:39:58.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fe63df94a5ebea-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:39:58.087 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.087 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.088 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.088 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.088 DEBUG:boto:Method: GET 2024-04-12T03:39:58.088 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.088 DEBUG:boto:Data: 2024-04-12T03:39:58.088 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.088 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.088 DEBUG:boto:Params: {} 2024-04-12T03:39:58.088 DEBUG:boto:Token: None 2024-04-12T03:39:58.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws= 2024-04-12T03:39:58.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws=', 'Content-Length': '0'} 2024-04-12T03:39:58.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b6953b19247cca3-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.090 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.090 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.090 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.090 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.091 DEBUG:boto:Method: GET 2024-04-12T03:39:58.091 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.091 DEBUG:boto:Data: 2024-04-12T03:39:58.091 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.091 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.091 DEBUG:boto:Params: {} 2024-04-12T03:39:58.091 DEBUG:boto:Token: None 2024-04-12T03:39:58.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053?versionId=null 2024-04-12T03:39:58.091 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws= 2024-04-12T03:39:58.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Jwy1hD1VAx5ZVcallhfTwyZ3Ws=', 'Content-Length': '0'} 2024-04-12T03:39:58.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20fae789ed36fe0-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.093 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.093 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.093 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.093 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.093 DEBUG:boto:Method: GET 2024-04-12T03:39:58.094 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.094 DEBUG:boto:Data: 2024-04-12T03:39:58.094 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.094 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.094 DEBUG:boto:Params: {} 2024-04-12T03:39:58.094 DEBUG:boto:Token: None 2024-04-12T03:39:58.094 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f412f60bc1670ff6-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.097 DEBUG:boto:b'dgbpxg-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.097 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.097 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.097 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.097 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.097 DEBUG:boto:Method: GET 2024-04-12T03:39:58.097 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.097 DEBUG:boto:Data: 2024-04-12T03:39:58.098 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.098 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.098 DEBUG:boto:Params: {} 2024-04-12T03:39:58.098 DEBUG:boto:Token: None 2024-04-12T03:39:58.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a07f8710a6fd2e16-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.102 DEBUG:boto:b'dgbpxg-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.102 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.102 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.102 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.103 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.103 DEBUG:boto:Data: 2024-04-12T03:39:58.103 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.103 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.103 DEBUG:boto:Params: {} 2024-04-12T03:39:58.103 DEBUG:boto:Token: None 2024-04-12T03:39:58.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A4Fh4jSe1GfhLnXdQUp2QlHJsvE= 2024-04-12T03:39:58.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A4Fh4jSe1GfhLnXdQUp2QlHJsvE=', 'Content-Length': '0'} 2024-04-12T03:39:58.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9918977bda04a5-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.105 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.105 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.105 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.105 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.105 DEBUG:boto:Data: 2024-04-12T03:39:58.105 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.106 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.106 DEBUG:boto:Params: {} 2024-04-12T03:39:58.106 DEBUG:boto:Token: None 2024-04-12T03:39:58.106 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A4Fh4jSe1GfhLnXdQUp2QlHJsvE= 2024-04-12T03:39:58.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A4Fh4jSe1GfhLnXdQUp2QlHJsvE=', 'Content-Length': '0'} 2024-04-12T03:39:58.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6ebd7ac2fbd717-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:39:58.108 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.109 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.109 DEBUG:boto:Method: GET 2024-04-12T03:39:58.109 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.109 DEBUG:boto:Data: 2024-04-12T03:39:58.109 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.109 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.109 DEBUG:boto:Params: {} 2024-04-12T03:39:58.109 DEBUG:boto:Token: None 2024-04-12T03:39:58.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.109 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Iv73JBjFCcHpPIE/QGUn01w/MI= 2024-04-12T03:39:58.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Iv73JBjFCcHpPIE/QGUn01w/MI=', 'Content-Length': '0'} 2024-04-12T03:39:58.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078102814f9301877-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.111 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.111 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.111 DEBUG:boto:Method: GET 2024-04-12T03:39:58.112 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.112 DEBUG:boto:Data: 2024-04-12T03:39:58.112 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.112 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.112 DEBUG:boto:Params: {} 2024-04-12T03:39:58.112 DEBUG:boto:Token: None 2024-04-12T03:39:58.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000053 2024-04-12T03:39:58.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Iv73JBjFCcHpPIE/QGUn01w/MI= 2024-04-12T03:39:58.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Iv73JBjFCcHpPIE/QGUn01w/MI=', 'Content-Length': '0'} 2024-04-12T03:39:58.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86c4c69da6b7110-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.114 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.114 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.114 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.114 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.114 DEBUG:boto:Method: GET 2024-04-12T03:39:58.115 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.115 DEBUG:boto:Data: 2024-04-12T03:39:58.115 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.115 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.115 DEBUG:boto:Params: {} 2024-04-12T03:39:58.115 DEBUG:boto:Token: None 2024-04-12T03:39:58.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfc7de6d56c68d0-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.118 DEBUG:boto:b'dgbpxg-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.118 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.118 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.118 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.118 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.118 DEBUG:boto:Method: GET 2024-04-12T03:39:58.118 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000053 2024-04-12T03:39:58.118 DEBUG:boto:Data: 2024-04-12T03:39:58.118 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.119 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.119 DEBUG:boto:Params: {} 2024-04-12T03:39:58.119 DEBUG:boto:Token: None 2024-04-12T03:39:58.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ea9cd22c13862e-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.122 DEBUG:boto:b'dgbpxg-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:39:43.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:39:58.123 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.123 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.123 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.123 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.123 DEBUG:boto:Method: GET 2024-04-12T03:39:58.123 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.123 DEBUG:boto:Data: 2024-04-12T03:39:58.123 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.124 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.124 DEBUG:boto:Params: {} 2024-04-12T03:39:58.124 DEBUG:boto:Token: None 2024-04-12T03:39:58.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs= 2024-04-12T03:39:58.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs=', 'Content-Length': '0'} 2024-04-12T03:39:58.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbcc699123516bf-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.126 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.126 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.126 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.126 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.126 DEBUG:boto:Method: GET 2024-04-12T03:39:58.127 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.127 DEBUG:boto:Data: 2024-04-12T03:39:58.127 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.127 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.127 DEBUG:boto:Params: {} 2024-04-12T03:39:58.127 DEBUG:boto:Token: None 2024-04-12T03:39:58.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs= 2024-04-12T03:39:58.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs=', 'Content-Length': '0'} 2024-04-12T03:39:58.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006815dadb4551074f-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.174 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.174 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.175 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.175 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.175 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.175 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.175 DEBUG:boto:Data: 2024-04-12T03:39:58.175 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.175 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.175 DEBUG:boto:Params: {} 2024-04-12T03:39:58.175 DEBUG:boto:Token: None 2024-04-12T03:39:58.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CBAHH3GG9WoYOd48ruwsXDWC4MI= 2024-04-12T03:39:58.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CBAHH3GG9WoYOd48ruwsXDWC4MI=', 'Content-Length': '0'} 2024-04-12T03:39:58.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8047159574c0841-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.177 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.177 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.177 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.177 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.177 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.178 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.178 DEBUG:boto:Data: 2024-04-12T03:39:58.178 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.178 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.178 DEBUG:boto:Params: {} 2024-04-12T03:39:58.178 DEBUG:boto:Token: None 2024-04-12T03:39:58.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CBAHH3GG9WoYOd48ruwsXDWC4MI= 2024-04-12T03:39:58.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CBAHH3GG9WoYOd48ruwsXDWC4MI=', 'Content-Length': '0'} 2024-04-12T03:39:58.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002796f229af8de866-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:39:58.180 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.180 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.180 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.180 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.181 DEBUG:boto:Method: GET 2024-04-12T03:39:58.181 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.181 DEBUG:boto:Data: 2024-04-12T03:39:58.181 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.181 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.181 DEBUG:boto:Params: {} 2024-04-12T03:39:58.181 DEBUG:boto:Token: None 2024-04-12T03:39:58.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs= 2024-04-12T03:39:58.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs=', 'Content-Length': '0'} 2024-04-12T03:39:58.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815e15d48a605d2a-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.183 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.183 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.183 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.183 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.183 DEBUG:boto:Method: GET 2024-04-12T03:39:58.184 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.184 DEBUG:boto:Data: 2024-04-12T03:39:58.184 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.184 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.184 DEBUG:boto:Params: {} 2024-04-12T03:39:58.184 DEBUG:boto:Token: None 2024-04-12T03:39:58.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055?versionId=null 2024-04-12T03:39:58.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs= 2024-04-12T03:39:58.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4+NI49MoC48nMoasHGfXkGmwQs=', 'Content-Length': '0'} 2024-04-12T03:39:58.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffc49ed98c60182-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.186 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.187 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.187 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.187 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.187 DEBUG:boto:Method: GET 2024-04-12T03:39:58.187 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.187 DEBUG:boto:Data: 2024-04-12T03:39:58.187 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.187 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.187 DEBUG:boto:Params: {} 2024-04-12T03:39:58.188 DEBUG:boto:Token: None 2024-04-12T03:39:58.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050302e22d58a97f7-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.230 DEBUG:boto:b'dgbpxg-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.231 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.231 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.231 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.231 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.231 DEBUG:boto:Method: GET 2024-04-12T03:39:58.231 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.231 DEBUG:boto:Data: 2024-04-12T03:39:58.231 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.231 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.231 DEBUG:boto:Params: {} 2024-04-12T03:39:58.231 DEBUG:boto:Token: None 2024-04-12T03:39:58.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050787449e654d487-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.235 DEBUG:boto:b'dgbpxg-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.236 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.236 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.236 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.236 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.236 DEBUG:boto:Data: 2024-04-12T03:39:58.236 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.236 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.236 DEBUG:boto:Params: {} 2024-04-12T03:39:58.236 DEBUG:boto:Token: None 2024-04-12T03:39:58.237 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.237 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B2iu7qwVajWswwmht6v9GOn7vq8= 2024-04-12T03:39:58.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B2iu7qwVajWswwmht6v9GOn7vq8=', 'Content-Length': '0'} 2024-04-12T03:39:58.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95269fe7e134e78-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.239 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.239 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.239 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.239 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.239 DEBUG:boto:Data: 2024-04-12T03:39:58.239 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.239 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.239 DEBUG:boto:Params: {} 2024-04-12T03:39:58.239 DEBUG:boto:Token: None 2024-04-12T03:39:58.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B2iu7qwVajWswwmht6v9GOn7vq8= 2024-04-12T03:39:58.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B2iu7qwVajWswwmht6v9GOn7vq8=', 'Content-Length': '0'} 2024-04-12T03:39:58.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448352a70142a625-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:39:58.241 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.242 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.242 DEBUG:boto:Method: GET 2024-04-12T03:39:58.242 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.242 DEBUG:boto:Data: 2024-04-12T03:39:58.242 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.242 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.242 DEBUG:boto:Params: {} 2024-04-12T03:39:58.242 DEBUG:boto:Token: None 2024-04-12T03:39:58.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PsrMdj4Dc6tiGkQJ+dbH11cKG/E= 2024-04-12T03:39:58.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PsrMdj4Dc6tiGkQJ+dbH11cKG/E=', 'Content-Length': '0'} 2024-04-12T03:39:58.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3bec3a7d379b6a-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.244 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.244 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.244 DEBUG:boto:Method: GET 2024-04-12T03:39:58.245 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.245 DEBUG:boto:Data: 2024-04-12T03:39:58.245 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.245 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.245 DEBUG:boto:Params: {} 2024-04-12T03:39:58.245 DEBUG:boto:Token: None 2024-04-12T03:39:58.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000055 2024-04-12T03:39:58.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PsrMdj4Dc6tiGkQJ+dbH11cKG/E= 2024-04-12T03:39:58.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PsrMdj4Dc6tiGkQJ+dbH11cKG/E=', 'Content-Length': '0'} 2024-04-12T03:39:58.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779d5da39a697eb2-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.247 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.247 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.247 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.247 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.247 DEBUG:boto:Method: GET 2024-04-12T03:39:58.247 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.247 DEBUG:boto:Data: 2024-04-12T03:39:58.248 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.248 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.248 DEBUG:boto:Params: {} 2024-04-12T03:39:58.248 DEBUG:boto:Token: None 2024-04-12T03:39:58.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f4c25d84365da5-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.250 DEBUG:boto:b'dgbpxg-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.251 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.251 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.251 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.251 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.251 DEBUG:boto:Method: GET 2024-04-12T03:39:58.251 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000055 2024-04-12T03:39:58.251 DEBUG:boto:Data: 2024-04-12T03:39:58.251 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.251 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.251 DEBUG:boto:Params: {} 2024-04-12T03:39:58.251 DEBUG:boto:Token: None 2024-04-12T03:39:58.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007b242c0f1b05855-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.298 DEBUG:boto:b'dgbpxg-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:39:40.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:39:58.299 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.299 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.299 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.299 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.299 DEBUG:boto:Method: GET 2024-04-12T03:39:58.299 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.299 DEBUG:boto:Data: 2024-04-12T03:39:58.299 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.299 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.299 DEBUG:boto:Params: {} 2024-04-12T03:39:58.300 DEBUG:boto:Token: None 2024-04-12T03:39:58.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E= 2024-04-12T03:39:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E=', 'Content-Length': '0'} 2024-04-12T03:39:58.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a9ac8834577cf8-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.302 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.302 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.302 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.302 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.302 DEBUG:boto:Method: GET 2024-04-12T03:39:58.302 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.302 DEBUG:boto:Data: 2024-04-12T03:39:58.302 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.303 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.303 DEBUG:boto:Params: {} 2024-04-12T03:39:58.303 DEBUG:boto:Token: None 2024-04-12T03:39:58.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E= 2024-04-12T03:39:58.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E=', 'Content-Length': '0'} 2024-04-12T03:39:58.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054883c08261acf9d-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.305 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.305 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.305 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.305 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.305 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.305 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.305 DEBUG:boto:Data: 2024-04-12T03:39:58.305 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.305 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.305 DEBUG:boto:Params: {} 2024-04-12T03:39:58.306 DEBUG:boto:Token: None 2024-04-12T03:39:58.306 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h+KcryyMK4zeT53k/GhCJVIBpLI= 2024-04-12T03:39:58.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h+KcryyMK4zeT53k/GhCJVIBpLI=', 'Content-Length': '0'} 2024-04-12T03:39:58.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25c81f845433402-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.308 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.308 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.308 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.308 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.308 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.308 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.308 DEBUG:boto:Data: 2024-04-12T03:39:58.308 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.308 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.308 DEBUG:boto:Params: {} 2024-04-12T03:39:58.308 DEBUG:boto:Token: None 2024-04-12T03:39:58.309 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h+KcryyMK4zeT53k/GhCJVIBpLI= 2024-04-12T03:39:58.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h+KcryyMK4zeT53k/GhCJVIBpLI=', 'Content-Length': '0'} 2024-04-12T03:39:58.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6ada53da9cd886-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:39:58.311 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.311 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.311 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.311 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.311 DEBUG:boto:Method: GET 2024-04-12T03:39:58.311 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.311 DEBUG:boto:Data: 2024-04-12T03:39:58.311 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.311 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.312 DEBUG:boto:Params: {} 2024-04-12T03:39:58.312 DEBUG:boto:Token: None 2024-04-12T03:39:58.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E= 2024-04-12T03:39:58.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E=', 'Content-Length': '0'} 2024-04-12T03:39:58.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646bfcf849abb6ef-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.314 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.314 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.314 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.314 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.314 DEBUG:boto:Method: GET 2024-04-12T03:39:58.314 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.315 DEBUG:boto:Data: 2024-04-12T03:39:58.315 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.315 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.315 DEBUG:boto:Params: {} 2024-04-12T03:39:58.315 DEBUG:boto:Token: None 2024-04-12T03:39:58.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056?versionId=null 2024-04-12T03:39:58.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E= 2024-04-12T03:39:58.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/3uhBmpOCTcmIdSkqUOwrNObf1E=', 'Content-Length': '0'} 2024-04-12T03:39:58.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad73224f0dd916f0-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.317 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.317 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.317 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.317 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.317 DEBUG:boto:Method: GET 2024-04-12T03:39:58.318 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.318 DEBUG:boto:Data: 2024-04-12T03:39:58.318 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.318 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.318 DEBUG:boto:Params: {} 2024-04-12T03:39:58.318 DEBUG:boto:Token: None 2024-04-12T03:39:58.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.318 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0dabc36ed8cfd6f-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.320 DEBUG:boto:b'dgbpxg-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.321 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.321 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.321 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.321 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.321 DEBUG:boto:Method: GET 2024-04-12T03:39:58.321 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.321 DEBUG:boto:Data: 2024-04-12T03:39:58.321 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.321 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.321 DEBUG:boto:Params: {} 2024-04-12T03:39:58.322 DEBUG:boto:Token: None 2024-04-12T03:39:58.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2d018302cc3c3b-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.325 DEBUG:boto:b'dgbpxg-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.326 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.326 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.326 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.326 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.326 DEBUG:boto:Data: 2024-04-12T03:39:58.326 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.326 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.326 DEBUG:boto:Params: {} 2024-04-12T03:39:58.327 DEBUG:boto:Token: None 2024-04-12T03:39:58.327 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.327 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QSHMuiSkQ8B1oQ/cvLIpnsZZq8A= 2024-04-12T03:39:58.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QSHMuiSkQ8B1oQ/cvLIpnsZZq8A=', 'Content-Length': '0'} 2024-04-12T03:39:58.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008adc6eb47060b7a7-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.328 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.329 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.329 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.329 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.329 DEBUG:boto:Data: 2024-04-12T03:39:58.329 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.329 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.329 DEBUG:boto:Params: {} 2024-04-12T03:39:58.329 DEBUG:boto:Token: None 2024-04-12T03:39:58.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.330 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QSHMuiSkQ8B1oQ/cvLIpnsZZq8A= 2024-04-12T03:39:58.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QSHMuiSkQ8B1oQ/cvLIpnsZZq8A=', 'Content-Length': '0'} 2024-04-12T03:39:58.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006971d19839291256-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:39:58.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.332 DEBUG:boto:Method: GET 2024-04-12T03:39:58.332 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.332 DEBUG:boto:Data: 2024-04-12T03:39:58.332 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.332 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.332 DEBUG:boto:Params: {} 2024-04-12T03:39:58.332 DEBUG:boto:Token: None 2024-04-12T03:39:58.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKXjMJZLcafcyQAIxj4Y9s3xgvk= 2024-04-12T03:39:58.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKXjMJZLcafcyQAIxj4Y9s3xgvk=', 'Content-Length': '0'} 2024-04-12T03:39:58.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05654846637a1fd-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.334 DEBUG:boto:Method: GET 2024-04-12T03:39:58.334 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.335 DEBUG:boto:Data: 2024-04-12T03:39:58.335 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.335 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.335 DEBUG:boto:Params: {} 2024-04-12T03:39:58.335 DEBUG:boto:Token: None 2024-04-12T03:39:58.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000056 2024-04-12T03:39:58.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKXjMJZLcafcyQAIxj4Y9s3xgvk= 2024-04-12T03:39:58.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKXjMJZLcafcyQAIxj4Y9s3xgvk=', 'Content-Length': '0'} 2024-04-12T03:39:58.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b75bbec8f5b29b-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.378 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.379 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.379 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.379 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.379 DEBUG:boto:Method: GET 2024-04-12T03:39:58.379 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.379 DEBUG:boto:Data: 2024-04-12T03:39:58.379 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.379 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.379 DEBUG:boto:Params: {} 2024-04-12T03:39:58.379 DEBUG:boto:Token: None 2024-04-12T03:39:58.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3e18c0229dae866-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.382 DEBUG:boto:b'dgbpxg-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.382 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.382 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.382 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.383 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.383 DEBUG:boto:Method: GET 2024-04-12T03:39:58.383 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000056 2024-04-12T03:39:58.383 DEBUG:boto:Data: 2024-04-12T03:39:58.383 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.383 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.383 DEBUG:boto:Params: {} 2024-04-12T03:39:58.383 DEBUG:boto:Token: None 2024-04-12T03:39:58.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b7d6c7f00cb44d-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.387 DEBUG:boto:b'dgbpxg-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:39:42.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:39:58.388 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.388 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.388 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.388 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.388 DEBUG:boto:Method: GET 2024-04-12T03:39:58.388 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.388 DEBUG:boto:Data: 2024-04-12T03:39:58.388 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.388 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.388 DEBUG:boto:Params: {} 2024-04-12T03:39:58.389 DEBUG:boto:Token: None 2024-04-12T03:39:58.389 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.389 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s= 2024-04-12T03:39:58.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s=', 'Content-Length': '0'} 2024-04-12T03:39:58.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3430710fe007291-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.391 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.391 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.391 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.391 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.391 DEBUG:boto:Method: GET 2024-04-12T03:39:58.391 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.391 DEBUG:boto:Data: 2024-04-12T03:39:58.391 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.391 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.392 DEBUG:boto:Params: {} 2024-04-12T03:39:58.392 DEBUG:boto:Token: None 2024-04-12T03:39:58.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s= 2024-04-12T03:39:58.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s=', 'Content-Length': '0'} 2024-04-12T03:39:58.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ff775a95ef3ba7-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.394 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.394 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.394 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.394 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.394 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.394 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.394 DEBUG:boto:Data: 2024-04-12T03:39:58.394 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.395 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.395 DEBUG:boto:Params: {} 2024-04-12T03:39:58.395 DEBUG:boto:Token: None 2024-04-12T03:39:58.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q6G/BPFrYpcEBFVsMIirr0sKRHQ= 2024-04-12T03:39:58.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q6G/BPFrYpcEBFVsMIirr0sKRHQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006571c93c0cdf8fa4-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.397 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.397 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.397 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.397 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.397 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.397 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.397 DEBUG:boto:Data: 2024-04-12T03:39:58.397 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.397 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.398 DEBUG:boto:Params: {} 2024-04-12T03:39:58.398 DEBUG:boto:Token: None 2024-04-12T03:39:58.398 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q6G/BPFrYpcEBFVsMIirr0sKRHQ= 2024-04-12T03:39:58.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q6G/BPFrYpcEBFVsMIirr0sKRHQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004877cb76bfbfc4ae-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:39:58.400 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.400 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.400 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.400 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.400 DEBUG:boto:Method: GET 2024-04-12T03:39:58.400 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.400 DEBUG:boto:Data: 2024-04-12T03:39:58.400 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.401 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.401 DEBUG:boto:Params: {} 2024-04-12T03:39:58.401 DEBUG:boto:Token: None 2024-04-12T03:39:58.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s= 2024-04-12T03:39:58.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s=', 'Content-Length': '0'} 2024-04-12T03:39:58.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3758e9907f6487e-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.446 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.446 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.447 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.447 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.447 DEBUG:boto:Method: GET 2024-04-12T03:39:58.447 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.447 DEBUG:boto:Data: 2024-04-12T03:39:58.447 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.447 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.447 DEBUG:boto:Params: {} 2024-04-12T03:39:58.447 DEBUG:boto:Token: None 2024-04-12T03:39:58.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057?versionId=null 2024-04-12T03:39:58.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s= 2024-04-12T03:39:58.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GCGvmFlhkoV+iQ4twcDqWP8ef7s=', 'Content-Length': '0'} 2024-04-12T03:39:58.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d7d5dfc4d575ab-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.450 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.450 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.450 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.450 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.450 DEBUG:boto:Method: GET 2024-04-12T03:39:58.450 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.450 DEBUG:boto:Data: 2024-04-12T03:39:58.451 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.451 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.451 DEBUG:boto:Params: {} 2024-04-12T03:39:58.451 DEBUG:boto:Token: None 2024-04-12T03:39:58.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ad7d372d39abb4-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.454 DEBUG:boto:b'dgbpxg-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.454 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.454 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.454 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.454 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.454 DEBUG:boto:Method: GET 2024-04-12T03:39:58.455 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.455 DEBUG:boto:Data: 2024-04-12T03:39:58.455 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.455 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.455 DEBUG:boto:Params: {} 2024-04-12T03:39:58.455 DEBUG:boto:Token: None 2024-04-12T03:39:58.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.455 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a9b1a058f43ad4-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.459 DEBUG:boto:b'dgbpxg-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.459 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.460 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.460 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.460 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.460 DEBUG:boto:Data: 2024-04-12T03:39:58.460 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.460 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.460 DEBUG:boto:Params: {} 2024-04-12T03:39:58.460 DEBUG:boto:Token: None 2024-04-12T03:39:58.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VFPqZxPA1b4hyFxegxGvN/guXSM= 2024-04-12T03:39:58.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VFPqZxPA1b4hyFxegxGvN/guXSM=', 'Content-Length': '0'} 2024-04-12T03:39:58.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057fc0544658cf425-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.462 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.462 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.462 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.463 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.463 DEBUG:boto:Data: 2024-04-12T03:39:58.463 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.463 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.463 DEBUG:boto:Params: {} 2024-04-12T03:39:58.463 DEBUG:boto:Token: None 2024-04-12T03:39:58.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VFPqZxPA1b4hyFxegxGvN/guXSM= 2024-04-12T03:39:58.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VFPqZxPA1b4hyFxegxGvN/guXSM=', 'Content-Length': '0'} 2024-04-12T03:39:58.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c6ef61f602c094-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:39:58.465 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.465 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.466 DEBUG:boto:Method: GET 2024-04-12T03:39:58.466 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.466 DEBUG:boto:Data: 2024-04-12T03:39:58.466 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.466 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.466 DEBUG:boto:Params: {} 2024-04-12T03:39:58.466 DEBUG:boto:Token: None 2024-04-12T03:39:58.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7sRvGpjPewIHtMhr5MoDDGLLxHk= 2024-04-12T03:39:58.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7sRvGpjPewIHtMhr5MoDDGLLxHk=', 'Content-Length': '0'} 2024-04-12T03:39:58.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde05775ef9e1e2f-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.468 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.468 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.468 DEBUG:boto:Method: GET 2024-04-12T03:39:58.469 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.469 DEBUG:boto:Data: 2024-04-12T03:39:58.469 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.469 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.469 DEBUG:boto:Params: {} 2024-04-12T03:39:58.469 DEBUG:boto:Token: None 2024-04-12T03:39:58.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000057 2024-04-12T03:39:58.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7sRvGpjPewIHtMhr5MoDDGLLxHk= 2024-04-12T03:39:58.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7sRvGpjPewIHtMhr5MoDDGLLxHk=', 'Content-Length': '0'} 2024-04-12T03:39:58.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38421c1a17b1f78-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.514 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.514 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.514 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.515 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.515 DEBUG:boto:Method: GET 2024-04-12T03:39:58.515 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.515 DEBUG:boto:Data: 2024-04-12T03:39:58.515 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.515 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.515 DEBUG:boto:Params: {} 2024-04-12T03:39:58.515 DEBUG:boto:Token: None 2024-04-12T03:39:58.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc99c74d35741a24-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.518 DEBUG:boto:b'dgbpxg-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.518 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.518 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.518 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.518 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.518 DEBUG:boto:Method: GET 2024-04-12T03:39:58.519 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000057 2024-04-12T03:39:58.519 DEBUG:boto:Data: 2024-04-12T03:39:58.519 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.519 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.519 DEBUG:boto:Params: {} 2024-04-12T03:39:58.519 DEBUG:boto:Token: None 2024-04-12T03:39:58.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebefa260d360572f-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.523 DEBUG:boto:b'dgbpxg-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:39:43.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:39:58.523 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.523 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.523 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.524 DEBUG:boto:Method: GET 2024-04-12T03:39:58.524 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.524 DEBUG:boto:Data: 2024-04-12T03:39:58.524 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.524 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.524 DEBUG:boto:Params: {} 2024-04-12T03:39:58.524 DEBUG:boto:Token: None 2024-04-12T03:39:58.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY= 2024-04-12T03:39:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY=', 'Content-Length': '0'} 2024-04-12T03:39:58.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ef264c4dba3cad-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.527 DEBUG:boto:Method: GET 2024-04-12T03:39:58.527 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.527 DEBUG:boto:Data: 2024-04-12T03:39:58.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.527 DEBUG:boto:Params: {} 2024-04-12T03:39:58.528 DEBUG:boto:Token: None 2024-04-12T03:39:58.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY= 2024-04-12T03:39:58.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY=', 'Content-Length': '0'} 2024-04-12T03:39:58.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007735da6e845971f6-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.529 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.530 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.530 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.530 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.530 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.530 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.530 DEBUG:boto:Data: 2024-04-12T03:39:58.530 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.530 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.530 DEBUG:boto:Params: {} 2024-04-12T03:39:58.530 DEBUG:boto:Token: None 2024-04-12T03:39:58.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:goPtj1AwNG9UFQQW1V5FfwdTOqw= 2024-04-12T03:39:58.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:goPtj1AwNG9UFQQW1V5FfwdTOqw=', 'Content-Length': '0'} 2024-04-12T03:39:58.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba1b4c9544176fcd-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.532 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.533 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.533 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.533 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.533 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.533 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.533 DEBUG:boto:Data: 2024-04-12T03:39:58.533 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.533 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.533 DEBUG:boto:Params: {} 2024-04-12T03:39:58.533 DEBUG:boto:Token: None 2024-04-12T03:39:58.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:goPtj1AwNG9UFQQW1V5FfwdTOqw= 2024-04-12T03:39:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:goPtj1AwNG9UFQQW1V5FfwdTOqw=', 'Content-Length': '0'} 2024-04-12T03:39:58.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5856c67ff0df991-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:39:58.536 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.536 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.536 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.536 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.536 DEBUG:boto:Method: GET 2024-04-12T03:39:58.536 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.536 DEBUG:boto:Data: 2024-04-12T03:39:58.536 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.536 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.536 DEBUG:boto:Params: {} 2024-04-12T03:39:58.537 DEBUG:boto:Token: None 2024-04-12T03:39:58.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY= 2024-04-12T03:39:58.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY=', 'Content-Length': '0'} 2024-04-12T03:39:58.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d73bc855cee879a-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.539 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.539 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.539 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.539 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.539 DEBUG:boto:Method: GET 2024-04-12T03:39:58.539 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.539 DEBUG:boto:Data: 2024-04-12T03:39:58.540 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.540 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.540 DEBUG:boto:Params: {} 2024-04-12T03:39:58.540 DEBUG:boto:Token: None 2024-04-12T03:39:58.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058?versionId=null 2024-04-12T03:39:58.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY= 2024-04-12T03:39:58.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGU2ZP3hR9A+yg0qzWziR2ToQlY=', 'Content-Length': '0'} 2024-04-12T03:39:58.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002f721865197876c-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.542 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.542 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.542 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.542 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.542 DEBUG:boto:Method: GET 2024-04-12T03:39:58.542 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.542 DEBUG:boto:Data: 2024-04-12T03:39:58.542 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.543 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.543 DEBUG:boto:Params: {} 2024-04-12T03:39:58.543 DEBUG:boto:Token: None 2024-04-12T03:39:58.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.543 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916b0a8f4f5adcaf-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.586 DEBUG:boto:b'dgbpxg-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.587 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.587 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.587 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.587 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.587 DEBUG:boto:Method: GET 2024-04-12T03:39:58.587 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.587 DEBUG:boto:Data: 2024-04-12T03:39:58.587 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.587 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.587 DEBUG:boto:Params: {} 2024-04-12T03:39:58.588 DEBUG:boto:Token: None 2024-04-12T03:39:58.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449a1bd5b0cb992d-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.591 DEBUG:boto:b'dgbpxg-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.592 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.592 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.592 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.592 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.592 DEBUG:boto:Data: 2024-04-12T03:39:58.592 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.592 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.592 DEBUG:boto:Params: {} 2024-04-12T03:39:58.592 DEBUG:boto:Token: None 2024-04-12T03:39:58.593 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.593 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8hm8zOrVWmZh4SZ9G/OObHiMMU= 2024-04-12T03:39:58.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8hm8zOrVWmZh4SZ9G/OObHiMMU=', 'Content-Length': '0'} 2024-04-12T03:39:58.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6097e093a40b748-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.594 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.595 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.595 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.595 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.595 DEBUG:boto:Data: 2024-04-12T03:39:58.595 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.595 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.595 DEBUG:boto:Params: {} 2024-04-12T03:39:58.595 DEBUG:boto:Token: None 2024-04-12T03:39:58.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8hm8zOrVWmZh4SZ9G/OObHiMMU= 2024-04-12T03:39:58.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8hm8zOrVWmZh4SZ9G/OObHiMMU=', 'Content-Length': '0'} 2024-04-12T03:39:58.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020fcb0875760c127-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:39:58.598 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.598 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.598 DEBUG:boto:Method: GET 2024-04-12T03:39:58.598 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.598 DEBUG:boto:Data: 2024-04-12T03:39:58.598 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.598 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.598 DEBUG:boto:Params: {} 2024-04-12T03:39:58.599 DEBUG:boto:Token: None 2024-04-12T03:39:58.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WLXVhUg1V2c6kQX4EGlykqbeAMY= 2024-04-12T03:39:58.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WLXVhUg1V2c6kQX4EGlykqbeAMY=', 'Content-Length': '0'} 2024-04-12T03:39:58.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000467236884eecaa2a-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.601 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.601 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.601 DEBUG:boto:Method: GET 2024-04-12T03:39:58.601 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.601 DEBUG:boto:Data: 2024-04-12T03:39:58.601 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.601 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.601 DEBUG:boto:Params: {} 2024-04-12T03:39:58.601 DEBUG:boto:Token: None 2024-04-12T03:39:58.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000058 2024-04-12T03:39:58.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WLXVhUg1V2c6kQX4EGlykqbeAMY= 2024-04-12T03:39:58.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WLXVhUg1V2c6kQX4EGlykqbeAMY=', 'Content-Length': '0'} 2024-04-12T03:39:58.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b093166851c340-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.650 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.650 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.651 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.651 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.651 DEBUG:boto:Method: GET 2024-04-12T03:39:58.651 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.651 DEBUG:boto:Data: 2024-04-12T03:39:58.651 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.651 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.651 DEBUG:boto:Params: {} 2024-04-12T03:39:58.651 DEBUG:boto:Token: None 2024-04-12T03:39:58.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a52be9c9397f8b-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.654 DEBUG:boto:b'dgbpxg-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.654 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.654 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.654 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.654 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.655 DEBUG:boto:Method: GET 2024-04-12T03:39:58.655 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000058 2024-04-12T03:39:58.655 DEBUG:boto:Data: 2024-04-12T03:39:58.655 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.655 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.655 DEBUG:boto:Params: {} 2024-04-12T03:39:58.655 DEBUG:boto:Token: None 2024-04-12T03:39:58.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157212a7128530eb-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.659 DEBUG:boto:b'dgbpxg-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:39:40.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:39:58.659 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.659 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.659 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.659 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.660 DEBUG:boto:Method: GET 2024-04-12T03:39:58.660 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.660 DEBUG:boto:Data: 2024-04-12T03:39:58.660 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.660 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.660 DEBUG:boto:Params: {} 2024-04-12T03:39:58.660 DEBUG:boto:Token: None 2024-04-12T03:39:58.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8= 2024-04-12T03:39:58.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8=', 'Content-Length': '0'} 2024-04-12T03:39:58.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924ca08a004f677f-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.662 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.662 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.663 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.663 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.663 DEBUG:boto:Method: GET 2024-04-12T03:39:58.663 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.663 DEBUG:boto:Data: 2024-04-12T03:39:58.663 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.663 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.663 DEBUG:boto:Params: {} 2024-04-12T03:39:58.663 DEBUG:boto:Token: None 2024-04-12T03:39:58.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8= 2024-04-12T03:39:58.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8=', 'Content-Length': '0'} 2024-04-12T03:39:58.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366d4ab328d22d83-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.665 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.665 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.666 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.666 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.666 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.666 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.666 DEBUG:boto:Data: 2024-04-12T03:39:58.666 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.666 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.666 DEBUG:boto:Params: {} 2024-04-12T03:39:58.666 DEBUG:boto:Token: None 2024-04-12T03:39:58.667 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qqTAC1oe4YZ4FzcI6nnJPygfiE0= 2024-04-12T03:39:58.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qqTAC1oe4YZ4FzcI6nnJPygfiE0=', 'Content-Length': '0'} 2024-04-12T03:39:58.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000802b02338e914c65-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.668 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.668 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.669 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.669 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.669 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.669 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.669 DEBUG:boto:Data: 2024-04-12T03:39:58.669 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.669 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.669 DEBUG:boto:Params: {} 2024-04-12T03:39:58.669 DEBUG:boto:Token: None 2024-04-12T03:39:58.670 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.670 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qqTAC1oe4YZ4FzcI6nnJPygfiE0= 2024-04-12T03:39:58.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qqTAC1oe4YZ4FzcI6nnJPygfiE0=', 'Content-Length': '0'} 2024-04-12T03:39:58.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b977e631f79eed-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:39:58.672 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.672 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.672 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.672 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.672 DEBUG:boto:Method: GET 2024-04-12T03:39:58.672 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.672 DEBUG:boto:Data: 2024-04-12T03:39:58.672 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.672 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.672 DEBUG:boto:Params: {} 2024-04-12T03:39:58.673 DEBUG:boto:Token: None 2024-04-12T03:39:58.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.673 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8= 2024-04-12T03:39:58.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8=', 'Content-Length': '0'} 2024-04-12T03:39:58.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b5bf50e7733c86-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.718 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.718 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.719 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.719 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.719 DEBUG:boto:Method: GET 2024-04-12T03:39:58.719 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.719 DEBUG:boto:Data: 2024-04-12T03:39:58.719 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.719 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.719 DEBUG:boto:Params: {} 2024-04-12T03:39:58.719 DEBUG:boto:Token: None 2024-04-12T03:39:58.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059?versionId=null 2024-04-12T03:39:58.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8= 2024-04-12T03:39:58.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+UGvhGsb8FXjkai7fxp7ZOW8Az8=', 'Content-Length': '0'} 2024-04-12T03:39:58.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50850db95cb7f6c-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.721 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.722 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.722 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.722 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.722 DEBUG:boto:Method: GET 2024-04-12T03:39:58.722 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.722 DEBUG:boto:Data: 2024-04-12T03:39:58.722 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.722 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.722 DEBUG:boto:Params: {} 2024-04-12T03:39:58.722 DEBUG:boto:Token: None 2024-04-12T03:39:58.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.723 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db6fd651a914481-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.725 DEBUG:boto:b'dgbpxg-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.725 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.725 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.725 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.725 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.725 DEBUG:boto:Method: GET 2024-04-12T03:39:58.725 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.726 DEBUG:boto:Data: 2024-04-12T03:39:58.726 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.726 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.726 DEBUG:boto:Params: {} 2024-04-12T03:39:58.726 DEBUG:boto:Token: None 2024-04-12T03:39:58.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004992b18b55444f2-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.730 DEBUG:boto:b'dgbpxg-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.730 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.730 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.731 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.731 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.731 DEBUG:boto:Data: 2024-04-12T03:39:58.731 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.731 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.731 DEBUG:boto:Params: {} 2024-04-12T03:39:58.731 DEBUG:boto:Token: None 2024-04-12T03:39:58.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+MheM694Z/k3n2IqSzaFIVQc+s= 2024-04-12T03:39:58.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+MheM694Z/k3n2IqSzaFIVQc+s=', 'Content-Length': '0'} 2024-04-12T03:39:58.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a626886dbef9853-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.733 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.733 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.733 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.733 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.733 DEBUG:boto:Data: 2024-04-12T03:39:58.734 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.734 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.734 DEBUG:boto:Params: {} 2024-04-12T03:39:58.734 DEBUG:boto:Token: None 2024-04-12T03:39:58.734 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+MheM694Z/k3n2IqSzaFIVQc+s= 2024-04-12T03:39:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+MheM694Z/k3n2IqSzaFIVQc+s=', 'Content-Length': '0'} 2024-04-12T03:39:58.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867229a2c85085eb-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:39:58.736 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.736 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.736 DEBUG:boto:Method: GET 2024-04-12T03:39:58.736 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.736 DEBUG:boto:Data: 2024-04-12T03:39:58.736 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.737 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.737 DEBUG:boto:Params: {} 2024-04-12T03:39:58.737 DEBUG:boto:Token: None 2024-04-12T03:39:58.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+TyflYkguEjB4KcN1RPefmZTFMw= 2024-04-12T03:39:58.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+TyflYkguEjB4KcN1RPefmZTFMw=', 'Content-Length': '0'} 2024-04-12T03:39:58.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684aa130d6101cbc-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.739 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.739 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.739 DEBUG:boto:Method: GET 2024-04-12T03:39:58.739 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.739 DEBUG:boto:Data: 2024-04-12T03:39:58.739 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.740 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.740 DEBUG:boto:Params: {} 2024-04-12T03:39:58.740 DEBUG:boto:Token: None 2024-04-12T03:39:58.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000059 2024-04-12T03:39:58.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+TyflYkguEjB4KcN1RPefmZTFMw= 2024-04-12T03:39:58.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+TyflYkguEjB4KcN1RPefmZTFMw=', 'Content-Length': '0'} 2024-04-12T03:39:58.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1569723128e3ca-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.742 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.742 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.742 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.742 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.742 DEBUG:boto:Method: GET 2024-04-12T03:39:58.742 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.742 DEBUG:boto:Data: 2024-04-12T03:39:58.743 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.743 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.743 DEBUG:boto:Params: {} 2024-04-12T03:39:58.743 DEBUG:boto:Token: None 2024-04-12T03:39:58.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.743 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bab0c7406d1538f-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.745 DEBUG:boto:b'dgbpxg-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.746 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.746 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.746 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.746 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.746 DEBUG:boto:Method: GET 2024-04-12T03:39:58.746 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000059 2024-04-12T03:39:58.746 DEBUG:boto:Data: 2024-04-12T03:39:58.746 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.746 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.746 DEBUG:boto:Params: {} 2024-04-12T03:39:58.747 DEBUG:boto:Token: None 2024-04-12T03:39:58.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85fcdf9b21bf71c-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.751 DEBUG:boto:b'dgbpxg-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:39:41.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:39:58.751 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.751 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.751 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.751 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.751 DEBUG:boto:Method: GET 2024-04-12T03:39:58.751 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.752 DEBUG:boto:Data: 2024-04-12T03:39:58.752 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.752 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.752 DEBUG:boto:Params: {} 2024-04-12T03:39:58.752 DEBUG:boto:Token: None 2024-04-12T03:39:58.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY= 2024-04-12T03:39:58.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY=', 'Content-Length': '0'} 2024-04-12T03:39:58.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d175399f8bb7da-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.754 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.754 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.754 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.754 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.755 DEBUG:boto:Method: GET 2024-04-12T03:39:58.755 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.755 DEBUG:boto:Data: 2024-04-12T03:39:58.755 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.755 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.755 DEBUG:boto:Params: {} 2024-04-12T03:39:58.755 DEBUG:boto:Token: None 2024-04-12T03:39:58.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY= 2024-04-12T03:39:58.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY=', 'Content-Length': '0'} 2024-04-12T03:39:58.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942e2ab9f537eb4d-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.798 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.798 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.798 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.799 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.799 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.799 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.799 DEBUG:boto:Data: 2024-04-12T03:39:58.799 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.799 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.799 DEBUG:boto:Params: {} 2024-04-12T03:39:58.799 DEBUG:boto:Token: None 2024-04-12T03:39:58.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjdlhngj8rkLGecQic+F+FSU+e8= 2024-04-12T03:39:58.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjdlhngj8rkLGecQic+F+FSU+e8=', 'Content-Length': '0'} 2024-04-12T03:39:58.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a182b43e948699-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.801 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.801 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.802 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.802 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.802 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.802 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.802 DEBUG:boto:Data: 2024-04-12T03:39:58.802 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.802 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.802 DEBUG:boto:Params: {} 2024-04-12T03:39:58.802 DEBUG:boto:Token: None 2024-04-12T03:39:58.802 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjdlhngj8rkLGecQic+F+FSU+e8= 2024-04-12T03:39:58.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjdlhngj8rkLGecQic+F+FSU+e8=', 'Content-Length': '0'} 2024-04-12T03:39:58.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f82cd950b2b50d-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:39:58.804 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.805 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.805 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.805 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.805 DEBUG:boto:Method: GET 2024-04-12T03:39:58.805 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.805 DEBUG:boto:Data: 2024-04-12T03:39:58.805 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.805 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.805 DEBUG:boto:Params: {} 2024-04-12T03:39:58.805 DEBUG:boto:Token: None 2024-04-12T03:39:58.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY= 2024-04-12T03:39:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY=', 'Content-Length': '0'} 2024-04-12T03:39:58.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f06a576de9a280f-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.807 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.808 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.808 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.808 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.808 DEBUG:boto:Method: GET 2024-04-12T03:39:58.808 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.808 DEBUG:boto:Data: 2024-04-12T03:39:58.808 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.808 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.808 DEBUG:boto:Params: {} 2024-04-12T03:39:58.808 DEBUG:boto:Token: None 2024-04-12T03:39:58.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060?versionId=null 2024-04-12T03:39:58.809 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY= 2024-04-12T03:39:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6NvAcfCLdKZss8R7f0tw7mTpCpY=', 'Content-Length': '0'} 2024-04-12T03:39:58.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ae6ee5e72ccce8-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.810 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.811 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.811 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.811 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.811 DEBUG:boto:Method: GET 2024-04-12T03:39:58.811 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.811 DEBUG:boto:Data: 2024-04-12T03:39:58.811 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.811 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.811 DEBUG:boto:Params: {} 2024-04-12T03:39:58.811 DEBUG:boto:Token: None 2024-04-12T03:39:58.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55b7de6bde36795-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.814 DEBUG:boto:b'dgbpxg-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.814 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.814 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.815 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.815 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.815 DEBUG:boto:Method: GET 2024-04-12T03:39:58.815 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.815 DEBUG:boto:Data: 2024-04-12T03:39:58.815 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.815 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.815 DEBUG:boto:Params: {} 2024-04-12T03:39:58.815 DEBUG:boto:Token: None 2024-04-12T03:39:58.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027974792db5b53c0-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.819 DEBUG:boto:b'dgbpxg-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.820 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.820 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.820 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.820 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.820 DEBUG:boto:Data: 2024-04-12T03:39:58.820 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.820 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.820 DEBUG:boto:Params: {} 2024-04-12T03:39:58.820 DEBUG:boto:Token: None 2024-04-12T03:39:58.821 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cmOZMCSbA9DUDSQEkUIjo1atQUI= 2024-04-12T03:39:58.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cmOZMCSbA9DUDSQEkUIjo1atQUI=', 'Content-Length': '0'} 2024-04-12T03:39:58.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ae5632c8810ca60-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.823 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.823 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.823 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.823 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.823 DEBUG:boto:Data: 2024-04-12T03:39:58.823 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.823 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.823 DEBUG:boto:Params: {} 2024-04-12T03:39:58.823 DEBUG:boto:Token: None 2024-04-12T03:39:58.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cmOZMCSbA9DUDSQEkUIjo1atQUI= 2024-04-12T03:39:58.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cmOZMCSbA9DUDSQEkUIjo1atQUI=', 'Content-Length': '0'} 2024-04-12T03:39:58.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddf5cb5bf02050c1-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:39:58.825 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.825 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.826 DEBUG:boto:Method: GET 2024-04-12T03:39:58.826 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.826 DEBUG:boto:Data: 2024-04-12T03:39:58.826 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.826 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.826 DEBUG:boto:Params: {} 2024-04-12T03:39:58.826 DEBUG:boto:Token: None 2024-04-12T03:39:58.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2bGC3hEYiIN8f+VgfqLAh/F/oVg= 2024-04-12T03:39:58.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2bGC3hEYiIN8f+VgfqLAh/F/oVg=', 'Content-Length': '0'} 2024-04-12T03:39:58.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0e2d8ca968602d8-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.829 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.829 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.829 DEBUG:boto:Method: GET 2024-04-12T03:39:58.829 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.829 DEBUG:boto:Data: 2024-04-12T03:39:58.829 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.829 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.829 DEBUG:boto:Params: {} 2024-04-12T03:39:58.829 DEBUG:boto:Token: None 2024-04-12T03:39:58.830 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000060 2024-04-12T03:39:58.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2bGC3hEYiIN8f+VgfqLAh/F/oVg= 2024-04-12T03:39:58.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2bGC3hEYiIN8f+VgfqLAh/F/oVg=', 'Content-Length': '0'} 2024-04-12T03:39:58.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbe9ddecfe486a2-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.874 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.874 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.875 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.875 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.875 DEBUG:boto:Method: GET 2024-04-12T03:39:58.875 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.875 DEBUG:boto:Data: 2024-04-12T03:39:58.875 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.875 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.875 DEBUG:boto:Params: {} 2024-04-12T03:39:58.875 DEBUG:boto:Token: None 2024-04-12T03:39:58.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd1969e4aaa0a18-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.878 DEBUG:boto:b'dgbpxg-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.878 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.878 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.878 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.878 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.878 DEBUG:boto:Method: GET 2024-04-12T03:39:58.878 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000060 2024-04-12T03:39:58.879 DEBUG:boto:Data: 2024-04-12T03:39:58.879 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.879 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.879 DEBUG:boto:Params: {} 2024-04-12T03:39:58.879 DEBUG:boto:Token: None 2024-04-12T03:39:58.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8aeeeda46ce59d2-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.883 DEBUG:boto:b'dgbpxg-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:39:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:39:58.883 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.883 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.884 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.884 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.884 DEBUG:boto:Method: GET 2024-04-12T03:39:58.884 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.884 DEBUG:boto:Data: 2024-04-12T03:39:58.884 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.884 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.884 DEBUG:boto:Params: {} 2024-04-12T03:39:58.884 DEBUG:boto:Token: None 2024-04-12T03:39:58.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.885 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A= 2024-04-12T03:39:58.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A=', 'Content-Length': '0'} 2024-04-12T03:39:58.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86a2ecdc4a80df6-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.887 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.887 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.887 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.887 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.887 DEBUG:boto:Method: GET 2024-04-12T03:39:58.887 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.887 DEBUG:boto:Data: 2024-04-12T03:39:58.887 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.887 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.888 DEBUG:boto:Params: {} 2024-04-12T03:39:58.888 DEBUG:boto:Token: None 2024-04-12T03:39:58.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A= 2024-04-12T03:39:58.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A=', 'Content-Length': '0'} 2024-04-12T03:39:58.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051bfd8d5cd6b8a4f-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.890 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.890 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.890 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.890 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.890 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.890 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.890 DEBUG:boto:Data: 2024-04-12T03:39:58.890 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.890 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.891 DEBUG:boto:Params: {} 2024-04-12T03:39:58.891 DEBUG:boto:Token: None 2024-04-12T03:39:58.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:77vQNOrpG1iVH7/mHBnWKTNulfM= 2024-04-12T03:39:58.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:77vQNOrpG1iVH7/mHBnWKTNulfM=', 'Content-Length': '0'} 2024-04-12T03:39:58.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81234848626cc64-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.892 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.893 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.893 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.893 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.893 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.893 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.893 DEBUG:boto:Data: 2024-04-12T03:39:58.893 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.893 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.893 DEBUG:boto:Params: {} 2024-04-12T03:39:58.893 DEBUG:boto:Token: None 2024-04-12T03:39:58.894 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.894 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:77vQNOrpG1iVH7/mHBnWKTNulfM= 2024-04-12T03:39:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:77vQNOrpG1iVH7/mHBnWKTNulfM=', 'Content-Length': '0'} 2024-04-12T03:39:58.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50c84f17da96d54-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:39:58.896 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.896 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.896 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.896 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.896 DEBUG:boto:Method: GET 2024-04-12T03:39:58.896 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.896 DEBUG:boto:Data: 2024-04-12T03:39:58.896 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.897 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.897 DEBUG:boto:Params: {} 2024-04-12T03:39:58.897 DEBUG:boto:Token: None 2024-04-12T03:39:58.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A= 2024-04-12T03:39:58.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A=', 'Content-Length': '0'} 2024-04-12T03:39:58.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027cdf9c0e858a0ea-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.942 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.942 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.942 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.942 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.943 DEBUG:boto:Method: GET 2024-04-12T03:39:58.943 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.943 DEBUG:boto:Data: 2024-04-12T03:39:58.943 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.943 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.943 DEBUG:boto:Params: {} 2024-04-12T03:39:58.943 DEBUG:boto:Token: None 2024-04-12T03:39:58.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061?versionId=null 2024-04-12T03:39:58.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A= 2024-04-12T03:39:58.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9fDQzvGlTpxYxyqrtN5k8LuR50A=', 'Content-Length': '0'} 2024-04-12T03:39:58.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332f4cc0fff3aa96-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.945 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.945 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.946 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.946 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.946 DEBUG:boto:Method: GET 2024-04-12T03:39:58.946 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.946 DEBUG:boto:Data: 2024-04-12T03:39:58.946 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.946 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.946 DEBUG:boto:Params: {} 2024-04-12T03:39:58.946 DEBUG:boto:Token: None 2024-04-12T03:39:58.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb064304445395a-006618ad0e-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.949 DEBUG:boto:b'dgbpxg-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.949 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:58.949 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:58.949 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.949 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.949 DEBUG:boto:Method: GET 2024-04-12T03:39:58.950 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:58.950 DEBUG:boto:Data: 2024-04-12T03:39:58.950 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.950 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.950 DEBUG:boto:Params: {} 2024-04-12T03:39:58.950 DEBUG:boto:Token: None 2024-04-12T03:39:58.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/ 2024-04-12T03:39:58.950 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE= 2024-04-12T03:39:58.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3cFKqRO9+62C7EIXSNRQRanBjE=', 'Content-Length': '0'} 2024-04-12T03:39:58.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e86a0986718a54e9-006618ad0e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.954 DEBUG:boto:b'dgbpxg-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:58.955 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.955 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.955 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.955 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.955 DEBUG:boto:Data: 2024-04-12T03:39:58.955 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.955 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.955 DEBUG:boto:Params: {} 2024-04-12T03:39:58.955 DEBUG:boto:Token: None 2024-04-12T03:39:58.955 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4BPh4gR9J1Fic53mFhZ/2m9ugoQ= 2024-04-12T03:39:58.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4BPh4gR9J1Fic53mFhZ/2m9ugoQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c942fc89754cf4-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.957 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.957 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.957 DEBUG:boto:Method: HEAD 2024-04-12T03:39:58.958 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.958 DEBUG:boto:Data: 2024-04-12T03:39:58.958 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.958 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.958 DEBUG:boto:Params: {} 2024-04-12T03:39:58.958 DEBUG:boto:Token: None 2024-04-12T03:39:58.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4BPh4gR9J1Fic53mFhZ/2m9ugoQ= 2024-04-12T03:39:58.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4BPh4gR9J1Fic53mFhZ/2m9ugoQ=', 'Content-Length': '0'} 2024-04-12T03:39:58.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c85cb4e95c76d2-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:39:58.960 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.960 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.960 DEBUG:boto:Method: GET 2024-04-12T03:39:58.960 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.960 DEBUG:boto:Data: 2024-04-12T03:39:58.961 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:58.961 DEBUG:boto:Port: 8001 2024-04-12T03:39:58.961 DEBUG:boto:Params: {} 2024-04-12T03:39:58.961 DEBUG:boto:Token: None 2024-04-12T03:39:58.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Va3qJ4aNZe6iPXbQzvxYu6c7RSc= 2024-04-12T03:39:58.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Va3qJ4aNZe6iPXbQzvxYu6c7RSc=', 'Content-Length': '0'} 2024-04-12T03:39:58.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793586dbed7d542a-006618ad0e-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:58.963 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.963 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.963 DEBUG:boto:Method: GET 2024-04-12T03:39:58.963 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.963 DEBUG:boto:Data: 2024-04-12T03:39:58.963 DEBUG:boto:Headers: {} 2024-04-12T03:39:58.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:58.963 DEBUG:boto:Port: 8000 2024-04-12T03:39:58.964 DEBUG:boto:Params: {} 2024-04-12T03:39:58.964 DEBUG:boto:Token: None 2024-04-12T03:39:58.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:58 GMT /dgbpxg-60/obj00000000000000000061 2024-04-12T03:39:58.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Va3qJ4aNZe6iPXbQzvxYu6c7RSc= 2024-04-12T03:39:58.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:58 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Va3qJ4aNZe6iPXbQzvxYu6c7RSc=', 'Content-Length': '0'} 2024-04-12T03:39:58.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78f7c327c0b8e2a-006618ad0e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.006 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.006 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.006 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.007 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.007 DEBUG:boto:Method: GET 2024-04-12T03:39:59.007 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.007 DEBUG:boto:Data: 2024-04-12T03:39:59.007 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.007 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.007 DEBUG:boto:Params: {} 2024-04-12T03:39:59.007 DEBUG:boto:Token: None 2024-04-12T03:39:59.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e22d205e7a68ed-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.009 DEBUG:boto:b'dgbpxg-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.010 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.010 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.010 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.010 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.010 DEBUG:boto:Method: GET 2024-04-12T03:39:59.010 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000061 2024-04-12T03:39:59.010 DEBUG:boto:Data: 2024-04-12T03:39:59.010 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.011 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.011 DEBUG:boto:Params: {} 2024-04-12T03:39:59.011 DEBUG:boto:Token: None 2024-04-12T03:39:59.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9f387e69f06648-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.014 DEBUG:boto:b'dgbpxg-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:39:42.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:39:59.014 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.015 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.015 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.015 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.015 DEBUG:boto:Method: GET 2024-04-12T03:39:59.015 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.015 DEBUG:boto:Data: 2024-04-12T03:39:59.015 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.015 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.015 DEBUG:boto:Params: {} 2024-04-12T03:39:59.015 DEBUG:boto:Token: None 2024-04-12T03:39:59.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis= 2024-04-12T03:39:59.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis=', 'Content-Length': '0'} 2024-04-12T03:39:59.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ebc9aed25e9cd6-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.018 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.018 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.018 DEBUG:boto:Method: GET 2024-04-12T03:39:59.018 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.018 DEBUG:boto:Data: 2024-04-12T03:39:59.018 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.018 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.018 DEBUG:boto:Params: {} 2024-04-12T03:39:59.019 DEBUG:boto:Token: None 2024-04-12T03:39:59.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis= 2024-04-12T03:39:59.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis=', 'Content-Length': '0'} 2024-04-12T03:39:59.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007027ca7cc8207c1-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.021 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.021 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.021 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.021 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.021 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.021 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.021 DEBUG:boto:Data: 2024-04-12T03:39:59.021 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.021 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.021 DEBUG:boto:Params: {} 2024-04-12T03:39:59.021 DEBUG:boto:Token: None 2024-04-12T03:39:59.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hBXw36/GSIskBTx77+LQ3Yoero4= 2024-04-12T03:39:59.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hBXw36/GSIskBTx77+LQ3Yoero4=', 'Content-Length': '0'} 2024-04-12T03:39:59.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186771cffd24b0ce-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.023 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.023 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.024 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.024 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.024 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.024 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.024 DEBUG:boto:Data: 2024-04-12T03:39:59.024 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.024 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.024 DEBUG:boto:Params: {} 2024-04-12T03:39:59.024 DEBUG:boto:Token: None 2024-04-12T03:39:59.024 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hBXw36/GSIskBTx77+LQ3Yoero4= 2024-04-12T03:39:59.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hBXw36/GSIskBTx77+LQ3Yoero4=', 'Content-Length': '0'} 2024-04-12T03:39:59.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6e3335f213333c-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:39:59.026 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.027 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.027 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.027 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.027 DEBUG:boto:Method: GET 2024-04-12T03:39:59.027 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.027 DEBUG:boto:Data: 2024-04-12T03:39:59.027 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.027 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.027 DEBUG:boto:Params: {} 2024-04-12T03:39:59.027 DEBUG:boto:Token: None 2024-04-12T03:39:59.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis= 2024-04-12T03:39:59.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis=', 'Content-Length': '0'} 2024-04-12T03:39:59.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddaf62a6d45b7b7-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.030 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.030 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.030 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.030 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.030 DEBUG:boto:Method: GET 2024-04-12T03:39:59.030 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.031 DEBUG:boto:Data: 2024-04-12T03:39:59.031 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.031 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.031 DEBUG:boto:Params: {} 2024-04-12T03:39:59.031 DEBUG:boto:Token: None 2024-04-12T03:39:59.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062?versionId=null 2024-04-12T03:39:59.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis= 2024-04-12T03:39:59.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bij7Mm7j4+XNF1mL250WqfhhAis=', 'Content-Length': '0'} 2024-04-12T03:39:59.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eab010c3ef9cf59-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.033 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.033 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.033 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.034 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.034 DEBUG:boto:Method: GET 2024-04-12T03:39:59.034 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.034 DEBUG:boto:Data: 2024-04-12T03:39:59.034 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.034 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.034 DEBUG:boto:Params: {} 2024-04-12T03:39:59.034 DEBUG:boto:Token: None 2024-04-12T03:39:59.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000594de593ad007afb-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.037 DEBUG:boto:b'dgbpxg-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.037 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.037 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.037 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.037 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.037 DEBUG:boto:Method: GET 2024-04-12T03:39:59.038 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.038 DEBUG:boto:Data: 2024-04-12T03:39:59.038 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.038 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.038 DEBUG:boto:Params: {} 2024-04-12T03:39:59.038 DEBUG:boto:Token: None 2024-04-12T03:39:59.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f58952d4bf18be90-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.042 DEBUG:boto:b'dgbpxg-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.042 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.042 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.042 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.042 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.042 DEBUG:boto:Data: 2024-04-12T03:39:59.042 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.043 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.043 DEBUG:boto:Params: {} 2024-04-12T03:39:59.043 DEBUG:boto:Token: None 2024-04-12T03:39:59.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Hg+QAow0s8bO3zLTBSMCZckpZo= 2024-04-12T03:39:59.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Hg+QAow0s8bO3zLTBSMCZckpZo=', 'Content-Length': '0'} 2024-04-12T03:39:59.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef013547b4a0c475-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.045 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.045 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.045 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.045 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.045 DEBUG:boto:Data: 2024-04-12T03:39:59.045 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.045 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.046 DEBUG:boto:Params: {} 2024-04-12T03:39:59.046 DEBUG:boto:Token: None 2024-04-12T03:39:59.046 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.046 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3Hg+QAow0s8bO3zLTBSMCZckpZo= 2024-04-12T03:39:59.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3Hg+QAow0s8bO3zLTBSMCZckpZo=', 'Content-Length': '0'} 2024-04-12T03:39:59.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000718b8188b59e07aa-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:39:59.048 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.048 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.048 DEBUG:boto:Method: GET 2024-04-12T03:39:59.048 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.048 DEBUG:boto:Data: 2024-04-12T03:39:59.049 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.049 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.049 DEBUG:boto:Params: {} 2024-04-12T03:39:59.049 DEBUG:boto:Token: None 2024-04-12T03:39:59.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.049 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3lpCHnS611OxnuL7IR+lYnJ9JaQ= 2024-04-12T03:39:59.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3lpCHnS611OxnuL7IR+lYnJ9JaQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b3d67060bafbc3-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.051 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.051 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.051 DEBUG:boto:Method: GET 2024-04-12T03:39:59.051 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.051 DEBUG:boto:Data: 2024-04-12T03:39:59.051 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.052 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.052 DEBUG:boto:Params: {} 2024-04-12T03:39:59.052 DEBUG:boto:Token: None 2024-04-12T03:39:59.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000062 2024-04-12T03:39:59.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3lpCHnS611OxnuL7IR+lYnJ9JaQ= 2024-04-12T03:39:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3lpCHnS611OxnuL7IR+lYnJ9JaQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d67406295cd0e7e-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.054 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.054 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.054 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.054 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.054 DEBUG:boto:Method: GET 2024-04-12T03:39:59.054 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.054 DEBUG:boto:Data: 2024-04-12T03:39:59.054 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.055 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.055 DEBUG:boto:Params: {} 2024-04-12T03:39:59.055 DEBUG:boto:Token: None 2024-04-12T03:39:59.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f017636b5ea687fb-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.057 DEBUG:boto:b'dgbpxg-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.058 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.058 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.058 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.058 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.058 DEBUG:boto:Method: GET 2024-04-12T03:39:59.058 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000062 2024-04-12T03:39:59.058 DEBUG:boto:Data: 2024-04-12T03:39:59.058 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.059 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.059 DEBUG:boto:Params: {} 2024-04-12T03:39:59.059 DEBUG:boto:Token: None 2024-04-12T03:39:59.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa16b463e8f7e91-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.062 DEBUG:boto:b'dgbpxg-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:39:43.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-12T03:39:59.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.063 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.063 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.063 DEBUG:boto:Method: GET 2024-04-12T03:39:59.063 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.063 DEBUG:boto:Data: 2024-04-12T03:39:59.064 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.064 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.064 DEBUG:boto:Params: {} 2024-04-12T03:39:59.064 DEBUG:boto:Token: None 2024-04-12T03:39:59.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI= 2024-04-12T03:39:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI=', 'Content-Length': '0'} 2024-04-12T03:39:59.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002802963c63a10f53-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.066 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.066 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.066 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.067 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.067 DEBUG:boto:Method: GET 2024-04-12T03:39:59.067 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.067 DEBUG:boto:Data: 2024-04-12T03:39:59.067 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.067 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.067 DEBUG:boto:Params: {} 2024-04-12T03:39:59.067 DEBUG:boto:Token: None 2024-04-12T03:39:59.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI= 2024-04-12T03:39:59.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI=', 'Content-Length': '0'} 2024-04-12T03:39:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419bfb7548c4d8b6-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.069 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.069 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.070 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.070 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.070 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.070 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.070 DEBUG:boto:Data: 2024-04-12T03:39:59.070 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.070 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.070 DEBUG:boto:Params: {} 2024-04-12T03:39:59.070 DEBUG:boto:Token: None 2024-04-12T03:39:59.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4mjjuavW1kKJsVLX/8qXT1DzjAM= 2024-04-12T03:39:59.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4mjjuavW1kKJsVLX/8qXT1DzjAM=', 'Content-Length': '0'} 2024-04-12T03:39:59.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ccfb128fbb65ef-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.073 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.073 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.073 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.073 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.073 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.073 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.073 DEBUG:boto:Data: 2024-04-12T03:39:59.073 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.073 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.073 DEBUG:boto:Params: {} 2024-04-12T03:39:59.073 DEBUG:boto:Token: None 2024-04-12T03:39:59.074 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4mjjuavW1kKJsVLX/8qXT1DzjAM= 2024-04-12T03:39:59.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4mjjuavW1kKJsVLX/8qXT1DzjAM=', 'Content-Length': '0'} 2024-04-12T03:39:59.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003feeb666a25833a6-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-12T03:39:59.076 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.076 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.076 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.076 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.076 DEBUG:boto:Method: GET 2024-04-12T03:39:59.076 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.076 DEBUG:boto:Data: 2024-04-12T03:39:59.076 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.076 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.077 DEBUG:boto:Params: {} 2024-04-12T03:39:59.077 DEBUG:boto:Token: None 2024-04-12T03:39:59.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.077 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI= 2024-04-12T03:39:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI=', 'Content-Length': '0'} 2024-04-12T03:39:59.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081529d85287ee7a3-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.079 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.079 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.079 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.079 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.079 DEBUG:boto:Method: GET 2024-04-12T03:39:59.079 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.079 DEBUG:boto:Data: 2024-04-12T03:39:59.079 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.079 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.080 DEBUG:boto:Params: {} 2024-04-12T03:39:59.080 DEBUG:boto:Token: None 2024-04-12T03:39:59.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065?versionId=null 2024-04-12T03:39:59.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI= 2024-04-12T03:39:59.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SXpOYnVs0uLYOmok92HcbXBX/TI=', 'Content-Length': '0'} 2024-04-12T03:39:59.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4c78aeede04797-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.082 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.082 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.082 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.082 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.082 DEBUG:boto:Method: GET 2024-04-12T03:39:59.082 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.082 DEBUG:boto:Data: 2024-04-12T03:39:59.082 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.082 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.083 DEBUG:boto:Params: {} 2024-04-12T03:39:59.083 DEBUG:boto:Token: None 2024-04-12T03:39:59.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e0c1cbca257e3d-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.126 DEBUG:boto:b'dgbpxg-60obj000000000000000000651000falseobj000000000000000000652024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.127 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.127 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.127 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.127 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.127 DEBUG:boto:Method: GET 2024-04-12T03:39:59.127 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.127 DEBUG:boto:Data: 2024-04-12T03:39:59.127 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.127 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.127 DEBUG:boto:Params: {} 2024-04-12T03:39:59.128 DEBUG:boto:Token: None 2024-04-12T03:39:59.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a498f731dada9a1-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.132 DEBUG:boto:b'dgbpxg-60obj000000000000000000651000falseobj000000000000000000652024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.132 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.132 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.132 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.132 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.132 DEBUG:boto:Data: 2024-04-12T03:39:59.132 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.132 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.133 DEBUG:boto:Params: {} 2024-04-12T03:39:59.133 DEBUG:boto:Token: None 2024-04-12T03:39:59.133 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzD9pLIqlDWO4pjWLPag29oMxw0= 2024-04-12T03:39:59.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzD9pLIqlDWO4pjWLPag29oMxw0=', 'Content-Length': '0'} 2024-04-12T03:39:59.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c343f1883647738-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.135 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.135 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.135 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.135 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.135 DEBUG:boto:Data: 2024-04-12T03:39:59.135 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.135 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.135 DEBUG:boto:Params: {} 2024-04-12T03:39:59.135 DEBUG:boto:Token: None 2024-04-12T03:39:59.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzD9pLIqlDWO4pjWLPag29oMxw0= 2024-04-12T03:39:59.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzD9pLIqlDWO4pjWLPag29oMxw0=', 'Content-Length': '0'} 2024-04-12T03:39:59.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d245c4127b9a5fb-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-12T03:39:59.137 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.138 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.138 DEBUG:boto:Method: GET 2024-04-12T03:39:59.138 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.138 DEBUG:boto:Data: 2024-04-12T03:39:59.138 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.138 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.138 DEBUG:boto:Params: {} 2024-04-12T03:39:59.138 DEBUG:boto:Token: None 2024-04-12T03:39:59.138 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YdpLjWqdixPCOp4GBcVsXYVMHLY= 2024-04-12T03:39:59.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YdpLjWqdixPCOp4GBcVsXYVMHLY=', 'Content-Length': '0'} 2024-04-12T03:39:59.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be37fd07a2d0f36-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.140 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.140 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.140 DEBUG:boto:Method: GET 2024-04-12T03:39:59.140 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.141 DEBUG:boto:Data: 2024-04-12T03:39:59.141 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.141 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.141 DEBUG:boto:Params: {} 2024-04-12T03:39:59.141 DEBUG:boto:Token: None 2024-04-12T03:39:59.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000065 2024-04-12T03:39:59.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YdpLjWqdixPCOp4GBcVsXYVMHLY= 2024-04-12T03:39:59.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YdpLjWqdixPCOp4GBcVsXYVMHLY=', 'Content-Length': '0'} 2024-04-12T03:39:59.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3084e13806e3dc-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.186 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.186 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.187 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.187 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.187 DEBUG:boto:Method: GET 2024-04-12T03:39:59.187 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.187 DEBUG:boto:Data: 2024-04-12T03:39:59.187 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.187 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.187 DEBUG:boto:Params: {} 2024-04-12T03:39:59.187 DEBUG:boto:Token: None 2024-04-12T03:39:59.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b60d851b8cb660-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.190 DEBUG:boto:b'dgbpxg-60obj000000000000000000651000falseobj000000000000000000652024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.190 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.190 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.191 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.191 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.191 DEBUG:boto:Method: GET 2024-04-12T03:39:59.191 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000065 2024-04-12T03:39:59.191 DEBUG:boto:Data: 2024-04-12T03:39:59.191 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.191 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.191 DEBUG:boto:Params: {} 2024-04-12T03:39:59.191 DEBUG:boto:Token: None 2024-04-12T03:39:59.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b798b8de38cc159-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.195 DEBUG:boto:b'dgbpxg-60obj000000000000000000651000falseobj000000000000000000652024-04-12T03:39:43.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:39:59.196 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.196 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.196 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.196 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.196 DEBUG:boto:Method: GET 2024-04-12T03:39:59.196 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.196 DEBUG:boto:Data: 2024-04-12T03:39:59.196 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.196 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.196 DEBUG:boto:Params: {} 2024-04-12T03:39:59.197 DEBUG:boto:Token: None 2024-04-12T03:39:59.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU= 2024-04-12T03:39:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU=', 'Content-Length': '0'} 2024-04-12T03:39:59.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e5e83222475250-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.199 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.199 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.199 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.199 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.199 DEBUG:boto:Method: GET 2024-04-12T03:39:59.199 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.199 DEBUG:boto:Data: 2024-04-12T03:39:59.200 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.200 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.200 DEBUG:boto:Params: {} 2024-04-12T03:39:59.200 DEBUG:boto:Token: None 2024-04-12T03:39:59.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU= 2024-04-12T03:39:59.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU=', 'Content-Length': '0'} 2024-04-12T03:39:59.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000961ce0d42b68207-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.202 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.202 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.202 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.202 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.203 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.203 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.203 DEBUG:boto:Data: 2024-04-12T03:39:59.203 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.203 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.203 DEBUG:boto:Params: {} 2024-04-12T03:39:59.203 DEBUG:boto:Token: None 2024-04-12T03:39:59.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4IzCnJbRVDnxAi/NiE/2an36M5E= 2024-04-12T03:39:59.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4IzCnJbRVDnxAi/NiE/2an36M5E=', 'Content-Length': '0'} 2024-04-12T03:39:59.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e846f3617337765-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.205 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.205 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.205 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.205 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.205 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.205 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.205 DEBUG:boto:Data: 2024-04-12T03:39:59.206 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.206 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.206 DEBUG:boto:Params: {} 2024-04-12T03:39:59.206 DEBUG:boto:Token: None 2024-04-12T03:39:59.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.206 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4IzCnJbRVDnxAi/NiE/2an36M5E= 2024-04-12T03:39:59.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4IzCnJbRVDnxAi/NiE/2an36M5E=', 'Content-Length': '0'} 2024-04-12T03:39:59.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c125f46c410f41d3-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:39:59.208 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.208 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.208 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.208 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.209 DEBUG:boto:Method: GET 2024-04-12T03:39:59.209 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.209 DEBUG:boto:Data: 2024-04-12T03:39:59.209 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.209 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.209 DEBUG:boto:Params: {} 2024-04-12T03:39:59.209 DEBUG:boto:Token: None 2024-04-12T03:39:59.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.209 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU= 2024-04-12T03:39:59.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU=', 'Content-Length': '0'} 2024-04-12T03:39:59.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83b25718545b810-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.254 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.254 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.255 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.255 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.255 DEBUG:boto:Method: GET 2024-04-12T03:39:59.255 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.255 DEBUG:boto:Data: 2024-04-12T03:39:59.255 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.255 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.255 DEBUG:boto:Params: {} 2024-04-12T03:39:59.255 DEBUG:boto:Token: None 2024-04-12T03:39:59.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076?versionId=null 2024-04-12T03:39:59.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU= 2024-04-12T03:39:59.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wIbvjQ305uGrCRoaN5k+jNYu9AU=', 'Content-Length': '0'} 2024-04-12T03:39:59.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc05f770e75c2e4f-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.258 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.258 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.258 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.258 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.258 DEBUG:boto:Method: GET 2024-04-12T03:39:59.258 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.258 DEBUG:boto:Data: 2024-04-12T03:39:59.258 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.258 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.259 DEBUG:boto:Params: {} 2024-04-12T03:39:59.259 DEBUG:boto:Token: None 2024-04-12T03:39:59.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b056cc4763c5871-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.261 DEBUG:boto:b'dgbpxg-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.261 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.262 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.262 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.262 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.262 DEBUG:boto:Method: GET 2024-04-12T03:39:59.262 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.262 DEBUG:boto:Data: 2024-04-12T03:39:59.262 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.262 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.262 DEBUG:boto:Params: {} 2024-04-12T03:39:59.263 DEBUG:boto:Token: None 2024-04-12T03:39:59.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074543183b5ae9bb9-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.266 DEBUG:boto:b'dgbpxg-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.267 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.267 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.267 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.267 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.267 DEBUG:boto:Data: 2024-04-12T03:39:59.267 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.267 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.267 DEBUG:boto:Params: {} 2024-04-12T03:39:59.267 DEBUG:boto:Token: None 2024-04-12T03:39:59.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jab742QWw15ssonfe+g41EwsKNE= 2024-04-12T03:39:59.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jab742QWw15ssonfe+g41EwsKNE=', 'Content-Length': '0'} 2024-04-12T03:39:59.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8534751459c824-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.269 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.269 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.269 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.270 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.270 DEBUG:boto:Data: 2024-04-12T03:39:59.270 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.270 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.270 DEBUG:boto:Params: {} 2024-04-12T03:39:59.270 DEBUG:boto:Token: None 2024-04-12T03:39:59.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.270 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jab742QWw15ssonfe+g41EwsKNE= 2024-04-12T03:39:59.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jab742QWw15ssonfe+g41EwsKNE=', 'Content-Length': '0'} 2024-04-12T03:39:59.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc148cb901f6ba4c-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:39:59.272 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.272 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.273 DEBUG:boto:Method: GET 2024-04-12T03:39:59.273 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.273 DEBUG:boto:Data: 2024-04-12T03:39:59.273 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.273 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.273 DEBUG:boto:Params: {} 2024-04-12T03:39:59.273 DEBUG:boto:Token: None 2024-04-12T03:39:59.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j9GsxNHJyNwkFL1bdG+R2ITBUBg= 2024-04-12T03:39:59.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j9GsxNHJyNwkFL1bdG+R2ITBUBg=', 'Content-Length': '0'} 2024-04-12T03:39:59.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e4b0e2fca1acd5-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.275 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.275 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.275 DEBUG:boto:Method: GET 2024-04-12T03:39:59.275 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.275 DEBUG:boto:Data: 2024-04-12T03:39:59.276 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.276 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.276 DEBUG:boto:Params: {} 2024-04-12T03:39:59.276 DEBUG:boto:Token: None 2024-04-12T03:39:59.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000076 2024-04-12T03:39:59.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j9GsxNHJyNwkFL1bdG+R2ITBUBg= 2024-04-12T03:39:59.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j9GsxNHJyNwkFL1bdG+R2ITBUBg=', 'Content-Length': '0'} 2024-04-12T03:39:59.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098654f8ae4ff870a-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.318 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.318 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.319 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.319 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.319 DEBUG:boto:Method: GET 2024-04-12T03:39:59.319 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.319 DEBUG:boto:Data: 2024-04-12T03:39:59.319 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.319 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.319 DEBUG:boto:Params: {} 2024-04-12T03:39:59.319 DEBUG:boto:Token: None 2024-04-12T03:39:59.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc439383a01d995-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.322 DEBUG:boto:b'dgbpxg-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.322 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.322 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.322 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.323 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.323 DEBUG:boto:Method: GET 2024-04-12T03:39:59.323 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000076 2024-04-12T03:39:59.323 DEBUG:boto:Data: 2024-04-12T03:39:59.323 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.323 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.323 DEBUG:boto:Params: {} 2024-04-12T03:39:59.323 DEBUG:boto:Token: None 2024-04-12T03:39:59.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2abd4c193af642-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.327 DEBUG:boto:b'dgbpxg-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:39:39.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:39:59.327 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.327 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.327 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.328 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.328 DEBUG:boto:Method: GET 2024-04-12T03:39:59.328 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.328 DEBUG:boto:Data: 2024-04-12T03:39:59.328 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.328 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.328 DEBUG:boto:Params: {} 2024-04-12T03:39:59.328 DEBUG:boto:Token: None 2024-04-12T03:39:59.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU= 2024-04-12T03:39:59.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU=', 'Content-Length': '0'} 2024-04-12T03:39:59.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092179415cb9a71c1-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.331 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.331 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.331 DEBUG:boto:Method: GET 2024-04-12T03:39:59.331 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.331 DEBUG:boto:Data: 2024-04-12T03:39:59.331 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.331 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.331 DEBUG:boto:Params: {} 2024-04-12T03:39:59.332 DEBUG:boto:Token: None 2024-04-12T03:39:59.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU= 2024-04-12T03:39:59.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU=', 'Content-Length': '0'} 2024-04-12T03:39:59.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008354d78960d3bada-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.334 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.334 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.334 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.334 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.334 DEBUG:boto:Data: 2024-04-12T03:39:59.334 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.335 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.335 DEBUG:boto:Params: {} 2024-04-12T03:39:59.335 DEBUG:boto:Token: None 2024-04-12T03:39:59.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:51dDwdEGmAdWCQHOCWW+OMqifMQ= 2024-04-12T03:39:59.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:51dDwdEGmAdWCQHOCWW+OMqifMQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7ba056becaa614-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.337 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.337 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.337 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.337 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.337 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.337 DEBUG:boto:Data: 2024-04-12T03:39:59.338 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.338 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.338 DEBUG:boto:Params: {} 2024-04-12T03:39:59.338 DEBUG:boto:Token: None 2024-04-12T03:39:59.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:51dDwdEGmAdWCQHOCWW+OMqifMQ= 2024-04-12T03:39:59.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:51dDwdEGmAdWCQHOCWW+OMqifMQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065351d9cef839670-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:39:59.340 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.340 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.340 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.340 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.340 DEBUG:boto:Method: GET 2024-04-12T03:39:59.340 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.340 DEBUG:boto:Data: 2024-04-12T03:39:59.341 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.341 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.341 DEBUG:boto:Params: {} 2024-04-12T03:39:59.341 DEBUG:boto:Token: None 2024-04-12T03:39:59.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU= 2024-04-12T03:39:59.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU=', 'Content-Length': '0'} 2024-04-12T03:39:59.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58bedc61521d4a4-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.343 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.343 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.343 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.343 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.343 DEBUG:boto:Method: GET 2024-04-12T03:39:59.343 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.343 DEBUG:boto:Data: 2024-04-12T03:39:59.343 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.344 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.344 DEBUG:boto:Params: {} 2024-04-12T03:39:59.344 DEBUG:boto:Token: None 2024-04-12T03:39:59.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133?versionId=null 2024-04-12T03:39:59.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU= 2024-04-12T03:39:59.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SVTKmYrx2fCaE4HhIeUeEUYWizU=', 'Content-Length': '0'} 2024-04-12T03:39:59.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c91c76c3b37244-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.346 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.346 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.346 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.346 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.346 DEBUG:boto:Method: GET 2024-04-12T03:39:59.346 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.347 DEBUG:boto:Data: 2024-04-12T03:39:59.347 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.347 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.347 DEBUG:boto:Params: {} 2024-04-12T03:39:59.347 DEBUG:boto:Token: None 2024-04-12T03:39:59.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d8e207ce5e8504-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.390 DEBUG:boto:b'dgbpxg-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.391 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.391 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.391 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.391 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.391 DEBUG:boto:Method: GET 2024-04-12T03:39:59.391 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.391 DEBUG:boto:Data: 2024-04-12T03:39:59.391 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.391 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.391 DEBUG:boto:Params: {} 2024-04-12T03:39:59.392 DEBUG:boto:Token: None 2024-04-12T03:39:59.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d471af6395b02f8-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.395 DEBUG:boto:b'dgbpxg-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.396 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.396 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.396 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.396 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.396 DEBUG:boto:Data: 2024-04-12T03:39:59.396 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.396 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.396 DEBUG:boto:Params: {} 2024-04-12T03:39:59.396 DEBUG:boto:Token: None 2024-04-12T03:39:59.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893w9a7Sjq0re1bBHmT5xlOtl8o= 2024-04-12T03:39:59.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893w9a7Sjq0re1bBHmT5xlOtl8o=', 'Content-Length': '0'} 2024-04-12T03:39:59.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e75942998cec372-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.398 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.398 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.398 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.398 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.399 DEBUG:boto:Data: 2024-04-12T03:39:59.399 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.399 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.399 DEBUG:boto:Params: {} 2024-04-12T03:39:59.399 DEBUG:boto:Token: None 2024-04-12T03:39:59.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:893w9a7Sjq0re1bBHmT5xlOtl8o= 2024-04-12T03:39:59.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:893w9a7Sjq0re1bBHmT5xlOtl8o=', 'Content-Length': '0'} 2024-04-12T03:39:59.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5951cda4ae0ee1b-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:39:59.401 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.401 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.401 DEBUG:boto:Method: GET 2024-04-12T03:39:59.401 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.401 DEBUG:boto:Data: 2024-04-12T03:39:59.401 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.401 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.401 DEBUG:boto:Params: {} 2024-04-12T03:39:59.402 DEBUG:boto:Token: None 2024-04-12T03:39:59.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmKzScRIWHTbFmDorMY6+7zFt2k= 2024-04-12T03:39:59.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmKzScRIWHTbFmDorMY6+7zFt2k=', 'Content-Length': '0'} 2024-04-12T03:39:59.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f076041fe645d06-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.404 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.404 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.404 DEBUG:boto:Method: GET 2024-04-12T03:39:59.404 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.404 DEBUG:boto:Data: 2024-04-12T03:39:59.404 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.404 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.404 DEBUG:boto:Params: {} 2024-04-12T03:39:59.404 DEBUG:boto:Token: None 2024-04-12T03:39:59.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000133 2024-04-12T03:39:59.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmKzScRIWHTbFmDorMY6+7zFt2k= 2024-04-12T03:39:59.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmKzScRIWHTbFmDorMY6+7zFt2k=', 'Content-Length': '0'} 2024-04-12T03:39:59.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e775cd9a3062efb-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.446 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.447 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.447 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.447 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.447 DEBUG:boto:Method: GET 2024-04-12T03:39:59.447 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.447 DEBUG:boto:Data: 2024-04-12T03:39:59.447 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.447 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.447 DEBUG:boto:Params: {} 2024-04-12T03:39:59.447 DEBUG:boto:Token: None 2024-04-12T03:39:59.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73a386c21b64bf0-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.450 DEBUG:boto:b'dgbpxg-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.450 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.450 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.450 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.450 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.451 DEBUG:boto:Method: GET 2024-04-12T03:39:59.451 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000133 2024-04-12T03:39:59.451 DEBUG:boto:Data: 2024-04-12T03:39:59.451 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.451 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.451 DEBUG:boto:Params: {} 2024-04-12T03:39:59.451 DEBUG:boto:Token: None 2024-04-12T03:39:59.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025961e552eeaed6b-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.455 DEBUG:boto:b'dgbpxg-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:39:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:39:59.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.455 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.455 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.456 DEBUG:boto:Method: GET 2024-04-12T03:39:59.456 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.456 DEBUG:boto:Data: 2024-04-12T03:39:59.456 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.456 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.456 DEBUG:boto:Params: {} 2024-04-12T03:39:59.456 DEBUG:boto:Token: None 2024-04-12T03:39:59.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8= 2024-04-12T03:39:59.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8=', 'Content-Length': '0'} 2024-04-12T03:39:59.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044bde5718b4eb46a-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.458 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.458 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.458 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.459 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.459 DEBUG:boto:Method: GET 2024-04-12T03:39:59.459 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.459 DEBUG:boto:Data: 2024-04-12T03:39:59.459 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.459 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.459 DEBUG:boto:Params: {} 2024-04-12T03:39:59.459 DEBUG:boto:Token: None 2024-04-12T03:39:59.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8= 2024-04-12T03:39:59.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8=', 'Content-Length': '0'} 2024-04-12T03:39:59.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89651cba7518170-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.461 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.461 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.461 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.462 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.462 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.462 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.462 DEBUG:boto:Data: 2024-04-12T03:39:59.462 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.462 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.462 DEBUG:boto:Params: {} 2024-04-12T03:39:59.462 DEBUG:boto:Token: None 2024-04-12T03:39:59.462 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E8HElxtGuD+ymzE+92ilseAdR8s= 2024-04-12T03:39:59.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E8HElxtGuD+ymzE+92ilseAdR8s=', 'Content-Length': '0'} 2024-04-12T03:39:59.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f19c2e38c2d1166-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.464 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.464 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.464 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.464 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.464 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.465 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.465 DEBUG:boto:Data: 2024-04-12T03:39:59.465 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.465 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.465 DEBUG:boto:Params: {} 2024-04-12T03:39:59.465 DEBUG:boto:Token: None 2024-04-12T03:39:59.465 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E8HElxtGuD+ymzE+92ilseAdR8s= 2024-04-12T03:39:59.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E8HElxtGuD+ymzE+92ilseAdR8s=', 'Content-Length': '0'} 2024-04-12T03:39:59.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bf489135713dcb-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:39:59.467 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.467 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.467 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.468 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.468 DEBUG:boto:Method: GET 2024-04-12T03:39:59.468 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.468 DEBUG:boto:Data: 2024-04-12T03:39:59.468 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.468 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.468 DEBUG:boto:Params: {} 2024-04-12T03:39:59.468 DEBUG:boto:Token: None 2024-04-12T03:39:59.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8= 2024-04-12T03:39:59.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8=', 'Content-Length': '0'} 2024-04-12T03:39:59.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d16641dbdb45819-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.470 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.470 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.471 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.471 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.471 DEBUG:boto:Method: GET 2024-04-12T03:39:59.471 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.471 DEBUG:boto:Data: 2024-04-12T03:39:59.471 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.471 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.471 DEBUG:boto:Params: {} 2024-04-12T03:39:59.471 DEBUG:boto:Token: None 2024-04-12T03:39:59.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162?versionId=null 2024-04-12T03:39:59.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8= 2024-04-12T03:39:59.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKwzL3SCG99d/WVW7bBaZf7Hxo8=', 'Content-Length': '0'} 2024-04-12T03:39:59.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46094240fbc049d-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.514 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.514 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.515 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.515 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.515 DEBUG:boto:Method: GET 2024-04-12T03:39:59.515 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.515 DEBUG:boto:Data: 2024-04-12T03:39:59.515 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.515 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.515 DEBUG:boto:Params: {} 2024-04-12T03:39:59.515 DEBUG:boto:Token: None 2024-04-12T03:39:59.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c7b6a89e34b6d9-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.518 DEBUG:boto:b'dgbpxg-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.518 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.519 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.519 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.519 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.519 DEBUG:boto:Method: GET 2024-04-12T03:39:59.519 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.519 DEBUG:boto:Data: 2024-04-12T03:39:59.519 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.519 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.519 DEBUG:boto:Params: {} 2024-04-12T03:39:59.520 DEBUG:boto:Token: None 2024-04-12T03:39:59.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000298938d139be0d50-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.523 DEBUG:boto:b'dgbpxg-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.524 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.524 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.524 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.524 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.524 DEBUG:boto:Data: 2024-04-12T03:39:59.524 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.524 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.525 DEBUG:boto:Params: {} 2024-04-12T03:39:59.525 DEBUG:boto:Token: None 2024-04-12T03:39:59.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GpnZzDsIApoRtXu9Tj36sOLygfA= 2024-04-12T03:39:59.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GpnZzDsIApoRtXu9Tj36sOLygfA=', 'Content-Length': '0'} 2024-04-12T03:39:59.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710c4a2a7782ebfc-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.527 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.527 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.527 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.527 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.527 DEBUG:boto:Data: 2024-04-12T03:39:59.527 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.527 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.527 DEBUG:boto:Params: {} 2024-04-12T03:39:59.528 DEBUG:boto:Token: None 2024-04-12T03:39:59.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GpnZzDsIApoRtXu9Tj36sOLygfA= 2024-04-12T03:39:59.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GpnZzDsIApoRtXu9Tj36sOLygfA=', 'Content-Length': '0'} 2024-04-12T03:39:59.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c287b9ddb1c29d-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:39:59.530 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.530 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.530 DEBUG:boto:Method: GET 2024-04-12T03:39:59.530 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.530 DEBUG:boto:Data: 2024-04-12T03:39:59.531 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.531 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.531 DEBUG:boto:Params: {} 2024-04-12T03:39:59.531 DEBUG:boto:Token: None 2024-04-12T03:39:59.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKS5rBxXUkLy+S8qIEc9se1FeKs= 2024-04-12T03:39:59.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKS5rBxXUkLy+S8qIEc9se1FeKs=', 'Content-Length': '0'} 2024-04-12T03:39:59.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ba0e66e92734a8-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.574 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.574 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.575 DEBUG:boto:Method: GET 2024-04-12T03:39:59.575 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.575 DEBUG:boto:Data: 2024-04-12T03:39:59.575 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.575 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.575 DEBUG:boto:Params: {} 2024-04-12T03:39:59.575 DEBUG:boto:Token: None 2024-04-12T03:39:59.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000162 2024-04-12T03:39:59.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKS5rBxXUkLy+S8qIEc9se1FeKs= 2024-04-12T03:39:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKS5rBxXUkLy+S8qIEc9se1FeKs=', 'Content-Length': '0'} 2024-04-12T03:39:59.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823d2b1072b9f3c1-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.577 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.577 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.578 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.578 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.578 DEBUG:boto:Method: GET 2024-04-12T03:39:59.578 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.578 DEBUG:boto:Data: 2024-04-12T03:39:59.578 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.578 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.578 DEBUG:boto:Params: {} 2024-04-12T03:39:59.578 DEBUG:boto:Token: None 2024-04-12T03:39:59.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0a3da636836db6-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.581 DEBUG:boto:b'dgbpxg-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.581 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.581 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.581 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.581 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.582 DEBUG:boto:Method: GET 2024-04-12T03:39:59.582 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000162 2024-04-12T03:39:59.582 DEBUG:boto:Data: 2024-04-12T03:39:59.582 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.582 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.582 DEBUG:boto:Params: {} 2024-04-12T03:39:59.582 DEBUG:boto:Token: None 2024-04-12T03:39:59.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c47f1332d391be23-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.586 DEBUG:boto:b'dgbpxg-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:39:42.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:39:59.586 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.587 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.587 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.587 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.587 DEBUG:boto:Method: GET 2024-04-12T03:39:59.587 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.587 DEBUG:boto:Data: 2024-04-12T03:39:59.587 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.587 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.587 DEBUG:boto:Params: {} 2024-04-12T03:39:59.587 DEBUG:boto:Token: None 2024-04-12T03:39:59.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo= 2024-04-12T03:39:59.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo=', 'Content-Length': '0'} 2024-04-12T03:39:59.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc27e8416f92e9c-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.590 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.590 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.590 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.590 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.590 DEBUG:boto:Method: GET 2024-04-12T03:39:59.591 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.591 DEBUG:boto:Data: 2024-04-12T03:39:59.591 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.591 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.591 DEBUG:boto:Params: {} 2024-04-12T03:39:59.591 DEBUG:boto:Token: None 2024-04-12T03:39:59.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo= 2024-04-12T03:39:59.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo=', 'Content-Length': '0'} 2024-04-12T03:39:59.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8736d8ee271a951-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.594 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.594 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.594 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.594 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.594 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.594 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.594 DEBUG:boto:Data: 2024-04-12T03:39:59.594 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.595 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.595 DEBUG:boto:Params: {} 2024-04-12T03:39:59.595 DEBUG:boto:Token: None 2024-04-12T03:39:59.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E4fPGl+7dsjxfsoSgIhMVg4C/rQ= 2024-04-12T03:39:59.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E4fPGl+7dsjxfsoSgIhMVg4C/rQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc6ec47bc699464-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.597 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.597 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.597 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.597 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.597 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.597 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.597 DEBUG:boto:Data: 2024-04-12T03:39:59.597 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.598 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.598 DEBUG:boto:Params: {} 2024-04-12T03:39:59.598 DEBUG:boto:Token: None 2024-04-12T03:39:59.598 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E4fPGl+7dsjxfsoSgIhMVg4C/rQ= 2024-04-12T03:39:59.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E4fPGl+7dsjxfsoSgIhMVg4C/rQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034d3b6579b823546-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:39:59.600 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.600 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.600 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.600 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.601 DEBUG:boto:Method: GET 2024-04-12T03:39:59.601 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.601 DEBUG:boto:Data: 2024-04-12T03:39:59.601 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.601 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.601 DEBUG:boto:Params: {} 2024-04-12T03:39:59.601 DEBUG:boto:Token: None 2024-04-12T03:39:59.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo= 2024-04-12T03:39:59.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo=', 'Content-Length': '0'} 2024-04-12T03:39:59.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003841caa5edb842fc-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.603 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.603 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.604 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.604 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.604 DEBUG:boto:Method: GET 2024-04-12T03:39:59.604 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.604 DEBUG:boto:Data: 2024-04-12T03:39:59.604 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.604 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.604 DEBUG:boto:Params: {} 2024-04-12T03:39:59.604 DEBUG:boto:Token: None 2024-04-12T03:39:59.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271?versionId=null 2024-04-12T03:39:59.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo= 2024-04-12T03:39:59.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jNZMzBIYkZySsTnif5pd49LF8zo=', 'Content-Length': '0'} 2024-04-12T03:39:59.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077861d242dafdce5-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.606 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.606 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.607 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.607 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.607 DEBUG:boto:Method: GET 2024-04-12T03:39:59.607 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.607 DEBUG:boto:Data: 2024-04-12T03:39:59.607 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.607 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.607 DEBUG:boto:Params: {} 2024-04-12T03:39:59.607 DEBUG:boto:Token: None 2024-04-12T03:39:59.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcea78e297a6f02e-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.610 DEBUG:boto:b'dgbpxg-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.610 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.610 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.610 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.610 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.610 DEBUG:boto:Method: GET 2024-04-12T03:39:59.611 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.611 DEBUG:boto:Data: 2024-04-12T03:39:59.611 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.611 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.611 DEBUG:boto:Params: {} 2024-04-12T03:39:59.611 DEBUG:boto:Token: None 2024-04-12T03:39:59.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b4348966fc446e-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.617 DEBUG:boto:b'dgbpxg-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.617 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.618 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.618 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.618 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.618 DEBUG:boto:Data: 2024-04-12T03:39:59.618 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.618 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.618 DEBUG:boto:Params: {} 2024-04-12T03:39:59.618 DEBUG:boto:Token: None 2024-04-12T03:39:59.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nPqTrx/LywgjHa7uOdw+FmCqYg0= 2024-04-12T03:39:59.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nPqTrx/LywgjHa7uOdw+FmCqYg0=', 'Content-Length': '0'} 2024-04-12T03:39:59.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1b39fc2c8dc9b4-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.620 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.620 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.620 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.621 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.621 DEBUG:boto:Data: 2024-04-12T03:39:59.621 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.621 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.621 DEBUG:boto:Params: {} 2024-04-12T03:39:59.621 DEBUG:boto:Token: None 2024-04-12T03:39:59.621 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nPqTrx/LywgjHa7uOdw+FmCqYg0= 2024-04-12T03:39:59.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nPqTrx/LywgjHa7uOdw+FmCqYg0=', 'Content-Length': '0'} 2024-04-12T03:39:59.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000405a5e2cd85146f7-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:39:59.623 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.623 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.623 DEBUG:boto:Method: GET 2024-04-12T03:39:59.624 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.624 DEBUG:boto:Data: 2024-04-12T03:39:59.624 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.624 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.624 DEBUG:boto:Params: {} 2024-04-12T03:39:59.624 DEBUG:boto:Token: None 2024-04-12T03:39:59.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ptVZ9f8c6pv0cGyJqmFFMDjF0PQ= 2024-04-12T03:39:59.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ptVZ9f8c6pv0cGyJqmFFMDjF0PQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f366298ca2ceb3e9-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.626 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.626 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.626 DEBUG:boto:Method: GET 2024-04-12T03:39:59.626 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.627 DEBUG:boto:Data: 2024-04-12T03:39:59.627 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.627 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.627 DEBUG:boto:Params: {} 2024-04-12T03:39:59.627 DEBUG:boto:Token: None 2024-04-12T03:39:59.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000271 2024-04-12T03:39:59.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ptVZ9f8c6pv0cGyJqmFFMDjF0PQ= 2024-04-12T03:39:59.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ptVZ9f8c6pv0cGyJqmFFMDjF0PQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697adf8eddec80c6-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.670 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.670 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.671 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.671 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.671 DEBUG:boto:Method: GET 2024-04-12T03:39:59.671 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.671 DEBUG:boto:Data: 2024-04-12T03:39:59.671 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.671 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.671 DEBUG:boto:Params: {} 2024-04-12T03:39:59.671 DEBUG:boto:Token: None 2024-04-12T03:39:59.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac76bb7a1c445d5b-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.674 DEBUG:boto:b'dgbpxg-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.674 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.674 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.675 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.675 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.675 DEBUG:boto:Method: GET 2024-04-12T03:39:59.675 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000271 2024-04-12T03:39:59.675 DEBUG:boto:Data: 2024-04-12T03:39:59.675 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.675 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.675 DEBUG:boto:Params: {} 2024-04-12T03:39:59.675 DEBUG:boto:Token: None 2024-04-12T03:39:59.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c682f414e4c46788-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.679 DEBUG:boto:b'dgbpxg-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:39:42.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:39:59.679 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.680 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.680 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.680 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.680 DEBUG:boto:Method: GET 2024-04-12T03:39:59.680 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.680 DEBUG:boto:Data: 2024-04-12T03:39:59.680 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.680 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.680 DEBUG:boto:Params: {} 2024-04-12T03:39:59.680 DEBUG:boto:Token: None 2024-04-12T03:39:59.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M= 2024-04-12T03:39:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M=', 'Content-Length': '0'} 2024-04-12T03:39:59.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a6b10db8ca6577-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.683 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.683 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.683 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.683 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.683 DEBUG:boto:Method: GET 2024-04-12T03:39:59.683 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.683 DEBUG:boto:Data: 2024-04-12T03:39:59.683 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.684 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.684 DEBUG:boto:Params: {} 2024-04-12T03:39:59.684 DEBUG:boto:Token: None 2024-04-12T03:39:59.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M= 2024-04-12T03:39:59.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M=', 'Content-Length': '0'} 2024-04-12T03:39:59.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe6bfffccf6f627-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.686 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.686 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.686 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.687 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.687 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.687 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.687 DEBUG:boto:Data: 2024-04-12T03:39:59.687 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.687 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.687 DEBUG:boto:Params: {} 2024-04-12T03:39:59.687 DEBUG:boto:Token: None 2024-04-12T03:39:59.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rk6B4clOeKuZlG9e4AL/lLXb51c= 2024-04-12T03:39:59.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rk6B4clOeKuZlG9e4AL/lLXb51c=', 'Content-Length': '0'} 2024-04-12T03:39:59.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21607af4050d1af-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.689 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.689 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.690 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.690 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.690 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.690 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.690 DEBUG:boto:Data: 2024-04-12T03:39:59.690 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.690 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.690 DEBUG:boto:Params: {} 2024-04-12T03:39:59.690 DEBUG:boto:Token: None 2024-04-12T03:39:59.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rk6B4clOeKuZlG9e4AL/lLXb51c= 2024-04-12T03:39:59.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rk6B4clOeKuZlG9e4AL/lLXb51c=', 'Content-Length': '0'} 2024-04-12T03:39:59.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930aa08a713ba9e7-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:39:59.693 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.693 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.693 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.693 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.693 DEBUG:boto:Method: GET 2024-04-12T03:39:59.693 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.693 DEBUG:boto:Data: 2024-04-12T03:39:59.693 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.693 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.693 DEBUG:boto:Params: {} 2024-04-12T03:39:59.694 DEBUG:boto:Token: None 2024-04-12T03:39:59.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.694 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M= 2024-04-12T03:39:59.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M=', 'Content-Length': '0'} 2024-04-12T03:39:59.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e13eac982ea7f7-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.696 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.696 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.696 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.696 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.696 DEBUG:boto:Method: GET 2024-04-12T03:39:59.696 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.696 DEBUG:boto:Data: 2024-04-12T03:39:59.697 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.697 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.697 DEBUG:boto:Params: {} 2024-04-12T03:39:59.697 DEBUG:boto:Token: None 2024-04-12T03:39:59.697 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283?versionId=null 2024-04-12T03:39:59.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M= 2024-04-12T03:39:59.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5eIKMXpDT40XR3DKoNyQG8DJB2M=', 'Content-Length': '0'} 2024-04-12T03:39:59.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c2e8f8138fee42-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.699 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.699 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.699 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.700 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.700 DEBUG:boto:Method: GET 2024-04-12T03:39:59.700 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.700 DEBUG:boto:Data: 2024-04-12T03:39:59.700 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.700 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.700 DEBUG:boto:Params: {} 2024-04-12T03:39:59.700 DEBUG:boto:Token: None 2024-04-12T03:39:59.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d84073077d83508-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.703 DEBUG:boto:b'dgbpxg-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.703 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.703 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.703 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.703 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.703 DEBUG:boto:Method: GET 2024-04-12T03:39:59.704 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.704 DEBUG:boto:Data: 2024-04-12T03:39:59.704 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.704 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.704 DEBUG:boto:Params: {} 2024-04-12T03:39:59.704 DEBUG:boto:Token: None 2024-04-12T03:39:59.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d4381a280db06bb-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.750 DEBUG:boto:b'dgbpxg-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.751 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.751 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.751 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.751 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.751 DEBUG:boto:Data: 2024-04-12T03:39:59.751 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.751 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.751 DEBUG:boto:Params: {} 2024-04-12T03:39:59.751 DEBUG:boto:Token: None 2024-04-12T03:39:59.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4yhYdQ8PyLhzENlP5+aKBKOW2Hw= 2024-04-12T03:39:59.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4yhYdQ8PyLhzENlP5+aKBKOW2Hw=', 'Content-Length': '0'} 2024-04-12T03:39:59.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056557540afb586f1-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.754 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.754 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.754 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.754 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.754 DEBUG:boto:Data: 2024-04-12T03:39:59.754 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.754 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.754 DEBUG:boto:Params: {} 2024-04-12T03:39:59.754 DEBUG:boto:Token: None 2024-04-12T03:39:59.754 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4yhYdQ8PyLhzENlP5+aKBKOW2Hw= 2024-04-12T03:39:59.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4yhYdQ8PyLhzENlP5+aKBKOW2Hw=', 'Content-Length': '0'} 2024-04-12T03:39:59.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005765189ffed45e7d-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:39:59.757 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.757 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.757 DEBUG:boto:Method: GET 2024-04-12T03:39:59.757 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.757 DEBUG:boto:Data: 2024-04-12T03:39:59.757 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.757 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.758 DEBUG:boto:Params: {} 2024-04-12T03:39:59.758 DEBUG:boto:Token: None 2024-04-12T03:39:59.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8gX+OunLMog98nLzLONTcekvgGw= 2024-04-12T03:39:59.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8gX+OunLMog98nLzLONTcekvgGw=', 'Content-Length': '0'} 2024-04-12T03:39:59.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1238c4ac5135f82-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.760 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.760 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.760 DEBUG:boto:Method: GET 2024-04-12T03:39:59.760 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.760 DEBUG:boto:Data: 2024-04-12T03:39:59.761 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.761 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.761 DEBUG:boto:Params: {} 2024-04-12T03:39:59.761 DEBUG:boto:Token: None 2024-04-12T03:39:59.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000283 2024-04-12T03:39:59.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8gX+OunLMog98nLzLONTcekvgGw= 2024-04-12T03:39:59.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8gX+OunLMog98nLzLONTcekvgGw=', 'Content-Length': '0'} 2024-04-12T03:39:59.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28bbfe69f9b3075-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.763 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.763 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.763 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.763 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.763 DEBUG:boto:Method: GET 2024-04-12T03:39:59.763 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.763 DEBUG:boto:Data: 2024-04-12T03:39:59.764 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.764 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.764 DEBUG:boto:Params: {} 2024-04-12T03:39:59.764 DEBUG:boto:Token: None 2024-04-12T03:39:59.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ce9890e7cc0aeb-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.766 DEBUG:boto:b'dgbpxg-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.767 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.767 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.767 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.767 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.767 DEBUG:boto:Method: GET 2024-04-12T03:39:59.767 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000283 2024-04-12T03:39:59.767 DEBUG:boto:Data: 2024-04-12T03:39:59.767 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.767 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.767 DEBUG:boto:Params: {} 2024-04-12T03:39:59.768 DEBUG:boto:Token: None 2024-04-12T03:39:59.768 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa255f394e8a6d30-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.772 DEBUG:boto:b'dgbpxg-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:39:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:39:59.772 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.772 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.772 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.772 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.772 DEBUG:boto:Method: GET 2024-04-12T03:39:59.773 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.773 DEBUG:boto:Data: 2024-04-12T03:39:59.773 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.773 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.773 DEBUG:boto:Params: {} 2024-04-12T03:39:59.773 DEBUG:boto:Token: None 2024-04-12T03:39:59.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME= 2024-04-12T03:39:59.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME=', 'Content-Length': '0'} 2024-04-12T03:39:59.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033dba6e463e5e40a-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.818 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.818 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.818 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.819 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.819 DEBUG:boto:Method: GET 2024-04-12T03:39:59.819 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.819 DEBUG:boto:Data: 2024-04-12T03:39:59.819 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.819 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.819 DEBUG:boto:Params: {} 2024-04-12T03:39:59.819 DEBUG:boto:Token: None 2024-04-12T03:39:59.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME= 2024-04-12T03:39:59.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME=', 'Content-Length': '0'} 2024-04-12T03:39:59.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9272a166fdf6f4b-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.822 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.822 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.822 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.822 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.822 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.822 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.822 DEBUG:boto:Data: 2024-04-12T03:39:59.822 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.823 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.823 DEBUG:boto:Params: {} 2024-04-12T03:39:59.823 DEBUG:boto:Token: None 2024-04-12T03:39:59.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZflzocH6eVp8VrqBVuL+r1ReOcc= 2024-04-12T03:39:59.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZflzocH6eVp8VrqBVuL+r1ReOcc=', 'Content-Length': '0'} 2024-04-12T03:39:59.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d447008fb47eee-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.825 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.825 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.825 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.825 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.825 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.825 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.825 DEBUG:boto:Data: 2024-04-12T03:39:59.825 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.826 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.826 DEBUG:boto:Params: {} 2024-04-12T03:39:59.826 DEBUG:boto:Token: None 2024-04-12T03:39:59.826 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZflzocH6eVp8VrqBVuL+r1ReOcc= 2024-04-12T03:39:59.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZflzocH6eVp8VrqBVuL+r1ReOcc=', 'Content-Length': '0'} 2024-04-12T03:39:59.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70ab45e2367aa3c-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:39:59.828 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.828 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.828 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.828 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.828 DEBUG:boto:Method: GET 2024-04-12T03:39:59.829 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.829 DEBUG:boto:Data: 2024-04-12T03:39:59.829 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.829 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.829 DEBUG:boto:Params: {} 2024-04-12T03:39:59.829 DEBUG:boto:Token: None 2024-04-12T03:39:59.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME= 2024-04-12T03:39:59.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME=', 'Content-Length': '0'} 2024-04-12T03:39:59.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d2f178f88cc709-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.831 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.831 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.831 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.831 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.831 DEBUG:boto:Method: GET 2024-04-12T03:39:59.832 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.832 DEBUG:boto:Data: 2024-04-12T03:39:59.832 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.832 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.832 DEBUG:boto:Params: {} 2024-04-12T03:39:59.832 DEBUG:boto:Token: None 2024-04-12T03:39:59.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317?versionId=null 2024-04-12T03:39:59.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME= 2024-04-12T03:39:59.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zOPA+dj9IClzAo7z2HLxoGV3FME=', 'Content-Length': '0'} 2024-04-12T03:39:59.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f59cb27f2ea6a56d-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.834 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.834 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.835 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.835 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.835 DEBUG:boto:Method: GET 2024-04-12T03:39:59.835 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.835 DEBUG:boto:Data: 2024-04-12T03:39:59.835 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.835 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.835 DEBUG:boto:Params: {} 2024-04-12T03:39:59.835 DEBUG:boto:Token: None 2024-04-12T03:39:59.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d408962c2f5756b2-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.838 DEBUG:boto:b'dgbpxg-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.838 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.838 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.839 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.839 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.839 DEBUG:boto:Method: GET 2024-04-12T03:39:59.839 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.839 DEBUG:boto:Data: 2024-04-12T03:39:59.839 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.839 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.839 DEBUG:boto:Params: {} 2024-04-12T03:39:59.839 DEBUG:boto:Token: None 2024-04-12T03:39:59.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ddf7db20000076-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.843 DEBUG:boto:b'dgbpxg-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.843 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.844 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.844 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.844 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.844 DEBUG:boto:Data: 2024-04-12T03:39:59.844 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.844 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.844 DEBUG:boto:Params: {} 2024-04-12T03:39:59.844 DEBUG:boto:Token: None 2024-04-12T03:39:59.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.845 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ECHZDVVQNjuOeA57VpZ6abLaYBw= 2024-04-12T03:39:59.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ECHZDVVQNjuOeA57VpZ6abLaYBw=', 'Content-Length': '0'} 2024-04-12T03:39:59.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d4397648c14c54-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.847 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.847 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.847 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.847 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.847 DEBUG:boto:Data: 2024-04-12T03:39:59.847 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.847 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.847 DEBUG:boto:Params: {} 2024-04-12T03:39:59.848 DEBUG:boto:Token: None 2024-04-12T03:39:59.848 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ECHZDVVQNjuOeA57VpZ6abLaYBw= 2024-04-12T03:39:59.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ECHZDVVQNjuOeA57VpZ6abLaYBw=', 'Content-Length': '0'} 2024-04-12T03:39:59.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784d45afd100f884-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:39:59.850 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.850 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.850 DEBUG:boto:Method: GET 2024-04-12T03:39:59.850 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.850 DEBUG:boto:Data: 2024-04-12T03:39:59.850 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.850 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.851 DEBUG:boto:Params: {} 2024-04-12T03:39:59.851 DEBUG:boto:Token: None 2024-04-12T03:39:59.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ds0TmsCtHxYyVtPjEaDeZdeqOcc= 2024-04-12T03:39:59.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ds0TmsCtHxYyVtPjEaDeZdeqOcc=', 'Content-Length': '0'} 2024-04-12T03:39:59.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77f74a0a38890e4-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.894 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.894 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.895 DEBUG:boto:Method: GET 2024-04-12T03:39:59.895 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.895 DEBUG:boto:Data: 2024-04-12T03:39:59.895 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.895 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.895 DEBUG:boto:Params: {} 2024-04-12T03:39:59.895 DEBUG:boto:Token: None 2024-04-12T03:39:59.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000317 2024-04-12T03:39:59.895 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ds0TmsCtHxYyVtPjEaDeZdeqOcc= 2024-04-12T03:39:59.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ds0TmsCtHxYyVtPjEaDeZdeqOcc=', 'Content-Length': '0'} 2024-04-12T03:39:59.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b47c229db1b64d6-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.897 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.898 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.898 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.898 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.898 DEBUG:boto:Method: GET 2024-04-12T03:39:59.898 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.898 DEBUG:boto:Data: 2024-04-12T03:39:59.898 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.898 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.898 DEBUG:boto:Params: {} 2024-04-12T03:39:59.899 DEBUG:boto:Token: None 2024-04-12T03:39:59.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.899 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44c0e6c89617d95-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.901 DEBUG:boto:b'dgbpxg-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.902 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.902 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.902 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.902 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.902 DEBUG:boto:Method: GET 2024-04-12T03:39:59.902 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000317 2024-04-12T03:39:59.902 DEBUG:boto:Data: 2024-04-12T03:39:59.902 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.902 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.902 DEBUG:boto:Params: {} 2024-04-12T03:39:59.903 DEBUG:boto:Token: None 2024-04-12T03:39:59.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9be135754edd80-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.906 DEBUG:boto:b'dgbpxg-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:39:42.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:39:59.907 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.907 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.907 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.907 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.907 DEBUG:boto:Method: GET 2024-04-12T03:39:59.907 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.907 DEBUG:boto:Data: 2024-04-12T03:39:59.907 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.908 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.908 DEBUG:boto:Params: {} 2024-04-12T03:39:59.908 DEBUG:boto:Token: None 2024-04-12T03:39:59.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk= 2024-04-12T03:39:59.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk=', 'Content-Length': '0'} 2024-04-12T03:39:59.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd14155d6ec0db41-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.910 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.910 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.911 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.911 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.911 DEBUG:boto:Method: GET 2024-04-12T03:39:59.911 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.911 DEBUG:boto:Data: 2024-04-12T03:39:59.911 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.911 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.911 DEBUG:boto:Params: {} 2024-04-12T03:39:59.911 DEBUG:boto:Token: None 2024-04-12T03:39:59.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk= 2024-04-12T03:39:59.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk=', 'Content-Length': '0'} 2024-04-12T03:39:59.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c68eff34928809-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.914 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.914 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.914 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.914 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.914 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.914 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.914 DEBUG:boto:Data: 2024-04-12T03:39:59.914 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.915 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.915 DEBUG:boto:Params: {} 2024-04-12T03:39:59.915 DEBUG:boto:Token: None 2024-04-12T03:39:59.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ztNeNy3cZtEA7zhVozhiHLd0YLY= 2024-04-12T03:39:59.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ztNeNy3cZtEA7zhVozhiHLd0YLY=', 'Content-Length': '0'} 2024-04-12T03:39:59.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa6391fd54aa49c-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.917 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.917 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.917 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.917 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.917 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.917 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.918 DEBUG:boto:Data: 2024-04-12T03:39:59.918 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.918 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.918 DEBUG:boto:Params: {} 2024-04-12T03:39:59.918 DEBUG:boto:Token: None 2024-04-12T03:39:59.918 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.918 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ztNeNy3cZtEA7zhVozhiHLd0YLY= 2024-04-12T03:39:59.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ztNeNy3cZtEA7zhVozhiHLd0YLY=', 'Content-Length': '0'} 2024-04-12T03:39:59.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110eff131adcc7a3-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:39:59.920 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.920 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.920 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.920 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.921 DEBUG:boto:Method: GET 2024-04-12T03:39:59.921 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.921 DEBUG:boto:Data: 2024-04-12T03:39:59.921 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.921 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.921 DEBUG:boto:Params: {} 2024-04-12T03:39:59.921 DEBUG:boto:Token: None 2024-04-12T03:39:59.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk= 2024-04-12T03:39:59.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk=', 'Content-Length': '0'} 2024-04-12T03:39:59.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632e1668c0cba1ae-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.923 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.923 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.923 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.924 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.924 DEBUG:boto:Method: GET 2024-04-12T03:39:59.924 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.924 DEBUG:boto:Data: 2024-04-12T03:39:59.924 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.924 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.924 DEBUG:boto:Params: {} 2024-04-12T03:39:59.924 DEBUG:boto:Token: None 2024-04-12T03:39:59.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423?versionId=null 2024-04-12T03:39:59.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk= 2024-04-12T03:39:59.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0MQeTb6HqniA2gryeVXdqa/sylk=', 'Content-Length': '0'} 2024-04-12T03:39:59.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e67df67315a5452-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.926 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.927 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.927 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.927 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.927 DEBUG:boto:Method: GET 2024-04-12T03:39:59.927 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.927 DEBUG:boto:Data: 2024-04-12T03:39:59.927 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.927 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.927 DEBUG:boto:Params: {} 2024-04-12T03:39:59.927 DEBUG:boto:Token: None 2024-04-12T03:39:59.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090d370d18fdc3127-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.930 DEBUG:boto:b'dgbpxg-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.930 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.931 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.931 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.931 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.931 DEBUG:boto:Method: GET 2024-04-12T03:39:59.931 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.931 DEBUG:boto:Data: 2024-04-12T03:39:59.931 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.931 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.931 DEBUG:boto:Params: {} 2024-04-12T03:39:59.932 DEBUG:boto:Token: None 2024-04-12T03:39:59.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b3ad365f9457dc-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.937 DEBUG:boto:b'dgbpxg-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.938 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.938 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.938 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.938 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.938 DEBUG:boto:Data: 2024-04-12T03:39:59.938 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.938 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.939 DEBUG:boto:Params: {} 2024-04-12T03:39:59.939 DEBUG:boto:Token: None 2024-04-12T03:39:59.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iaxTOnEBhUaTFIoDggi/KBVxYVQ= 2024-04-12T03:39:59.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iaxTOnEBhUaTFIoDggi/KBVxYVQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f27e22a6a774e6-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.941 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.941 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.941 DEBUG:boto:Method: HEAD 2024-04-12T03:39:59.941 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.941 DEBUG:boto:Data: 2024-04-12T03:39:59.941 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.941 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.942 DEBUG:boto:Params: {} 2024-04-12T03:39:59.942 DEBUG:boto:Token: None 2024-04-12T03:39:59.942 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iaxTOnEBhUaTFIoDggi/KBVxYVQ= 2024-04-12T03:39:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iaxTOnEBhUaTFIoDggi/KBVxYVQ=', 'Content-Length': '0'} 2024-04-12T03:39:59.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016aec78f7ea476e5-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:39:59.944 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.944 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.944 DEBUG:boto:Method: GET 2024-04-12T03:39:59.944 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.944 DEBUG:boto:Data: 2024-04-12T03:39:59.945 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.945 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.945 DEBUG:boto:Params: {} 2024-04-12T03:39:59.945 DEBUG:boto:Token: None 2024-04-12T03:39:59.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqFE9EIjWiFY15/wcOpHPt+DLdI= 2024-04-12T03:39:59.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqFE9EIjWiFY15/wcOpHPt+DLdI=', 'Content-Length': '0'} 2024-04-12T03:39:59.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939ccf61697539a3-006618ad0f-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.947 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.947 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.947 DEBUG:boto:Method: GET 2024-04-12T03:39:59.947 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.947 DEBUG:boto:Data: 2024-04-12T03:39:59.948 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.948 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.948 DEBUG:boto:Params: {} 2024-04-12T03:39:59.948 DEBUG:boto:Token: None 2024-04-12T03:39:59.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/obj00000000000000000423 2024-04-12T03:39:59.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqFE9EIjWiFY15/wcOpHPt+DLdI= 2024-04-12T03:39:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqFE9EIjWiFY15/wcOpHPt+DLdI=', 'Content-Length': '0'} 2024-04-12T03:39:59.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f869beadeb3ab19-006618ad0f-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.990 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.990 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.991 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.991 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.991 DEBUG:boto:Method: GET 2024-04-12T03:39:59.991 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.991 DEBUG:boto:Data: 2024-04-12T03:39:59.991 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:39:59.991 DEBUG:boto:Port: 8001 2024-04-12T03:39:59.991 DEBUG:boto:Params: {} 2024-04-12T03:39:59.991 DEBUG:boto:Token: None 2024-04-12T03:39:59.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d58bfdaadd493ff6-006618ad0f-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.994 DEBUG:boto:b'dgbpxg-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.994 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:39:59.994 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:39:59.994 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.994 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.994 DEBUG:boto:Method: GET 2024-04-12T03:39:59.995 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000423 2024-04-12T03:39:59.995 DEBUG:boto:Data: 2024-04-12T03:39:59.995 DEBUG:boto:Headers: {} 2024-04-12T03:39:59.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:39:59.995 DEBUG:boto:Port: 8000 2024-04-12T03:39:59.995 DEBUG:boto:Params: {} 2024-04-12T03:39:59.995 DEBUG:boto:Token: None 2024-04-12T03:39:59.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:59 GMT /dgbpxg-60/ 2024-04-12T03:39:59.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI= 2024-04-12T03:39:59.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:59 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PyZ9CjV+7rnzSdfFUduKl49xuQI=', 'Content-Length': '0'} 2024-04-12T03:39:59.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000636378b0a187733-006618ad0f-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:39:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:59.999 DEBUG:boto:b'dgbpxg-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:39:42.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:39:59.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:40:00.000 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.000 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.000 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.000 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.000 DEBUG:boto:Method: GET 2024-04-12T03:40:00.000 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.000 DEBUG:boto:Data: 2024-04-12T03:40:00.000 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.000 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.000 DEBUG:boto:Params: {} 2024-04-12T03:40:00.001 DEBUG:boto:Token: None 2024-04-12T03:40:00.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ= 2024-04-12T03:40:00.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f986c27a167c7759-006618ad10-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.003 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.003 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.003 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.003 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.003 DEBUG:boto:Method: GET 2024-04-12T03:40:00.004 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.004 DEBUG:boto:Data: 2024-04-12T03:40:00.004 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.004 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.004 DEBUG:boto:Params: {} 2024-04-12T03:40:00.004 DEBUG:boto:Token: None 2024-04-12T03:40:00.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ= 2024-04-12T03:40:00.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0930d2a4a79791-006618ad10-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.006 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.006 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.007 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.007 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.007 DEBUG:boto:Method: HEAD 2024-04-12T03:40:00.007 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.007 DEBUG:boto:Data: 2024-04-12T03:40:00.007 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.007 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.007 DEBUG:boto:Params: {} 2024-04-12T03:40:00.007 DEBUG:boto:Token: None 2024-04-12T03:40:00.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jfl2lcS3rlb/XwVuPFBh1nx4OgQ= 2024-04-12T03:40:00.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jfl2lcS3rlb/XwVuPFBh1nx4OgQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e99c309e740cf84-006618ad10-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.009 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.009 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.009 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.010 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.010 DEBUG:boto:Method: HEAD 2024-04-12T03:40:00.010 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.010 DEBUG:boto:Data: 2024-04-12T03:40:00.010 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.010 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.010 DEBUG:boto:Params: {} 2024-04-12T03:40:00.010 DEBUG:boto:Token: None 2024-04-12T03:40:00.010 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Jfl2lcS3rlb/XwVuPFBh1nx4OgQ= 2024-04-12T03:40:00.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Jfl2lcS3rlb/XwVuPFBh1nx4OgQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa53070f389f733b-006618ad10-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:40:00.013 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.013 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.013 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.013 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.013 DEBUG:boto:Method: GET 2024-04-12T03:40:00.013 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.013 DEBUG:boto:Data: 2024-04-12T03:40:00.013 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.013 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.013 DEBUG:boto:Params: {} 2024-04-12T03:40:00.014 DEBUG:boto:Token: None 2024-04-12T03:40:00.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ= 2024-04-12T03:40:00.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5d6d006dc92056-006618ad10-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.058 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.059 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.059 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.059 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.059 DEBUG:boto:Method: GET 2024-04-12T03:40:00.059 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.059 DEBUG:boto:Data: 2024-04-12T03:40:00.059 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.059 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.059 DEBUG:boto:Params: {} 2024-04-12T03:40:00.060 DEBUG:boto:Token: None 2024-04-12T03:40:00.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467?versionId=null 2024-04-12T03:40:00.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ= 2024-04-12T03:40:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8q3WYJUI5LUDSZu6LCXJANxradQ=', 'Content-Length': '0'} 2024-04-12T03:40:00.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24aaac11ef9d90a-006618ad10-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.062 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:40:00.062 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:40:00.063 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.063 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.063 DEBUG:boto:Method: GET 2024-04-12T03:40:00.063 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.063 DEBUG:boto:Data: 2024-04-12T03:40:00.063 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.063 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.063 DEBUG:boto:Params: {} 2024-04-12T03:40:00.063 DEBUG:boto:Token: None 2024-04-12T03:40:00.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/ 2024-04-12T03:40:00.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8= 2024-04-12T03:40:00.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8=', 'Content-Length': '0'} 2024-04-12T03:40:00.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb53ef5219ef342-006618ad10-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.066 DEBUG:boto:b'dgbpxg-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:40:00.067 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:40:00.067 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:40:00.067 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.067 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.067 DEBUG:boto:Method: GET 2024-04-12T03:40:00.067 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.067 DEBUG:boto:Data: 2024-04-12T03:40:00.067 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.068 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.068 DEBUG:boto:Params: {} 2024-04-12T03:40:00.068 DEBUG:boto:Token: None 2024-04-12T03:40:00.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/ 2024-04-12T03:40:00.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8= 2024-04-12T03:40:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8=', 'Content-Length': '0'} 2024-04-12T03:40:00.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20da08dd11b84d0-006618ad10-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.072 DEBUG:boto:b'dgbpxg-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:40:00.072 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.072 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.072 DEBUG:boto:Method: HEAD 2024-04-12T03:40:00.072 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.072 DEBUG:boto:Data: 2024-04-12T03:40:00.072 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.073 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.073 DEBUG:boto:Params: {} 2024-04-12T03:40:00.073 DEBUG:boto:Token: None 2024-04-12T03:40:00.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8w1lygyC+j/kczWx+XA8pJ4yMBw= 2024-04-12T03:40:00.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8w1lygyC+j/kczWx+XA8pJ4yMBw=', 'Content-Length': '0'} 2024-04-12T03:40:00.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3e2a3c9997e8da-006618ad10-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.075 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.075 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.075 DEBUG:boto:Method: HEAD 2024-04-12T03:40:00.075 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.075 DEBUG:boto:Data: 2024-04-12T03:40:00.075 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.075 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.075 DEBUG:boto:Params: {} 2024-04-12T03:40:00.075 DEBUG:boto:Token: None 2024-04-12T03:40:00.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8w1lygyC+j/kczWx+XA8pJ4yMBw= 2024-04-12T03:40:00.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8w1lygyC+j/kczWx+XA8pJ4yMBw=', 'Content-Length': '0'} 2024-04-12T03:40:00.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436ad704a8eac08c-006618ad10-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:40:00.078 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.078 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.078 DEBUG:boto:Method: GET 2024-04-12T03:40:00.078 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.078 DEBUG:boto:Data: 2024-04-12T03:40:00.078 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.079 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.079 DEBUG:boto:Params: {} 2024-04-12T03:40:00.079 DEBUG:boto:Token: None 2024-04-12T03:40:00.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LPAUrsZEWJXYFjRAatqyWQekPAI= 2024-04-12T03:40:00.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LPAUrsZEWJXYFjRAatqyWQekPAI=', 'Content-Length': '0'} 2024-04-12T03:40:00.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-60:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090cc39c0dc98d4db-006618ad10-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.122 DEBUG:boto:path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.122 DEBUG:boto:auth_path=/dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.123 DEBUG:boto:Method: GET 2024-04-12T03:40:00.123 DEBUG:boto:Path: /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.123 DEBUG:boto:Data: 2024-04-12T03:40:00.123 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.123 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.123 DEBUG:boto:Params: {} 2024-04-12T03:40:00.123 DEBUG:boto:Token: None 2024-04-12T03:40:00.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/obj00000000000000000467 2024-04-12T03:40:00.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LPAUrsZEWJXYFjRAatqyWQekPAI= 2024-04-12T03:40:00.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LPAUrsZEWJXYFjRAatqyWQekPAI=', 'Content-Length': '0'} 2024-04-12T03:40:00.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028b9cb88d8dc5c06-006618ad10-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.125 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:40:00.126 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:40:00.126 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.126 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.126 DEBUG:boto:Method: GET 2024-04-12T03:40:00.126 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.126 DEBUG:boto:Data: 2024-04-12T03:40:00.126 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:40:00.126 DEBUG:boto:Port: 8001 2024-04-12T03:40:00.126 DEBUG:boto:Params: {} 2024-04-12T03:40:00.127 DEBUG:boto:Token: None 2024-04-12T03:40:00.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/ 2024-04-12T03:40:00.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8= 2024-04-12T03:40:00.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8=', 'Content-Length': '0'} 2024-04-12T03:40:00.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea64d512da32130e-006618ad10-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.129 DEBUG:boto:b'dgbpxg-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:40:00.130 DEBUG:boto:path=/dgbpxg-60/ 2024-04-12T03:40:00.130 DEBUG:boto:auth_path=/dgbpxg-60/ 2024-04-12T03:40:00.130 DEBUG:boto:path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.130 DEBUG:boto:auth_path=/dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.130 DEBUG:boto:Method: GET 2024-04-12T03:40:00.130 DEBUG:boto:Path: /dgbpxg-60/?prefix=obj00000000000000000467 2024-04-12T03:40:00.130 DEBUG:boto:Data: 2024-04-12T03:40:00.130 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.130 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.130 DEBUG:boto:Params: {} 2024-04-12T03:40:00.131 DEBUG:boto:Token: None 2024-04-12T03:40:00.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-60/ 2024-04-12T03:40:00.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8= 2024-04-12T03:40:00.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nOvnXnZQrBwdJg0JaDvhxz2+Em8=', 'Content-Length': '0'} 2024-04-12T03:40:00.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14cdbf394c8f888-006618ad10-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.134 DEBUG:boto:b'dgbpxg-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:39:42.586Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:40:00.135 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-60 zones={a2, a1} 2024-04-12T03:40:00.135 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2024-04-12T03:40:00.137 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:00.138 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:00.138 DEBUG:boto:Method: PUT 2024-04-12T03:40:00.138 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:00.138 DEBUG:boto:Data: 2024-04-12T03:40:00.138 DEBUG:boto:Headers: {} 2024-04-12T03:40:00.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:00.138 DEBUG:boto:Port: 8000 2024-04-12T03:40:00.138 DEBUG:boto:Params: {} 2024-04-12T03:40:00.139 DEBUG:boto:Token: None 2024-04-12T03:40:00.139 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:40:00 GMT /dgbpxg-61/ 2024-04-12T03:40:00.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pSzy6yIuV11t57gKqD/u4+eqqUI= 2024-04-12T03:40:00.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:00 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pSzy6yIuV11t57gKqD/u4+eqqUI=', 'Content-Length': '0'} 2024-04-12T03:40:00.153 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fd88c56e46a6ea5-006618ad10-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:00.153 DEBUG:rgw_multi.tests:created bucket=dgbpxg-61 2024-04-12T03:40:00.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:00.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:00.154 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:00.203 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:00.203 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:00.234 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:40:00.246 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:40:00.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893200.151669_1230.1", 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:40:00.151669Z" 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.248 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:40:00.249 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:00.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:40:00.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:00.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:00.250 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:00.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:00.326+0000 7f7a2569b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:00.327+0000 7f7a2569b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:00.340 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:00.340 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:00.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:00Z" 2024-04-12T03:40:00.346 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:00.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:00Z" } 2024-04-12T03:40:00.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:00.348 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:00.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:00.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:05.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:05.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:05.350 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:05.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:05.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:05.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:05.432+0000 7fe28ee3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:05.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:05.433+0000 7fe28ee3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:05.447 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:05.447 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:05.448 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:05.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:05.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:05.452 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:05.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:05.453 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:05Z" 2024-04-12T03:40:05.453 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:05.453 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:05Z" } 2024-04-12T03:40:05.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:05.454 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:05.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:05.454 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:10.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:10.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:10.456 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:10.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:10.543+0000 7f8781ec6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:10.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:10.544+0000 7f8781ec6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:10.559 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:10.559 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:10.559 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:10.559 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:10.559 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:10.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:10.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:10.563 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:10.564 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:10.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:10.565 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:10Z" 2024-04-12T03:40:10.565 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:10.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:10Z" } 2024-04-12T03:40:10.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:10.566 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:10.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:10.566 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:15.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:15.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:15.568 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:15.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:15.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:15.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:15.657+0000 7fee7df44780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:15.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:15.658+0000 7fee7df44780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:15.673 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:15Z" 2024-04-12T03:40:15.679 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:15.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:15Z" } 2024-04-12T03:40:15.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:15.680 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:15.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:15.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:20.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:20.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:20.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:20.767+0000 7fbc419bd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:20.768+0000 7fbc419bd780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:20.784 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:20.784 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:20.785 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:20.786 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:20.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:20.789 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:20Z" 2024-04-12T03:40:20.790 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:20.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:20Z" } 2024-04-12T03:40:20.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:20.792 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:20.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:20.792 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:25.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:25.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:25.793 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:25.882+0000 7fdcdbdb4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:25.883+0000 7fdcdbdb4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:25.898 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:25.898 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:25.898 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:25.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:25.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:25.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:25.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:25.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:25Z" 2024-04-12T03:40:25.904 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:25.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:25Z" } 2024-04-12T03:40:25.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:25.906 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:25.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:25.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:30.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:30.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:30.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:30.997+0000 7ff0b322f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:30.998+0000 7ff0b322f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:30Z" 2024-04-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:31.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:30Z" } 2024-04-12T03:40:31.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:31.019 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:31.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:31.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:36.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:36.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:36.021 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:36.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:36.108+0000 7f5453ad9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:36.109+0000 7f5453ad9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.124 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:36.126 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:36.127 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:36.128 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:36.129 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:36Z" 2024-04-12T03:40:36.130 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:36.130 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:36Z" } 2024-04-12T03:40:36.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:36.131 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:36.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:36.131 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:41.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:41.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:41.132 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:41.221+0000 7fa430ffb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:41.222+0000 7fa430ffb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:41.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:41.238 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:41.239 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:41.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893160.697788_1162.1", 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:41.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:20.697788Z", 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:41.242 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:41.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:41.243 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:41Z" 2024-04-12T03:40:41.243 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:41.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893160.697788_1162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:20.697788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:41Z" } 2024-04-12T03:40:41.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893160.697788_1162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:20.697788Z', 'realm_epoch': 2}}] 2024-04-12T03:40:41.244 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:41.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893160.697788_1162.1'} 2024-04-12T03:40:41.244 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893160.697788_1162.1 2024-04-12T03:40:46.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:46.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:46.245 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:46.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:46.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:46.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:46.330+0000 7fe2270ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:46.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:46.331+0000 7fe2270ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.345 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:46.346 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:46.347 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:46.348 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893178.746217_1172.1", 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:46.349 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:39:38.746217Z", 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:46Z" 2024-04-12T03:40:46.350 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:46.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893178.746217_1172.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:38.746217Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:46Z" } 2024-04-12T03:40:46.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893178.746217_1172.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:38.746217Z', 'realm_epoch': 2}}] 2024-04-12T03:40:46.352 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:46.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893178.746217_1172.1'} 2024-04-12T03:40:46.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893200.151669_1230.1 target=1_1712893178.746217_1172.1 2024-04-12T03:40:51.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:51.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:40:51.353 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:40:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:51.410 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:51.442+0000 7fcc2b310780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:51.443+0000 7fcc2b310780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:40:51.457 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:40:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:40:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:40:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893200.151669_1230.1", 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:40:00.151669Z", 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:40:51Z" 2024-04-12T03:40:51.463 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:40:51.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893200.151669_1230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:40:00.151669Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:40:51Z" } 2024-04-12T03:40:51.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893200.151669_1230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:40:00.151669Z', 'realm_epoch': 2}}] 2024-04-12T03:40:51.464 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:51.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893200.151669_1230.1'} 2024-04-12T03:40:51.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:40:51.465 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:51.465 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:51.465 DEBUG:boto:Method: HEAD 2024-04-12T03:40:51.465 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:51.465 DEBUG:boto:Data: 2024-04-12T03:40:51.465 DEBUG:boto:Headers: {} 2024-04-12T03:40:51.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.465 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.465 DEBUG:boto:Params: {} 2024-04-12T03:40:51.466 DEBUG:boto:Token: None 2024-04-12T03:40:51.466 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/ 2024-04-12T03:40:51.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY= 2024-04-12T03:40:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY=', 'Content-Length': '0'} 2024-04-12T03:40:51.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059cd0f49641a6a74-006618ad43-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.470 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:40:51.470 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:40:51.470 DEBUG:boto:Method: PUT 2024-04-12T03:40:51.470 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:40:51.471 DEBUG:boto:Data: 2024-04-12T03:40:51.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:40:51.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.471 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.471 DEBUG:boto:Params: {} 2024-04-12T03:40:51.471 DEBUG:boto:Token: None 2024-04-12T03:40:51.471 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/obj11 2024-04-12T03:40:51.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gYHJFDINz/PMtHyIlBh+02b8GRY= 2024-04-12T03:40:51.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gYHJFDINz/PMtHyIlBh+02b8GRY='} 2024-04-12T03:40:51.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f03e9a94fb36b6a2-006618ad43-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.478 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:51.478 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:51.478 DEBUG:boto:Method: HEAD 2024-04-12T03:40:51.478 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:51.478 DEBUG:boto:Data: 2024-04-12T03:40:51.478 DEBUG:boto:Headers: {} 2024-04-12T03:40:51.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.478 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.478 DEBUG:boto:Params: {} 2024-04-12T03:40:51.479 DEBUG:boto:Token: None 2024-04-12T03:40:51.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/ 2024-04-12T03:40:51.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY= 2024-04-12T03:40:51.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY=', 'Content-Length': '0'} 2024-04-12T03:40:51.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9c7370084bee993-006618ad43-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.481 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:40:51.481 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:40:51.481 DEBUG:boto:Method: PUT 2024-04-12T03:40:51.481 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:40:51.481 DEBUG:boto:Data: 2024-04-12T03:40:51.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:40:51.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.482 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.482 DEBUG:boto:Params: {} 2024-04-12T03:40:51.482 DEBUG:boto:Token: None 2024-04-12T03:40:51.482 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/obj22 2024-04-12T03:40:51.482 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sIKW0PL6PLxH/wRgj/n0s7RJxUU= 2024-04-12T03:40:51.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sIKW0PL6PLxH/wRgj/n0s7RJxUU='} 2024-04-12T03:40:51.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074a754c058bec10b-006618ad43-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.486 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:51.486 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:51.486 DEBUG:boto:Method: HEAD 2024-04-12T03:40:51.486 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:51.486 DEBUG:boto:Data: 2024-04-12T03:40:51.486 DEBUG:boto:Headers: {} 2024-04-12T03:40:51.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.486 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.486 DEBUG:boto:Params: {} 2024-04-12T03:40:51.486 DEBUG:boto:Token: None 2024-04-12T03:40:51.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/ 2024-04-12T03:40:51.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY= 2024-04-12T03:40:51.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY=', 'Content-Length': '0'} 2024-04-12T03:40:51.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073317a1a9dca998f-006618ad43-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.489 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:40:51.489 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:40:51.489 DEBUG:boto:Method: PUT 2024-04-12T03:40:51.489 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:40:51.489 DEBUG:boto:Data: 2024-04-12T03:40:51.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:40:51.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.489 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.490 DEBUG:boto:Params: {} 2024-04-12T03:40:51.490 DEBUG:boto:Token: None 2024-04-12T03:40:51.490 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/obj33 2024-04-12T03:40:51.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8LonxZnjLb4LodVBQLV4NOVDY6g= 2024-04-12T03:40:51.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8LonxZnjLb4LodVBQLV4NOVDY6g='} 2024-04-12T03:40:51.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039f067fac4b43e48-006618ad43-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.495 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:51.495 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:51.495 DEBUG:boto:Method: HEAD 2024-04-12T03:40:51.495 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:51.495 DEBUG:boto:Data: 2024-04-12T03:40:51.495 DEBUG:boto:Headers: {} 2024-04-12T03:40:51.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.495 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.496 DEBUG:boto:Params: {} 2024-04-12T03:40:51.496 DEBUG:boto:Token: None 2024-04-12T03:40:51.496 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/ 2024-04-12T03:40:51.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY= 2024-04-12T03:40:51.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY=', 'Content-Length': '0'} 2024-04-12T03:40:51.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013fbae8d9113338c-006618ad43-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.498 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:40:51.498 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:40:51.498 DEBUG:boto:Method: PUT 2024-04-12T03:40:51.498 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:40:51.498 DEBUG:boto:Data: 2024-04-12T03:40:51.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:40:51.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.499 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.499 DEBUG:boto:Params: {} 2024-04-12T03:40:51.499 DEBUG:boto:Token: None 2024-04-12T03:40:51.499 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/obj44 2024-04-12T03:40:51.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ylr/0Ae+/YdN9/2SCGOa9oNW32s= 2024-04-12T03:40:51.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ylr/0Ae+/YdN9/2SCGOa9oNW32s='} 2024-04-12T03:40:51.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034312f9ac194cca3-006618ad43-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.543 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:40:51.543 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:40:51.543 DEBUG:boto:Method: HEAD 2024-04-12T03:40:51.543 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:40:51.543 DEBUG:boto:Data: 2024-04-12T03:40:51.543 DEBUG:boto:Headers: {} 2024-04-12T03:40:51.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.543 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.543 DEBUG:boto:Params: {} 2024-04-12T03:40:51.543 DEBUG:boto:Token: None 2024-04-12T03:40:51.544 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/ 2024-04-12T03:40:51.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY= 2024-04-12T03:40:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLi+UbaB/bd+sqOsnKW8V8sF6KY=', 'Content-Length': '0'} 2024-04-12T03:40:51.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000281da165422802a3-006618ad43-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.546 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:40:51.546 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:40:51.546 DEBUG:boto:Method: PUT 2024-04-12T03:40:51.546 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:40:51.546 DEBUG:boto:Data: 2024-04-12T03:40:51.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:40:51.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:40:51.546 DEBUG:boto:Port: 8000 2024-04-12T03:40:51.546 DEBUG:boto:Params: {} 2024-04-12T03:40:51.547 DEBUG:boto:Token: None 2024-04-12T03:40:51.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:40:51 GMT /dgbpxg-61/obj55 2024-04-12T03:40:51.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e2lLW1/4CxnJECxmOX/imu/0uJo= 2024-04-12T03:40:51.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:40:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e2lLW1/4CxnJECxmOX/imu/0uJo='} 2024-04-12T03:40:51.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009471c2f01fa58d2e-006618ad43-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:40:51.551 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:40:51.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:40:51.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:40:51.551 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:40:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:51.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:51.644+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:40:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:40:56.645+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:41:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:01.645+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:41:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:06.646+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:41:11.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:11.647+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:41:16.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:16.647+0000 7f767d2ee780 1 waiting to reach incremental sync.. 2024-04-12T03:41:21.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:21.649+0000 7f767d2ee780 1 bucket sync caught up with source: 2024-04-12T03:41:21.650 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.56.6, 00000000002.101.6, 00000000002.90.6] 2024-04-12T03:41:21.650 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.56.6, 00000000002.101.6, 00000000002.90.6] 2024-04-12T03:41:21.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:21.649+0000 7f767d2ee780 0 bucket checkpoint complete 2024-04-12T03:41:21.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:41:21.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:41:21.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:41:21.668 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:41:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:21.764+0000 7f0b5dc45780 1 waiting to reach incremental sync.. 2024-04-12T03:41:26.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:26.764+0000 7f0b5dc45780 1 waiting to reach incremental sync.. 2024-04-12T03:41:27.887 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:41:27.887+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:41:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:31.766+0000 7f0b5dc45780 1 bucket sync caught up with source: 2024-04-12T03:41:31.767 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.419.6, 00000000002.2228.6, 00000000002.1481.6] 2024-04-12T03:41:31.767 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.419.6, 00000000002.2228.6, 00000000002.1481.6] 2024-04-12T03:41:31.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:31.766+0000 7f0b5dc45780 0 bucket checkpoint complete 2024-04-12T03:41:31.784 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:41:31.784 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.784 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.784 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.785 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:31.785 DEBUG:boto:Data: 2024-04-12T03:41:31.785 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.785 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.785 DEBUG:boto:Params: {} 2024-04-12T03:41:31.785 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:41:31.785 DEBUG:boto:Token: None 2024-04-12T03:41:31.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okzFPkAfGNLNPhIC9XIW0OVbD7E= 2024-04-12T03:41:31.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okzFPkAfGNLNPhIC9XIW0OVbD7E=', 'Content-Length': '0'} 2024-04-12T03:41:31.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034f4561fd6b3fedd-006618ad6b-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.799 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.799 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.799 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.799 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:31.799 DEBUG:boto:Data: 2024-04-12T03:41:31.800 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.800 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.800 DEBUG:boto:Params: {} 2024-04-12T03:41:31.800 DEBUG:boto:Token: None 2024-04-12T03:41:31.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:okzFPkAfGNLNPhIC9XIW0OVbD7E= 2024-04-12T03:41:31.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:okzFPkAfGNLNPhIC9XIW0OVbD7E=', 'Content-Length': '0'} 2024-04-12T03:41:31.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7fc7f2b44127ced-006618ad6b-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:41:31.803 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.803 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.803 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.803 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.803 DEBUG:boto:Method: GET 2024-04-12T03:41:31.804 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.804 DEBUG:boto:Data: 2024-04-12T03:41:31.804 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.804 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.804 DEBUG:boto:Params: {} 2024-04-12T03:41:31.804 DEBUG:boto:Token: None 2024-04-12T03:41:31.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/?versions 2024-04-12T03:41:31.805 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg= 2024-04-12T03:41:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg=', 'Content-Length': '0'} 2024-04-12T03:41:31.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052ea28084a2d479a-006618ad6b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.807 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:41:31.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:41:31.808 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.808 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.809 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.809 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.809 DEBUG:boto:Method: GET 2024-04-12T03:41:31.809 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.809 DEBUG:boto:Data: 2024-04-12T03:41:31.809 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.809 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.809 DEBUG:boto:Params: {} 2024-04-12T03:41:31.809 DEBUG:boto:Token: None 2024-04-12T03:41:31.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/?versions 2024-04-12T03:41:31.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg= 2024-04-12T03:41:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg=', 'Content-Length': '0'} 2024-04-12T03:41:31.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30496c7f87fa8cf-006618ad6b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.812 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:41:31.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:41:31.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:41:31.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:41:31.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:41:31.813 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.813 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.813 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.813 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.813 DEBUG:boto:Method: GET 2024-04-12T03:41:31.813 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.813 DEBUG:boto:Data: 2024-04-12T03:41:31.814 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.814 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.814 DEBUG:boto:Params: {} 2024-04-12T03:41:31.814 DEBUG:boto:Token: None 2024-04-12T03:41:31.814 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/?versions 2024-04-12T03:41:31.814 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg= 2024-04-12T03:41:31.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg=', 'Content-Length': '0'} 2024-04-12T03:41:31.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df91afc62d12d8b1-006618ad6b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.816 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.817 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.817 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.817 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.817 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.817 DEBUG:boto:Method: GET 2024-04-12T03:41:31.817 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:41:31.817 DEBUG:boto:Data: 2024-04-12T03:41:31.818 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.818 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.818 DEBUG:boto:Params: {} 2024-04-12T03:41:31.818 DEBUG:boto:Token: None 2024-04-12T03:41:31.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/?versions 2024-04-12T03:41:31.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg= 2024-04-12T03:41:31.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sKKmUlNijaLMajWDqLWRhGT25Rg=', 'Content-Length': '0'} 2024-04-12T03:41:31.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007052c58253009aae-006618ad6b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.820 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:41:31.821 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.821 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.821 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.821 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.822 DEBUG:boto:Method: GET 2024-04-12T03:41:31.822 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.822 DEBUG:boto:Data: 2024-04-12T03:41:31.822 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.822 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.822 DEBUG:boto:Params: {} 2024-04-12T03:41:31.822 DEBUG:boto:Token: None 2024-04-12T03:41:31.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM= 2024-04-12T03:41:31.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM=', 'Content-Length': '0'} 2024-04-12T03:41:31.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b03eb4350a3ea3-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.825 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.825 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.825 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.825 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.825 DEBUG:boto:Method: GET 2024-04-12T03:41:31.825 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.825 DEBUG:boto:Data: 2024-04-12T03:41:31.825 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.825 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.825 DEBUG:boto:Params: {} 2024-04-12T03:41:31.826 DEBUG:boto:Token: None 2024-04-12T03:41:31.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM= 2024-04-12T03:41:31.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM=', 'Content-Length': '0'} 2024-04-12T03:41:31.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001641639064da11bf-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.828 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.828 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.828 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.828 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.828 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.829 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.829 DEBUG:boto:Data: 2024-04-12T03:41:31.829 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.829 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.829 DEBUG:boto:Params: {} 2024-04-12T03:41:31.829 DEBUG:boto:Token: None 2024-04-12T03:41:31.829 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1v374L3GF0Mq2F/F8XuGnmG/eBA= 2024-04-12T03:41:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1v374L3GF0Mq2F/F8XuGnmG/eBA=', 'Content-Length': '0'} 2024-04-12T03:41:31.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006286965bd9d9f640-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.831 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.831 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.831 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.831 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.832 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.832 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.832 DEBUG:boto:Data: 2024-04-12T03:41:31.832 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.832 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.832 DEBUG:boto:Params: {} 2024-04-12T03:41:31.832 DEBUG:boto:Token: None 2024-04-12T03:41:31.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1v374L3GF0Mq2F/F8XuGnmG/eBA= 2024-04-12T03:41:31.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1v374L3GF0Mq2F/F8XuGnmG/eBA=', 'Content-Length': '0'} 2024-04-12T03:41:31.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3ba786d9f9d7d7-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:41:31.834 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.835 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.835 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.835 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.835 DEBUG:boto:Method: GET 2024-04-12T03:41:31.835 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.835 DEBUG:boto:Data: 2024-04-12T03:41:31.835 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.835 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.835 DEBUG:boto:Params: {} 2024-04-12T03:41:31.835 DEBUG:boto:Token: None 2024-04-12T03:41:31.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM= 2024-04-12T03:41:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM=', 'Content-Length': '0'} 2024-04-12T03:41:31.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ed611c23506467-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.878 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.878 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.878 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.879 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.879 DEBUG:boto:Method: GET 2024-04-12T03:41:31.879 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.879 DEBUG:boto:Data: 2024-04-12T03:41:31.879 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.879 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.879 DEBUG:boto:Params: {} 2024-04-12T03:41:31.879 DEBUG:boto:Token: None 2024-04-12T03:41:31.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:41:31.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM= 2024-04-12T03:41:31.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwJkP5n8fIBGHWM2ylayN46nAjM=', 'Content-Length': '0'} 2024-04-12T03:41:31.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d80f9d1071f8feb-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.882 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.882 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.882 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.882 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.882 DEBUG:boto:Method: GET 2024-04-12T03:41:31.882 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.882 DEBUG:boto:Data: 2024-04-12T03:41:31.882 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.883 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.883 DEBUG:boto:Params: {} 2024-04-12T03:41:31.883 DEBUG:boto:Token: None 2024-04-12T03:41:31.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs= 2024-04-12T03:41:31.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs=', 'Content-Length': '0'} 2024-04-12T03:41:31.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224fbfd93a3a2637-006618ad6b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.886 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.886 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.886 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.886 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.886 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.886 DEBUG:boto:Method: GET 2024-04-12T03:41:31.886 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.887 DEBUG:boto:Data: 2024-04-12T03:41:31.887 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.887 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.887 DEBUG:boto:Params: {} 2024-04-12T03:41:31.887 DEBUG:boto:Token: None 2024-04-12T03:41:31.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs= 2024-04-12T03:41:31.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs=', 'Content-Length': '0'} 2024-04-12T03:41:31.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ad2e382eb95d63-006618ad6b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.890 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.890 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.890 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.890 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.890 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:41:31.891 DEBUG:boto:Data: 2024-04-12T03:41:31.891 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.891 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.891 DEBUG:boto:Params: {} 2024-04-12T03:41:31.891 DEBUG:boto:Token: None 2024-04-12T03:41:31.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11 2024-04-12T03:41:31.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEWAE9KzMOGv1S5x3wJ0VeyL4jg= 2024-04-12T03:41:31.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEWAE9KzMOGv1S5x3wJ0VeyL4jg=', 'Content-Length': '0'} 2024-04-12T03:41:31.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8f80bd065c9d9c-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.893 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.893 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.893 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.893 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:41:31.893 DEBUG:boto:Data: 2024-04-12T03:41:31.893 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.894 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.894 DEBUG:boto:Params: {} 2024-04-12T03:41:31.894 DEBUG:boto:Token: None 2024-04-12T03:41:31.894 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11 2024-04-12T03:41:31.894 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEWAE9KzMOGv1S5x3wJ0VeyL4jg= 2024-04-12T03:41:31.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEWAE9KzMOGv1S5x3wJ0VeyL4jg=', 'Content-Length': '0'} 2024-04-12T03:41:31.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005506253ab8d6dbc-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:41:31.896 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.896 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.897 DEBUG:boto:Method: GET 2024-04-12T03:41:31.897 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:41:31.897 DEBUG:boto:Data: 2024-04-12T03:41:31.897 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.897 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.897 DEBUG:boto:Params: {} 2024-04-12T03:41:31.897 DEBUG:boto:Token: None 2024-04-12T03:41:31.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11 2024-04-12T03:41:31.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TPD6hUJv9UzXdfk6SzIcjvlFrb8= 2024-04-12T03:41:31.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TPD6hUJv9UzXdfk6SzIcjvlFrb8=', 'Content-Length': '0'} 2024-04-12T03:41:31.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307f07cb05b5d6b8-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.899 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:41:31.899 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:41:31.899 DEBUG:boto:Method: GET 2024-04-12T03:41:31.900 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:41:31.900 DEBUG:boto:Data: 2024-04-12T03:41:31.900 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.900 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.900 DEBUG:boto:Params: {} 2024-04-12T03:41:31.900 DEBUG:boto:Token: None 2024-04-12T03:41:31.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj11 2024-04-12T03:41:31.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TPD6hUJv9UzXdfk6SzIcjvlFrb8= 2024-04-12T03:41:31.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TPD6hUJv9UzXdfk6SzIcjvlFrb8=', 'Content-Length': '0'} 2024-04-12T03:41:31.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271efbc229c9b780-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.902 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.902 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.903 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.903 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.903 DEBUG:boto:Method: GET 2024-04-12T03:41:31.903 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.903 DEBUG:boto:Data: 2024-04-12T03:41:31.903 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.903 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.903 DEBUG:boto:Params: {} 2024-04-12T03:41:31.903 DEBUG:boto:Token: None 2024-04-12T03:41:31.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs= 2024-04-12T03:41:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs=', 'Content-Length': '0'} 2024-04-12T03:41:31.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff261003bb368502-006618ad6b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.906 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.906 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.906 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.906 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.907 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.907 DEBUG:boto:Method: GET 2024-04-12T03:41:31.907 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:41:31.907 DEBUG:boto:Data: 2024-04-12T03:41:31.907 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.907 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.907 DEBUG:boto:Params: {} 2024-04-12T03:41:31.907 DEBUG:boto:Token: None 2024-04-12T03:41:31.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs= 2024-04-12T03:41:31.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs=', 'Content-Length': '0'} 2024-04-12T03:41:31.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004835256811856c8-006618ad6b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.950 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:31.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:41:31.951 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.951 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.951 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.951 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.951 DEBUG:boto:Method: GET 2024-04-12T03:41:31.951 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.951 DEBUG:boto:Data: 2024-04-12T03:41:31.951 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.951 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.951 DEBUG:boto:Params: {} 2024-04-12T03:41:31.951 DEBUG:boto:Token: None 2024-04-12T03:41:31.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q= 2024-04-12T03:41:31.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q=', 'Content-Length': '0'} 2024-04-12T03:41:31.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd019d8a7650619-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.954 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.954 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.954 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.954 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.954 DEBUG:boto:Method: GET 2024-04-12T03:41:31.954 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.954 DEBUG:boto:Data: 2024-04-12T03:41:31.955 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.955 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.955 DEBUG:boto:Params: {} 2024-04-12T03:41:31.955 DEBUG:boto:Token: None 2024-04-12T03:41:31.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q= 2024-04-12T03:41:31.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q=', 'Content-Length': '0'} 2024-04-12T03:41:31.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb3a81d78f5ea29f-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.957 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.957 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.957 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.957 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.957 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.958 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.958 DEBUG:boto:Data: 2024-04-12T03:41:31.958 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.958 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.958 DEBUG:boto:Params: {} 2024-04-12T03:41:31.958 DEBUG:boto:Token: None 2024-04-12T03:41:31.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U6QD/FBHDlfo+YUyjnSzAwxatGw= 2024-04-12T03:41:31.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U6QD/FBHDlfo+YUyjnSzAwxatGw=', 'Content-Length': '0'} 2024-04-12T03:41:31.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d13ea6618ba7a8-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.960 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.960 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.961 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.961 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.961 DEBUG:boto:Method: HEAD 2024-04-12T03:41:31.961 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.961 DEBUG:boto:Data: 2024-04-12T03:41:31.961 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.961 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.961 DEBUG:boto:Params: {} 2024-04-12T03:41:31.961 DEBUG:boto:Token: None 2024-04-12T03:41:31.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.962 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U6QD/FBHDlfo+YUyjnSzAwxatGw= 2024-04-12T03:41:31.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U6QD/FBHDlfo+YUyjnSzAwxatGw=', 'Content-Length': '0'} 2024-04-12T03:41:31.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dded74b5a96052bd-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:41:31.964 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.964 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.964 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.964 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.964 DEBUG:boto:Method: GET 2024-04-12T03:41:31.964 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.964 DEBUG:boto:Data: 2024-04-12T03:41:31.964 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.964 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.964 DEBUG:boto:Params: {} 2024-04-12T03:41:31.965 DEBUG:boto:Token: None 2024-04-12T03:41:31.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.965 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q= 2024-04-12T03:41:31.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q=', 'Content-Length': '0'} 2024-04-12T03:41:31.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b69606518ca8e14-006618ad6b-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.967 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:31.967 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:31.967 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.967 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.967 DEBUG:boto:Method: GET 2024-04-12T03:41:31.967 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.967 DEBUG:boto:Data: 2024-04-12T03:41:31.967 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:31.968 DEBUG:boto:Port: 8000 2024-04-12T03:41:31.968 DEBUG:boto:Params: {} 2024-04-12T03:41:31.968 DEBUG:boto:Token: None 2024-04-12T03:41:31.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:41:31.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q= 2024-04-12T03:41:31.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SHd+fxFG0vjc3Pt4Xsi/krBYl7Q=', 'Content-Length': '0'} 2024-04-12T03:41:31.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d6d9718160a9d4-006618ad6b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:31.970 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:31.970 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:31.970 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:31.970 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:31.971 DEBUG:boto:Method: GET 2024-04-12T03:41:31.971 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:41:31.971 DEBUG:boto:Data: 2024-04-12T03:41:31.971 DEBUG:boto:Headers: {} 2024-04-12T03:41:31.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:31.971 DEBUG:boto:Port: 8001 2024-04-12T03:41:31.971 DEBUG:boto:Params: {} 2024-04-12T03:41:31.971 DEBUG:boto:Token: None 2024-04-12T03:41:31.971 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:31 GMT /dgbpxg-61/ 2024-04-12T03:41:31.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs= 2024-04-12T03:41:31.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MAY4cmdAKgfWnRce6/lsE9i0Obs=', 'Content-Length': '0'} 2024-04-12T03:41:31.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf85e6da220b7e9-006618ad6b-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.014 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.014 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.015 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.015 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.015 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.015 DEBUG:boto:Method: GET 2024-04-12T03:41:32.015 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.015 DEBUG:boto:Data: 2024-04-12T03:41:32.015 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.015 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.015 DEBUG:boto:Params: {} 2024-04-12T03:41:32.016 DEBUG:boto:Token: None 2024-04-12T03:41:32.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d99021a5a4f91c3-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.018 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.018 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:32.018 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:32.018 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.019 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:41:32.019 DEBUG:boto:Data: 2024-04-12T03:41:32.019 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.019 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.019 DEBUG:boto:Params: {} 2024-04-12T03:41:32.019 DEBUG:boto:Token: None 2024-04-12T03:41:32.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj22 2024-04-12T03:41:32.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a1Vg9ZE9sG9YRks7LZVXwfC74I0= 2024-04-12T03:41:32.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a1Vg9ZE9sG9YRks7LZVXwfC74I0=', 'Content-Length': '0'} 2024-04-12T03:41:32.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113ed1b6217913f2-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.021 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:32.021 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:32.021 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.022 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:41:32.022 DEBUG:boto:Data: 2024-04-12T03:41:32.022 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.022 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.022 DEBUG:boto:Params: {} 2024-04-12T03:41:32.022 DEBUG:boto:Token: None 2024-04-12T03:41:32.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj22 2024-04-12T03:41:32.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a1Vg9ZE9sG9YRks7LZVXwfC74I0= 2024-04-12T03:41:32.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a1Vg9ZE9sG9YRks7LZVXwfC74I0=', 'Content-Length': '0'} 2024-04-12T03:41:32.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008862c119f55fbf3-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:41:32.024 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:32.024 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:32.024 DEBUG:boto:Method: GET 2024-04-12T03:41:32.024 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:41:32.024 DEBUG:boto:Data: 2024-04-12T03:41:32.025 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.025 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.025 DEBUG:boto:Params: {} 2024-04-12T03:41:32.025 DEBUG:boto:Token: None 2024-04-12T03:41:32.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj22 2024-04-12T03:41:32.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPvz0qmpSFW+y60uDNU7QRIuoLs= 2024-04-12T03:41:32.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPvz0qmpSFW+y60uDNU7QRIuoLs=', 'Content-Length': '0'} 2024-04-12T03:41:32.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd6252daaef1197-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.027 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:41:32.027 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:41:32.027 DEBUG:boto:Method: GET 2024-04-12T03:41:32.027 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:41:32.028 DEBUG:boto:Data: 2024-04-12T03:41:32.028 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.028 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.028 DEBUG:boto:Params: {} 2024-04-12T03:41:32.028 DEBUG:boto:Token: None 2024-04-12T03:41:32.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj22 2024-04-12T03:41:32.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPvz0qmpSFW+y60uDNU7QRIuoLs= 2024-04-12T03:41:32.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPvz0qmpSFW+y60uDNU7QRIuoLs=', 'Content-Length': '0'} 2024-04-12T03:41:32.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c72b4dd2bb5a9321-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.030 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.030 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.030 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.031 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.031 DEBUG:boto:Method: GET 2024-04-12T03:41:32.031 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.031 DEBUG:boto:Data: 2024-04-12T03:41:32.031 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.031 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.031 DEBUG:boto:Params: {} 2024-04-12T03:41:32.031 DEBUG:boto:Token: None 2024-04-12T03:41:32.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9fd55dfa668d0c-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.033 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.034 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.034 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.034 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.034 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.034 DEBUG:boto:Method: GET 2024-04-12T03:41:32.034 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:41:32.034 DEBUG:boto:Data: 2024-04-12T03:41:32.034 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.035 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.035 DEBUG:boto:Params: {} 2024-04-12T03:41:32.035 DEBUG:boto:Token: None 2024-04-12T03:41:32.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fea52ef422d3c2d-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.037 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:41:32.038 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.038 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.038 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.038 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.038 DEBUG:boto:Method: GET 2024-04-12T03:41:32.038 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.038 DEBUG:boto:Data: 2024-04-12T03:41:32.038 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.038 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.038 DEBUG:boto:Params: {} 2024-04-12T03:41:32.038 DEBUG:boto:Token: None 2024-04-12T03:41:32.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc= 2024-04-12T03:41:32.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc=', 'Content-Length': '0'} 2024-04-12T03:41:32.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954803668090d3cb-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.040 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.041 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.041 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.041 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.041 DEBUG:boto:Method: GET 2024-04-12T03:41:32.041 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.041 DEBUG:boto:Data: 2024-04-12T03:41:32.041 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.041 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.041 DEBUG:boto:Params: {} 2024-04-12T03:41:32.041 DEBUG:boto:Token: None 2024-04-12T03:41:32.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc= 2024-04-12T03:41:32.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc=', 'Content-Length': '0'} 2024-04-12T03:41:32.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1569da935f1c8b-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.043 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.044 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.044 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.044 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.044 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.044 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.044 DEBUG:boto:Data: 2024-04-12T03:41:32.044 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.044 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.044 DEBUG:boto:Params: {} 2024-04-12T03:41:32.045 DEBUG:boto:Token: None 2024-04-12T03:41:32.045 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FCC9+/802IYIg2nPdN8E8nIlyHA= 2024-04-12T03:41:32.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FCC9+/802IYIg2nPdN8E8nIlyHA=', 'Content-Length': '0'} 2024-04-12T03:41:32.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d21b053a8f853d0-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.047 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.047 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.047 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.047 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.047 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.047 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.047 DEBUG:boto:Data: 2024-04-12T03:41:32.047 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.047 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.047 DEBUG:boto:Params: {} 2024-04-12T03:41:32.048 DEBUG:boto:Token: None 2024-04-12T03:41:32.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FCC9+/802IYIg2nPdN8E8nIlyHA= 2024-04-12T03:41:32.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FCC9+/802IYIg2nPdN8E8nIlyHA=', 'Content-Length': '0'} 2024-04-12T03:41:32.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc9a5857fffea97-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:41:32.050 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.050 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.050 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.050 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.050 DEBUG:boto:Method: GET 2024-04-12T03:41:32.050 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.050 DEBUG:boto:Data: 2024-04-12T03:41:32.050 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.050 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.050 DEBUG:boto:Params: {} 2024-04-12T03:41:32.051 DEBUG:boto:Token: None 2024-04-12T03:41:32.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc= 2024-04-12T03:41:32.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc=', 'Content-Length': '0'} 2024-04-12T03:41:32.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c5df52d47c80ad-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.098 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.098 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.098 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.099 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.099 DEBUG:boto:Method: GET 2024-04-12T03:41:32.099 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.099 DEBUG:boto:Data: 2024-04-12T03:41:32.099 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.099 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.099 DEBUG:boto:Params: {} 2024-04-12T03:41:32.099 DEBUG:boto:Token: None 2024-04-12T03:41:32.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:41:32.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc= 2024-04-12T03:41:32.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dsC58Vv0oNjTAjt5eAmgoPo5Wkc=', 'Content-Length': '0'} 2024-04-12T03:41:32.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2454fe4ffee0b3e-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.101 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.101 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.101 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.102 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.102 DEBUG:boto:Method: GET 2024-04-12T03:41:32.102 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.102 DEBUG:boto:Data: 2024-04-12T03:41:32.102 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.102 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.102 DEBUG:boto:Params: {} 2024-04-12T03:41:32.102 DEBUG:boto:Token: None 2024-04-12T03:41:32.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb4833921582f71-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.105 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.105 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.105 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.105 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.105 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.105 DEBUG:boto:Method: GET 2024-04-12T03:41:32.106 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.106 DEBUG:boto:Data: 2024-04-12T03:41:32.106 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.106 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.106 DEBUG:boto:Params: {} 2024-04-12T03:41:32.106 DEBUG:boto:Token: None 2024-04-12T03:41:32.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017e1c06decba43f1-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.108 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.109 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.109 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.109 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.109 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:41:32.109 DEBUG:boto:Data: 2024-04-12T03:41:32.109 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.109 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.109 DEBUG:boto:Params: {} 2024-04-12T03:41:32.109 DEBUG:boto:Token: None 2024-04-12T03:41:32.109 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33 2024-04-12T03:41:32.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPsCrBJRIDyXsnnD2qyV732oUGk= 2024-04-12T03:41:32.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPsCrBJRIDyXsnnD2qyV732oUGk=', 'Content-Length': '0'} 2024-04-12T03:41:32.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f70498f4c3521c-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.111 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.111 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.111 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.111 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:41:32.112 DEBUG:boto:Data: 2024-04-12T03:41:32.112 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.112 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.112 DEBUG:boto:Params: {} 2024-04-12T03:41:32.112 DEBUG:boto:Token: None 2024-04-12T03:41:32.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33 2024-04-12T03:41:32.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SPsCrBJRIDyXsnnD2qyV732oUGk= 2024-04-12T03:41:32.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SPsCrBJRIDyXsnnD2qyV732oUGk=', 'Content-Length': '0'} 2024-04-12T03:41:32.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c73923add3cd4e3-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:41:32.114 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.114 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.114 DEBUG:boto:Method: GET 2024-04-12T03:41:32.114 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:41:32.115 DEBUG:boto:Data: 2024-04-12T03:41:32.115 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.115 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.115 DEBUG:boto:Params: {} 2024-04-12T03:41:32.115 DEBUG:boto:Token: None 2024-04-12T03:41:32.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33 2024-04-12T03:41:32.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8cpfkExl0VD5dIoNNELJVuzhdZY= 2024-04-12T03:41:32.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8cpfkExl0VD5dIoNNELJVuzhdZY=', 'Content-Length': '0'} 2024-04-12T03:41:32.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201b00d8193d6aaa-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.117 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:41:32.117 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:41:32.117 DEBUG:boto:Method: GET 2024-04-12T03:41:32.117 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:41:32.117 DEBUG:boto:Data: 2024-04-12T03:41:32.117 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.117 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.118 DEBUG:boto:Params: {} 2024-04-12T03:41:32.118 DEBUG:boto:Token: None 2024-04-12T03:41:32.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj33 2024-04-12T03:41:32.118 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8cpfkExl0VD5dIoNNELJVuzhdZY= 2024-04-12T03:41:32.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8cpfkExl0VD5dIoNNELJVuzhdZY=', 'Content-Length': '0'} 2024-04-12T03:41:32.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eed9472a3a3cd745-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.120 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.120 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.120 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.120 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.120 DEBUG:boto:Method: GET 2024-04-12T03:41:32.120 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.120 DEBUG:boto:Data: 2024-04-12T03:41:32.120 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.120 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.120 DEBUG:boto:Params: {} 2024-04-12T03:41:32.121 DEBUG:boto:Token: None 2024-04-12T03:41:32.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf431e0bc1c9f6e8-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.123 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.123 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.123 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.124 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.124 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.124 DEBUG:boto:Method: GET 2024-04-12T03:41:32.124 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:41:32.124 DEBUG:boto:Data: 2024-04-12T03:41:32.124 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.124 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.124 DEBUG:boto:Params: {} 2024-04-12T03:41:32.124 DEBUG:boto:Token: None 2024-04-12T03:41:32.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043751730eab7d14e-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.126 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:41:32.127 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.127 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.127 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.127 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.127 DEBUG:boto:Method: GET 2024-04-12T03:41:32.127 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.127 DEBUG:boto:Data: 2024-04-12T03:41:32.128 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.128 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.128 DEBUG:boto:Params: {} 2024-04-12T03:41:32.128 DEBUG:boto:Token: None 2024-04-12T03:41:32.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY= 2024-04-12T03:41:32.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY=', 'Content-Length': '0'} 2024-04-12T03:41:32.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31b68294c3d23ff-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.170 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.170 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.170 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.171 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.171 DEBUG:boto:Method: GET 2024-04-12T03:41:32.171 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.171 DEBUG:boto:Data: 2024-04-12T03:41:32.171 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.171 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.171 DEBUG:boto:Params: {} 2024-04-12T03:41:32.171 DEBUG:boto:Token: None 2024-04-12T03:41:32.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY= 2024-04-12T03:41:32.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY=', 'Content-Length': '0'} 2024-04-12T03:41:32.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d5e9d992802af4-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.174 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.174 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.174 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.174 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.174 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.174 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.174 DEBUG:boto:Data: 2024-04-12T03:41:32.174 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.175 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.175 DEBUG:boto:Params: {} 2024-04-12T03:41:32.175 DEBUG:boto:Token: None 2024-04-12T03:41:32.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RVm18+6AJZvFEfdxnhK65jOsxiw= 2024-04-12T03:41:32.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RVm18+6AJZvFEfdxnhK65jOsxiw=', 'Content-Length': '0'} 2024-04-12T03:41:32.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1582210dceb4f03-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.177 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.177 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.177 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.177 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.177 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.177 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.177 DEBUG:boto:Data: 2024-04-12T03:41:32.178 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.178 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.178 DEBUG:boto:Params: {} 2024-04-12T03:41:32.178 DEBUG:boto:Token: None 2024-04-12T03:41:32.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RVm18+6AJZvFEfdxnhK65jOsxiw= 2024-04-12T03:41:32.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RVm18+6AJZvFEfdxnhK65jOsxiw=', 'Content-Length': '0'} 2024-04-12T03:41:32.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed23b625d28bde5d-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:41:32.180 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.180 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.180 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.180 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.180 DEBUG:boto:Method: GET 2024-04-12T03:41:32.181 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.181 DEBUG:boto:Data: 2024-04-12T03:41:32.181 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.181 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.181 DEBUG:boto:Params: {} 2024-04-12T03:41:32.181 DEBUG:boto:Token: None 2024-04-12T03:41:32.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY= 2024-04-12T03:41:32.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY=', 'Content-Length': '0'} 2024-04-12T03:41:32.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de02540d5db2a89d-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.183 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.183 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.183 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.183 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.183 DEBUG:boto:Method: GET 2024-04-12T03:41:32.184 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.184 DEBUG:boto:Data: 2024-04-12T03:41:32.184 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.184 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.184 DEBUG:boto:Params: {} 2024-04-12T03:41:32.184 DEBUG:boto:Token: None 2024-04-12T03:41:32.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:41:32.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY= 2024-04-12T03:41:32.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9op38w8OS8BbwAVa3n+XIgLtXdY=', 'Content-Length': '0'} 2024-04-12T03:41:32.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033cc2177ee22cd1d-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.186 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.186 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.187 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.187 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.187 DEBUG:boto:Method: GET 2024-04-12T03:41:32.187 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.187 DEBUG:boto:Data: 2024-04-12T03:41:32.187 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.187 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.187 DEBUG:boto:Params: {} 2024-04-12T03:41:32.187 DEBUG:boto:Token: None 2024-04-12T03:41:32.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c7df48730af939-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.190 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.190 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.191 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.191 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.191 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.191 DEBUG:boto:Method: GET 2024-04-12T03:41:32.191 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.191 DEBUG:boto:Data: 2024-04-12T03:41:32.191 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.191 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.191 DEBUG:boto:Params: {} 2024-04-12T03:41:32.192 DEBUG:boto:Token: None 2024-04-12T03:41:32.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064df3749e1d24774-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.234 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.235 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.235 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.235 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.235 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:41:32.235 DEBUG:boto:Data: 2024-04-12T03:41:32.235 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.235 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.235 DEBUG:boto:Params: {} 2024-04-12T03:41:32.235 DEBUG:boto:Token: None 2024-04-12T03:41:32.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44 2024-04-12T03:41:32.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rh5GUkgs1fnISM3VuzxTxMv+5fo= 2024-04-12T03:41:32.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rh5GUkgs1fnISM3VuzxTxMv+5fo=', 'Content-Length': '0'} 2024-04-12T03:41:32.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024047bfe34bf79a9-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.237 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.237 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.238 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.238 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:41:32.238 DEBUG:boto:Data: 2024-04-12T03:41:32.238 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.238 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.238 DEBUG:boto:Params: {} 2024-04-12T03:41:32.238 DEBUG:boto:Token: None 2024-04-12T03:41:32.238 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44 2024-04-12T03:41:32.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rh5GUkgs1fnISM3VuzxTxMv+5fo= 2024-04-12T03:41:32.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rh5GUkgs1fnISM3VuzxTxMv+5fo=', 'Content-Length': '0'} 2024-04-12T03:41:32.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930ea5b201085dd6-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:41:32.240 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.240 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.241 DEBUG:boto:Method: GET 2024-04-12T03:41:32.241 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:41:32.241 DEBUG:boto:Data: 2024-04-12T03:41:32.241 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.241 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.241 DEBUG:boto:Params: {} 2024-04-12T03:41:32.241 DEBUG:boto:Token: None 2024-04-12T03:41:32.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44 2024-04-12T03:41:32.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QniJQ9ugVxsx7Gm7oVPQXvclRuY= 2024-04-12T03:41:32.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QniJQ9ugVxsx7Gm7oVPQXvclRuY=', 'Content-Length': '0'} 2024-04-12T03:41:32.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f527bb16323c659b-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.243 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:41:32.243 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:41:32.243 DEBUG:boto:Method: GET 2024-04-12T03:41:32.243 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:41:32.243 DEBUG:boto:Data: 2024-04-12T03:41:32.243 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.244 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.244 DEBUG:boto:Params: {} 2024-04-12T03:41:32.244 DEBUG:boto:Token: None 2024-04-12T03:41:32.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj44 2024-04-12T03:41:32.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QniJQ9ugVxsx7Gm7oVPQXvclRuY= 2024-04-12T03:41:32.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QniJQ9ugVxsx7Gm7oVPQXvclRuY=', 'Content-Length': '0'} 2024-04-12T03:41:32.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0efa3bf0615863-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.246 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.246 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.246 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.246 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.246 DEBUG:boto:Method: GET 2024-04-12T03:41:32.246 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.247 DEBUG:boto:Data: 2024-04-12T03:41:32.247 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.247 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.247 DEBUG:boto:Params: {} 2024-04-12T03:41:32.247 DEBUG:boto:Token: None 2024-04-12T03:41:32.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea8937fc0353e58d-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.294 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.295 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.295 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.295 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.295 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.295 DEBUG:boto:Method: GET 2024-04-12T03:41:32.295 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:41:32.295 DEBUG:boto:Data: 2024-04-12T03:41:32.295 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.295 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.295 DEBUG:boto:Params: {} 2024-04-12T03:41:32.296 DEBUG:boto:Token: None 2024-04-12T03:41:32.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66ce93e56d8ddd1-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.298 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:41:32.299 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.299 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.299 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.299 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.299 DEBUG:boto:Method: GET 2024-04-12T03:41:32.299 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.299 DEBUG:boto:Data: 2024-04-12T03:41:32.299 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.299 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.299 DEBUG:boto:Params: {} 2024-04-12T03:41:32.300 DEBUG:boto:Token: None 2024-04-12T03:41:32.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE= 2024-04-12T03:41:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE=', 'Content-Length': '0'} 2024-04-12T03:41:32.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763bda57ce09e6f5-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.302 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.302 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.302 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.302 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.302 DEBUG:boto:Method: GET 2024-04-12T03:41:32.302 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.302 DEBUG:boto:Data: 2024-04-12T03:41:32.302 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.302 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.302 DEBUG:boto:Params: {} 2024-04-12T03:41:32.303 DEBUG:boto:Token: None 2024-04-12T03:41:32.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE= 2024-04-12T03:41:32.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE=', 'Content-Length': '0'} 2024-04-12T03:41:32.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d70f567304ffb6-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.305 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.305 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.305 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.305 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.305 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.305 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.305 DEBUG:boto:Data: 2024-04-12T03:41:32.305 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.305 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.305 DEBUG:boto:Params: {} 2024-04-12T03:41:32.306 DEBUG:boto:Token: None 2024-04-12T03:41:32.306 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jpDIBAXcuAowUO3OY2acOeXE/a0= 2024-04-12T03:41:32.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jpDIBAXcuAowUO3OY2acOeXE/a0=', 'Content-Length': '0'} 2024-04-12T03:41:32.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0486762e1d90229-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.308 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.308 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.308 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.308 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.308 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.308 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.308 DEBUG:boto:Data: 2024-04-12T03:41:32.308 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.308 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.308 DEBUG:boto:Params: {} 2024-04-12T03:41:32.309 DEBUG:boto:Token: None 2024-04-12T03:41:32.309 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jpDIBAXcuAowUO3OY2acOeXE/a0= 2024-04-12T03:41:32.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jpDIBAXcuAowUO3OY2acOeXE/a0=', 'Content-Length': '0'} 2024-04-12T03:41:32.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b3e41b6bbab8ba-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:41:32.311 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.311 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.311 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.311 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.311 DEBUG:boto:Method: GET 2024-04-12T03:41:32.311 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.311 DEBUG:boto:Data: 2024-04-12T03:41:32.311 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.312 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.312 DEBUG:boto:Params: {} 2024-04-12T03:41:32.312 DEBUG:boto:Token: None 2024-04-12T03:41:32.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE= 2024-04-12T03:41:32.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE=', 'Content-Length': '0'} 2024-04-12T03:41:32.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8d087d762082fd-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.314 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.314 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.314 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.314 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.314 DEBUG:boto:Method: GET 2024-04-12T03:41:32.314 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.314 DEBUG:boto:Data: 2024-04-12T03:41:32.314 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.315 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.315 DEBUG:boto:Params: {} 2024-04-12T03:41:32.315 DEBUG:boto:Token: None 2024-04-12T03:41:32.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:41:32.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE= 2024-04-12T03:41:32.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gd83XnTT3A746NctRFuK6sR3kKE=', 'Content-Length': '0'} 2024-04-12T03:41:32.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8a402554cc787c-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.358 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.359 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.359 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.359 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.359 DEBUG:boto:Method: GET 2024-04-12T03:41:32.359 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.359 DEBUG:boto:Data: 2024-04-12T03:41:32.359 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.359 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.359 DEBUG:boto:Params: {} 2024-04-12T03:41:32.359 DEBUG:boto:Token: None 2024-04-12T03:41:32.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f6232fbb2ad0a5-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.362 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.362 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.362 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.362 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.363 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.363 DEBUG:boto:Method: GET 2024-04-12T03:41:32.363 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.363 DEBUG:boto:Data: 2024-04-12T03:41:32.363 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.363 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.363 DEBUG:boto:Params: {} 2024-04-12T03:41:32.363 DEBUG:boto:Token: None 2024-04-12T03:41:32.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590807b0d8432e15-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.365 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.366 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.366 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.366 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.366 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:41:32.366 DEBUG:boto:Data: 2024-04-12T03:41:32.366 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.366 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.366 DEBUG:boto:Params: {} 2024-04-12T03:41:32.367 DEBUG:boto:Token: None 2024-04-12T03:41:32.367 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55 2024-04-12T03:41:32.367 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CQQGl+wOx8HXEe5VOunR/esEjrM= 2024-04-12T03:41:32.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CQQGl+wOx8HXEe5VOunR/esEjrM=', 'Content-Length': '0'} 2024-04-12T03:41:32.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ea89feda69b8d7-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.368 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.369 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.369 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.369 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:41:32.369 DEBUG:boto:Data: 2024-04-12T03:41:32.369 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.369 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.369 DEBUG:boto:Params: {} 2024-04-12T03:41:32.369 DEBUG:boto:Token: None 2024-04-12T03:41:32.369 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55 2024-04-12T03:41:32.370 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CQQGl+wOx8HXEe5VOunR/esEjrM= 2024-04-12T03:41:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CQQGl+wOx8HXEe5VOunR/esEjrM=', 'Content-Length': '0'} 2024-04-12T03:41:32.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f05ce60305450e-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:41:32.371 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.371 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.372 DEBUG:boto:Method: GET 2024-04-12T03:41:32.372 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:41:32.372 DEBUG:boto:Data: 2024-04-12T03:41:32.372 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.372 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.372 DEBUG:boto:Params: {} 2024-04-12T03:41:32.372 DEBUG:boto:Token: None 2024-04-12T03:41:32.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55 2024-04-12T03:41:32.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X6pkB8kvJAF7ws6BPstDHYpmiLc= 2024-04-12T03:41:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X6pkB8kvJAF7ws6BPstDHYpmiLc=', 'Content-Length': '0'} 2024-04-12T03:41:32.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece0bf987b0630b1-006618ad6c-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.374 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:41:32.374 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:41:32.374 DEBUG:boto:Method: GET 2024-04-12T03:41:32.374 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:41:32.375 DEBUG:boto:Data: 2024-04-12T03:41:32.375 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.375 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.375 DEBUG:boto:Params: {} 2024-04-12T03:41:32.375 DEBUG:boto:Token: None 2024-04-12T03:41:32.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj55 2024-04-12T03:41:32.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X6pkB8kvJAF7ws6BPstDHYpmiLc= 2024-04-12T03:41:32.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X6pkB8kvJAF7ws6BPstDHYpmiLc=', 'Content-Length': '0'} 2024-04-12T03:41:32.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d55c890809a354-006618ad6c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.377 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.377 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.378 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.378 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.378 DEBUG:boto:Method: GET 2024-04-12T03:41:32.378 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.378 DEBUG:boto:Data: 2024-04-12T03:41:32.378 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:41:32.378 DEBUG:boto:Port: 8001 2024-04-12T03:41:32.378 DEBUG:boto:Params: {} 2024-04-12T03:41:32.378 DEBUG:boto:Token: None 2024-04-12T03:41:32.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9633fd2e05c3757-006618ad6c-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.381 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.381 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.381 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.381 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.381 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.381 DEBUG:boto:Method: GET 2024-04-12T03:41:32.382 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:41:32.382 DEBUG:boto:Data: 2024-04-12T03:41:32.382 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.382 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.382 DEBUG:boto:Params: {} 2024-04-12T03:41:32.382 DEBUG:boto:Token: None 2024-04-12T03:41:32.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY= 2024-04-12T03:41:32.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iM5QcAJgtLmKDdmZdZwl5yODOMY=', 'Content-Length': '0'} 2024-04-12T03:41:32.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd14f30ae070764-006618ad6c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.384 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:41:32.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:41:32.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:32.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:32.385 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:41:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:41:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2024-04-12T03:41:32.522 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.522 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.522 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.522 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:32.522 DEBUG:boto:Data: 2024-04-12T03:41:32.522 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.523 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.523 DEBUG:boto:Params: {} 2024-04-12T03:41:32.523 DEBUG:boto:Token: None 2024-04-12T03:41:32.523 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc= 2024-04-12T03:41:32.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc=', 'Content-Length': '0'} 2024-04-12T03:41:32.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008703767dcca4cece-006618ad6c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.526 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:41:32.526 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:41:32.526 DEBUG:boto:Method: PUT 2024-04-12T03:41:32.526 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:41:32.526 DEBUG:boto:Data: 2024-04-12T03:41:32.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:41:32.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.527 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.527 DEBUG:boto:Params: {} 2024-04-12T03:41:32.527 DEBUG:boto:Token: None 2024-04-12T03:41:32.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj17 2024-04-12T03:41:32.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWIVOPpYWozmyNjVvWXn6djd7NU= 2024-04-12T03:41:32.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWIVOPpYWozmyNjVvWXn6djd7NU='} 2024-04-12T03:41:32.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac4fb6ce99994120-006618ad6c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.533 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.533 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.533 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.533 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:32.533 DEBUG:boto:Data: 2024-04-12T03:41:32.533 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.533 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.533 DEBUG:boto:Params: {} 2024-04-12T03:41:32.533 DEBUG:boto:Token: None 2024-04-12T03:41:32.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc= 2024-04-12T03:41:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc=', 'Content-Length': '0'} 2024-04-12T03:41:32.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba03ac28a62b8ad8-006618ad6c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.537 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:41:32.537 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:41:32.537 DEBUG:boto:Method: PUT 2024-04-12T03:41:32.537 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:41:32.537 DEBUG:boto:Data: 2024-04-12T03:41:32.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:41:32.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.537 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.537 DEBUG:boto:Params: {} 2024-04-12T03:41:32.537 DEBUG:boto:Token: None 2024-04-12T03:41:32.538 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj34 2024-04-12T03:41:32.538 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tStoULxF2LvYhgZBj2mGISpqp8U= 2024-04-12T03:41:32.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tStoULxF2LvYhgZBj2mGISpqp8U='} 2024-04-12T03:41:32.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008da7386f8439bb92-006618ad6c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.583 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.583 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.583 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.583 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:32.583 DEBUG:boto:Data: 2024-04-12T03:41:32.583 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.583 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.583 DEBUG:boto:Params: {} 2024-04-12T03:41:32.584 DEBUG:boto:Token: None 2024-04-12T03:41:32.584 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc= 2024-04-12T03:41:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc=', 'Content-Length': '0'} 2024-04-12T03:41:32.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdd41bab097be40b-006618ad6c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.587 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:41:32.587 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:41:32.587 DEBUG:boto:Method: PUT 2024-04-12T03:41:32.587 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:41:32.587 DEBUG:boto:Data: 2024-04-12T03:41:32.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:41:32.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.588 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.588 DEBUG:boto:Params: {} 2024-04-12T03:41:32.588 DEBUG:boto:Token: None 2024-04-12T03:41:32.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj51 2024-04-12T03:41:32.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IFdGz4IGcyi0SYEzHrsGACGcAcE= 2024-04-12T03:41:32.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IFdGz4IGcyi0SYEzHrsGACGcAcE='} 2024-04-12T03:41:32.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b6bab6c25d8810d-006618ad6c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.594 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.594 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.595 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.595 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:32.595 DEBUG:boto:Data: 2024-04-12T03:41:32.595 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.595 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.595 DEBUG:boto:Params: {} 2024-04-12T03:41:32.595 DEBUG:boto:Token: None 2024-04-12T03:41:32.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc= 2024-04-12T03:41:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc=', 'Content-Length': '0'} 2024-04-12T03:41:32.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ebe964bea00b727-006618ad6c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.598 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:41:32.598 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:41:32.599 DEBUG:boto:Method: PUT 2024-04-12T03:41:32.599 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:41:32.599 DEBUG:boto:Data: 2024-04-12T03:41:32.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:41:32.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.599 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.599 DEBUG:boto:Params: {} 2024-04-12T03:41:32.599 DEBUG:boto:Token: None 2024-04-12T03:41:32.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj68 2024-04-12T03:41:32.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0SfeDhofH4395zhZN1P3fr1Mgjk= 2024-04-12T03:41:32.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0SfeDhofH4395zhZN1P3fr1Mgjk='} 2024-04-12T03:41:32.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c844db07200256b-006618ad6c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.643 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:41:32.643 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:41:32.643 DEBUG:boto:Method: HEAD 2024-04-12T03:41:32.643 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:41:32.643 DEBUG:boto:Data: 2024-04-12T03:41:32.643 DEBUG:boto:Headers: {} 2024-04-12T03:41:32.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.643 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.643 DEBUG:boto:Params: {} 2024-04-12T03:41:32.644 DEBUG:boto:Token: None 2024-04-12T03:41:32.644 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/ 2024-04-12T03:41:32.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc= 2024-04-12T03:41:32.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ev9hDRh8sUkF+chViV9368QLUDc=', 'Content-Length': '0'} 2024-04-12T03:41:32.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a7f2a0bdb540e6-006618ad6c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.647 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:41:32.647 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:41:32.647 DEBUG:boto:Method: PUT 2024-04-12T03:41:32.647 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:41:32.647 DEBUG:boto:Data: 2024-04-12T03:41:32.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:41:32.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:41:32.647 DEBUG:boto:Port: 8000 2024-04-12T03:41:32.647 DEBUG:boto:Params: {} 2024-04-12T03:41:32.647 DEBUG:boto:Token: None 2024-04-12T03:41:32.648 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:41:32 GMT /dgbpxg-61/obj85 2024-04-12T03:41:32.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BBfbS6/gZIfZSQ7aAslEptpklNo= 2024-04-12T03:41:32.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:41:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BBfbS6/gZIfZSQ7aAslEptpklNo='} 2024-04-12T03:41:32.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb8e74d63cfaa924-006618ad6c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:32.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:41:32.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:41:32.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:41:32.654 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:41:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:32.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:32.742+0000 7f8d80983780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:41:37.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:37.744+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:41:37.745 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-12T03:41:37.745 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:41:42.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:42.744+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:41:42.746 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:41:42.746 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:41:47.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:47.746+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:41:47.748 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:41:47.748 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:41:52.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:52.748+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:41:52.749 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:41:52.749 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:41:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:41:57.750+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:41:57.751 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:41:57.751 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:02.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:02.751+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:02.752 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:02.753 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:07.753+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:12.754+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:12.755 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:12.755 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:17.756+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:17.757 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:17.757 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:22.757+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:22.759 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:22.759 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:27.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:27.758+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:27.760 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:27.760 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:32.759+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:32.761 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:32.762 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:37.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:37.762+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:37.764 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:37.764 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:42.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:42.763+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:42.765 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:42.765 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:47.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:47.765+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:47.766 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:47.766 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:52.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:52.766+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:52.770 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:52.770 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:42:57.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:42:57.768+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:42:57.770 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:42:57.770 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:02.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:02.770+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:02.771 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:02.771 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:07.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:07.770+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:07.773 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:07.773 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:12.772+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:12.774 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:12.774 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:17.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:17.775+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:17.776 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:17.776 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:22.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:22.776+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:22.777 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:22.777 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:27.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:27.778+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:27.779 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:27.779 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:32.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:32.779+0000 7f8d80983780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:32.781 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, , , , ] 2024-04-12T03:43:32.781 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:37.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:37.781+0000 7f8d80983780 1 bucket sync caught up with source: 2024-04-12T03:43:37.782 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:37.782 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.56.6, , , , , , 00000000001.104.6, , 00000000001.75.6, , , , 00000000001.101.6, 00000000001.49.6, , , ] 2024-04-12T03:43:37.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:37.781+0000 7f8d80983780 0 bucket checkpoint complete 2024-04-12T03:43:37.800 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:43:37.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:43:37.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:43:37.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:43:37.862 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:37.863 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:37.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:37.900+0000 7f2fdd54a780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:37.901 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1066.6, , , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:37.901 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1066.6, 00000000001.2091.6, , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:42.901+0000 7f2fdd54a780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:42.902 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1066.6, , , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:42.902 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1066.6, 00000000001.2091.6, , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:47.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:47.902+0000 7f2fdd54a780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:47.904 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1066.6, , , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:47.904 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1066.6, 00000000001.2091.6, , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:52.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:52.903+0000 7f2fdd54a780 1 bucket sync caught up with source: 2024-04-12T03:43:52.905 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1066.6, 00000000001.2091.6, , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:52.905 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1066.6, 00000000001.2091.6, , , 00000000001.434.6, , , 00000000001.419.6, 00000000003.2230.6, 00000000003.1483.6] 2024-04-12T03:43:52.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:52.903+0000 7f2fdd54a780 0 bucket checkpoint complete 2024-04-12T03:43:52.919 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:43:52.920 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.920 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.920 DEBUG:boto:Method: HEAD 2024-04-12T03:43:52.920 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:52.920 DEBUG:boto:Data: 2024-04-12T03:43:52.920 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.920 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.920 DEBUG:boto:Params: {} 2024-04-12T03:43:52.921 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:43:52.921 DEBUG:boto:Token: None 2024-04-12T03:43:52.921 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/ 2024-04-12T03:43:52.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z36DSVQhDzBbRK32/KF/8z4B7yI= 2024-04-12T03:43:52.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z36DSVQhDzBbRK32/KF/8z4B7yI=', 'Content-Length': '0'} 2024-04-12T03:43:52.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e11b6a6c196a1c4-006618adf8-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.934 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.934 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.934 DEBUG:boto:Method: HEAD 2024-04-12T03:43:52.934 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:52.934 DEBUG:boto:Data: 2024-04-12T03:43:52.934 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.935 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.935 DEBUG:boto:Params: {} 2024-04-12T03:43:52.935 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:43:52.935 DEBUG:boto:Token: None 2024-04-12T03:43:52.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/ 2024-04-12T03:43:52.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z36DSVQhDzBbRK32/KF/8z4B7yI= 2024-04-12T03:43:52.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z36DSVQhDzBbRK32/KF/8z4B7yI=', 'Content-Length': '0'} 2024-04-12T03:43:52.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008641f4838f324dc4-006618adf8-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.949 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:43:52.949 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.949 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.949 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.949 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.950 DEBUG:boto:Method: GET 2024-04-12T03:43:52.950 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.950 DEBUG:boto:Data: 2024-04-12T03:43:52.950 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.950 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.950 DEBUG:boto:Params: {} 2024-04-12T03:43:52.951 DEBUG:boto:Token: None 2024-04-12T03:43:52.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/?versions 2024-04-12T03:43:52.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8= 2024-04-12T03:43:52.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8=', 'Content-Length': '0'} 2024-04-12T03:43:52.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f846359bb1cdc87-006618adf8-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.954 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:52.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:43:52.956 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:43:52.957 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.957 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.957 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.957 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.957 DEBUG:boto:Method: GET 2024-04-12T03:43:52.957 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.957 DEBUG:boto:Data: 2024-04-12T03:43:52.957 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.958 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.958 DEBUG:boto:Params: {} 2024-04-12T03:43:52.958 DEBUG:boto:Token: None 2024-04-12T03:43:52.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/?versions 2024-04-12T03:43:52.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8= 2024-04-12T03:43:52.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8=', 'Content-Length': '0'} 2024-04-12T03:43:52.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4d784850f49f76-006618adf8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.961 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:43:52.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:43:52.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:43:52.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:43:52.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:43:52.963 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.963 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.963 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.963 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.963 DEBUG:boto:Method: GET 2024-04-12T03:43:52.963 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.963 DEBUG:boto:Data: 2024-04-12T03:43:52.963 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.964 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.964 DEBUG:boto:Params: {} 2024-04-12T03:43:52.964 DEBUG:boto:Token: None 2024-04-12T03:43:52.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/?versions 2024-04-12T03:43:52.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8= 2024-04-12T03:43:52.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8=', 'Content-Length': '0'} 2024-04-12T03:43:52.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078737520e6518255-006618adf8-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.966 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:52.967 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.967 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.967 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.968 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.968 DEBUG:boto:Method: GET 2024-04-12T03:43:52.968 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:43:52.968 DEBUG:boto:Data: 2024-04-12T03:43:52.968 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.969 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.969 DEBUG:boto:Params: {} 2024-04-12T03:43:52.969 DEBUG:boto:Token: None 2024-04-12T03:43:52.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/?versions 2024-04-12T03:43:52.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8= 2024-04-12T03:43:52.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mNqhZhTz5q8gviCM7L4rulHSb8=', 'Content-Length': '0'} 2024-04-12T03:43:52.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b44dc3b7e9a48152-006618adf8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.972 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:52.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:43:52.973 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.973 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.973 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.973 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.973 DEBUG:boto:Method: GET 2024-04-12T03:43:52.973 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.974 DEBUG:boto:Data: 2024-04-12T03:43:52.974 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.974 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.974 DEBUG:boto:Params: {} 2024-04-12T03:43:52.974 DEBUG:boto:Token: None 2024-04-12T03:43:52.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM= 2024-04-12T03:43:52.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM=', 'Content-Length': '0'} 2024-04-12T03:43:52.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f43c053b53fc9d-006618adf8-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.976 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.976 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.976 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.977 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.977 DEBUG:boto:Method: GET 2024-04-12T03:43:52.977 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.977 DEBUG:boto:Data: 2024-04-12T03:43:52.977 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.977 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.977 DEBUG:boto:Params: {} 2024-04-12T03:43:52.977 DEBUG:boto:Token: None 2024-04-12T03:43:52.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.978 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM= 2024-04-12T03:43:52.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM=', 'Content-Length': '0'} 2024-04-12T03:43:52.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3182965222e71ae-006618adf8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.980 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.980 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.980 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.980 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.980 DEBUG:boto:Method: HEAD 2024-04-12T03:43:52.981 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.981 DEBUG:boto:Data: 2024-04-12T03:43:52.981 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.981 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.981 DEBUG:boto:Params: {} 2024-04-12T03:43:52.981 DEBUG:boto:Token: None 2024-04-12T03:43:52.981 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V6e025VeCgQIHobZdUD0KWvIXLY= 2024-04-12T03:43:52.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V6e025VeCgQIHobZdUD0KWvIXLY=', 'Content-Length': '0'} 2024-04-12T03:43:52.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ddfd84ace639908-006618adf8-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.984 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.984 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.984 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.984 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.984 DEBUG:boto:Method: HEAD 2024-04-12T03:43:52.984 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.984 DEBUG:boto:Data: 2024-04-12T03:43:52.985 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.985 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.985 DEBUG:boto:Params: {} 2024-04-12T03:43:52.985 DEBUG:boto:Token: None 2024-04-12T03:43:52.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V6e025VeCgQIHobZdUD0KWvIXLY= 2024-04-12T03:43:52.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V6e025VeCgQIHobZdUD0KWvIXLY=', 'Content-Length': '0'} 2024-04-12T03:43:52.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48bbc03d19d7d9a-006618adf8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:43:52.987 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.987 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.987 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.987 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.988 DEBUG:boto:Method: GET 2024-04-12T03:43:52.988 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.988 DEBUG:boto:Data: 2024-04-12T03:43:52.988 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.988 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.988 DEBUG:boto:Params: {} 2024-04-12T03:43:52.988 DEBUG:boto:Token: None 2024-04-12T03:43:52.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM= 2024-04-12T03:43:52.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM=', 'Content-Length': '0'} 2024-04-12T03:43:52.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd619ff353153a0b-006618adf8-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.990 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:52.991 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:52.991 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.991 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.991 DEBUG:boto:Method: GET 2024-04-12T03:43:52.991 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.991 DEBUG:boto:Data: 2024-04-12T03:43:52.991 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.991 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.991 DEBUG:boto:Params: {} 2024-04-12T03:43:52.992 DEBUG:boto:Token: None 2024-04-12T03:43:52.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:43:52.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM= 2024-04-12T03:43:52.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:312fwmumdOGWrZcApttXZQD74LM=', 'Content-Length': '0'} 2024-04-12T03:43:52.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9aa811eab5f9fb8-006618adf8-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.994 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.994 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.994 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.994 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.994 DEBUG:boto:Method: GET 2024-04-12T03:43:52.994 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.994 DEBUG:boto:Data: 2024-04-12T03:43:52.994 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:52.995 DEBUG:boto:Port: 8001 2024-04-12T03:43:52.995 DEBUG:boto:Params: {} 2024-04-12T03:43:52.995 DEBUG:boto:Token: None 2024-04-12T03:43:52.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/ 2024-04-12T03:43:52.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N0qX2pVfM2XFLr3G1SR+SCLkP1Y= 2024-04-12T03:43:52.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N0qX2pVfM2XFLr3G1SR+SCLkP1Y=', 'Content-Length': '0'} 2024-04-12T03:43:52.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3219162151c84f8-006618adf8-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:52.997 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:52.998 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:52.998 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:52.998 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.998 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.998 DEBUG:boto:Method: GET 2024-04-12T03:43:52.998 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:43:52.999 DEBUG:boto:Data: 2024-04-12T03:43:52.999 DEBUG:boto:Headers: {} 2024-04-12T03:43:52.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:52.999 DEBUG:boto:Port: 8000 2024-04-12T03:43:52.999 DEBUG:boto:Params: {} 2024-04-12T03:43:52.999 DEBUG:boto:Token: None 2024-04-12T03:43:52.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:52 GMT /dgbpxg-61/ 2024-04-12T03:43:52.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N0qX2pVfM2XFLr3G1SR+SCLkP1Y= 2024-04-12T03:43:52.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N0qX2pVfM2XFLr3G1SR+SCLkP1Y=', 'Content-Length': '0'} 2024-04-12T03:43:53.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4d2197fc7f3ad6-006618adf8-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.002 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.002 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:53.002 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:53.002 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.002 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:43:53.002 DEBUG:boto:Data: 2024-04-12T03:43:53.003 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.003 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.003 DEBUG:boto:Params: {} 2024-04-12T03:43:53.003 DEBUG:boto:Token: None 2024-04-12T03:43:53.003 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj11 2024-04-12T03:43:53.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqtfReZr8adJNJ7rlv5DwVXbklI= 2024-04-12T03:43:53.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqtfReZr8adJNJ7rlv5DwVXbklI=', 'Content-Length': '0'} 2024-04-12T03:43:53.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a5321e8a69e66b-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.005 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:53.005 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:53.005 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.005 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:43:53.005 DEBUG:boto:Data: 2024-04-12T03:43:53.005 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.005 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.006 DEBUG:boto:Params: {} 2024-04-12T03:43:53.006 DEBUG:boto:Token: None 2024-04-12T03:43:53.006 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj11 2024-04-12T03:43:53.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqtfReZr8adJNJ7rlv5DwVXbklI= 2024-04-12T03:43:53.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqtfReZr8adJNJ7rlv5DwVXbklI=', 'Content-Length': '0'} 2024-04-12T03:43:53.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f9bbef3d5fac12-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:43:53.008 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:53.008 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:53.008 DEBUG:boto:Method: GET 2024-04-12T03:43:53.008 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:43:53.008 DEBUG:boto:Data: 2024-04-12T03:43:53.008 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.008 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.008 DEBUG:boto:Params: {} 2024-04-12T03:43:53.009 DEBUG:boto:Token: None 2024-04-12T03:43:53.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj11 2024-04-12T03:43:53.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b6JUA9cKpGv7ZEhvchMzgBGc9Ts= 2024-04-12T03:43:53.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b6JUA9cKpGv7ZEhvchMzgBGc9Ts=', 'Content-Length': '0'} 2024-04-12T03:43:53.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890c2c64d83e9aaa-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.011 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:43:53.011 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:43:53.011 DEBUG:boto:Method: GET 2024-04-12T03:43:53.011 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:43:53.011 DEBUG:boto:Data: 2024-04-12T03:43:53.011 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.012 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.012 DEBUG:boto:Params: {} 2024-04-12T03:43:53.012 DEBUG:boto:Token: None 2024-04-12T03:43:53.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj11 2024-04-12T03:43:53.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b6JUA9cKpGv7ZEhvchMzgBGc9Ts= 2024-04-12T03:43:53.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b6JUA9cKpGv7ZEhvchMzgBGc9Ts=', 'Content-Length': '0'} 2024-04-12T03:43:53.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f6469601518ba5-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.014 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.014 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.014 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.014 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.014 DEBUG:boto:Method: GET 2024-04-12T03:43:53.014 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.014 DEBUG:boto:Data: 2024-04-12T03:43:53.014 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.015 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.015 DEBUG:boto:Params: {} 2024-04-12T03:43:53.015 DEBUG:boto:Token: None 2024-04-12T03:43:53.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005150158cfc2ebbc2-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.058 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.059 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.059 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.059 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.059 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.059 DEBUG:boto:Method: GET 2024-04-12T03:43:53.059 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:43:53.059 DEBUG:boto:Data: 2024-04-12T03:43:53.060 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.060 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.060 DEBUG:boto:Params: {} 2024-04-12T03:43:53.060 DEBUG:boto:Token: None 2024-04-12T03:43:53.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307cc33edbfa623e-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.064 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:43:53.064 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.064 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.064 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.065 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.065 DEBUG:boto:Method: GET 2024-04-12T03:43:53.065 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.065 DEBUG:boto:Data: 2024-04-12T03:43:53.065 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.065 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.065 DEBUG:boto:Params: {} 2024-04-12T03:43:53.065 DEBUG:boto:Token: None 2024-04-12T03:43:53.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis= 2024-04-12T03:43:53.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis=', 'Content-Length': '0'} 2024-04-12T03:43:53.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f265a4fa168dd1-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.068 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.068 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.068 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.068 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.068 DEBUG:boto:Method: GET 2024-04-12T03:43:53.068 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.069 DEBUG:boto:Data: 2024-04-12T03:43:53.069 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.069 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.069 DEBUG:boto:Params: {} 2024-04-12T03:43:53.069 DEBUG:boto:Token: None 2024-04-12T03:43:53.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis= 2024-04-12T03:43:53.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis=', 'Content-Length': '0'} 2024-04-12T03:43:53.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf371866779725b-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.114 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.115 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.115 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.115 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.115 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.115 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.115 DEBUG:boto:Data: 2024-04-12T03:43:53.115 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.115 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.115 DEBUG:boto:Params: {} 2024-04-12T03:43:53.116 DEBUG:boto:Token: None 2024-04-12T03:43:53.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dGh/xd8yMNz3mJ4H4bybaSpBTCk= 2024-04-12T03:43:53.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dGh/xd8yMNz3mJ4H4bybaSpBTCk=', 'Content-Length': '0'} 2024-04-12T03:43:53.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e0d56aa236eb23-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.118 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.118 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.118 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.118 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.118 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.118 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.119 DEBUG:boto:Data: 2024-04-12T03:43:53.119 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.119 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.119 DEBUG:boto:Params: {} 2024-04-12T03:43:53.119 DEBUG:boto:Token: None 2024-04-12T03:43:53.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dGh/xd8yMNz3mJ4H4bybaSpBTCk= 2024-04-12T03:43:53.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dGh/xd8yMNz3mJ4H4bybaSpBTCk=', 'Content-Length': '0'} 2024-04-12T03:43:53.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298078d1bcf6ceec-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:43:53.121 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.121 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.121 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.122 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.122 DEBUG:boto:Method: GET 2024-04-12T03:43:53.122 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.122 DEBUG:boto:Data: 2024-04-12T03:43:53.122 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.122 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.122 DEBUG:boto:Params: {} 2024-04-12T03:43:53.122 DEBUG:boto:Token: None 2024-04-12T03:43:53.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis= 2024-04-12T03:43:53.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis=', 'Content-Length': '0'} 2024-04-12T03:43:53.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b7e9ab0bbd6f0d-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.125 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.125 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.125 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.125 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.125 DEBUG:boto:Method: GET 2024-04-12T03:43:53.125 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.125 DEBUG:boto:Data: 2024-04-12T03:43:53.125 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.125 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.126 DEBUG:boto:Params: {} 2024-04-12T03:43:53.126 DEBUG:boto:Token: None 2024-04-12T03:43:53.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:43:53.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis= 2024-04-12T03:43:53.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fYrnW21crqz6lYtxc2X0lCMRJis=', 'Content-Length': '0'} 2024-04-12T03:43:53.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843da42a44d54553-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.128 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.128 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.128 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.128 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.128 DEBUG:boto:Method: GET 2024-04-12T03:43:53.128 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.129 DEBUG:boto:Data: 2024-04-12T03:43:53.129 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.129 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.129 DEBUG:boto:Params: {} 2024-04-12T03:43:53.129 DEBUG:boto:Token: None 2024-04-12T03:43:53.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1abb6eeba4bf1c-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.132 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.132 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.133 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.133 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.133 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.133 DEBUG:boto:Method: GET 2024-04-12T03:43:53.133 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.133 DEBUG:boto:Data: 2024-04-12T03:43:53.133 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.133 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.133 DEBUG:boto:Params: {} 2024-04-12T03:43:53.134 DEBUG:boto:Token: None 2024-04-12T03:43:53.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9b988e4e0cbd5d-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.137 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.137 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.137 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.137 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.137 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:43:53.137 DEBUG:boto:Data: 2024-04-12T03:43:53.138 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.138 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.138 DEBUG:boto:Params: {} 2024-04-12T03:43:53.138 DEBUG:boto:Token: None 2024-04-12T03:43:53.138 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17 2024-04-12T03:43:53.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T2rLTMPaEPbKr+LB9kf4hbWkm1g= 2024-04-12T03:43:53.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T2rLTMPaEPbKr+LB9kf4hbWkm1g=', 'Content-Length': '0'} 2024-04-12T03:43:53.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29d2c856f3aa806-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.140 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.140 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.140 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.140 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:43:53.141 DEBUG:boto:Data: 2024-04-12T03:43:53.141 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.141 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.141 DEBUG:boto:Params: {} 2024-04-12T03:43:53.141 DEBUG:boto:Token: None 2024-04-12T03:43:53.141 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17 2024-04-12T03:43:53.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T2rLTMPaEPbKr+LB9kf4hbWkm1g= 2024-04-12T03:43:53.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T2rLTMPaEPbKr+LB9kf4hbWkm1g=', 'Content-Length': '0'} 2024-04-12T03:43:53.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005edd34a61de3d388-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:43:53.143 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.144 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.144 DEBUG:boto:Method: GET 2024-04-12T03:43:53.144 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:43:53.144 DEBUG:boto:Data: 2024-04-12T03:43:53.144 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.144 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.144 DEBUG:boto:Params: {} 2024-04-12T03:43:53.144 DEBUG:boto:Token: None 2024-04-12T03:43:53.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17 2024-04-12T03:43:53.145 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bihqfneUznaW3m9rwMH86d4oXhw= 2024-04-12T03:43:53.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bihqfneUznaW3m9rwMH86d4oXhw=', 'Content-Length': '0'} 2024-04-12T03:43:53.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df4e535fa17512c-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.147 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:43:53.147 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:43:53.147 DEBUG:boto:Method: GET 2024-04-12T03:43:53.147 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:43:53.147 DEBUG:boto:Data: 2024-04-12T03:43:53.147 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.147 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.147 DEBUG:boto:Params: {} 2024-04-12T03:43:53.147 DEBUG:boto:Token: None 2024-04-12T03:43:53.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj17 2024-04-12T03:43:53.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bihqfneUznaW3m9rwMH86d4oXhw= 2024-04-12T03:43:53.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bihqfneUznaW3m9rwMH86d4oXhw=', 'Content-Length': '0'} 2024-04-12T03:43:53.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c740a203ac3bcc-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.150 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.150 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.150 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.150 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.150 DEBUG:boto:Method: GET 2024-04-12T03:43:53.150 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.150 DEBUG:boto:Data: 2024-04-12T03:43:53.150 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.151 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.151 DEBUG:boto:Params: {} 2024-04-12T03:43:53.151 DEBUG:boto:Token: None 2024-04-12T03:43:53.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b865cd7965a70db-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.194 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.195 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.195 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.195 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.195 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.195 DEBUG:boto:Method: GET 2024-04-12T03:43:53.195 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:43:53.196 DEBUG:boto:Data: 2024-04-12T03:43:53.196 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.196 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.196 DEBUG:boto:Params: {} 2024-04-12T03:43:53.196 DEBUG:boto:Token: None 2024-04-12T03:43:53.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003460a1f816d172b5-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.199 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:43:53.200 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.200 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.200 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.200 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.200 DEBUG:boto:Method: GET 2024-04-12T03:43:53.200 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.200 DEBUG:boto:Data: 2024-04-12T03:43:53.200 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.200 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.200 DEBUG:boto:Params: {} 2024-04-12T03:43:53.201 DEBUG:boto:Token: None 2024-04-12T03:43:53.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ= 2024-04-12T03:43:53.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003172b3427ce38ebd-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.203 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.203 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.203 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.203 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.204 DEBUG:boto:Method: GET 2024-04-12T03:43:53.204 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.204 DEBUG:boto:Data: 2024-04-12T03:43:53.204 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.204 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.204 DEBUG:boto:Params: {} 2024-04-12T03:43:53.204 DEBUG:boto:Token: None 2024-04-12T03:43:53.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ= 2024-04-12T03:43:53.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ccf853a852ad834-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.246 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.247 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.247 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.247 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.247 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.247 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.247 DEBUG:boto:Data: 2024-04-12T03:43:53.247 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.247 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.248 DEBUG:boto:Params: {} 2024-04-12T03:43:53.248 DEBUG:boto:Token: None 2024-04-12T03:43:53.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tL7G7htgRmnQayPFuToiHxQtYtM= 2024-04-12T03:43:53.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tL7G7htgRmnQayPFuToiHxQtYtM=', 'Content-Length': '0'} 2024-04-12T03:43:53.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a716e81e05692b40-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.250 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.251 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.251 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.251 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.251 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.251 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.251 DEBUG:boto:Data: 2024-04-12T03:43:53.251 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.251 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.251 DEBUG:boto:Params: {} 2024-04-12T03:43:53.252 DEBUG:boto:Token: None 2024-04-12T03:43:53.252 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tL7G7htgRmnQayPFuToiHxQtYtM= 2024-04-12T03:43:53.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tL7G7htgRmnQayPFuToiHxQtYtM=', 'Content-Length': '0'} 2024-04-12T03:43:53.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162447d8f7412816-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:43:53.254 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.255 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.255 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.255 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.255 DEBUG:boto:Method: GET 2024-04-12T03:43:53.255 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.255 DEBUG:boto:Data: 2024-04-12T03:43:53.255 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.255 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.255 DEBUG:boto:Params: {} 2024-04-12T03:43:53.256 DEBUG:boto:Token: None 2024-04-12T03:43:53.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ= 2024-04-12T03:43:53.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1a5324d62b5c9e-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.258 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.258 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.258 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.258 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.259 DEBUG:boto:Method: GET 2024-04-12T03:43:53.259 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.259 DEBUG:boto:Data: 2024-04-12T03:43:53.259 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.259 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.259 DEBUG:boto:Params: {} 2024-04-12T03:43:53.259 DEBUG:boto:Token: None 2024-04-12T03:43:53.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:43:53.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ= 2024-04-12T03:43:53.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IlgQBKROGcVWo4QZEFE20blNcaQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e892de41e233c5c6-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.261 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.262 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.262 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.262 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.262 DEBUG:boto:Method: GET 2024-04-12T03:43:53.262 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.262 DEBUG:boto:Data: 2024-04-12T03:43:53.262 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.262 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.262 DEBUG:boto:Params: {} 2024-04-12T03:43:53.263 DEBUG:boto:Token: None 2024-04-12T03:43:53.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e98fc24c1404324-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.265 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.265 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.266 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.266 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.266 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.266 DEBUG:boto:Method: GET 2024-04-12T03:43:53.266 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.266 DEBUG:boto:Data: 2024-04-12T03:43:53.266 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.266 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.266 DEBUG:boto:Params: {} 2024-04-12T03:43:53.267 DEBUG:boto:Token: None 2024-04-12T03:43:53.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c3288c542c63b1-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.270 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.270 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.270 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.271 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.271 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:43:53.271 DEBUG:boto:Data: 2024-04-12T03:43:53.271 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.271 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.271 DEBUG:boto:Params: {} 2024-04-12T03:43:53.271 DEBUG:boto:Token: None 2024-04-12T03:43:53.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22 2024-04-12T03:43:53.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7WO6Fx7EwMxdCrMYLHwjoMhLBVs= 2024-04-12T03:43:53.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7WO6Fx7EwMxdCrMYLHwjoMhLBVs=', 'Content-Length': '0'} 2024-04-12T03:43:53.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb1f0d037acbd1e-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.274 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.274 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.274 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.274 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:43:53.274 DEBUG:boto:Data: 2024-04-12T03:43:53.274 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.274 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.274 DEBUG:boto:Params: {} 2024-04-12T03:43:53.275 DEBUG:boto:Token: None 2024-04-12T03:43:53.275 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22 2024-04-12T03:43:53.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7WO6Fx7EwMxdCrMYLHwjoMhLBVs= 2024-04-12T03:43:53.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7WO6Fx7EwMxdCrMYLHwjoMhLBVs=', 'Content-Length': '0'} 2024-04-12T03:43:53.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3735cfe87f27609-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:43:53.277 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.277 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.277 DEBUG:boto:Method: GET 2024-04-12T03:43:53.277 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:43:53.277 DEBUG:boto:Data: 2024-04-12T03:43:53.277 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.278 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.278 DEBUG:boto:Params: {} 2024-04-12T03:43:53.278 DEBUG:boto:Token: None 2024-04-12T03:43:53.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22 2024-04-12T03:43:53.278 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PoFVoARv0dFaZNTEBfncb/Cylk8= 2024-04-12T03:43:53.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PoFVoARv0dFaZNTEBfncb/Cylk8=', 'Content-Length': '0'} 2024-04-12T03:43:53.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008123011cbdb237bd-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.280 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:43:53.280 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:43:53.280 DEBUG:boto:Method: GET 2024-04-12T03:43:53.280 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:43:53.280 DEBUG:boto:Data: 2024-04-12T03:43:53.281 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.281 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.281 DEBUG:boto:Params: {} 2024-04-12T03:43:53.281 DEBUG:boto:Token: None 2024-04-12T03:43:53.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj22 2024-04-12T03:43:53.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PoFVoARv0dFaZNTEBfncb/Cylk8= 2024-04-12T03:43:53.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PoFVoARv0dFaZNTEBfncb/Cylk8=', 'Content-Length': '0'} 2024-04-12T03:43:53.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d492503851d1bc-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.283 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.283 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.283 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.284 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.284 DEBUG:boto:Method: GET 2024-04-12T03:43:53.284 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.284 DEBUG:boto:Data: 2024-04-12T03:43:53.284 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.284 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.284 DEBUG:boto:Params: {} 2024-04-12T03:43:53.284 DEBUG:boto:Token: None 2024-04-12T03:43:53.285 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.285 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ae8d6c8cd36fc4b-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.287 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.289 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.289 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.289 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.290 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.290 DEBUG:boto:Method: GET 2024-04-12T03:43:53.290 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:43:53.290 DEBUG:boto:Data: 2024-04-12T03:43:53.290 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.290 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.290 DEBUG:boto:Params: {} 2024-04-12T03:43:53.290 DEBUG:boto:Token: None 2024-04-12T03:43:53.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62c806b8e977d56-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.293 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:43:53.294 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.294 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.294 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.294 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.294 DEBUG:boto:Method: GET 2024-04-12T03:43:53.295 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.295 DEBUG:boto:Data: 2024-04-12T03:43:53.295 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.295 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.295 DEBUG:boto:Params: {} 2024-04-12T03:43:53.295 DEBUG:boto:Token: None 2024-04-12T03:43:53.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I= 2024-04-12T03:43:53.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I=', 'Content-Length': '0'} 2024-04-12T03:43:53.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6ce50efab3b2d3-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.297 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.298 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.298 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.298 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.298 DEBUG:boto:Method: GET 2024-04-12T03:43:53.298 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.298 DEBUG:boto:Data: 2024-04-12T03:43:53.298 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.298 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.298 DEBUG:boto:Params: {} 2024-04-12T03:43:53.299 DEBUG:boto:Token: None 2024-04-12T03:43:53.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I= 2024-04-12T03:43:53.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I=', 'Content-Length': '0'} 2024-04-12T03:43:53.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b89f7f66fc72b6d-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.301 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.301 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.301 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.301 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.301 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.301 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.301 DEBUG:boto:Data: 2024-04-12T03:43:53.302 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.302 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.302 DEBUG:boto:Params: {} 2024-04-12T03:43:53.302 DEBUG:boto:Token: None 2024-04-12T03:43:53.302 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mLMTz4gpqmnMJYiZulv9WVkEBn0= 2024-04-12T03:43:53.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mLMTz4gpqmnMJYiZulv9WVkEBn0=', 'Content-Length': '0'} 2024-04-12T03:43:53.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec54edd3c6433ae-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.304 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.304 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.304 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.304 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.304 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.304 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.305 DEBUG:boto:Data: 2024-04-12T03:43:53.305 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.305 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.305 DEBUG:boto:Params: {} 2024-04-12T03:43:53.305 DEBUG:boto:Token: None 2024-04-12T03:43:53.305 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mLMTz4gpqmnMJYiZulv9WVkEBn0= 2024-04-12T03:43:53.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mLMTz4gpqmnMJYiZulv9WVkEBn0=', 'Content-Length': '0'} 2024-04-12T03:43:53.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006388b5bcba85ac53-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:43:53.307 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.307 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.307 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.307 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.307 DEBUG:boto:Method: GET 2024-04-12T03:43:53.308 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.308 DEBUG:boto:Data: 2024-04-12T03:43:53.308 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.308 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.308 DEBUG:boto:Params: {} 2024-04-12T03:43:53.308 DEBUG:boto:Token: None 2024-04-12T03:43:53.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I= 2024-04-12T03:43:53.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I=', 'Content-Length': '0'} 2024-04-12T03:43:53.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f209c2cc7584bf04-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.310 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.310 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.310 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.310 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.311 DEBUG:boto:Method: GET 2024-04-12T03:43:53.311 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.311 DEBUG:boto:Data: 2024-04-12T03:43:53.311 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.311 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.311 DEBUG:boto:Params: {} 2024-04-12T03:43:53.311 DEBUG:boto:Token: None 2024-04-12T03:43:53.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:43:53.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I= 2024-04-12T03:43:53.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DQd0rIAJbIvTzFS5oEgVOjG5Q1I=', 'Content-Length': '0'} 2024-04-12T03:43:53.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ba25ecbf37946b-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.354 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.355 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.355 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.355 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.355 DEBUG:boto:Method: GET 2024-04-12T03:43:53.355 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.355 DEBUG:boto:Data: 2024-04-12T03:43:53.355 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.356 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.356 DEBUG:boto:Params: {} 2024-04-12T03:43:53.356 DEBUG:boto:Token: None 2024-04-12T03:43:53.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e04a230ac566604-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.359 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.359 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.359 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.360 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.360 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.360 DEBUG:boto:Method: GET 2024-04-12T03:43:53.360 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.360 DEBUG:boto:Data: 2024-04-12T03:43:53.360 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.360 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.360 DEBUG:boto:Params: {} 2024-04-12T03:43:53.361 DEBUG:boto:Token: None 2024-04-12T03:43:53.361 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.361 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085bba4aa2d82ebdb-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.364 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.364 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.364 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.364 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.365 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:43:53.365 DEBUG:boto:Data: 2024-04-12T03:43:53.365 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.365 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.365 DEBUG:boto:Params: {} 2024-04-12T03:43:53.365 DEBUG:boto:Token: None 2024-04-12T03:43:53.365 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33 2024-04-12T03:43:53.365 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ruaJ57WlP/d2r4K5ZwkAms/MNlQ= 2024-04-12T03:43:53.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ruaJ57WlP/d2r4K5ZwkAms/MNlQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000786a20991d5fc9d-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.367 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.367 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.367 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.368 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:43:53.368 DEBUG:boto:Data: 2024-04-12T03:43:53.368 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.368 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.368 DEBUG:boto:Params: {} 2024-04-12T03:43:53.368 DEBUG:boto:Token: None 2024-04-12T03:43:53.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33 2024-04-12T03:43:53.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ruaJ57WlP/d2r4K5ZwkAms/MNlQ= 2024-04-12T03:43:53.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ruaJ57WlP/d2r4K5ZwkAms/MNlQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844a3c14ba362ea9-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:43:53.371 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.371 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.371 DEBUG:boto:Method: GET 2024-04-12T03:43:53.371 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:43:53.372 DEBUG:boto:Data: 2024-04-12T03:43:53.372 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.372 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.372 DEBUG:boto:Params: {} 2024-04-12T03:43:53.372 DEBUG:boto:Token: None 2024-04-12T03:43:53.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33 2024-04-12T03:43:53.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AP8QEdtKmbH5//G4xEhiSAUc9lY= 2024-04-12T03:43:53.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AP8QEdtKmbH5//G4xEhiSAUc9lY=', 'Content-Length': '0'} 2024-04-12T03:43:53.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193af54ef692b15d-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.415 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:43:53.416 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:43:53.416 DEBUG:boto:Method: GET 2024-04-12T03:43:53.416 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:43:53.416 DEBUG:boto:Data: 2024-04-12T03:43:53.416 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.416 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.416 DEBUG:boto:Params: {} 2024-04-12T03:43:53.417 DEBUG:boto:Token: None 2024-04-12T03:43:53.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj33 2024-04-12T03:43:53.417 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AP8QEdtKmbH5//G4xEhiSAUc9lY= 2024-04-12T03:43:53.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AP8QEdtKmbH5//G4xEhiSAUc9lY=', 'Content-Length': '0'} 2024-04-12T03:43:53.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071d8deb4e2a8bc57-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.419 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.419 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.419 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.420 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.420 DEBUG:boto:Method: GET 2024-04-12T03:43:53.420 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.420 DEBUG:boto:Data: 2024-04-12T03:43:53.420 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.420 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.420 DEBUG:boto:Params: {} 2024-04-12T03:43:53.421 DEBUG:boto:Token: None 2024-04-12T03:43:53.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026f0d93e00ca3129-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.423 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.424 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.424 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.424 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.424 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.424 DEBUG:boto:Method: GET 2024-04-12T03:43:53.424 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:43:53.424 DEBUG:boto:Data: 2024-04-12T03:43:53.424 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.425 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.425 DEBUG:boto:Params: {} 2024-04-12T03:43:53.425 DEBUG:boto:Token: None 2024-04-12T03:43:53.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9211e317608d9e4-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.470 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:43:53.471 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.471 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.471 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.471 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.472 DEBUG:boto:Method: GET 2024-04-12T03:43:53.472 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.472 DEBUG:boto:Data: 2024-04-12T03:43:53.472 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.472 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.472 DEBUG:boto:Params: {} 2024-04-12T03:43:53.472 DEBUG:boto:Token: None 2024-04-12T03:43:53.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU= 2024-04-12T03:43:53.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU=', 'Content-Length': '0'} 2024-04-12T03:43:53.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce5df305baab196-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.475 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.475 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.475 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.475 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.475 DEBUG:boto:Method: GET 2024-04-12T03:43:53.475 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.475 DEBUG:boto:Data: 2024-04-12T03:43:53.476 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.476 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.476 DEBUG:boto:Params: {} 2024-04-12T03:43:53.476 DEBUG:boto:Token: None 2024-04-12T03:43:53.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU= 2024-04-12T03:43:53.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU=', 'Content-Length': '0'} 2024-04-12T03:43:53.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063391a37e6eda88f-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.478 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.479 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.479 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.479 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.479 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.479 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.479 DEBUG:boto:Data: 2024-04-12T03:43:53.479 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.479 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.479 DEBUG:boto:Params: {} 2024-04-12T03:43:53.479 DEBUG:boto:Token: None 2024-04-12T03:43:53.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Aqw/1o3Qs9t9RDt4Rn/DRRBwBZY= 2024-04-12T03:43:53.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Aqw/1o3Qs9t9RDt4Rn/DRRBwBZY=', 'Content-Length': '0'} 2024-04-12T03:43:53.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371e62bbfb4f7a2d-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.482 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.482 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.482 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.482 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.482 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.482 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.482 DEBUG:boto:Data: 2024-04-12T03:43:53.482 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.483 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.483 DEBUG:boto:Params: {} 2024-04-12T03:43:53.483 DEBUG:boto:Token: None 2024-04-12T03:43:53.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Aqw/1o3Qs9t9RDt4Rn/DRRBwBZY= 2024-04-12T03:43:53.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Aqw/1o3Qs9t9RDt4Rn/DRRBwBZY=', 'Content-Length': '0'} 2024-04-12T03:43:53.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04f527dae276686-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:43:53.485 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.485 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.485 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.485 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.486 DEBUG:boto:Method: GET 2024-04-12T03:43:53.486 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.486 DEBUG:boto:Data: 2024-04-12T03:43:53.486 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.486 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.486 DEBUG:boto:Params: {} 2024-04-12T03:43:53.486 DEBUG:boto:Token: None 2024-04-12T03:43:53.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.486 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU= 2024-04-12T03:43:53.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU=', 'Content-Length': '0'} 2024-04-12T03:43:53.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d9d762ec335462-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.488 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.488 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.489 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.489 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.489 DEBUG:boto:Method: GET 2024-04-12T03:43:53.489 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.489 DEBUG:boto:Data: 2024-04-12T03:43:53.489 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.489 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.489 DEBUG:boto:Params: {} 2024-04-12T03:43:53.489 DEBUG:boto:Token: None 2024-04-12T03:43:53.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:43:53.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU= 2024-04-12T03:43:53.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WEWWqlRwunUMGSwR5mbybtq1NU=', 'Content-Length': '0'} 2024-04-12T03:43:53.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b764de63a113f03-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.494 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.494 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.494 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.495 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.495 DEBUG:boto:Method: GET 2024-04-12T03:43:53.495 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.495 DEBUG:boto:Data: 2024-04-12T03:43:53.496 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.496 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.496 DEBUG:boto:Params: {} 2024-04-12T03:43:53.496 DEBUG:boto:Token: None 2024-04-12T03:43:53.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032bad6fc460611d2-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.499 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.500 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.500 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.500 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.500 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.500 DEBUG:boto:Method: GET 2024-04-12T03:43:53.500 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.500 DEBUG:boto:Data: 2024-04-12T03:43:53.500 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.500 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.501 DEBUG:boto:Params: {} 2024-04-12T03:43:53.501 DEBUG:boto:Token: None 2024-04-12T03:43:53.501 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.501 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005077142731bd2167-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.504 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.504 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.504 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.505 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.505 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:43:53.505 DEBUG:boto:Data: 2024-04-12T03:43:53.505 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.505 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.505 DEBUG:boto:Params: {} 2024-04-12T03:43:53.505 DEBUG:boto:Token: None 2024-04-12T03:43:53.505 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34 2024-04-12T03:43:53.505 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vuoxop/byd52o6tIV/H/tMQ0uAQ= 2024-04-12T03:43:53.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vuoxop/byd52o6tIV/H/tMQ0uAQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8aa0751a32069c-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.507 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.507 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.507 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.507 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:43:53.508 DEBUG:boto:Data: 2024-04-12T03:43:53.508 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.508 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.508 DEBUG:boto:Params: {} 2024-04-12T03:43:53.508 DEBUG:boto:Token: None 2024-04-12T03:43:53.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34 2024-04-12T03:43:53.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vuoxop/byd52o6tIV/H/tMQ0uAQ= 2024-04-12T03:43:53.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vuoxop/byd52o6tIV/H/tMQ0uAQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf621659240af18-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:43:53.510 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.510 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.510 DEBUG:boto:Method: GET 2024-04-12T03:43:53.510 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:43:53.511 DEBUG:boto:Data: 2024-04-12T03:43:53.511 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.511 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.511 DEBUG:boto:Params: {} 2024-04-12T03:43:53.511 DEBUG:boto:Token: None 2024-04-12T03:43:53.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34 2024-04-12T03:43:53.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RFhB74Qc+iVZ+yFBlRruYTZXNGM= 2024-04-12T03:43:53.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RFhB74Qc+iVZ+yFBlRruYTZXNGM=', 'Content-Length': '0'} 2024-04-12T03:43:53.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000642f40c923d392f-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.554 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:43:53.555 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:43:53.555 DEBUG:boto:Method: GET 2024-04-12T03:43:53.555 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:43:53.555 DEBUG:boto:Data: 2024-04-12T03:43:53.555 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.555 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.555 DEBUG:boto:Params: {} 2024-04-12T03:43:53.556 DEBUG:boto:Token: None 2024-04-12T03:43:53.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj34 2024-04-12T03:43:53.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RFhB74Qc+iVZ+yFBlRruYTZXNGM= 2024-04-12T03:43:53.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RFhB74Qc+iVZ+yFBlRruYTZXNGM=', 'Content-Length': '0'} 2024-04-12T03:43:53.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d759c78b336319-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.558 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.558 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.559 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.559 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.559 DEBUG:boto:Method: GET 2024-04-12T03:43:53.559 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.559 DEBUG:boto:Data: 2024-04-12T03:43:53.559 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.559 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.559 DEBUG:boto:Params: {} 2024-04-12T03:43:53.559 DEBUG:boto:Token: None 2024-04-12T03:43:53.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9867d3212f425a7-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.562 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.562 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.562 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.563 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.563 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.563 DEBUG:boto:Method: GET 2024-04-12T03:43:53.563 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:43:53.563 DEBUG:boto:Data: 2024-04-12T03:43:53.563 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.563 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.563 DEBUG:boto:Params: {} 2024-04-12T03:43:53.563 DEBUG:boto:Token: None 2024-04-12T03:43:53.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070cebae06564a259-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.567 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:43:53.567 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.567 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.567 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.567 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.568 DEBUG:boto:Method: GET 2024-04-12T03:43:53.568 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.568 DEBUG:boto:Data: 2024-04-12T03:43:53.568 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.568 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.568 DEBUG:boto:Params: {} 2024-04-12T03:43:53.568 DEBUG:boto:Token: None 2024-04-12T03:43:53.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.569 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg= 2024-04-12T03:43:53.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg=', 'Content-Length': '0'} 2024-04-12T03:43:53.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487827136ebb9711-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.570 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.571 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.571 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.571 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.571 DEBUG:boto:Method: GET 2024-04-12T03:43:53.571 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.571 DEBUG:boto:Data: 2024-04-12T03:43:53.571 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.571 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.571 DEBUG:boto:Params: {} 2024-04-12T03:43:53.572 DEBUG:boto:Token: None 2024-04-12T03:43:53.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg= 2024-04-12T03:43:53.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg=', 'Content-Length': '0'} 2024-04-12T03:43:53.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1db6dee92b5a00-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.614 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.615 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.615 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.615 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.615 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.615 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.615 DEBUG:boto:Data: 2024-04-12T03:43:53.615 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.615 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.616 DEBUG:boto:Params: {} 2024-04-12T03:43:53.616 DEBUG:boto:Token: None 2024-04-12T03:43:53.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b1IY69ULzraSGyyz+3xBllw6ImQ= 2024-04-12T03:43:53.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b1IY69ULzraSGyyz+3xBllw6ImQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000087d5c129a5cefb-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.618 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.618 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.618 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.619 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.619 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.619 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.619 DEBUG:boto:Data: 2024-04-12T03:43:53.619 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.619 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.619 DEBUG:boto:Params: {} 2024-04-12T03:43:53.619 DEBUG:boto:Token: None 2024-04-12T03:43:53.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b1IY69ULzraSGyyz+3xBllw6ImQ= 2024-04-12T03:43:53.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b1IY69ULzraSGyyz+3xBllw6ImQ=', 'Content-Length': '0'} 2024-04-12T03:43:53.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f15706b54b9d2b2-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:43:53.622 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.622 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.622 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.622 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.622 DEBUG:boto:Method: GET 2024-04-12T03:43:53.622 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.622 DEBUG:boto:Data: 2024-04-12T03:43:53.622 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.622 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.622 DEBUG:boto:Params: {} 2024-04-12T03:43:53.623 DEBUG:boto:Token: None 2024-04-12T03:43:53.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg= 2024-04-12T03:43:53.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg=', 'Content-Length': '0'} 2024-04-12T03:43:53.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c4648ebe3a464e-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.625 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.625 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.625 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.625 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.625 DEBUG:boto:Method: GET 2024-04-12T03:43:53.625 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.625 DEBUG:boto:Data: 2024-04-12T03:43:53.625 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.626 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.626 DEBUG:boto:Params: {} 2024-04-12T03:43:53.626 DEBUG:boto:Token: None 2024-04-12T03:43:53.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:43:53.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg= 2024-04-12T03:43:53.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9NX9ktpmc8bVqAfZk8VHlo6iJPg=', 'Content-Length': '0'} 2024-04-12T03:43:53.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbc0e3a69a6684f-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.628 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.628 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.628 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.629 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.629 DEBUG:boto:Method: GET 2024-04-12T03:43:53.629 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.629 DEBUG:boto:Data: 2024-04-12T03:43:53.629 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.629 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.629 DEBUG:boto:Params: {} 2024-04-12T03:43:53.629 DEBUG:boto:Token: None 2024-04-12T03:43:53.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.630 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094372d4e9424a7c3-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.632 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.632 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.632 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.632 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.632 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.633 DEBUG:boto:Method: GET 2024-04-12T03:43:53.633 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.633 DEBUG:boto:Data: 2024-04-12T03:43:53.633 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.633 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.633 DEBUG:boto:Params: {} 2024-04-12T03:43:53.633 DEBUG:boto:Token: None 2024-04-12T03:43:53.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000999f1fbc553f0b1b-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.636 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.637 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.637 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.637 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.637 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:43:53.637 DEBUG:boto:Data: 2024-04-12T03:43:53.637 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.638 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.638 DEBUG:boto:Params: {} 2024-04-12T03:43:53.638 DEBUG:boto:Token: None 2024-04-12T03:43:53.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44 2024-04-12T03:43:53.638 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SpSx5d5e8GGKJ6VYsp93d+pvCpk= 2024-04-12T03:43:53.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SpSx5d5e8GGKJ6VYsp93d+pvCpk=', 'Content-Length': '0'} 2024-04-12T03:43:53.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc98fcd9eda0075e-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.640 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.640 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.640 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.640 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:43:53.640 DEBUG:boto:Data: 2024-04-12T03:43:53.640 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.641 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.641 DEBUG:boto:Params: {} 2024-04-12T03:43:53.641 DEBUG:boto:Token: None 2024-04-12T03:43:53.641 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44 2024-04-12T03:43:53.641 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SpSx5d5e8GGKJ6VYsp93d+pvCpk= 2024-04-12T03:43:53.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SpSx5d5e8GGKJ6VYsp93d+pvCpk=', 'Content-Length': '0'} 2024-04-12T03:43:53.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da7fbd0806ab2cec-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:43:53.643 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.643 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.643 DEBUG:boto:Method: GET 2024-04-12T03:43:53.643 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:43:53.644 DEBUG:boto:Data: 2024-04-12T03:43:53.644 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.644 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.644 DEBUG:boto:Params: {} 2024-04-12T03:43:53.644 DEBUG:boto:Token: None 2024-04-12T03:43:53.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44 2024-04-12T03:43:53.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vfbJloSSOE8b0JcfNhz81bufSkY= 2024-04-12T03:43:53.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vfbJloSSOE8b0JcfNhz81bufSkY=', 'Content-Length': '0'} 2024-04-12T03:43:53.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca51eafc66f08281-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.646 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:43:53.646 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:43:53.646 DEBUG:boto:Method: GET 2024-04-12T03:43:53.646 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:43:53.647 DEBUG:boto:Data: 2024-04-12T03:43:53.647 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.647 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.647 DEBUG:boto:Params: {} 2024-04-12T03:43:53.647 DEBUG:boto:Token: None 2024-04-12T03:43:53.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj44 2024-04-12T03:43:53.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vfbJloSSOE8b0JcfNhz81bufSkY= 2024-04-12T03:43:53.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vfbJloSSOE8b0JcfNhz81bufSkY=', 'Content-Length': '0'} 2024-04-12T03:43:53.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c50e755f5c991c-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.690 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.691 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.691 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.691 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.691 DEBUG:boto:Method: GET 2024-04-12T03:43:53.691 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.691 DEBUG:boto:Data: 2024-04-12T03:43:53.691 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.691 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.692 DEBUG:boto:Params: {} 2024-04-12T03:43:53.692 DEBUG:boto:Token: None 2024-04-12T03:43:53.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd72b0d1b3a60657-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.698 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.699 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.699 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.699 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.699 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.699 DEBUG:boto:Method: GET 2024-04-12T03:43:53.699 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:43:53.699 DEBUG:boto:Data: 2024-04-12T03:43:53.699 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.699 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.699 DEBUG:boto:Params: {} 2024-04-12T03:43:53.700 DEBUG:boto:Token: None 2024-04-12T03:43:53.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feed2ee70f93dd0b-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.703 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:43:53.703 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.703 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.704 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.704 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.704 DEBUG:boto:Method: GET 2024-04-12T03:43:53.704 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.704 DEBUG:boto:Data: 2024-04-12T03:43:53.704 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.704 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.704 DEBUG:boto:Params: {} 2024-04-12T03:43:53.704 DEBUG:boto:Token: None 2024-04-12T03:43:53.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0= 2024-04-12T03:43:53.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0=', 'Content-Length': '0'} 2024-04-12T03:43:53.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4833d749685d9b-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.707 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.707 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.707 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.707 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.707 DEBUG:boto:Method: GET 2024-04-12T03:43:53.707 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.707 DEBUG:boto:Data: 2024-04-12T03:43:53.707 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.707 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.708 DEBUG:boto:Params: {} 2024-04-12T03:43:53.708 DEBUG:boto:Token: None 2024-04-12T03:43:53.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0= 2024-04-12T03:43:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0=', 'Content-Length': '0'} 2024-04-12T03:43:53.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2971060cd37d91-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.710 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.710 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.710 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.710 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.710 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.710 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.710 DEBUG:boto:Data: 2024-04-12T03:43:53.711 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.711 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.711 DEBUG:boto:Params: {} 2024-04-12T03:43:53.711 DEBUG:boto:Token: None 2024-04-12T03:43:53.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/BXZgWjt/QD5o481o68J6Y0Pdi8= 2024-04-12T03:43:53.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BXZgWjt/QD5o481o68J6Y0Pdi8=', 'Content-Length': '0'} 2024-04-12T03:43:53.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d15709460f665c-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.713 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.713 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.713 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.713 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.713 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.713 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.713 DEBUG:boto:Data: 2024-04-12T03:43:53.713 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.714 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.714 DEBUG:boto:Params: {} 2024-04-12T03:43:53.714 DEBUG:boto:Token: None 2024-04-12T03:43:53.714 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/BXZgWjt/QD5o481o68J6Y0Pdi8= 2024-04-12T03:43:53.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BXZgWjt/QD5o481o68J6Y0Pdi8=', 'Content-Length': '0'} 2024-04-12T03:43:53.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e5eec235a59444-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:43:53.716 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.716 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.716 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.716 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.717 DEBUG:boto:Method: GET 2024-04-12T03:43:53.717 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.717 DEBUG:boto:Data: 2024-04-12T03:43:53.717 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.717 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.717 DEBUG:boto:Params: {} 2024-04-12T03:43:53.717 DEBUG:boto:Token: None 2024-04-12T03:43:53.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0= 2024-04-12T03:43:53.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0=', 'Content-Length': '0'} 2024-04-12T03:43:53.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4169ef141e29e49-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.762 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.763 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.763 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.763 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.763 DEBUG:boto:Method: GET 2024-04-12T03:43:53.763 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.764 DEBUG:boto:Data: 2024-04-12T03:43:53.764 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.764 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.764 DEBUG:boto:Params: {} 2024-04-12T03:43:53.764 DEBUG:boto:Token: None 2024-04-12T03:43:53.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:43:53.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0= 2024-04-12T03:43:53.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ta1oypg0AYZd9kW4MJEyqeSZAB0=', 'Content-Length': '0'} 2024-04-12T03:43:53.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000be4df198e55eb8-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.766 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.766 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.767 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.767 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.767 DEBUG:boto:Method: GET 2024-04-12T03:43:53.767 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.767 DEBUG:boto:Data: 2024-04-12T03:43:53.767 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.767 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.767 DEBUG:boto:Params: {} 2024-04-12T03:43:53.767 DEBUG:boto:Token: None 2024-04-12T03:43:53.768 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084cce776bdaedf83-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.770 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.770 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.771 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.771 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.771 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.771 DEBUG:boto:Method: GET 2024-04-12T03:43:53.771 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.771 DEBUG:boto:Data: 2024-04-12T03:43:53.771 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.771 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.771 DEBUG:boto:Params: {} 2024-04-12T03:43:53.771 DEBUG:boto:Token: None 2024-04-12T03:43:53.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009aa1569541f36e9-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.775 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.775 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.775 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.775 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.775 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:43:53.775 DEBUG:boto:Data: 2024-04-12T03:43:53.776 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.776 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.776 DEBUG:boto:Params: {} 2024-04-12T03:43:53.776 DEBUG:boto:Token: None 2024-04-12T03:43:53.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51 2024-04-12T03:43:53.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Np+EYxuYR27aRUww92q61c35tk= 2024-04-12T03:43:53.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Np+EYxuYR27aRUww92q61c35tk=', 'Content-Length': '0'} 2024-04-12T03:43:53.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1fb4ed10bfc7f2a-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.778 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.778 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.778 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.778 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:43:53.778 DEBUG:boto:Data: 2024-04-12T03:43:53.779 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.779 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.779 DEBUG:boto:Params: {} 2024-04-12T03:43:53.779 DEBUG:boto:Token: None 2024-04-12T03:43:53.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51 2024-04-12T03:43:53.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Np+EYxuYR27aRUww92q61c35tk= 2024-04-12T03:43:53.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Np+EYxuYR27aRUww92q61c35tk=', 'Content-Length': '0'} 2024-04-12T03:43:53.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace940bed79132c3-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:43:53.781 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.781 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.781 DEBUG:boto:Method: GET 2024-04-12T03:43:53.781 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:43:53.781 DEBUG:boto:Data: 2024-04-12T03:43:53.781 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.782 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.782 DEBUG:boto:Params: {} 2024-04-12T03:43:53.782 DEBUG:boto:Token: None 2024-04-12T03:43:53.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51 2024-04-12T03:43:53.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NbGTcWJoyrX6Gl7R7xHiefeYZFg= 2024-04-12T03:43:53.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NbGTcWJoyrX6Gl7R7xHiefeYZFg=', 'Content-Length': '0'} 2024-04-12T03:43:53.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f085b3d7f31eba-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.784 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:43:53.785 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:43:53.785 DEBUG:boto:Method: GET 2024-04-12T03:43:53.785 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:43:53.785 DEBUG:boto:Data: 2024-04-12T03:43:53.785 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.785 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.785 DEBUG:boto:Params: {} 2024-04-12T03:43:53.785 DEBUG:boto:Token: None 2024-04-12T03:43:53.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj51 2024-04-12T03:43:53.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NbGTcWJoyrX6Gl7R7xHiefeYZFg= 2024-04-12T03:43:53.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NbGTcWJoyrX6Gl7R7xHiefeYZFg=', 'Content-Length': '0'} 2024-04-12T03:43:53.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1882d3740b7ee9-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.830 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.831 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.831 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.831 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.831 DEBUG:boto:Method: GET 2024-04-12T03:43:53.831 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.831 DEBUG:boto:Data: 2024-04-12T03:43:53.831 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.831 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.831 DEBUG:boto:Params: {} 2024-04-12T03:43:53.832 DEBUG:boto:Token: None 2024-04-12T03:43:53.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7184f095d7c84d6-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.834 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.835 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.835 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.835 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.835 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.835 DEBUG:boto:Method: GET 2024-04-12T03:43:53.835 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:43:53.835 DEBUG:boto:Data: 2024-04-12T03:43:53.835 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.835 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.836 DEBUG:boto:Params: {} 2024-04-12T03:43:53.836 DEBUG:boto:Token: None 2024-04-12T03:43:53.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a174567d3405f9a-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.839 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:43:53.839 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.840 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.840 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.840 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.840 DEBUG:boto:Method: GET 2024-04-12T03:43:53.840 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.840 DEBUG:boto:Data: 2024-04-12T03:43:53.840 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.840 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.840 DEBUG:boto:Params: {} 2024-04-12T03:43:53.841 DEBUG:boto:Token: None 2024-04-12T03:43:53.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM= 2024-04-12T03:43:53.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM=', 'Content-Length': '0'} 2024-04-12T03:43:53.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007837d7a2537e4007-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.843 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.843 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.843 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.843 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.843 DEBUG:boto:Method: GET 2024-04-12T03:43:53.844 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.844 DEBUG:boto:Data: 2024-04-12T03:43:53.844 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.844 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.844 DEBUG:boto:Params: {} 2024-04-12T03:43:53.844 DEBUG:boto:Token: None 2024-04-12T03:43:53.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM= 2024-04-12T03:43:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM=', 'Content-Length': '0'} 2024-04-12T03:43:53.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af0009cb9b1f8ab-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.846 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.846 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.846 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.847 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.847 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.847 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.847 DEBUG:boto:Data: 2024-04-12T03:43:53.847 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.847 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.847 DEBUG:boto:Params: {} 2024-04-12T03:43:53.847 DEBUG:boto:Token: None 2024-04-12T03:43:53.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2OKVUV3ShiRYa9448BgzGtdq2XI= 2024-04-12T03:43:53.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2OKVUV3ShiRYa9448BgzGtdq2XI=', 'Content-Length': '0'} 2024-04-12T03:43:53.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029816c83b9f15661-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.849 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.849 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.849 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.850 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.850 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.850 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.850 DEBUG:boto:Data: 2024-04-12T03:43:53.850 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.850 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.850 DEBUG:boto:Params: {} 2024-04-12T03:43:53.850 DEBUG:boto:Token: None 2024-04-12T03:43:53.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2OKVUV3ShiRYa9448BgzGtdq2XI= 2024-04-12T03:43:53.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2OKVUV3ShiRYa9448BgzGtdq2XI=', 'Content-Length': '0'} 2024-04-12T03:43:53.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009acb687d5546d719-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:43:53.853 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.853 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.853 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.853 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.853 DEBUG:boto:Method: GET 2024-04-12T03:43:53.853 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.853 DEBUG:boto:Data: 2024-04-12T03:43:53.853 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.853 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.854 DEBUG:boto:Params: {} 2024-04-12T03:43:53.854 DEBUG:boto:Token: None 2024-04-12T03:43:53.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM= 2024-04-12T03:43:53.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM=', 'Content-Length': '0'} 2024-04-12T03:43:53.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec97d5c3164a582f-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.898 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.899 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.899 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.899 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.899 DEBUG:boto:Method: GET 2024-04-12T03:43:53.899 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.899 DEBUG:boto:Data: 2024-04-12T03:43:53.899 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.899 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.899 DEBUG:boto:Params: {} 2024-04-12T03:43:53.900 DEBUG:boto:Token: None 2024-04-12T03:43:53.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:43:53.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM= 2024-04-12T03:43:53.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uDwgfjJV/+stzzv03dS/sMUDTZM=', 'Content-Length': '0'} 2024-04-12T03:43:53.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f1bab0f8688b22-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.902 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.902 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.902 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.903 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.903 DEBUG:boto:Method: GET 2024-04-12T03:43:53.903 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.903 DEBUG:boto:Data: 2024-04-12T03:43:53.903 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.903 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.903 DEBUG:boto:Params: {} 2024-04-12T03:43:53.903 DEBUG:boto:Token: None 2024-04-12T03:43:53.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c47039de398565-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.906 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.906 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.907 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.907 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.907 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.907 DEBUG:boto:Method: GET 2024-04-12T03:43:53.907 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.907 DEBUG:boto:Data: 2024-04-12T03:43:53.907 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.907 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.907 DEBUG:boto:Params: {} 2024-04-12T03:43:53.907 DEBUG:boto:Token: None 2024-04-12T03:43:53.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000197165a6e1cc3305-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.910 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.911 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.911 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.911 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.911 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:43:53.911 DEBUG:boto:Data: 2024-04-12T03:43:53.911 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.911 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.911 DEBUG:boto:Params: {} 2024-04-12T03:43:53.912 DEBUG:boto:Token: None 2024-04-12T03:43:53.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55 2024-04-12T03:43:53.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L9CNnGFAD5ewzotkIMl/3JBeHj0= 2024-04-12T03:43:53.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9CNnGFAD5ewzotkIMl/3JBeHj0=', 'Content-Length': '0'} 2024-04-12T03:43:53.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc57805a78023ec-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.914 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.914 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.914 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.914 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:43:53.914 DEBUG:boto:Data: 2024-04-12T03:43:53.914 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.915 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.915 DEBUG:boto:Params: {} 2024-04-12T03:43:53.915 DEBUG:boto:Token: None 2024-04-12T03:43:53.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55 2024-04-12T03:43:53.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L9CNnGFAD5ewzotkIMl/3JBeHj0= 2024-04-12T03:43:53.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9CNnGFAD5ewzotkIMl/3JBeHj0=', 'Content-Length': '0'} 2024-04-12T03:43:53.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6b3b7a8c71ae57-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:43:53.917 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.917 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.918 DEBUG:boto:Method: GET 2024-04-12T03:43:53.918 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:43:53.918 DEBUG:boto:Data: 2024-04-12T03:43:53.918 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.918 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.918 DEBUG:boto:Params: {} 2024-04-12T03:43:53.918 DEBUG:boto:Token: None 2024-04-12T03:43:53.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55 2024-04-12T03:43:53.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A+oUszWqbURj08lKtd7+R7myzR8= 2024-04-12T03:43:53.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A+oUszWqbURj08lKtd7+R7myzR8=', 'Content-Length': '0'} 2024-04-12T03:43:53.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e44815e2e06321-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.920 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:43:53.921 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:43:53.921 DEBUG:boto:Method: GET 2024-04-12T03:43:53.921 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:43:53.921 DEBUG:boto:Data: 2024-04-12T03:43:53.921 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.921 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.921 DEBUG:boto:Params: {} 2024-04-12T03:43:53.921 DEBUG:boto:Token: None 2024-04-12T03:43:53.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj55 2024-04-12T03:43:53.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A+oUszWqbURj08lKtd7+R7myzR8= 2024-04-12T03:43:53.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A+oUszWqbURj08lKtd7+R7myzR8=', 'Content-Length': '0'} 2024-04-12T03:43:53.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd19350dcf86f3e-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.967 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.967 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.967 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.967 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.968 DEBUG:boto:Method: GET 2024-04-12T03:43:53.968 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.968 DEBUG:boto:Data: 2024-04-12T03:43:53.968 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.968 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.968 DEBUG:boto:Params: {} 2024-04-12T03:43:53.968 DEBUG:boto:Token: None 2024-04-12T03:43:53.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f74597ba1a4dcc-006618adf9-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.971 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.971 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:53.971 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:53.971 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.971 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.972 DEBUG:boto:Method: GET 2024-04-12T03:43:53.972 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:43:53.972 DEBUG:boto:Data: 2024-04-12T03:43:53.972 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.972 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.972 DEBUG:boto:Params: {} 2024-04-12T03:43:53.972 DEBUG:boto:Token: None 2024-04-12T03:43:53.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/ 2024-04-12T03:43:53.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js= 2024-04-12T03:43:53.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lee17uO+kJh+5MGws9Pj4uTL9Js=', 'Content-Length': '0'} 2024-04-12T03:43:53.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b56eceee2602e5f-006618adf9-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.975 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:53.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:43:53.975 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:53.975 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:53.976 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.976 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.976 DEBUG:boto:Method: GET 2024-04-12T03:43:53.976 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.976 DEBUG:boto:Data: 2024-04-12T03:43:53.976 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.976 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.976 DEBUG:boto:Params: {} 2024-04-12T03:43:53.976 DEBUG:boto:Token: None 2024-04-12T03:43:53.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E= 2024-04-12T03:43:53.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E=', 'Content-Length': '0'} 2024-04-12T03:43:53.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d27c1530575585-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.979 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:53.979 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:53.979 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.979 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.979 DEBUG:boto:Method: GET 2024-04-12T03:43:53.979 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.979 DEBUG:boto:Data: 2024-04-12T03:43:53.979 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.979 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.979 DEBUG:boto:Params: {} 2024-04-12T03:43:53.980 DEBUG:boto:Token: None 2024-04-12T03:43:53.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E= 2024-04-12T03:43:53.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E=', 'Content-Length': '0'} 2024-04-12T03:43:53.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dfab7e4f693b798-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.982 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:53.982 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:53.982 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.982 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.982 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.982 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.982 DEBUG:boto:Data: 2024-04-12T03:43:53.982 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.983 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.983 DEBUG:boto:Params: {} 2024-04-12T03:43:53.983 DEBUG:boto:Token: None 2024-04-12T03:43:53.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrWpJsFvenetTOExwVR1ozL2d24= 2024-04-12T03:43:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrWpJsFvenetTOExwVR1ozL2d24=', 'Content-Length': '0'} 2024-04-12T03:43:53.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c475bdb30d5c9f-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.985 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:53.985 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:53.985 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.985 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.985 DEBUG:boto:Method: HEAD 2024-04-12T03:43:53.985 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.985 DEBUG:boto:Data: 2024-04-12T03:43:53.986 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:53.986 DEBUG:boto:Port: 8000 2024-04-12T03:43:53.986 DEBUG:boto:Params: {} 2024-04-12T03:43:53.986 DEBUG:boto:Token: None 2024-04-12T03:43:53.986 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.986 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xrWpJsFvenetTOExwVR1ozL2d24= 2024-04-12T03:43:53.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xrWpJsFvenetTOExwVR1ozL2d24=', 'Content-Length': '0'} 2024-04-12T03:43:53.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65c229b3c5b103a-006618adf9-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:53.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:43:53.988 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:53.988 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:53.988 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.988 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.989 DEBUG:boto:Method: GET 2024-04-12T03:43:53.989 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.989 DEBUG:boto:Data: 2024-04-12T03:43:53.989 DEBUG:boto:Headers: {} 2024-04-12T03:43:53.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:53.989 DEBUG:boto:Port: 8001 2024-04-12T03:43:53.989 DEBUG:boto:Params: {} 2024-04-12T03:43:53.989 DEBUG:boto:Token: None 2024-04-12T03:43:53.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:53 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:53.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E= 2024-04-12T03:43:53.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4e5iZXX7t5YVIwv0yE1lFDt3c7E=', 'Content-Length': '0'} 2024-04-12T03:43:53.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008146e431a5459af4-006618adf9-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.034 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:54.035 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:54.035 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:54.035 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:54.035 DEBUG:boto:Method: GET 2024-04-12T03:43:54.035 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:54.035 DEBUG:boto:Data: 2024-04-12T03:43:54.035 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.035 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.035 DEBUG:boto:Params: {} 2024-04-12T03:43:54.036 DEBUG:boto:Token: None 2024-04-12T03:43:54.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:43:54.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Y+d50eGNj5PKgjFjWx1cReHr2I= 2024-04-12T03:43:54.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Y+d50eGNj5PKgjFjWx1cReHr2I=', 'Content-Length': '0'} 2024-04-12T03:43:54.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083277cac9e38a16f-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.038 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.039 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.039 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.039 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.039 DEBUG:boto:Method: GET 2024-04-12T03:43:54.039 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.039 DEBUG:boto:Data: 2024-04-12T03:43:54.039 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.039 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.039 DEBUG:boto:Params: {} 2024-04-12T03:43:54.039 DEBUG:boto:Token: None 2024-04-12T03:43:54.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716f83d9dab6679d-006618adfa-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.042 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.042 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.043 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.043 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.043 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.043 DEBUG:boto:Method: GET 2024-04-12T03:43:54.043 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.043 DEBUG:boto:Data: 2024-04-12T03:43:54.043 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.043 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.043 DEBUG:boto:Params: {} 2024-04-12T03:43:54.044 DEBUG:boto:Token: None 2024-04-12T03:43:54.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478b46817357f5ea-006618adfa-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.046 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.047 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:54.047 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:54.047 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.047 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:43:54.047 DEBUG:boto:Data: 2024-04-12T03:43:54.047 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.047 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.048 DEBUG:boto:Params: {} 2024-04-12T03:43:54.048 DEBUG:boto:Token: None 2024-04-12T03:43:54.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj68 2024-04-12T03:43:54.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KD7ACcv5tjrI+9gvZSrf/B/R68Y= 2024-04-12T03:43:54.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KD7ACcv5tjrI+9gvZSrf/B/R68Y=', 'Content-Length': '0'} 2024-04-12T03:43:54.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391eefb9579df602-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.050 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:54.050 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:54.050 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.050 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:43:54.050 DEBUG:boto:Data: 2024-04-12T03:43:54.050 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.051 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.051 DEBUG:boto:Params: {} 2024-04-12T03:43:54.051 DEBUG:boto:Token: None 2024-04-12T03:43:54.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj68 2024-04-12T03:43:54.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KD7ACcv5tjrI+9gvZSrf/B/R68Y= 2024-04-12T03:43:54.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KD7ACcv5tjrI+9gvZSrf/B/R68Y=', 'Content-Length': '0'} 2024-04-12T03:43:54.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b83b1d7b81402d8-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:43:54.053 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:54.053 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:54.053 DEBUG:boto:Method: GET 2024-04-12T03:43:54.053 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:43:54.053 DEBUG:boto:Data: 2024-04-12T03:43:54.053 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.053 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.053 DEBUG:boto:Params: {} 2024-04-12T03:43:54.054 DEBUG:boto:Token: None 2024-04-12T03:43:54.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj68 2024-04-12T03:43:54.054 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vv6/K9HS0x6TB3uvVVlOYkloD+E= 2024-04-12T03:43:54.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vv6/K9HS0x6TB3uvVVlOYkloD+E=', 'Content-Length': '0'} 2024-04-12T03:43:54.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b774fe11e77d06-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.056 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:43:54.056 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:43:54.056 DEBUG:boto:Method: GET 2024-04-12T03:43:54.056 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:43:54.056 DEBUG:boto:Data: 2024-04-12T03:43:54.056 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.056 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.056 DEBUG:boto:Params: {} 2024-04-12T03:43:54.057 DEBUG:boto:Token: None 2024-04-12T03:43:54.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj68 2024-04-12T03:43:54.057 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vv6/K9HS0x6TB3uvVVlOYkloD+E= 2024-04-12T03:43:54.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vv6/K9HS0x6TB3uvVVlOYkloD+E=', 'Content-Length': '0'} 2024-04-12T03:43:54.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050472da3a0ef34b3-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.106 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.107 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.107 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.107 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.107 DEBUG:boto:Method: GET 2024-04-12T03:43:54.107 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.107 DEBUG:boto:Data: 2024-04-12T03:43:54.107 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.107 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.107 DEBUG:boto:Params: {} 2024-04-12T03:43:54.108 DEBUG:boto:Token: None 2024-04-12T03:43:54.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1a6d80b5acb6b4d-006618adfa-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.110 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.111 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.111 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.111 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.111 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.111 DEBUG:boto:Method: GET 2024-04-12T03:43:54.111 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:43:54.111 DEBUG:boto:Data: 2024-04-12T03:43:54.111 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.112 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.112 DEBUG:boto:Params: {} 2024-04-12T03:43:54.112 DEBUG:boto:Token: None 2024-04-12T03:43:54.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ff035a5a6e4dcfd-006618adfa-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.115 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:43:54.115 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.115 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.116 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.116 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.116 DEBUG:boto:Method: GET 2024-04-12T03:43:54.116 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.116 DEBUG:boto:Data: 2024-04-12T03:43:54.116 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.116 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.116 DEBUG:boto:Params: {} 2024-04-12T03:43:54.116 DEBUG:boto:Token: None 2024-04-12T03:43:54.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0= 2024-04-12T03:43:54.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0=', 'Content-Length': '0'} 2024-04-12T03:43:54.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246a3342deee815e-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.119 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.120 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.120 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.120 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.120 DEBUG:boto:Method: GET 2024-04-12T03:43:54.120 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.120 DEBUG:boto:Data: 2024-04-12T03:43:54.120 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.120 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.120 DEBUG:boto:Params: {} 2024-04-12T03:43:54.120 DEBUG:boto:Token: None 2024-04-12T03:43:54.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0= 2024-04-12T03:43:54.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0=', 'Content-Length': '0'} 2024-04-12T03:43:54.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9da8f8690a322cc-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.123 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.123 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.123 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.123 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.123 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.123 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.123 DEBUG:boto:Data: 2024-04-12T03:43:54.123 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.124 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.124 DEBUG:boto:Params: {} 2024-04-12T03:43:54.124 DEBUG:boto:Token: None 2024-04-12T03:43:54.124 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C00hiuIMvAal0D7bcjpLiRX/NYQ= 2024-04-12T03:43:54.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C00hiuIMvAal0D7bcjpLiRX/NYQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b31f4cc4cde7bd-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.126 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.126 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.126 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.126 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.126 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.126 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.126 DEBUG:boto:Data: 2024-04-12T03:43:54.127 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.127 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.127 DEBUG:boto:Params: {} 2024-04-12T03:43:54.127 DEBUG:boto:Token: None 2024-04-12T03:43:54.127 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C00hiuIMvAal0D7bcjpLiRX/NYQ= 2024-04-12T03:43:54.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C00hiuIMvAal0D7bcjpLiRX/NYQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005006c8d41abbe1f6-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:43:54.129 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.129 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.130 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.130 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.130 DEBUG:boto:Method: GET 2024-04-12T03:43:54.130 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.130 DEBUG:boto:Data: 2024-04-12T03:43:54.130 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.130 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.130 DEBUG:boto:Params: {} 2024-04-12T03:43:54.130 DEBUG:boto:Token: None 2024-04-12T03:43:54.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0= 2024-04-12T03:43:54.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0=', 'Content-Length': '0'} 2024-04-12T03:43:54.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a19a6511385b9cdd-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.133 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.133 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.133 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.133 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.133 DEBUG:boto:Method: GET 2024-04-12T03:43:54.133 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.133 DEBUG:boto:Data: 2024-04-12T03:43:54.133 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.134 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.134 DEBUG:boto:Params: {} 2024-04-12T03:43:54.134 DEBUG:boto:Token: None 2024-04-12T03:43:54.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:43:54.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0= 2024-04-12T03:43:54.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iRaYg6J6qb75fyGu4q4BM24GBP0=', 'Content-Length': '0'} 2024-04-12T03:43:54.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5ef79d982a7ee8-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.178 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.179 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.179 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.179 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.179 DEBUG:boto:Method: GET 2024-04-12T03:43:54.179 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.179 DEBUG:boto:Data: 2024-04-12T03:43:54.179 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.179 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.179 DEBUG:boto:Params: {} 2024-04-12T03:43:54.180 DEBUG:boto:Token: None 2024-04-12T03:43:54.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592b4301e9ccc6a3-006618adfa-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.182 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.183 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.183 DEBUG:boto:Method: GET 2024-04-12T03:43:54.183 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.183 DEBUG:boto:Data: 2024-04-12T03:43:54.183 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.183 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.183 DEBUG:boto:Params: {} 2024-04-12T03:43:54.184 DEBUG:boto:Token: None 2024-04-12T03:43:54.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c07babceecd0ccf-006618adfa-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.186 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.187 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.187 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.187 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.187 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:43:54.187 DEBUG:boto:Data: 2024-04-12T03:43:54.187 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.187 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.187 DEBUG:boto:Params: {} 2024-04-12T03:43:54.187 DEBUG:boto:Token: None 2024-04-12T03:43:54.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85 2024-04-12T03:43:54.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR0QC/1+Z8nPMnpgLBIqtNtUDNY= 2024-04-12T03:43:54.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR0QC/1+Z8nPMnpgLBIqtNtUDNY=', 'Content-Length': '0'} 2024-04-12T03:43:54.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f42bdee587c1ab6e-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.190 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.190 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.190 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.190 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:43:54.190 DEBUG:boto:Data: 2024-04-12T03:43:54.190 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.191 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.191 DEBUG:boto:Params: {} 2024-04-12T03:43:54.191 DEBUG:boto:Token: None 2024-04-12T03:43:54.191 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85 2024-04-12T03:43:54.191 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR0QC/1+Z8nPMnpgLBIqtNtUDNY= 2024-04-12T03:43:54.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR0QC/1+Z8nPMnpgLBIqtNtUDNY=', 'Content-Length': '0'} 2024-04-12T03:43:54.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77425eaf6fe06e3-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:43:54.193 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.193 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.193 DEBUG:boto:Method: GET 2024-04-12T03:43:54.193 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:43:54.194 DEBUG:boto:Data: 2024-04-12T03:43:54.194 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.194 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.194 DEBUG:boto:Params: {} 2024-04-12T03:43:54.194 DEBUG:boto:Token: None 2024-04-12T03:43:54.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85 2024-04-12T03:43:54.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j5rSNIpAfNgCOXFWfil50ShiKBM= 2024-04-12T03:43:54.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j5rSNIpAfNgCOXFWfil50ShiKBM=', 'Content-Length': '0'} 2024-04-12T03:43:54.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e3333abee45c42-006618adfa-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.196 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:43:54.196 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:43:54.196 DEBUG:boto:Method: GET 2024-04-12T03:43:54.196 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:43:54.197 DEBUG:boto:Data: 2024-04-12T03:43:54.197 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.197 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.197 DEBUG:boto:Params: {} 2024-04-12T03:43:54.197 DEBUG:boto:Token: None 2024-04-12T03:43:54.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj85 2024-04-12T03:43:54.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j5rSNIpAfNgCOXFWfil50ShiKBM= 2024-04-12T03:43:54.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j5rSNIpAfNgCOXFWfil50ShiKBM=', 'Content-Length': '0'} 2024-04-12T03:43:54.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca3e89207675299-006618adfa-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.199 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.199 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.199 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.200 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.200 DEBUG:boto:Method: GET 2024-04-12T03:43:54.200 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.200 DEBUG:boto:Data: 2024-04-12T03:43:54.200 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:43:54.200 DEBUG:boto:Port: 8001 2024-04-12T03:43:54.200 DEBUG:boto:Params: {} 2024-04-12T03:43:54.200 DEBUG:boto:Token: None 2024-04-12T03:43:54.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006beb13f07d298bf5-006618adfa-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.203 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.203 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.203 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.203 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.203 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.204 DEBUG:boto:Method: GET 2024-04-12T03:43:54.204 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:43:54.204 DEBUG:boto:Data: 2024-04-12T03:43:54.204 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.204 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.204 DEBUG:boto:Params: {} 2024-04-12T03:43:54.204 DEBUG:boto:Token: None 2024-04-12T03:43:54.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ= 2024-04-12T03:43:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2IH4DikD7DEyaxDCPpFC6m9cAQ=', 'Content-Length': '0'} 2024-04-12T03:43:54.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9c197f122cbf3f-006618adfa-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.250 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:43:54.251 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:43:54.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:43:54.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:43:54.252 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:43:54.303 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:54.304 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:54.346 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:43:54.346 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:43:54.352 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2024-04-12T03:43:54.395 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.395 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.395 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.395 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:54.395 DEBUG:boto:Data: 2024-04-12T03:43:54.396 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.396 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.396 DEBUG:boto:Params: {} 2024-04-12T03:43:54.396 DEBUG:boto:Token: None 2024-04-12T03:43:54.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8= 2024-04-12T03:43:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8=', 'Content-Length': '0'} 2024-04-12T03:43:54.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c44ab8bf7a26bfb0-006618adfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.401 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:43:54.401 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:43:54.401 DEBUG:boto:Method: PUT 2024-04-12T03:43:54.401 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:43:54.401 DEBUG:boto:Data: 2024-04-12T03:43:54.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:43:54.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.401 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.401 DEBUG:boto:Params: {} 2024-04-12T03:43:54.401 DEBUG:boto:Token: None 2024-04-12T03:43:54.402 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj19 2024-04-12T03:43:54.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XAmMcHQXePZNxUSmo+KrVtzXM0M= 2024-04-12T03:43:54.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XAmMcHQXePZNxUSmo+KrVtzXM0M='} 2024-04-12T03:43:54.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd66ddd86345c024-006618adfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.408 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.408 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.408 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.408 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:54.408 DEBUG:boto:Data: 2024-04-12T03:43:54.408 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.409 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.409 DEBUG:boto:Params: {} 2024-04-12T03:43:54.409 DEBUG:boto:Token: None 2024-04-12T03:43:54.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8= 2024-04-12T03:43:54.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8=', 'Content-Length': '0'} 2024-04-12T03:43:54.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027b280ebd1927296-006618adfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.412 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:43:54.412 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:43:54.412 DEBUG:boto:Method: PUT 2024-04-12T03:43:54.412 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:43:54.412 DEBUG:boto:Data: 2024-04-12T03:43:54.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:43:54.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.413 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.413 DEBUG:boto:Params: {} 2024-04-12T03:43:54.413 DEBUG:boto:Token: None 2024-04-12T03:43:54.413 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj38 2024-04-12T03:43:54.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7LFKZm14bCjkkSyHV3oWBU9uIVA= 2024-04-12T03:43:54.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7LFKZm14bCjkkSyHV3oWBU9uIVA='} 2024-04-12T03:43:54.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074252fb09a28d9e5-006618adfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.463 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.463 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.463 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.463 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:54.463 DEBUG:boto:Data: 2024-04-12T03:43:54.463 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.463 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.464 DEBUG:boto:Params: {} 2024-04-12T03:43:54.464 DEBUG:boto:Token: None 2024-04-12T03:43:54.464 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8= 2024-04-12T03:43:54.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8=', 'Content-Length': '0'} 2024-04-12T03:43:54.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca91480965df0f84-006618adfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.467 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:43:54.467 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:43:54.467 DEBUG:boto:Method: PUT 2024-04-12T03:43:54.467 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:43:54.467 DEBUG:boto:Data: 2024-04-12T03:43:54.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:43:54.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.468 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.468 DEBUG:boto:Params: {} 2024-04-12T03:43:54.468 DEBUG:boto:Token: None 2024-04-12T03:43:54.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj57 2024-04-12T03:43:54.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+v0DUIO3h1y5kaLpcz3jbqlRIPI= 2024-04-12T03:43:54.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+v0DUIO3h1y5kaLpcz3jbqlRIPI='} 2024-04-12T03:43:54.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b07b853638e1e5d2-006618adfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.472 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.472 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.472 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.472 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:54.472 DEBUG:boto:Data: 2024-04-12T03:43:54.472 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.472 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.473 DEBUG:boto:Params: {} 2024-04-12T03:43:54.473 DEBUG:boto:Token: None 2024-04-12T03:43:54.473 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8= 2024-04-12T03:43:54.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8=', 'Content-Length': '0'} 2024-04-12T03:43:54.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000296ee08036ef12cb-006618adfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.476 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:43:54.476 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:43:54.476 DEBUG:boto:Method: PUT 2024-04-12T03:43:54.476 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:43:54.476 DEBUG:boto:Data: 2024-04-12T03:43:54.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:43:54.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.476 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.477 DEBUG:boto:Params: {} 2024-04-12T03:43:54.477 DEBUG:boto:Token: None 2024-04-12T03:43:54.477 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj76 2024-04-12T03:43:54.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WYhym/Qygy3B7Fhq56MtPhSB5fk= 2024-04-12T03:43:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WYhym/Qygy3B7Fhq56MtPhSB5fk='} 2024-04-12T03:43:54.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000085c48a4a3d97746-006618adfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.519 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:43:54.519 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:43:54.519 DEBUG:boto:Method: HEAD 2024-04-12T03:43:54.519 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:43:54.519 DEBUG:boto:Data: 2024-04-12T03:43:54.519 DEBUG:boto:Headers: {} 2024-04-12T03:43:54.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.519 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.520 DEBUG:boto:Params: {} 2024-04-12T03:43:54.520 DEBUG:boto:Token: None 2024-04-12T03:43:54.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/ 2024-04-12T03:43:54.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8= 2024-04-12T03:43:54.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t1MNVfUxIE8JJSuZM5fFuC2TMO8=', 'Content-Length': '0'} 2024-04-12T03:43:54.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8a339aca5b5a1b6-006618adfa-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.523 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:43:54.523 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:43:54.523 DEBUG:boto:Method: PUT 2024-04-12T03:43:54.523 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:43:54.523 DEBUG:boto:Data: 2024-04-12T03:43:54.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:43:54.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:43:54.523 DEBUG:boto:Port: 8000 2024-04-12T03:43:54.524 DEBUG:boto:Params: {} 2024-04-12T03:43:54.524 DEBUG:boto:Token: None 2024-04-12T03:43:54.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:43:54 GMT /dgbpxg-61/obj95 2024-04-12T03:43:54.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R6NSgCoH43JR9Xiv3gWppeZ5VeM= 2024-04-12T03:43:54.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:43:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R6NSgCoH43JR9Xiv3gWppeZ5VeM='} 2024-04-12T03:43:54.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3e672d84d646fbe-006618adfa-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:43:54.528 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:43:54.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:43:54.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:43:54.528 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:43:54.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:54.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:54.616+0000 7fe1a114a780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:43:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:43:59.618+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:43:59.620 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T03:43:59.620 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:04.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:04.620+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:04.622 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T03:44:04.622 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:09.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:09.622+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:09.623 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T03:44:09.623 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:14.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:14.623+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:14.625 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T03:44:14.625 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:19.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:19.625+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:19.627 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:19.627 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:24.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:24.627+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:24.628 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:24.628 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:29.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:29.628+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:29.630 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:29.630 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:34.629+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:34.631 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:34.631 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:39.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:39.631+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:39.633 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:39.633 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:44.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:44.633+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:44.635 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:44.635 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:49.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:49.635+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:49.638 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:49.638 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:54.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:54.637+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:54.638 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000002.66.6, , , , , , , , , , , ] 2024-04-12T03:44:54.638 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:59.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:44:59.638+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:44:59.640 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:44:59.640 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:04.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:04.640+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:04.641 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:04.641 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:09.641+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:09.643 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:09.643 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:14.643+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:14.644 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:14.644 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:19.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:19.643+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:19.645 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:19.645 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:24.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:24.645+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:24.647 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:24.647 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:29.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:29.648+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:29.648 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:29.648 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:34.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:34.649+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:34.650 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:34.650 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:39.651+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:39.651 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:39.652 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:44.652+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:44.653 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:44.653 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:49.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:49.654+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:49.654 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:49.655 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:54.656+0000 7fe1a114a780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:54.656 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:54.656 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:59.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:59.656+0000 7fe1a114a780 1 bucket sync caught up with source: 2024-04-12T03:45:59.657 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:59.658 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.49.6, 00000000004.70.6, , , , , , , , , , , ] 2024-04-12T03:45:59.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:59.656+0000 7fe1a114a780 0 bucket checkpoint complete 2024-04-12T03:45:59.675 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:45:59.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:45:59.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:45:59.676 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:45:59.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:59.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:59.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:45:59.774+0000 7fc7c3948780 1 waiting for incremental sync to catch up: 2024-04-12T03:45:59.775 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, , 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:45:59.775 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:04.775+0000 7fc7c3948780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:04.776 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, , 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:04.777 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:09.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:09.778+0000 7fc7c3948780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:09.778 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, , 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:09.778 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:14.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:14.779+0000 7fc7c3948780 1 bucket sync caught up with source: 2024-04-12T03:46:14.779 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:14.779 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:46:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:14.779+0000 7fc7c3948780 0 bucket checkpoint complete 2024-04-12T03:46:14.796 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:46:14.796 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.797 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.797 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.797 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:14.797 DEBUG:boto:Data: 2024-04-12T03:46:14.797 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.797 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.797 DEBUG:boto:Params: {} 2024-04-12T03:46:14.798 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:46:14.798 DEBUG:boto:Token: None 2024-04-12T03:46:14.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/ 2024-04-12T03:46:14.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sV6Hc7zYEQnPdE1P7rgHPvLJ3lo= 2024-04-12T03:46:14.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sV6Hc7zYEQnPdE1P7rgHPvLJ3lo=', 'Content-Length': '0'} 2024-04-12T03:46:14.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009467a9b9f1d3a188-006618ae86-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.829 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.829 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.829 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.830 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:14.830 DEBUG:boto:Data: 2024-04-12T03:46:14.830 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.830 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.830 DEBUG:boto:Params: {} 2024-04-12T03:46:14.830 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:46:14.830 DEBUG:boto:Token: None 2024-04-12T03:46:14.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/ 2024-04-12T03:46:14.831 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sV6Hc7zYEQnPdE1P7rgHPvLJ3lo= 2024-04-12T03:46:14.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sV6Hc7zYEQnPdE1P7rgHPvLJ3lo=', 'Content-Length': '0'} 2024-04-12T03:46:14.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038aeefcc6747b40d-006618ae86-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:46:14.844 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.845 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.845 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.845 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.845 DEBUG:boto:Method: GET 2024-04-12T03:46:14.845 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.845 DEBUG:boto:Data: 2024-04-12T03:46:14.845 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.845 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.845 DEBUG:boto:Params: {} 2024-04-12T03:46:14.845 DEBUG:boto:Token: None 2024-04-12T03:46:14.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/?versions 2024-04-12T03:46:14.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w= 2024-04-12T03:46:14.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w=', 'Content-Length': '0'} 2024-04-12T03:46:14.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7be1615624332f8-006618ae86-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.848 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:46:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:46:14.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:46:14.851 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.851 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.852 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.852 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.852 DEBUG:boto:Method: GET 2024-04-12T03:46:14.852 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.852 DEBUG:boto:Data: 2024-04-12T03:46:14.852 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.852 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.852 DEBUG:boto:Params: {} 2024-04-12T03:46:14.852 DEBUG:boto:Token: None 2024-04-12T03:46:14.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/?versions 2024-04-12T03:46:14.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w= 2024-04-12T03:46:14.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w=', 'Content-Length': '0'} 2024-04-12T03:46:14.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000547efdad688679eb-006618ae86-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.855 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:46:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:46:14.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:46:14.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:46:14.858 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.858 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.858 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.858 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.858 DEBUG:boto:Method: GET 2024-04-12T03:46:14.858 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.858 DEBUG:boto:Data: 2024-04-12T03:46:14.859 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.859 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.859 DEBUG:boto:Params: {} 2024-04-12T03:46:14.859 DEBUG:boto:Token: None 2024-04-12T03:46:14.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/?versions 2024-04-12T03:46:14.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w= 2024-04-12T03:46:14.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w=', 'Content-Length': '0'} 2024-04-12T03:46:14.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1ba7a13ac93b70-006618ae86-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.862 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.863 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.863 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.863 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.863 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.863 DEBUG:boto:Method: GET 2024-04-12T03:46:14.863 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:46:14.863 DEBUG:boto:Data: 2024-04-12T03:46:14.863 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.863 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.864 DEBUG:boto:Params: {} 2024-04-12T03:46:14.864 DEBUG:boto:Token: None 2024-04-12T03:46:14.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/?versions 2024-04-12T03:46:14.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w= 2024-04-12T03:46:14.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+afZpA7R/plcpyUNZEq9r1qhV6w=', 'Content-Length': '0'} 2024-04-12T03:46:14.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10c96ee1ee3342f-006618ae86-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.867 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:46:14.868 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.868 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.868 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.868 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.869 DEBUG:boto:Method: GET 2024-04-12T03:46:14.869 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.869 DEBUG:boto:Data: 2024-04-12T03:46:14.869 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.869 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.869 DEBUG:boto:Params: {} 2024-04-12T03:46:14.869 DEBUG:boto:Token: None 2024-04-12T03:46:14.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.869 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio= 2024-04-12T03:46:14.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio=', 'Content-Length': '0'} 2024-04-12T03:46:14.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d651bde7c7d20963-006618ae86-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.872 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.872 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.872 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.872 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.872 DEBUG:boto:Method: GET 2024-04-12T03:46:14.872 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.872 DEBUG:boto:Data: 2024-04-12T03:46:14.872 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.873 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.873 DEBUG:boto:Params: {} 2024-04-12T03:46:14.873 DEBUG:boto:Token: None 2024-04-12T03:46:14.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio= 2024-04-12T03:46:14.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio=', 'Content-Length': '0'} 2024-04-12T03:46:14.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f2e01df4e6ec8b-006618ae86-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.922 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.923 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.923 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.923 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.923 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.923 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.923 DEBUG:boto:Data: 2024-04-12T03:46:14.923 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.923 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.923 DEBUG:boto:Params: {} 2024-04-12T03:46:14.924 DEBUG:boto:Token: None 2024-04-12T03:46:14.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ye50zGB2WOh/7mZfHhm4vaCLx7Y= 2024-04-12T03:46:14.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ye50zGB2WOh/7mZfHhm4vaCLx7Y=', 'Content-Length': '0'} 2024-04-12T03:46:14.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c23d1f956ceb351-006618ae86-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.926 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.926 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.926 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.927 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.927 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.927 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.927 DEBUG:boto:Data: 2024-04-12T03:46:14.927 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.927 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.927 DEBUG:boto:Params: {} 2024-04-12T03:46:14.927 DEBUG:boto:Token: None 2024-04-12T03:46:14.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ye50zGB2WOh/7mZfHhm4vaCLx7Y= 2024-04-12T03:46:14.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ye50zGB2WOh/7mZfHhm4vaCLx7Y=', 'Content-Length': '0'} 2024-04-12T03:46:14.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b731131110f3ff02-006618ae86-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:46:14.930 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.930 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.930 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.930 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.931 DEBUG:boto:Method: GET 2024-04-12T03:46:14.931 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.931 DEBUG:boto:Data: 2024-04-12T03:46:14.931 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.931 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.931 DEBUG:boto:Params: {} 2024-04-12T03:46:14.931 DEBUG:boto:Token: None 2024-04-12T03:46:14.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio= 2024-04-12T03:46:14.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio=', 'Content-Length': '0'} 2024-04-12T03:46:14.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e75472931768fa-006618ae86-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.934 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.934 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.934 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.934 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.934 DEBUG:boto:Method: GET 2024-04-12T03:46:14.934 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.935 DEBUG:boto:Data: 2024-04-12T03:46:14.935 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.935 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.935 DEBUG:boto:Params: {} 2024-04-12T03:46:14.935 DEBUG:boto:Token: None 2024-04-12T03:46:14.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:46:14.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio= 2024-04-12T03:46:14.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxYIuSGPvMCLt3a1PtFkPsLgcio=', 'Content-Length': '0'} 2024-04-12T03:46:14.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052edeca6b8e138ef-006618ae86-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.939 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.939 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.939 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.940 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.940 DEBUG:boto:Method: GET 2024-04-12T03:46:14.940 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.940 DEBUG:boto:Data: 2024-04-12T03:46:14.940 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.940 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.940 DEBUG:boto:Params: {} 2024-04-12T03:46:14.940 DEBUG:boto:Token: None 2024-04-12T03:46:14.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/ 2024-04-12T03:46:14.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wby+YAykR4kq/3atBiQb9gR+cKs= 2024-04-12T03:46:14.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wby+YAykR4kq/3atBiQb9gR+cKs=', 'Content-Length': '0'} 2024-04-12T03:46:14.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af7f95d2848ec78d-006618ae86-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.943 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.944 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:14.944 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:14.944 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.944 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.944 DEBUG:boto:Method: GET 2024-04-12T03:46:14.944 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:46:14.944 DEBUG:boto:Data: 2024-04-12T03:46:14.944 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.945 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.945 DEBUG:boto:Params: {} 2024-04-12T03:46:14.945 DEBUG:boto:Token: None 2024-04-12T03:46:14.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/ 2024-04-12T03:46:14.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wby+YAykR4kq/3atBiQb9gR+cKs= 2024-04-12T03:46:14.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wby+YAykR4kq/3atBiQb9gR+cKs=', 'Content-Length': '0'} 2024-04-12T03:46:14.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a2b10e8fa1a6a6e-006618ae86-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.948 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:14.949 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.949 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.949 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.949 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:46:14.949 DEBUG:boto:Data: 2024-04-12T03:46:14.949 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.950 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.950 DEBUG:boto:Params: {} 2024-04-12T03:46:14.950 DEBUG:boto:Token: None 2024-04-12T03:46:14.950 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11 2024-04-12T03:46:14.950 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uUCFAZKP+TOeo4CxFanoh89dyXE= 2024-04-12T03:46:14.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uUCFAZKP+TOeo4CxFanoh89dyXE=', 'Content-Length': '0'} 2024-04-12T03:46:14.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031456fd7464e50a1-006618ae86-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.952 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.952 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.952 DEBUG:boto:Method: HEAD 2024-04-12T03:46:14.952 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:46:14.952 DEBUG:boto:Data: 2024-04-12T03:46:14.952 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.952 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.953 DEBUG:boto:Params: {} 2024-04-12T03:46:14.953 DEBUG:boto:Token: None 2024-04-12T03:46:14.953 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11 2024-04-12T03:46:14.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uUCFAZKP+TOeo4CxFanoh89dyXE= 2024-04-12T03:46:14.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uUCFAZKP+TOeo4CxFanoh89dyXE=', 'Content-Length': '0'} 2024-04-12T03:46:14.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f18ef989fd4757-006618ae86-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:46:14.955 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.955 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.955 DEBUG:boto:Method: GET 2024-04-12T03:46:14.955 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:46:14.955 DEBUG:boto:Data: 2024-04-12T03:46:14.956 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:14.956 DEBUG:boto:Port: 8001 2024-04-12T03:46:14.956 DEBUG:boto:Params: {} 2024-04-12T03:46:14.956 DEBUG:boto:Token: None 2024-04-12T03:46:14.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11 2024-04-12T03:46:14.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JJZ6ySizu8D7DRV0Z9AwerEdTFA= 2024-04-12T03:46:14.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JJZ6ySizu8D7DRV0Z9AwerEdTFA=', 'Content-Length': '0'} 2024-04-12T03:46:14.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032bbbf0f3d0ae051-006618ae86-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:14.958 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:46:14.958 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:46:14.958 DEBUG:boto:Method: GET 2024-04-12T03:46:14.958 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:46:14.958 DEBUG:boto:Data: 2024-04-12T03:46:14.959 DEBUG:boto:Headers: {} 2024-04-12T03:46:14.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:14.959 DEBUG:boto:Port: 8000 2024-04-12T03:46:14.959 DEBUG:boto:Params: {} 2024-04-12T03:46:14.959 DEBUG:boto:Token: None 2024-04-12T03:46:14.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:14 GMT /dgbpxg-61/obj11 2024-04-12T03:46:14.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JJZ6ySizu8D7DRV0Z9AwerEdTFA= 2024-04-12T03:46:14.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:14 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JJZ6ySizu8D7DRV0Z9AwerEdTFA=', 'Content-Length': '0'} 2024-04-12T03:46:14.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb1f5925a1b1041d-006618ae86-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.002 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.003 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.003 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.003 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.003 DEBUG:boto:Method: GET 2024-04-12T03:46:15.003 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.003 DEBUG:boto:Data: 2024-04-12T03:46:15.003 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.003 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.003 DEBUG:boto:Params: {} 2024-04-12T03:46:15.004 DEBUG:boto:Token: None 2024-04-12T03:46:15.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df031ee90fd230a-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.006 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.007 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.007 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.007 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.007 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.007 DEBUG:boto:Method: GET 2024-04-12T03:46:15.007 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:46:15.007 DEBUG:boto:Data: 2024-04-12T03:46:15.008 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.008 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.008 DEBUG:boto:Params: {} 2024-04-12T03:46:15.008 DEBUG:boto:Token: None 2024-04-12T03:46:15.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b2154d3263e87a-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.011 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:46:15.012 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.012 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.012 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.012 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.012 DEBUG:boto:Method: GET 2024-04-12T03:46:15.012 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.013 DEBUG:boto:Data: 2024-04-12T03:46:15.013 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.013 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.013 DEBUG:boto:Params: {} 2024-04-12T03:46:15.013 DEBUG:boto:Token: None 2024-04-12T03:46:15.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0= 2024-04-12T03:46:15.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0=', 'Content-Length': '0'} 2024-04-12T03:46:15.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a6337d9c66dd29-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.016 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.016 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.016 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.016 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.016 DEBUG:boto:Method: GET 2024-04-12T03:46:15.016 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.016 DEBUG:boto:Data: 2024-04-12T03:46:15.016 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.016 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.016 DEBUG:boto:Params: {} 2024-04-12T03:46:15.017 DEBUG:boto:Token: None 2024-04-12T03:46:15.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0= 2024-04-12T03:46:15.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0=', 'Content-Length': '0'} 2024-04-12T03:46:15.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c9e146fca63eb2-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.019 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.019 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.019 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.019 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.019 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.019 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.019 DEBUG:boto:Data: 2024-04-12T03:46:15.020 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.020 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.020 DEBUG:boto:Params: {} 2024-04-12T03:46:15.020 DEBUG:boto:Token: None 2024-04-12T03:46:15.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i164HstWJfQTVbp6gSnW9tZWkvE= 2024-04-12T03:46:15.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i164HstWJfQTVbp6gSnW9tZWkvE=', 'Content-Length': '0'} 2024-04-12T03:46:15.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949881d6fe88f11c-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.022 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.022 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.022 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.022 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.022 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.023 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.023 DEBUG:boto:Data: 2024-04-12T03:46:15.023 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.023 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.023 DEBUG:boto:Params: {} 2024-04-12T03:46:15.023 DEBUG:boto:Token: None 2024-04-12T03:46:15.023 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i164HstWJfQTVbp6gSnW9tZWkvE= 2024-04-12T03:46:15.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i164HstWJfQTVbp6gSnW9tZWkvE=', 'Content-Length': '0'} 2024-04-12T03:46:15.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001719303a10f5f81b-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:46:15.025 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.025 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.025 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.025 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.025 DEBUG:boto:Method: GET 2024-04-12T03:46:15.026 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.026 DEBUG:boto:Data: 2024-04-12T03:46:15.026 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.026 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.026 DEBUG:boto:Params: {} 2024-04-12T03:46:15.026 DEBUG:boto:Token: None 2024-04-12T03:46:15.026 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.026 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0= 2024-04-12T03:46:15.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0=', 'Content-Length': '0'} 2024-04-12T03:46:15.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50788eaef413fee-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.070 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.070 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.071 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.071 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.071 DEBUG:boto:Method: GET 2024-04-12T03:46:15.071 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.071 DEBUG:boto:Data: 2024-04-12T03:46:15.071 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.071 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.071 DEBUG:boto:Params: {} 2024-04-12T03:46:15.072 DEBUG:boto:Token: None 2024-04-12T03:46:15.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:46:15.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0= 2024-04-12T03:46:15.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yncOwvTs8C06q6eRNFpk0r4uKr0=', 'Content-Length': '0'} 2024-04-12T03:46:15.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335f6bc3adbae900-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.074 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.074 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.074 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.074 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.074 DEBUG:boto:Method: GET 2024-04-12T03:46:15.075 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.075 DEBUG:boto:Data: 2024-04-12T03:46:15.075 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.075 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.075 DEBUG:boto:Params: {} 2024-04-12T03:46:15.075 DEBUG:boto:Token: None 2024-04-12T03:46:15.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c448445737df9ff8-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.078 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.078 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.078 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.078 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.078 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.079 DEBUG:boto:Method: GET 2024-04-12T03:46:15.079 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.079 DEBUG:boto:Data: 2024-04-12T03:46:15.079 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.079 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.079 DEBUG:boto:Params: {} 2024-04-12T03:46:15.079 DEBUG:boto:Token: None 2024-04-12T03:46:15.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c94bdbc6018ba24-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.082 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.083 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.083 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.083 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.083 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:46:15.083 DEBUG:boto:Data: 2024-04-12T03:46:15.083 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.083 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.084 DEBUG:boto:Params: {} 2024-04-12T03:46:15.084 DEBUG:boto:Token: None 2024-04-12T03:46:15.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17 2024-04-12T03:46:15.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZMiIj4dawYhQ3W4Rmf4IOLG1Dk= 2024-04-12T03:46:15.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZMiIj4dawYhQ3W4Rmf4IOLG1Dk=', 'Content-Length': '0'} 2024-04-12T03:46:15.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007574b5d8accc5c73-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.086 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.086 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.086 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.086 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:46:15.086 DEBUG:boto:Data: 2024-04-12T03:46:15.086 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.086 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.086 DEBUG:boto:Params: {} 2024-04-12T03:46:15.087 DEBUG:boto:Token: None 2024-04-12T03:46:15.087 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17 2024-04-12T03:46:15.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZMiIj4dawYhQ3W4Rmf4IOLG1Dk= 2024-04-12T03:46:15.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZMiIj4dawYhQ3W4Rmf4IOLG1Dk=', 'Content-Length': '0'} 2024-04-12T03:46:15.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56812ae11a72726-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:46:15.089 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.089 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.089 DEBUG:boto:Method: GET 2024-04-12T03:46:15.089 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:46:15.089 DEBUG:boto:Data: 2024-04-12T03:46:15.090 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.090 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.090 DEBUG:boto:Params: {} 2024-04-12T03:46:15.090 DEBUG:boto:Token: None 2024-04-12T03:46:15.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17 2024-04-12T03:46:15.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RiDTNOayuT7+m9FE0X9/eK0zr0w= 2024-04-12T03:46:15.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RiDTNOayuT7+m9FE0X9/eK0zr0w=', 'Content-Length': '0'} 2024-04-12T03:46:15.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef8a00cedc7bbd5-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.092 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:46:15.092 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:46:15.093 DEBUG:boto:Method: GET 2024-04-12T03:46:15.093 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:46:15.093 DEBUG:boto:Data: 2024-04-12T03:46:15.093 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.093 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.093 DEBUG:boto:Params: {} 2024-04-12T03:46:15.093 DEBUG:boto:Token: None 2024-04-12T03:46:15.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj17 2024-04-12T03:46:15.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RiDTNOayuT7+m9FE0X9/eK0zr0w= 2024-04-12T03:46:15.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RiDTNOayuT7+m9FE0X9/eK0zr0w=', 'Content-Length': '0'} 2024-04-12T03:46:15.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000accb0e8b5b265541-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.096 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.096 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.096 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.096 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.096 DEBUG:boto:Method: GET 2024-04-12T03:46:15.096 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.096 DEBUG:boto:Data: 2024-04-12T03:46:15.096 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.096 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.097 DEBUG:boto:Params: {} 2024-04-12T03:46:15.097 DEBUG:boto:Token: None 2024-04-12T03:46:15.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e2cc0f19fbf383-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.100 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.100 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.100 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.100 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.100 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.100 DEBUG:boto:Method: GET 2024-04-12T03:46:15.101 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:46:15.101 DEBUG:boto:Data: 2024-04-12T03:46:15.101 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.101 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.101 DEBUG:boto:Params: {} 2024-04-12T03:46:15.101 DEBUG:boto:Token: None 2024-04-12T03:46:15.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7944d747483caab-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.104 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:46:15.105 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.105 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.105 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.105 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.105 DEBUG:boto:Method: GET 2024-04-12T03:46:15.105 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.105 DEBUG:boto:Data: 2024-04-12T03:46:15.105 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.105 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.105 DEBUG:boto:Params: {} 2024-04-12T03:46:15.106 DEBUG:boto:Token: None 2024-04-12T03:46:15.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4= 2024-04-12T03:46:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4=', 'Content-Length': '0'} 2024-04-12T03:46:15.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7e538ee146b336-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.108 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.108 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.108 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.108 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.109 DEBUG:boto:Method: GET 2024-04-12T03:46:15.109 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.109 DEBUG:boto:Data: 2024-04-12T03:46:15.109 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.109 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.109 DEBUG:boto:Params: {} 2024-04-12T03:46:15.109 DEBUG:boto:Token: None 2024-04-12T03:46:15.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.109 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4= 2024-04-12T03:46:15.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4=', 'Content-Length': '0'} 2024-04-12T03:46:15.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fc79d50d559bf5-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.111 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.112 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.112 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.112 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.112 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.112 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.112 DEBUG:boto:Data: 2024-04-12T03:46:15.112 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.112 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.112 DEBUG:boto:Params: {} 2024-04-12T03:46:15.112 DEBUG:boto:Token: None 2024-04-12T03:46:15.113 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XdxGoX8Q6mgBxU9jumVdQRCQrb8= 2024-04-12T03:46:15.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XdxGoX8Q6mgBxU9jumVdQRCQrb8=', 'Content-Length': '0'} 2024-04-12T03:46:15.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049420ed37aafd560-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.114 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.115 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.115 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.115 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.115 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.115 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.115 DEBUG:boto:Data: 2024-04-12T03:46:15.115 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.115 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.115 DEBUG:boto:Params: {} 2024-04-12T03:46:15.116 DEBUG:boto:Token: None 2024-04-12T03:46:15.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XdxGoX8Q6mgBxU9jumVdQRCQrb8= 2024-04-12T03:46:15.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XdxGoX8Q6mgBxU9jumVdQRCQrb8=', 'Content-Length': '0'} 2024-04-12T03:46:15.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02bf97aed62d073-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:46:15.118 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.118 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.118 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.118 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.118 DEBUG:boto:Method: GET 2024-04-12T03:46:15.118 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.119 DEBUG:boto:Data: 2024-04-12T03:46:15.119 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.119 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.119 DEBUG:boto:Params: {} 2024-04-12T03:46:15.119 DEBUG:boto:Token: None 2024-04-12T03:46:15.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4= 2024-04-12T03:46:15.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4=', 'Content-Length': '0'} 2024-04-12T03:46:15.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b14a8bec6528f94-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.121 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.121 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.121 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.121 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.122 DEBUG:boto:Method: GET 2024-04-12T03:46:15.122 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.122 DEBUG:boto:Data: 2024-04-12T03:46:15.122 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.122 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.122 DEBUG:boto:Params: {} 2024-04-12T03:46:15.122 DEBUG:boto:Token: None 2024-04-12T03:46:15.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:46:15.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4= 2024-04-12T03:46:15.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sULNZogTciUpO1CMv5IKQVtp+u4=', 'Content-Length': '0'} 2024-04-12T03:46:15.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe13e3849f30aa33-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.166 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.166 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.167 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.167 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.167 DEBUG:boto:Method: GET 2024-04-12T03:46:15.167 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.167 DEBUG:boto:Data: 2024-04-12T03:46:15.167 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.167 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.167 DEBUG:boto:Params: {} 2024-04-12T03:46:15.167 DEBUG:boto:Token: None 2024-04-12T03:46:15.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001feb57c227f0481d-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.170 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.170 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.171 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.171 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.171 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.171 DEBUG:boto:Method: GET 2024-04-12T03:46:15.171 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.171 DEBUG:boto:Data: 2024-04-12T03:46:15.171 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.171 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.171 DEBUG:boto:Params: {} 2024-04-12T03:46:15.172 DEBUG:boto:Token: None 2024-04-12T03:46:15.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810e4d219f9cc01a-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.175 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.175 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.175 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.176 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.176 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:46:15.176 DEBUG:boto:Data: 2024-04-12T03:46:15.176 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.176 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.176 DEBUG:boto:Params: {} 2024-04-12T03:46:15.176 DEBUG:boto:Token: None 2024-04-12T03:46:15.176 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19 2024-04-12T03:46:15.177 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aF5ovByaBHvWVBNbF8Cj3/Gfstg= 2024-04-12T03:46:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aF5ovByaBHvWVBNbF8Cj3/Gfstg=', 'Content-Length': '0'} 2024-04-12T03:46:15.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0cf66da7d9f5a8-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.178 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.178 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.179 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.179 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:46:15.179 DEBUG:boto:Data: 2024-04-12T03:46:15.179 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.179 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.179 DEBUG:boto:Params: {} 2024-04-12T03:46:15.179 DEBUG:boto:Token: None 2024-04-12T03:46:15.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19 2024-04-12T03:46:15.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aF5ovByaBHvWVBNbF8Cj3/Gfstg= 2024-04-12T03:46:15.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aF5ovByaBHvWVBNbF8Cj3/Gfstg=', 'Content-Length': '0'} 2024-04-12T03:46:15.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021278618cf0bf1e2-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:46:15.182 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.182 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.182 DEBUG:boto:Method: GET 2024-04-12T03:46:15.182 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:46:15.182 DEBUG:boto:Data: 2024-04-12T03:46:15.182 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.182 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.182 DEBUG:boto:Params: {} 2024-04-12T03:46:15.182 DEBUG:boto:Token: None 2024-04-12T03:46:15.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19 2024-04-12T03:46:15.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+HBZgVkINU6LcreE7LPb1kpveLc= 2024-04-12T03:46:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+HBZgVkINU6LcreE7LPb1kpveLc=', 'Content-Length': '0'} 2024-04-12T03:46:15.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb4fc5be774bad3-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.185 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:46:15.185 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:46:15.185 DEBUG:boto:Method: GET 2024-04-12T03:46:15.185 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:46:15.185 DEBUG:boto:Data: 2024-04-12T03:46:15.186 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.186 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.186 DEBUG:boto:Params: {} 2024-04-12T03:46:15.186 DEBUG:boto:Token: None 2024-04-12T03:46:15.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj19 2024-04-12T03:46:15.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+HBZgVkINU6LcreE7LPb1kpveLc= 2024-04-12T03:46:15.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+HBZgVkINU6LcreE7LPb1kpveLc=', 'Content-Length': '0'} 2024-04-12T03:46:15.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c010913c0152ac-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.188 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.188 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.188 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.189 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.189 DEBUG:boto:Method: GET 2024-04-12T03:46:15.189 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.189 DEBUG:boto:Data: 2024-04-12T03:46:15.189 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.189 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.189 DEBUG:boto:Params: {} 2024-04-12T03:46:15.189 DEBUG:boto:Token: None 2024-04-12T03:46:15.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73516d5200a61ae-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.192 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.192 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.192 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.193 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.193 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.193 DEBUG:boto:Method: GET 2024-04-12T03:46:15.193 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:46:15.193 DEBUG:boto:Data: 2024-04-12T03:46:15.193 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.193 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.193 DEBUG:boto:Params: {} 2024-04-12T03:46:15.193 DEBUG:boto:Token: None 2024-04-12T03:46:15.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b6e1dab4706c7a-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.196 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:46:15.197 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.197 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.197 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.197 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.197 DEBUG:boto:Method: GET 2024-04-12T03:46:15.197 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.197 DEBUG:boto:Data: 2024-04-12T03:46:15.198 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.198 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.198 DEBUG:boto:Params: {} 2024-04-12T03:46:15.198 DEBUG:boto:Token: None 2024-04-12T03:46:15.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.198 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw= 2024-04-12T03:46:15.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw=', 'Content-Length': '0'} 2024-04-12T03:46:15.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077724a2602fca3bf-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.200 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.200 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.200 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.200 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.201 DEBUG:boto:Method: GET 2024-04-12T03:46:15.201 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.201 DEBUG:boto:Data: 2024-04-12T03:46:15.201 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.201 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.201 DEBUG:boto:Params: {} 2024-04-12T03:46:15.201 DEBUG:boto:Token: None 2024-04-12T03:46:15.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw= 2024-04-12T03:46:15.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw=', 'Content-Length': '0'} 2024-04-12T03:46:15.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffbaa8a67068ac4-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.203 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.203 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.203 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.204 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.204 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.204 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.204 DEBUG:boto:Data: 2024-04-12T03:46:15.204 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.204 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.204 DEBUG:boto:Params: {} 2024-04-12T03:46:15.204 DEBUG:boto:Token: None 2024-04-12T03:46:15.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.205 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xfNFqBtSMwRqacucmmKISOoHHoc= 2024-04-12T03:46:15.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xfNFqBtSMwRqacucmmKISOoHHoc=', 'Content-Length': '0'} 2024-04-12T03:46:15.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f8168ffb44a647-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.206 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.207 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.207 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.207 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.207 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.207 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.207 DEBUG:boto:Data: 2024-04-12T03:46:15.207 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.207 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.207 DEBUG:boto:Params: {} 2024-04-12T03:46:15.207 DEBUG:boto:Token: None 2024-04-12T03:46:15.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xfNFqBtSMwRqacucmmKISOoHHoc= 2024-04-12T03:46:15.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xfNFqBtSMwRqacucmmKISOoHHoc=', 'Content-Length': '0'} 2024-04-12T03:46:15.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162eec32ec6eb825-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:46:15.209 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.210 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.210 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.210 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.210 DEBUG:boto:Method: GET 2024-04-12T03:46:15.210 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.210 DEBUG:boto:Data: 2024-04-12T03:46:15.210 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.210 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.210 DEBUG:boto:Params: {} 2024-04-12T03:46:15.211 DEBUG:boto:Token: None 2024-04-12T03:46:15.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw= 2024-04-12T03:46:15.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw=', 'Content-Length': '0'} 2024-04-12T03:46:15.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f491e77f5bfb96-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.258 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.258 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.258 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.259 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.259 DEBUG:boto:Method: GET 2024-04-12T03:46:15.259 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.259 DEBUG:boto:Data: 2024-04-12T03:46:15.259 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.259 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.259 DEBUG:boto:Params: {} 2024-04-12T03:46:15.259 DEBUG:boto:Token: None 2024-04-12T03:46:15.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:46:15.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw= 2024-04-12T03:46:15.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3wETjPY0k93xlG+2wag4LAcxEWw=', 'Content-Length': '0'} 2024-04-12T03:46:15.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed3a0a8f464b62f-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.261 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.262 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.262 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.262 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.262 DEBUG:boto:Method: GET 2024-04-12T03:46:15.262 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.262 DEBUG:boto:Data: 2024-04-12T03:46:15.262 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.262 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.262 DEBUG:boto:Params: {} 2024-04-12T03:46:15.263 DEBUG:boto:Token: None 2024-04-12T03:46:15.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac77a03911bdc49-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.265 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.265 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.266 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.266 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.266 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.266 DEBUG:boto:Method: GET 2024-04-12T03:46:15.266 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.266 DEBUG:boto:Data: 2024-04-12T03:46:15.266 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.266 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.266 DEBUG:boto:Params: {} 2024-04-12T03:46:15.267 DEBUG:boto:Token: None 2024-04-12T03:46:15.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f635dd97bbfaa49f-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.269 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.270 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.270 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.270 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.270 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:46:15.270 DEBUG:boto:Data: 2024-04-12T03:46:15.270 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.270 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.270 DEBUG:boto:Params: {} 2024-04-12T03:46:15.271 DEBUG:boto:Token: None 2024-04-12T03:46:15.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22 2024-04-12T03:46:15.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hQeu8kgTXToZrlFucullxPz6wbc= 2024-04-12T03:46:15.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hQeu8kgTXToZrlFucullxPz6wbc=', 'Content-Length': '0'} 2024-04-12T03:46:15.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008754b74cf84bb99d-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.273 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.273 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.273 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.273 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:46:15.273 DEBUG:boto:Data: 2024-04-12T03:46:15.273 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.273 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.273 DEBUG:boto:Params: {} 2024-04-12T03:46:15.274 DEBUG:boto:Token: None 2024-04-12T03:46:15.274 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22 2024-04-12T03:46:15.274 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hQeu8kgTXToZrlFucullxPz6wbc= 2024-04-12T03:46:15.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hQeu8kgTXToZrlFucullxPz6wbc=', 'Content-Length': '0'} 2024-04-12T03:46:15.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def4440af170c15c-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:46:15.276 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.276 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.276 DEBUG:boto:Method: GET 2024-04-12T03:46:15.276 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:46:15.276 DEBUG:boto:Data: 2024-04-12T03:46:15.276 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.276 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.277 DEBUG:boto:Params: {} 2024-04-12T03:46:15.277 DEBUG:boto:Token: None 2024-04-12T03:46:15.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22 2024-04-12T03:46:15.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2fNIaBNeuHTAXMhC30OIoGS31E= 2024-04-12T03:46:15.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2fNIaBNeuHTAXMhC30OIoGS31E=', 'Content-Length': '0'} 2024-04-12T03:46:15.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906b8104421af87b-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.279 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:46:15.279 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:46:15.279 DEBUG:boto:Method: GET 2024-04-12T03:46:15.279 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:46:15.279 DEBUG:boto:Data: 2024-04-12T03:46:15.279 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.279 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.279 DEBUG:boto:Params: {} 2024-04-12T03:46:15.279 DEBUG:boto:Token: None 2024-04-12T03:46:15.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj22 2024-04-12T03:46:15.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m2fNIaBNeuHTAXMhC30OIoGS31E= 2024-04-12T03:46:15.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2fNIaBNeuHTAXMhC30OIoGS31E=', 'Content-Length': '0'} 2024-04-12T03:46:15.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc00d909c751a8ec-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.282 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.282 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.282 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.282 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.282 DEBUG:boto:Method: GET 2024-04-12T03:46:15.282 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.282 DEBUG:boto:Data: 2024-04-12T03:46:15.282 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.283 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.283 DEBUG:boto:Params: {} 2024-04-12T03:46:15.283 DEBUG:boto:Token: None 2024-04-12T03:46:15.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b015e3de19f44dc-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.285 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.286 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.286 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.286 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.286 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.286 DEBUG:boto:Method: GET 2024-04-12T03:46:15.286 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:46:15.286 DEBUG:boto:Data: 2024-04-12T03:46:15.286 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.286 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.287 DEBUG:boto:Params: {} 2024-04-12T03:46:15.287 DEBUG:boto:Token: None 2024-04-12T03:46:15.287 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.287 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d678109170716c-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.290 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:46:15.291 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.291 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.291 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.291 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.291 DEBUG:boto:Method: GET 2024-04-12T03:46:15.291 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.291 DEBUG:boto:Data: 2024-04-12T03:46:15.291 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.292 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.292 DEBUG:boto:Params: {} 2024-04-12T03:46:15.292 DEBUG:boto:Token: None 2024-04-12T03:46:15.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ= 2024-04-12T03:46:15.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052708ac5eb8cca64-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.294 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.294 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.294 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.294 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.295 DEBUG:boto:Method: GET 2024-04-12T03:46:15.295 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.295 DEBUG:boto:Data: 2024-04-12T03:46:15.295 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.295 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.295 DEBUG:boto:Params: {} 2024-04-12T03:46:15.295 DEBUG:boto:Token: None 2024-04-12T03:46:15.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ= 2024-04-12T03:46:15.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6737861d40f2ac-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.297 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.297 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.297 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.298 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.298 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.298 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.298 DEBUG:boto:Data: 2024-04-12T03:46:15.298 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.298 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.298 DEBUG:boto:Params: {} 2024-04-12T03:46:15.298 DEBUG:boto:Token: None 2024-04-12T03:46:15.298 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.299 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE1uXCGKn5VRCtp602RN3UCX+DI= 2024-04-12T03:46:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE1uXCGKn5VRCtp602RN3UCX+DI=', 'Content-Length': '0'} 2024-04-12T03:46:15.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b665e5a8e099f4-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.300 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.300 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.300 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.301 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.301 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.301 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.301 DEBUG:boto:Data: 2024-04-12T03:46:15.301 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.301 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.301 DEBUG:boto:Params: {} 2024-04-12T03:46:15.301 DEBUG:boto:Token: None 2024-04-12T03:46:15.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.301 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE1uXCGKn5VRCtp602RN3UCX+DI= 2024-04-12T03:46:15.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE1uXCGKn5VRCtp602RN3UCX+DI=', 'Content-Length': '0'} 2024-04-12T03:46:15.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a27e89dbbc4d81-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:46:15.303 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.304 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.304 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.304 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.304 DEBUG:boto:Method: GET 2024-04-12T03:46:15.304 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.304 DEBUG:boto:Data: 2024-04-12T03:46:15.304 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.304 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.305 DEBUG:boto:Params: {} 2024-04-12T03:46:15.305 DEBUG:boto:Token: None 2024-04-12T03:46:15.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ= 2024-04-12T03:46:15.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fad0f75f6f1a550-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.307 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.307 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.307 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.307 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.307 DEBUG:boto:Method: GET 2024-04-12T03:46:15.307 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.307 DEBUG:boto:Data: 2024-04-12T03:46:15.307 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.308 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.308 DEBUG:boto:Params: {} 2024-04-12T03:46:15.308 DEBUG:boto:Token: None 2024-04-12T03:46:15.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:46:15.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ= 2024-04-12T03:46:15.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/z4F02GPeJktbiALs8Q2URu0rQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003837a1d9b7e813fa-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.310 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.310 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.310 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.310 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.310 DEBUG:boto:Method: GET 2024-04-12T03:46:15.310 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.311 DEBUG:boto:Data: 2024-04-12T03:46:15.311 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.311 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.311 DEBUG:boto:Params: {} 2024-04-12T03:46:15.311 DEBUG:boto:Token: None 2024-04-12T03:46:15.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e159b98f8de7cb44-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.354 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.355 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.355 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.355 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.355 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.355 DEBUG:boto:Method: GET 2024-04-12T03:46:15.356 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.356 DEBUG:boto:Data: 2024-04-12T03:46:15.357 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.357 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.357 DEBUG:boto:Params: {} 2024-04-12T03:46:15.358 DEBUG:boto:Token: None 2024-04-12T03:46:15.358 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.358 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f5bafd57b7b7607-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.361 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.361 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.361 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.362 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.362 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:46:15.362 DEBUG:boto:Data: 2024-04-12T03:46:15.362 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.362 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.362 DEBUG:boto:Params: {} 2024-04-12T03:46:15.362 DEBUG:boto:Token: None 2024-04-12T03:46:15.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33 2024-04-12T03:46:15.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WgBGhcIyGJV2bL3Z7V+xNAOnJ+8= 2024-04-12T03:46:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WgBGhcIyGJV2bL3Z7V+xNAOnJ+8=', 'Content-Length': '0'} 2024-04-12T03:46:15.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5a9b8aedcb862d-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.364 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.364 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.365 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.365 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:46:15.365 DEBUG:boto:Data: 2024-04-12T03:46:15.365 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.365 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.365 DEBUG:boto:Params: {} 2024-04-12T03:46:15.365 DEBUG:boto:Token: None 2024-04-12T03:46:15.365 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33 2024-04-12T03:46:15.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WgBGhcIyGJV2bL3Z7V+xNAOnJ+8= 2024-04-12T03:46:15.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WgBGhcIyGJV2bL3Z7V+xNAOnJ+8=', 'Content-Length': '0'} 2024-04-12T03:46:15.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac56ecb5cc440aed-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:46:15.368 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.368 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.368 DEBUG:boto:Method: GET 2024-04-12T03:46:15.368 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:46:15.368 DEBUG:boto:Data: 2024-04-12T03:46:15.368 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.368 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.368 DEBUG:boto:Params: {} 2024-04-12T03:46:15.368 DEBUG:boto:Token: None 2024-04-12T03:46:15.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33 2024-04-12T03:46:15.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KDIonmkqN1S2WBPp77VPGA2Pvco= 2024-04-12T03:46:15.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KDIonmkqN1S2WBPp77VPGA2Pvco=', 'Content-Length': '0'} 2024-04-12T03:46:15.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3cb50444307019-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.371 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:46:15.371 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:46:15.371 DEBUG:boto:Method: GET 2024-04-12T03:46:15.371 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:46:15.371 DEBUG:boto:Data: 2024-04-12T03:46:15.371 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.371 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.371 DEBUG:boto:Params: {} 2024-04-12T03:46:15.371 DEBUG:boto:Token: None 2024-04-12T03:46:15.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj33 2024-04-12T03:46:15.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KDIonmkqN1S2WBPp77VPGA2Pvco= 2024-04-12T03:46:15.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KDIonmkqN1S2WBPp77VPGA2Pvco=', 'Content-Length': '0'} 2024-04-12T03:46:15.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374fa129bc7dd02c-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.373 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.374 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.374 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.374 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.374 DEBUG:boto:Method: GET 2024-04-12T03:46:15.374 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.374 DEBUG:boto:Data: 2024-04-12T03:46:15.374 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.374 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.374 DEBUG:boto:Params: {} 2024-04-12T03:46:15.375 DEBUG:boto:Token: None 2024-04-12T03:46:15.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2f8b21cdf474b1-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.377 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.377 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.378 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.378 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.378 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.378 DEBUG:boto:Method: GET 2024-04-12T03:46:15.378 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:46:15.378 DEBUG:boto:Data: 2024-04-12T03:46:15.378 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.378 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.378 DEBUG:boto:Params: {} 2024-04-12T03:46:15.379 DEBUG:boto:Token: None 2024-04-12T03:46:15.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003e90163dd1096eb-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.381 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:46:15.382 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.382 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.382 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.382 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.382 DEBUG:boto:Method: GET 2024-04-12T03:46:15.382 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.382 DEBUG:boto:Data: 2024-04-12T03:46:15.383 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.383 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.383 DEBUG:boto:Params: {} 2024-04-12T03:46:15.383 DEBUG:boto:Token: None 2024-04-12T03:46:15.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38= 2024-04-12T03:46:15.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38=', 'Content-Length': '0'} 2024-04-12T03:46:15.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7c9d639692ca79-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.385 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.385 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.385 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.385 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.386 DEBUG:boto:Method: GET 2024-04-12T03:46:15.386 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.386 DEBUG:boto:Data: 2024-04-12T03:46:15.386 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.386 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.386 DEBUG:boto:Params: {} 2024-04-12T03:46:15.386 DEBUG:boto:Token: None 2024-04-12T03:46:15.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38= 2024-04-12T03:46:15.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38=', 'Content-Length': '0'} 2024-04-12T03:46:15.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748682b883c7f299-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.388 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.389 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.389 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.389 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.389 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.389 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.389 DEBUG:boto:Data: 2024-04-12T03:46:15.389 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.390 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.390 DEBUG:boto:Params: {} 2024-04-12T03:46:15.390 DEBUG:boto:Token: None 2024-04-12T03:46:15.390 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fv7pn5cHwuRZ/unk5q63gb30Q5I= 2024-04-12T03:46:15.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fv7pn5cHwuRZ/unk5q63gb30Q5I=', 'Content-Length': '0'} 2024-04-12T03:46:15.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683e172567cc9482-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.392 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.393 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.393 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.393 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.393 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.393 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.393 DEBUG:boto:Data: 2024-04-12T03:46:15.393 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.393 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.394 DEBUG:boto:Params: {} 2024-04-12T03:46:15.394 DEBUG:boto:Token: None 2024-04-12T03:46:15.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.394 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fv7pn5cHwuRZ/unk5q63gb30Q5I= 2024-04-12T03:46:15.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fv7pn5cHwuRZ/unk5q63gb30Q5I=', 'Content-Length': '0'} 2024-04-12T03:46:15.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b49eeba9889c53b-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:46:15.396 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.396 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.396 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.396 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.397 DEBUG:boto:Method: GET 2024-04-12T03:46:15.397 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.397 DEBUG:boto:Data: 2024-04-12T03:46:15.397 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.397 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.397 DEBUG:boto:Params: {} 2024-04-12T03:46:15.397 DEBUG:boto:Token: None 2024-04-12T03:46:15.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38= 2024-04-12T03:46:15.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38=', 'Content-Length': '0'} 2024-04-12T03:46:15.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f052561d9d2854-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.399 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.400 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.400 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.400 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.400 DEBUG:boto:Method: GET 2024-04-12T03:46:15.400 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.400 DEBUG:boto:Data: 2024-04-12T03:46:15.400 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.400 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.400 DEBUG:boto:Params: {} 2024-04-12T03:46:15.401 DEBUG:boto:Token: None 2024-04-12T03:46:15.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:46:15.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38= 2024-04-12T03:46:15.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rI6b+UbPryy9Yv5GVi8ebTbgF38=', 'Content-Length': '0'} 2024-04-12T03:46:15.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2261ae79ee4f83-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.403 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.403 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.403 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.403 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.403 DEBUG:boto:Method: GET 2024-04-12T03:46:15.404 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.404 DEBUG:boto:Data: 2024-04-12T03:46:15.404 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.404 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.404 DEBUG:boto:Params: {} 2024-04-12T03:46:15.404 DEBUG:boto:Token: None 2024-04-12T03:46:15.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782ad527a776d911-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.407 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.407 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.407 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.407 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.408 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.408 DEBUG:boto:Method: GET 2024-04-12T03:46:15.408 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.408 DEBUG:boto:Data: 2024-04-12T03:46:15.408 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.408 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.408 DEBUG:boto:Params: {} 2024-04-12T03:46:15.408 DEBUG:boto:Token: None 2024-04-12T03:46:15.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a316843baa2fa1d-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.411 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.412 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.412 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.412 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.412 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:46:15.412 DEBUG:boto:Data: 2024-04-12T03:46:15.412 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.412 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.412 DEBUG:boto:Params: {} 2024-04-12T03:46:15.412 DEBUG:boto:Token: None 2024-04-12T03:46:15.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34 2024-04-12T03:46:15.413 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsmVYmKixYII5Zfm2gFKg6a/uMM= 2024-04-12T03:46:15.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsmVYmKixYII5Zfm2gFKg6a/uMM=', 'Content-Length': '0'} 2024-04-12T03:46:15.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef0f71c181b0adc8-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.415 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.415 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.415 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.415 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:46:15.415 DEBUG:boto:Data: 2024-04-12T03:46:15.415 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.415 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.415 DEBUG:boto:Params: {} 2024-04-12T03:46:15.416 DEBUG:boto:Token: None 2024-04-12T03:46:15.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34 2024-04-12T03:46:15.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsmVYmKixYII5Zfm2gFKg6a/uMM= 2024-04-12T03:46:15.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsmVYmKixYII5Zfm2gFKg6a/uMM=', 'Content-Length': '0'} 2024-04-12T03:46:15.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006021a54981500e98-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:46:15.418 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.418 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.418 DEBUG:boto:Method: GET 2024-04-12T03:46:15.418 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:46:15.418 DEBUG:boto:Data: 2024-04-12T03:46:15.418 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.418 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.418 DEBUG:boto:Params: {} 2024-04-12T03:46:15.419 DEBUG:boto:Token: None 2024-04-12T03:46:15.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34 2024-04-12T03:46:15.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FcO2t4/o0oX2kuXd1IeFnvlL/+0= 2024-04-12T03:46:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FcO2t4/o0oX2kuXd1IeFnvlL/+0=', 'Content-Length': '0'} 2024-04-12T03:46:15.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d977f6ad4005eff-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.462 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:46:15.463 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:46:15.463 DEBUG:boto:Method: GET 2024-04-12T03:46:15.463 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:46:15.463 DEBUG:boto:Data: 2024-04-12T03:46:15.463 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.463 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.463 DEBUG:boto:Params: {} 2024-04-12T03:46:15.463 DEBUG:boto:Token: None 2024-04-12T03:46:15.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj34 2024-04-12T03:46:15.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FcO2t4/o0oX2kuXd1IeFnvlL/+0= 2024-04-12T03:46:15.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FcO2t4/o0oX2kuXd1IeFnvlL/+0=', 'Content-Length': '0'} 2024-04-12T03:46:15.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006853daf091480e46-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.466 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.466 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.466 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.466 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.466 DEBUG:boto:Method: GET 2024-04-12T03:46:15.466 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.467 DEBUG:boto:Data: 2024-04-12T03:46:15.467 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.467 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.467 DEBUG:boto:Params: {} 2024-04-12T03:46:15.467 DEBUG:boto:Token: None 2024-04-12T03:46:15.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070147d2861f9a26a-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.469 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.470 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.470 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.470 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.470 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.470 DEBUG:boto:Method: GET 2024-04-12T03:46:15.470 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:46:15.470 DEBUG:boto:Data: 2024-04-12T03:46:15.471 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.471 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.471 DEBUG:boto:Params: {} 2024-04-12T03:46:15.471 DEBUG:boto:Token: None 2024-04-12T03:46:15.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982878b6b17af184-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.474 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:46:15.474 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.475 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.475 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.475 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.475 DEBUG:boto:Method: GET 2024-04-12T03:46:15.475 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.475 DEBUG:boto:Data: 2024-04-12T03:46:15.475 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.475 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.475 DEBUG:boto:Params: {} 2024-04-12T03:46:15.475 DEBUG:boto:Token: None 2024-04-12T03:46:15.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo= 2024-04-12T03:46:15.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo=', 'Content-Length': '0'} 2024-04-12T03:46:15.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd8784b1c21c790-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.478 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.478 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.478 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.478 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.478 DEBUG:boto:Method: GET 2024-04-12T03:46:15.478 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.478 DEBUG:boto:Data: 2024-04-12T03:46:15.478 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.479 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.479 DEBUG:boto:Params: {} 2024-04-12T03:46:15.479 DEBUG:boto:Token: None 2024-04-12T03:46:15.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo= 2024-04-12T03:46:15.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo=', 'Content-Length': '0'} 2024-04-12T03:46:15.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22798dd42af530d-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.481 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.481 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.481 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.481 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.482 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.482 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.482 DEBUG:boto:Data: 2024-04-12T03:46:15.482 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.482 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.482 DEBUG:boto:Params: {} 2024-04-12T03:46:15.482 DEBUG:boto:Token: None 2024-04-12T03:46:15.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+iZg+WM6IU06qf40MwU4radoI0= 2024-04-12T03:46:15.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+iZg+WM6IU06qf40MwU4radoI0=', 'Content-Length': '0'} 2024-04-12T03:46:15.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227051e56e6081d0-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.484 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.485 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.485 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.485 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.485 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.485 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.485 DEBUG:boto:Data: 2024-04-12T03:46:15.485 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.485 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.485 DEBUG:boto:Params: {} 2024-04-12T03:46:15.486 DEBUG:boto:Token: None 2024-04-12T03:46:15.486 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.486 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z+iZg+WM6IU06qf40MwU4radoI0= 2024-04-12T03:46:15.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z+iZg+WM6IU06qf40MwU4radoI0=', 'Content-Length': '0'} 2024-04-12T03:46:15.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078359d2daeefbcd8-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:46:15.488 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.488 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.488 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.488 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.488 DEBUG:boto:Method: GET 2024-04-12T03:46:15.488 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.488 DEBUG:boto:Data: 2024-04-12T03:46:15.488 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.489 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.489 DEBUG:boto:Params: {} 2024-04-12T03:46:15.489 DEBUG:boto:Token: None 2024-04-12T03:46:15.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.489 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo= 2024-04-12T03:46:15.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo=', 'Content-Length': '0'} 2024-04-12T03:46:15.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa5430c1716e45a-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.491 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.491 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.491 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.491 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.492 DEBUG:boto:Method: GET 2024-04-12T03:46:15.492 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.492 DEBUG:boto:Data: 2024-04-12T03:46:15.492 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.492 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.492 DEBUG:boto:Params: {} 2024-04-12T03:46:15.492 DEBUG:boto:Token: None 2024-04-12T03:46:15.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:46:15.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo= 2024-04-12T03:46:15.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sGlaEGUryXRNagVoSXsYSptHobo=', 'Content-Length': '0'} 2024-04-12T03:46:15.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30d66419d35d848-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.495 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.495 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.495 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.495 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.495 DEBUG:boto:Method: GET 2024-04-12T03:46:15.495 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.495 DEBUG:boto:Data: 2024-04-12T03:46:15.496 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.496 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.496 DEBUG:boto:Params: {} 2024-04-12T03:46:15.496 DEBUG:boto:Token: None 2024-04-12T03:46:15.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2b6e0b1dde72d7-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.498 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.498 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.499 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.499 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.499 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.499 DEBUG:boto:Method: GET 2024-04-12T03:46:15.499 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.499 DEBUG:boto:Data: 2024-04-12T03:46:15.499 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.499 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.499 DEBUG:boto:Params: {} 2024-04-12T03:46:15.500 DEBUG:boto:Token: None 2024-04-12T03:46:15.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd5fbb413a93684-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.502 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.503 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.503 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.503 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.503 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:46:15.503 DEBUG:boto:Data: 2024-04-12T03:46:15.503 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.503 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.503 DEBUG:boto:Params: {} 2024-04-12T03:46:15.504 DEBUG:boto:Token: None 2024-04-12T03:46:15.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38 2024-04-12T03:46:15.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rY86RQzqDCHczolSLQZ9KG4W12s= 2024-04-12T03:46:15.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rY86RQzqDCHczolSLQZ9KG4W12s=', 'Content-Length': '0'} 2024-04-12T03:46:15.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc297b001e0f5d0-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.506 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.506 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.506 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.506 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:46:15.506 DEBUG:boto:Data: 2024-04-12T03:46:15.506 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.506 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.507 DEBUG:boto:Params: {} 2024-04-12T03:46:15.507 DEBUG:boto:Token: None 2024-04-12T03:46:15.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38 2024-04-12T03:46:15.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rY86RQzqDCHczolSLQZ9KG4W12s= 2024-04-12T03:46:15.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rY86RQzqDCHczolSLQZ9KG4W12s=', 'Content-Length': '0'} 2024-04-12T03:46:15.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ed1522e65c63b4-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:46:15.509 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.509 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.509 DEBUG:boto:Method: GET 2024-04-12T03:46:15.509 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:46:15.509 DEBUG:boto:Data: 2024-04-12T03:46:15.509 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.509 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.510 DEBUG:boto:Params: {} 2024-04-12T03:46:15.510 DEBUG:boto:Token: None 2024-04-12T03:46:15.510 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38 2024-04-12T03:46:15.510 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jl/OWAbBKYx3EONvPIKhLv1rVxs= 2024-04-12T03:46:15.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jl/OWAbBKYx3EONvPIKhLv1rVxs=', 'Content-Length': '0'} 2024-04-12T03:46:15.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f6d1d616b93c491-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.512 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:46:15.512 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:46:15.512 DEBUG:boto:Method: GET 2024-04-12T03:46:15.512 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:46:15.512 DEBUG:boto:Data: 2024-04-12T03:46:15.512 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.513 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.513 DEBUG:boto:Params: {} 2024-04-12T03:46:15.513 DEBUG:boto:Token: None 2024-04-12T03:46:15.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj38 2024-04-12T03:46:15.513 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jl/OWAbBKYx3EONvPIKhLv1rVxs= 2024-04-12T03:46:15.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jl/OWAbBKYx3EONvPIKhLv1rVxs=', 'Content-Length': '0'} 2024-04-12T03:46:15.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a2ad4d1fb52928-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.515 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.515 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.515 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.515 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.516 DEBUG:boto:Method: GET 2024-04-12T03:46:15.516 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.516 DEBUG:boto:Data: 2024-04-12T03:46:15.516 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.516 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.516 DEBUG:boto:Params: {} 2024-04-12T03:46:15.516 DEBUG:boto:Token: None 2024-04-12T03:46:15.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbce914f3e15216f-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.519 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.519 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.519 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.519 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.519 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.519 DEBUG:boto:Method: GET 2024-04-12T03:46:15.520 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:46:15.520 DEBUG:boto:Data: 2024-04-12T03:46:15.520 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.520 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.520 DEBUG:boto:Params: {} 2024-04-12T03:46:15.520 DEBUG:boto:Token: None 2024-04-12T03:46:15.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005763859f30269ed7-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.566 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:46:15.567 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.567 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.567 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.567 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.567 DEBUG:boto:Method: GET 2024-04-12T03:46:15.567 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.568 DEBUG:boto:Data: 2024-04-12T03:46:15.568 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.568 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.568 DEBUG:boto:Params: {} 2024-04-12T03:46:15.568 DEBUG:boto:Token: None 2024-04-12T03:46:15.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI= 2024-04-12T03:46:15.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI=', 'Content-Length': '0'} 2024-04-12T03:46:15.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0280f882b8a7b50-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.571 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.571 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.571 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.571 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.571 DEBUG:boto:Method: GET 2024-04-12T03:46:15.571 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.571 DEBUG:boto:Data: 2024-04-12T03:46:15.571 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.572 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.572 DEBUG:boto:Params: {} 2024-04-12T03:46:15.572 DEBUG:boto:Token: None 2024-04-12T03:46:15.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI= 2024-04-12T03:46:15.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI=', 'Content-Length': '0'} 2024-04-12T03:46:15.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303c81f167fcd915-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.574 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.574 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.574 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.575 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.575 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.575 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.575 DEBUG:boto:Data: 2024-04-12T03:46:15.575 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.575 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.575 DEBUG:boto:Params: {} 2024-04-12T03:46:15.575 DEBUG:boto:Token: None 2024-04-12T03:46:15.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OAcrstyV/QqsGGDblZhmc0akRB8= 2024-04-12T03:46:15.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OAcrstyV/QqsGGDblZhmc0akRB8=', 'Content-Length': '0'} 2024-04-12T03:46:15.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c930ee33b20c943e-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.577 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.577 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.577 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.578 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.578 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.578 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.578 DEBUG:boto:Data: 2024-04-12T03:46:15.578 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.578 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.578 DEBUG:boto:Params: {} 2024-04-12T03:46:15.578 DEBUG:boto:Token: None 2024-04-12T03:46:15.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OAcrstyV/QqsGGDblZhmc0akRB8= 2024-04-12T03:46:15.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OAcrstyV/QqsGGDblZhmc0akRB8=', 'Content-Length': '0'} 2024-04-12T03:46:15.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7879f42d3e7cb3-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:46:15.581 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.581 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.581 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.581 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.581 DEBUG:boto:Method: GET 2024-04-12T03:46:15.581 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.581 DEBUG:boto:Data: 2024-04-12T03:46:15.581 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.582 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.582 DEBUG:boto:Params: {} 2024-04-12T03:46:15.582 DEBUG:boto:Token: None 2024-04-12T03:46:15.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI= 2024-04-12T03:46:15.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI=', 'Content-Length': '0'} 2024-04-12T03:46:15.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e711ae5ad75874cf-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.630 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.631 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.631 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.631 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.631 DEBUG:boto:Method: GET 2024-04-12T03:46:15.631 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.631 DEBUG:boto:Data: 2024-04-12T03:46:15.631 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.631 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.631 DEBUG:boto:Params: {} 2024-04-12T03:46:15.632 DEBUG:boto:Token: None 2024-04-12T03:46:15.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:46:15.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI= 2024-04-12T03:46:15.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XOsbvBzis9f/Bx7Y3KW7G5EyAGI=', 'Content-Length': '0'} 2024-04-12T03:46:15.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006622f875eb41e51e-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.634 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.634 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.634 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.634 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.635 DEBUG:boto:Method: GET 2024-04-12T03:46:15.635 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.635 DEBUG:boto:Data: 2024-04-12T03:46:15.635 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.635 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.635 DEBUG:boto:Params: {} 2024-04-12T03:46:15.635 DEBUG:boto:Token: None 2024-04-12T03:46:15.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a4e6ae1bb5ce7f6-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.638 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.638 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.639 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.639 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.639 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.639 DEBUG:boto:Method: GET 2024-04-12T03:46:15.639 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.639 DEBUG:boto:Data: 2024-04-12T03:46:15.639 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.639 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.639 DEBUG:boto:Params: {} 2024-04-12T03:46:15.640 DEBUG:boto:Token: None 2024-04-12T03:46:15.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce6a45650c3e2269-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.644 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.644 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.644 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.644 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.644 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:46:15.644 DEBUG:boto:Data: 2024-04-12T03:46:15.645 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.645 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.645 DEBUG:boto:Params: {} 2024-04-12T03:46:15.645 DEBUG:boto:Token: None 2024-04-12T03:46:15.645 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44 2024-04-12T03:46:15.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0FIaN7ihIljLJlQ9T7r91Yl5HdA= 2024-04-12T03:46:15.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0FIaN7ihIljLJlQ9T7r91Yl5HdA=', 'Content-Length': '0'} 2024-04-12T03:46:15.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b51e2420c74f3d0-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.647 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.647 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.647 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.647 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:46:15.647 DEBUG:boto:Data: 2024-04-12T03:46:15.647 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.648 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.648 DEBUG:boto:Params: {} 2024-04-12T03:46:15.648 DEBUG:boto:Token: None 2024-04-12T03:46:15.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44 2024-04-12T03:46:15.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0FIaN7ihIljLJlQ9T7r91Yl5HdA= 2024-04-12T03:46:15.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0FIaN7ihIljLJlQ9T7r91Yl5HdA=', 'Content-Length': '0'} 2024-04-12T03:46:15.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741b6cd918a24110-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:46:15.650 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.650 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.651 DEBUG:boto:Method: GET 2024-04-12T03:46:15.651 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:46:15.651 DEBUG:boto:Data: 2024-04-12T03:46:15.651 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.651 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.651 DEBUG:boto:Params: {} 2024-04-12T03:46:15.651 DEBUG:boto:Token: None 2024-04-12T03:46:15.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44 2024-04-12T03:46:15.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPrYmQ1lNUFR6XnLoJSvBEdQ6pg= 2024-04-12T03:46:15.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPrYmQ1lNUFR6XnLoJSvBEdQ6pg=', 'Content-Length': '0'} 2024-04-12T03:46:15.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014018c9913654565-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.653 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:46:15.653 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:46:15.653 DEBUG:boto:Method: GET 2024-04-12T03:46:15.654 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:46:15.654 DEBUG:boto:Data: 2024-04-12T03:46:15.654 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.654 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.654 DEBUG:boto:Params: {} 2024-04-12T03:46:15.654 DEBUG:boto:Token: None 2024-04-12T03:46:15.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj44 2024-04-12T03:46:15.654 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPrYmQ1lNUFR6XnLoJSvBEdQ6pg= 2024-04-12T03:46:15.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPrYmQ1lNUFR6XnLoJSvBEdQ6pg=', 'Content-Length': '0'} 2024-04-12T03:46:15.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1fd91af14db5cef-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.656 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.656 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.657 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.657 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.657 DEBUG:boto:Method: GET 2024-04-12T03:46:15.657 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.657 DEBUG:boto:Data: 2024-04-12T03:46:15.657 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.657 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.657 DEBUG:boto:Params: {} 2024-04-12T03:46:15.657 DEBUG:boto:Token: None 2024-04-12T03:46:15.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3da26c13ef8c04-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.660 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.660 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.660 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.660 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.661 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.661 DEBUG:boto:Method: GET 2024-04-12T03:46:15.661 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:46:15.661 DEBUG:boto:Data: 2024-04-12T03:46:15.661 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.661 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.661 DEBUG:boto:Params: {} 2024-04-12T03:46:15.661 DEBUG:boto:Token: None 2024-04-12T03:46:15.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.662 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c2a3332dc6e062-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.664 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:46:15.665 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.665 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.665 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.665 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.665 DEBUG:boto:Method: GET 2024-04-12T03:46:15.665 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.665 DEBUG:boto:Data: 2024-04-12T03:46:15.665 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.665 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.665 DEBUG:boto:Params: {} 2024-04-12T03:46:15.666 DEBUG:boto:Token: None 2024-04-12T03:46:15.666 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.666 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw= 2024-04-12T03:46:15.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw=', 'Content-Length': '0'} 2024-04-12T03:46:15.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225a5683a862f30b-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.668 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.668 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.668 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.668 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.668 DEBUG:boto:Method: GET 2024-04-12T03:46:15.668 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.668 DEBUG:boto:Data: 2024-04-12T03:46:15.669 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.669 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.669 DEBUG:boto:Params: {} 2024-04-12T03:46:15.669 DEBUG:boto:Token: None 2024-04-12T03:46:15.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw= 2024-04-12T03:46:15.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw=', 'Content-Length': '0'} 2024-04-12T03:46:15.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7db7f31578b9087-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.671 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.671 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.672 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.672 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.672 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.672 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.672 DEBUG:boto:Data: 2024-04-12T03:46:15.672 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.672 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.672 DEBUG:boto:Params: {} 2024-04-12T03:46:15.672 DEBUG:boto:Token: None 2024-04-12T03:46:15.673 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.673 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hKFLd1N5qDtexY6mRVig1PjSn2s= 2024-04-12T03:46:15.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hKFLd1N5qDtexY6mRVig1PjSn2s=', 'Content-Length': '0'} 2024-04-12T03:46:15.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9c1b799b2f6e41-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.675 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.675 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.675 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.675 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.675 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.675 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.675 DEBUG:boto:Data: 2024-04-12T03:46:15.676 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.676 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.676 DEBUG:boto:Params: {} 2024-04-12T03:46:15.676 DEBUG:boto:Token: None 2024-04-12T03:46:15.676 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.676 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hKFLd1N5qDtexY6mRVig1PjSn2s= 2024-04-12T03:46:15.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hKFLd1N5qDtexY6mRVig1PjSn2s=', 'Content-Length': '0'} 2024-04-12T03:46:15.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609c32995b0c9690-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:46:15.679 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.679 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.679 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.679 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.679 DEBUG:boto:Method: GET 2024-04-12T03:46:15.680 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.680 DEBUG:boto:Data: 2024-04-12T03:46:15.680 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.680 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.680 DEBUG:boto:Params: {} 2024-04-12T03:46:15.681 DEBUG:boto:Token: None 2024-04-12T03:46:15.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw= 2024-04-12T03:46:15.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw=', 'Content-Length': '0'} 2024-04-12T03:46:15.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d858fd26136b777-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.726 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.727 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.727 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.727 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.727 DEBUG:boto:Method: GET 2024-04-12T03:46:15.727 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.727 DEBUG:boto:Data: 2024-04-12T03:46:15.727 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.727 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.727 DEBUG:boto:Params: {} 2024-04-12T03:46:15.728 DEBUG:boto:Token: None 2024-04-12T03:46:15.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:46:15.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw= 2024-04-12T03:46:15.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vp2qtt/7KzaVfNra6HYInmUvvhw=', 'Content-Length': '0'} 2024-04-12T03:46:15.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69cdc4dee17573b-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.730 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.730 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.730 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.730 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.730 DEBUG:boto:Method: GET 2024-04-12T03:46:15.731 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.731 DEBUG:boto:Data: 2024-04-12T03:46:15.731 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.731 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.731 DEBUG:boto:Params: {} 2024-04-12T03:46:15.731 DEBUG:boto:Token: None 2024-04-12T03:46:15.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b88dd3b5ec7c05e-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.734 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.734 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.734 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.734 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.734 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.735 DEBUG:boto:Method: GET 2024-04-12T03:46:15.735 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.735 DEBUG:boto:Data: 2024-04-12T03:46:15.735 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.735 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.735 DEBUG:boto:Params: {} 2024-04-12T03:46:15.735 DEBUG:boto:Token: None 2024-04-12T03:46:15.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007402d30a7ad11fc6-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.739 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.739 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.739 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.739 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.739 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:46:15.739 DEBUG:boto:Data: 2024-04-12T03:46:15.740 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.740 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.740 DEBUG:boto:Params: {} 2024-04-12T03:46:15.740 DEBUG:boto:Token: None 2024-04-12T03:46:15.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51 2024-04-12T03:46:15.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3La6klp9ER25GF4Sg+X67DMfzvE= 2024-04-12T03:46:15.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3La6klp9ER25GF4Sg+X67DMfzvE=', 'Content-Length': '0'} 2024-04-12T03:46:15.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c96373381d4226-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.742 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.742 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.742 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.742 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:46:15.743 DEBUG:boto:Data: 2024-04-12T03:46:15.743 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.743 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.743 DEBUG:boto:Params: {} 2024-04-12T03:46:15.743 DEBUG:boto:Token: None 2024-04-12T03:46:15.743 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51 2024-04-12T03:46:15.743 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3La6klp9ER25GF4Sg+X67DMfzvE= 2024-04-12T03:46:15.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3La6klp9ER25GF4Sg+X67DMfzvE=', 'Content-Length': '0'} 2024-04-12T03:46:15.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21e5c2505f15c2e-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:46:15.745 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.745 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.745 DEBUG:boto:Method: GET 2024-04-12T03:46:15.746 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:46:15.746 DEBUG:boto:Data: 2024-04-12T03:46:15.746 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.746 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.746 DEBUG:boto:Params: {} 2024-04-12T03:46:15.746 DEBUG:boto:Token: None 2024-04-12T03:46:15.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51 2024-04-12T03:46:15.746 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BmPROyut1J0FE2LWgJTUchfMYb4= 2024-04-12T03:46:15.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BmPROyut1J0FE2LWgJTUchfMYb4=', 'Content-Length': '0'} 2024-04-12T03:46:15.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d94a1d03bc87f4-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.748 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:46:15.748 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:46:15.748 DEBUG:boto:Method: GET 2024-04-12T03:46:15.748 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:46:15.749 DEBUG:boto:Data: 2024-04-12T03:46:15.749 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.749 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.749 DEBUG:boto:Params: {} 2024-04-12T03:46:15.749 DEBUG:boto:Token: None 2024-04-12T03:46:15.749 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj51 2024-04-12T03:46:15.749 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BmPROyut1J0FE2LWgJTUchfMYb4= 2024-04-12T03:46:15.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BmPROyut1J0FE2LWgJTUchfMYb4=', 'Content-Length': '0'} 2024-04-12T03:46:15.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adbda00941b79d00-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.751 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.751 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.751 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.752 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.752 DEBUG:boto:Method: GET 2024-04-12T03:46:15.752 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.752 DEBUG:boto:Data: 2024-04-12T03:46:15.752 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.752 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.752 DEBUG:boto:Params: {} 2024-04-12T03:46:15.752 DEBUG:boto:Token: None 2024-04-12T03:46:15.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.753 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bead5b44d3d381b8-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.755 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.755 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.755 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.755 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.755 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.756 DEBUG:boto:Method: GET 2024-04-12T03:46:15.756 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:46:15.756 DEBUG:boto:Data: 2024-04-12T03:46:15.756 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.756 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.756 DEBUG:boto:Params: {} 2024-04-12T03:46:15.756 DEBUG:boto:Token: None 2024-04-12T03:46:15.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a480702d46aa79a-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.759 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:46:15.759 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.760 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.760 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.760 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.760 DEBUG:boto:Method: GET 2024-04-12T03:46:15.760 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.760 DEBUG:boto:Data: 2024-04-12T03:46:15.760 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.760 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.760 DEBUG:boto:Params: {} 2024-04-12T03:46:15.761 DEBUG:boto:Token: None 2024-04-12T03:46:15.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU= 2024-04-12T03:46:15.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU=', 'Content-Length': '0'} 2024-04-12T03:46:15.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000625898a5c3ba7d75-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.763 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.763 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.763 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.763 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.763 DEBUG:boto:Method: GET 2024-04-12T03:46:15.763 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.763 DEBUG:boto:Data: 2024-04-12T03:46:15.763 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.763 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.764 DEBUG:boto:Params: {} 2024-04-12T03:46:15.765 DEBUG:boto:Token: None 2024-04-12T03:46:15.765 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU= 2024-04-12T03:46:15.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU=', 'Content-Length': '0'} 2024-04-12T03:46:15.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a76c3340c38b7e2-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.814 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.814 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.815 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.815 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.815 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.815 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.815 DEBUG:boto:Data: 2024-04-12T03:46:15.815 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.815 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.815 DEBUG:boto:Params: {} 2024-04-12T03:46:15.815 DEBUG:boto:Token: None 2024-04-12T03:46:15.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UmTKee+QFp/77R5BuydgoHeeSuk= 2024-04-12T03:46:15.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UmTKee+QFp/77R5BuydgoHeeSuk=', 'Content-Length': '0'} 2024-04-12T03:46:15.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f0e4aa36a54f7eb-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.818 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.818 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.818 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.818 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.818 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.818 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.818 DEBUG:boto:Data: 2024-04-12T03:46:15.818 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.818 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.818 DEBUG:boto:Params: {} 2024-04-12T03:46:15.819 DEBUG:boto:Token: None 2024-04-12T03:46:15.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UmTKee+QFp/77R5BuydgoHeeSuk= 2024-04-12T03:46:15.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UmTKee+QFp/77R5BuydgoHeeSuk=', 'Content-Length': '0'} 2024-04-12T03:46:15.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0d5dd768e533d9-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:46:15.821 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.821 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.821 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.821 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.821 DEBUG:boto:Method: GET 2024-04-12T03:46:15.821 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.821 DEBUG:boto:Data: 2024-04-12T03:46:15.821 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.821 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.821 DEBUG:boto:Params: {} 2024-04-12T03:46:15.822 DEBUG:boto:Token: None 2024-04-12T03:46:15.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU= 2024-04-12T03:46:15.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU=', 'Content-Length': '0'} 2024-04-12T03:46:15.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015542e0f3de79682-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.824 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.824 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.824 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.824 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.824 DEBUG:boto:Method: GET 2024-04-12T03:46:15.824 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.824 DEBUG:boto:Data: 2024-04-12T03:46:15.824 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.824 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.824 DEBUG:boto:Params: {} 2024-04-12T03:46:15.825 DEBUG:boto:Token: None 2024-04-12T03:46:15.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:46:15.825 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU= 2024-04-12T03:46:15.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EKdPUvYpC8wIP7Z/8A7gg5oPrCU=', 'Content-Length': '0'} 2024-04-12T03:46:15.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33a4c5423ecd7e7-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.827 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.827 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.827 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.827 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.827 DEBUG:boto:Method: GET 2024-04-12T03:46:15.827 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.827 DEBUG:boto:Data: 2024-04-12T03:46:15.827 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.827 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.828 DEBUG:boto:Params: {} 2024-04-12T03:46:15.828 DEBUG:boto:Token: None 2024-04-12T03:46:15.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af0ae0f92a4ee52-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.830 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.831 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.831 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.831 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.831 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.831 DEBUG:boto:Method: GET 2024-04-12T03:46:15.831 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.831 DEBUG:boto:Data: 2024-04-12T03:46:15.831 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.831 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.832 DEBUG:boto:Params: {} 2024-04-12T03:46:15.832 DEBUG:boto:Token: None 2024-04-12T03:46:15.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008893282b4819f50f-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.835 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.835 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.835 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.835 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.836 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:46:15.836 DEBUG:boto:Data: 2024-04-12T03:46:15.836 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.836 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.836 DEBUG:boto:Params: {} 2024-04-12T03:46:15.836 DEBUG:boto:Token: None 2024-04-12T03:46:15.837 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55 2024-04-12T03:46:15.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dr3ESH7oBCWitChXKjnmWKzd1Pg= 2024-04-12T03:46:15.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dr3ESH7oBCWitChXKjnmWKzd1Pg=', 'Content-Length': '0'} 2024-04-12T03:46:15.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc186b847e1743a3-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.838 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.839 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.839 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.839 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:46:15.839 DEBUG:boto:Data: 2024-04-12T03:46:15.839 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.839 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.839 DEBUG:boto:Params: {} 2024-04-12T03:46:15.839 DEBUG:boto:Token: None 2024-04-12T03:46:15.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55 2024-04-12T03:46:15.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dr3ESH7oBCWitChXKjnmWKzd1Pg= 2024-04-12T03:46:15.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dr3ESH7oBCWitChXKjnmWKzd1Pg=', 'Content-Length': '0'} 2024-04-12T03:46:15.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec390ec7fb126240-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:46:15.842 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.842 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.842 DEBUG:boto:Method: GET 2024-04-12T03:46:15.842 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:46:15.842 DEBUG:boto:Data: 2024-04-12T03:46:15.842 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.843 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.843 DEBUG:boto:Params: {} 2024-04-12T03:46:15.843 DEBUG:boto:Token: None 2024-04-12T03:46:15.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55 2024-04-12T03:46:15.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M2lYAC1X+TLqWgqojpguMPEMQ88= 2024-04-12T03:46:15.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M2lYAC1X+TLqWgqojpguMPEMQ88=', 'Content-Length': '0'} 2024-04-12T03:46:15.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6845433d996300-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.886 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:46:15.887 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:46:15.887 DEBUG:boto:Method: GET 2024-04-12T03:46:15.887 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:46:15.887 DEBUG:boto:Data: 2024-04-12T03:46:15.887 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.887 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.887 DEBUG:boto:Params: {} 2024-04-12T03:46:15.887 DEBUG:boto:Token: None 2024-04-12T03:46:15.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj55 2024-04-12T03:46:15.888 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M2lYAC1X+TLqWgqojpguMPEMQ88= 2024-04-12T03:46:15.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M2lYAC1X+TLqWgqojpguMPEMQ88=', 'Content-Length': '0'} 2024-04-12T03:46:15.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f4d0afcd8e4f57a-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.890 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.890 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.890 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.891 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.891 DEBUG:boto:Method: GET 2024-04-12T03:46:15.891 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.891 DEBUG:boto:Data: 2024-04-12T03:46:15.891 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.891 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.891 DEBUG:boto:Params: {} 2024-04-12T03:46:15.891 DEBUG:boto:Token: None 2024-04-12T03:46:15.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9cf90b90d11dff8-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.894 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.895 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.895 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.895 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.895 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.895 DEBUG:boto:Method: GET 2024-04-12T03:46:15.895 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:46:15.895 DEBUG:boto:Data: 2024-04-12T03:46:15.896 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.896 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.896 DEBUG:boto:Params: {} 2024-04-12T03:46:15.896 DEBUG:boto:Token: None 2024-04-12T03:46:15.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643ff14532f3bc04-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.899 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:46:15.900 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.900 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.900 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.900 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.900 DEBUG:boto:Method: GET 2024-04-12T03:46:15.900 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.901 DEBUG:boto:Data: 2024-04-12T03:46:15.901 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.901 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.902 DEBUG:boto:Params: {} 2024-04-12T03:46:15.902 DEBUG:boto:Token: None 2024-04-12T03:46:15.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac= 2024-04-12T03:46:15.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac=', 'Content-Length': '0'} 2024-04-12T03:46:15.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b070796a8eb367e-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.904 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.904 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.904 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.904 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.904 DEBUG:boto:Method: GET 2024-04-12T03:46:15.905 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.905 DEBUG:boto:Data: 2024-04-12T03:46:15.905 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.905 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.905 DEBUG:boto:Params: {} 2024-04-12T03:46:15.905 DEBUG:boto:Token: None 2024-04-12T03:46:15.905 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.905 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac= 2024-04-12T03:46:15.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac=', 'Content-Length': '0'} 2024-04-12T03:46:15.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975297a8817d76f7-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.907 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.907 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.907 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.908 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.908 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.908 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.908 DEBUG:boto:Data: 2024-04-12T03:46:15.908 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.908 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.908 DEBUG:boto:Params: {} 2024-04-12T03:46:15.908 DEBUG:boto:Token: None 2024-04-12T03:46:15.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wRPOAAiWyeYhLIA1zWxS0SfJbXo= 2024-04-12T03:46:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wRPOAAiWyeYhLIA1zWxS0SfJbXo=', 'Content-Length': '0'} 2024-04-12T03:46:15.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2a29129737b8c1-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.910 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.910 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.910 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.910 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.911 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.911 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.911 DEBUG:boto:Data: 2024-04-12T03:46:15.911 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.911 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.911 DEBUG:boto:Params: {} 2024-04-12T03:46:15.911 DEBUG:boto:Token: None 2024-04-12T03:46:15.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wRPOAAiWyeYhLIA1zWxS0SfJbXo= 2024-04-12T03:46:15.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wRPOAAiWyeYhLIA1zWxS0SfJbXo=', 'Content-Length': '0'} 2024-04-12T03:46:15.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba86a498d15b722-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:46:15.914 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.914 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.914 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.914 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.914 DEBUG:boto:Method: GET 2024-04-12T03:46:15.914 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.914 DEBUG:boto:Data: 2024-04-12T03:46:15.915 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.915 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.915 DEBUG:boto:Params: {} 2024-04-12T03:46:15.915 DEBUG:boto:Token: None 2024-04-12T03:46:15.915 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac= 2024-04-12T03:46:15.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac=', 'Content-Length': '0'} 2024-04-12T03:46:15.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4cd89b650e4a6d7-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.917 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.918 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.918 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.918 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.918 DEBUG:boto:Method: GET 2024-04-12T03:46:15.918 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.918 DEBUG:boto:Data: 2024-04-12T03:46:15.918 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.918 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.918 DEBUG:boto:Params: {} 2024-04-12T03:46:15.919 DEBUG:boto:Token: None 2024-04-12T03:46:15.919 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:46:15.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac= 2024-04-12T03:46:15.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0La7wPxlRB0PBU0S37WTlVS8zac=', 'Content-Length': '0'} 2024-04-12T03:46:15.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10c802df3380ea0-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.921 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.921 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.921 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.921 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.921 DEBUG:boto:Method: GET 2024-04-12T03:46:15.922 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.922 DEBUG:boto:Data: 2024-04-12T03:46:15.922 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.922 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.922 DEBUG:boto:Params: {} 2024-04-12T03:46:15.922 DEBUG:boto:Token: None 2024-04-12T03:46:15.922 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfe86776064d417-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.925 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.925 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.925 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.925 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.925 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.925 DEBUG:boto:Method: GET 2024-04-12T03:46:15.926 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.926 DEBUG:boto:Data: 2024-04-12T03:46:15.926 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.926 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.926 DEBUG:boto:Params: {} 2024-04-12T03:46:15.926 DEBUG:boto:Token: None 2024-04-12T03:46:15.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.926 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12c12a6844bfeac-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.929 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.929 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.929 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.930 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.930 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:46:15.930 DEBUG:boto:Data: 2024-04-12T03:46:15.930 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.930 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.930 DEBUG:boto:Params: {} 2024-04-12T03:46:15.930 DEBUG:boto:Token: None 2024-04-12T03:46:15.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57 2024-04-12T03:46:15.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fkhvyGrr9ihXLZq0An8xgbhKAT0= 2024-04-12T03:46:15.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fkhvyGrr9ihXLZq0An8xgbhKAT0=', 'Content-Length': '0'} 2024-04-12T03:46:15.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041046fb390726dac-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.932 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.932 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.932 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.932 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:46:15.932 DEBUG:boto:Data: 2024-04-12T03:46:15.933 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.933 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.933 DEBUG:boto:Params: {} 2024-04-12T03:46:15.933 DEBUG:boto:Token: None 2024-04-12T03:46:15.933 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57 2024-04-12T03:46:15.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fkhvyGrr9ihXLZq0An8xgbhKAT0= 2024-04-12T03:46:15.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fkhvyGrr9ihXLZq0An8xgbhKAT0=', 'Content-Length': '0'} 2024-04-12T03:46:15.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007464f24ecd2e3778-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:46:15.935 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.935 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.935 DEBUG:boto:Method: GET 2024-04-12T03:46:15.936 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:46:15.936 DEBUG:boto:Data: 2024-04-12T03:46:15.936 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.936 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.936 DEBUG:boto:Params: {} 2024-04-12T03:46:15.936 DEBUG:boto:Token: None 2024-04-12T03:46:15.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57 2024-04-12T03:46:15.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:texdPoVLTk0NlOm2DaRZrS+sKbQ= 2024-04-12T03:46:15.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:texdPoVLTk0NlOm2DaRZrS+sKbQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c172a7ac16f4d2a2-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.938 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:46:15.938 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:46:15.938 DEBUG:boto:Method: GET 2024-04-12T03:46:15.939 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:46:15.939 DEBUG:boto:Data: 2024-04-12T03:46:15.939 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.939 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.939 DEBUG:boto:Params: {} 2024-04-12T03:46:15.939 DEBUG:boto:Token: None 2024-04-12T03:46:15.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj57 2024-04-12T03:46:15.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:texdPoVLTk0NlOm2DaRZrS+sKbQ= 2024-04-12T03:46:15.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:texdPoVLTk0NlOm2DaRZrS+sKbQ=', 'Content-Length': '0'} 2024-04-12T03:46:15.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005eebb7ae0f228a4-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.982 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.983 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.983 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.983 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.983 DEBUG:boto:Method: GET 2024-04-12T03:46:15.983 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.983 DEBUG:boto:Data: 2024-04-12T03:46:15.983 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.983 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.983 DEBUG:boto:Params: {} 2024-04-12T03:46:15.984 DEBUG:boto:Token: None 2024-04-12T03:46:15.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080362387c3d6eab8-006618ae87-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.986 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.987 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:15.987 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:15.987 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.987 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.987 DEBUG:boto:Method: GET 2024-04-12T03:46:15.987 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:46:15.988 DEBUG:boto:Data: 2024-04-12T03:46:15.988 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.988 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.988 DEBUG:boto:Params: {} 2024-04-12T03:46:15.988 DEBUG:boto:Token: None 2024-04-12T03:46:15.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/ 2024-04-12T03:46:15.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY= 2024-04-12T03:46:15.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSC9nXB/KUa0PNA/AJ7AFye6xbY=', 'Content-Length': '0'} 2024-04-12T03:46:15.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b694a91ed99fa9f-006618ae87-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.991 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:15.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:46:15.991 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:15.991 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:15.991 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.992 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.992 DEBUG:boto:Method: GET 2024-04-12T03:46:15.992 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.992 DEBUG:boto:Data: 2024-04-12T03:46:15.992 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.992 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.992 DEBUG:boto:Params: {} 2024-04-12T03:46:15.992 DEBUG:boto:Token: None 2024-04-12T03:46:15.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s10/HGTgTZm8Nr+JVAFq1oCAcy0= 2024-04-12T03:46:15.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s10/HGTgTZm8Nr+JVAFq1oCAcy0=', 'Content-Length': '0'} 2024-04-12T03:46:15.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006318c5e2526d8737-006618ae87-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.995 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:15.995 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:15.995 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.995 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.995 DEBUG:boto:Method: GET 2024-04-12T03:46:15.995 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.995 DEBUG:boto:Data: 2024-04-12T03:46:15.995 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:15.995 DEBUG:boto:Port: 8000 2024-04-12T03:46:15.996 DEBUG:boto:Params: {} 2024-04-12T03:46:15.996 DEBUG:boto:Token: None 2024-04-12T03:46:15.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s10/HGTgTZm8Nr+JVAFq1oCAcy0= 2024-04-12T03:46:15.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s10/HGTgTZm8Nr+JVAFq1oCAcy0=', 'Content-Length': '0'} 2024-04-12T03:46:15.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001160c35509ed490b-006618ae87-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:15.998 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:15.998 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:15.998 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.998 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.998 DEBUG:boto:Method: HEAD 2024-04-12T03:46:15.998 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:15.998 DEBUG:boto:Data: 2024-04-12T03:46:15.999 DEBUG:boto:Headers: {} 2024-04-12T03:46:15.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:15.999 DEBUG:boto:Port: 8001 2024-04-12T03:46:15.999 DEBUG:boto:Params: {} 2024-04-12T03:46:15.999 DEBUG:boto:Token: None 2024-04-12T03:46:15.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:15 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cSJWitPf0IhJqFV7CpfPExkBdUE= 2024-04-12T03:46:16.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cSJWitPf0IhJqFV7CpfPExkBdUE=', 'Content-Length': '0'} 2024-04-12T03:46:16.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008808165a75c91cbc-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.001 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.002 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.002 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.002 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.002 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.002 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.002 DEBUG:boto:Data: 2024-04-12T03:46:16.002 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.002 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.003 DEBUG:boto:Params: {} 2024-04-12T03:46:16.003 DEBUG:boto:Token: None 2024-04-12T03:46:16.003 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bv5hAXeDwFRx4fC8BUTU5NSy4jM= 2024-04-12T03:46:16.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bv5hAXeDwFRx4fC8BUTU5NSy4jM=', 'Content-Length': '0'} 2024-04-12T03:46:16.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e68c026649daca8-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:46:16.005 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.005 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.005 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.005 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.005 DEBUG:boto:Method: GET 2024-04-12T03:46:16.005 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.006 DEBUG:boto:Data: 2024-04-12T03:46:16.006 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.006 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.006 DEBUG:boto:Params: {} 2024-04-12T03:46:16.006 DEBUG:boto:Token: None 2024-04-12T03:46:16.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ke94bkig9dEB3h9snydi1Jv6HwM= 2024-04-12T03:46:16.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ke94bkig9dEB3h9snydi1Jv6HwM=', 'Content-Length': '0'} 2024-04-12T03:46:16.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf5209e8c8cbe9c9-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.054 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.055 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.055 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.055 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.055 DEBUG:boto:Method: GET 2024-04-12T03:46:16.055 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.055 DEBUG:boto:Data: 2024-04-12T03:46:16.055 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.055 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.056 DEBUG:boto:Params: {} 2024-04-12T03:46:16.056 DEBUG:boto:Token: None 2024-04-12T03:46:16.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:46:16.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ke94bkig9dEB3h9snydi1Jv6HwM= 2024-04-12T03:46:16.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ke94bkig9dEB3h9snydi1Jv6HwM=', 'Content-Length': '0'} 2024-04-12T03:46:16.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ecd2d5785839518-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.058 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.058 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.058 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.059 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.059 DEBUG:boto:Method: GET 2024-04-12T03:46:16.059 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.059 DEBUG:boto:Data: 2024-04-12T03:46:16.059 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.059 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.059 DEBUG:boto:Params: {} 2024-04-12T03:46:16.059 DEBUG:boto:Token: None 2024-04-12T03:46:16.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477a8d02df965a22-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.062 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.062 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.062 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.062 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.062 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.063 DEBUG:boto:Method: GET 2024-04-12T03:46:16.063 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.063 DEBUG:boto:Data: 2024-04-12T03:46:16.063 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.063 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.063 DEBUG:boto:Params: {} 2024-04-12T03:46:16.063 DEBUG:boto:Token: None 2024-04-12T03:46:16.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b287f5eb2e271b3e-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.067 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.067 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.067 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.067 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.067 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:46:16.067 DEBUG:boto:Data: 2024-04-12T03:46:16.067 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.068 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.068 DEBUG:boto:Params: {} 2024-04-12T03:46:16.068 DEBUG:boto:Token: None 2024-04-12T03:46:16.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68 2024-04-12T03:46:16.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iL7RMquDO9F2x8CXbWPx9rC9jeI= 2024-04-12T03:46:16.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iL7RMquDO9F2x8CXbWPx9rC9jeI=', 'Content-Length': '0'} 2024-04-12T03:46:16.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280c95a2843614d9-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.070 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.070 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.070 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.070 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:46:16.070 DEBUG:boto:Data: 2024-04-12T03:46:16.070 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.071 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.071 DEBUG:boto:Params: {} 2024-04-12T03:46:16.071 DEBUG:boto:Token: None 2024-04-12T03:46:16.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68 2024-04-12T03:46:16.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iL7RMquDO9F2x8CXbWPx9rC9jeI= 2024-04-12T03:46:16.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iL7RMquDO9F2x8CXbWPx9rC9jeI=', 'Content-Length': '0'} 2024-04-12T03:46:16.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b551eba7d547ddbd-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:46:16.073 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.073 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.074 DEBUG:boto:Method: GET 2024-04-12T03:46:16.074 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:46:16.074 DEBUG:boto:Data: 2024-04-12T03:46:16.074 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.074 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.074 DEBUG:boto:Params: {} 2024-04-12T03:46:16.074 DEBUG:boto:Token: None 2024-04-12T03:46:16.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68 2024-04-12T03:46:16.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xILk1L/HfoKTrFEooL36EgnXAOE= 2024-04-12T03:46:16.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xILk1L/HfoKTrFEooL36EgnXAOE=', 'Content-Length': '0'} 2024-04-12T03:46:16.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003472292d1412a719-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.076 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:46:16.077 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:46:16.077 DEBUG:boto:Method: GET 2024-04-12T03:46:16.077 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:46:16.077 DEBUG:boto:Data: 2024-04-12T03:46:16.077 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.077 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.077 DEBUG:boto:Params: {} 2024-04-12T03:46:16.077 DEBUG:boto:Token: None 2024-04-12T03:46:16.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj68 2024-04-12T03:46:16.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xILk1L/HfoKTrFEooL36EgnXAOE= 2024-04-12T03:46:16.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xILk1L/HfoKTrFEooL36EgnXAOE=', 'Content-Length': '0'} 2024-04-12T03:46:16.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d508675ee156e977-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.122 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.123 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.123 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.123 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.123 DEBUG:boto:Method: GET 2024-04-12T03:46:16.123 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.123 DEBUG:boto:Data: 2024-04-12T03:46:16.123 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.123 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.123 DEBUG:boto:Params: {} 2024-04-12T03:46:16.124 DEBUG:boto:Token: None 2024-04-12T03:46:16.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292e04f2f3d674c7-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.126 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.127 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.127 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.127 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.127 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.127 DEBUG:boto:Method: GET 2024-04-12T03:46:16.127 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:46:16.127 DEBUG:boto:Data: 2024-04-12T03:46:16.127 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.127 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.127 DEBUG:boto:Params: {} 2024-04-12T03:46:16.128 DEBUG:boto:Token: None 2024-04-12T03:46:16.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7582a979d545e2-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.131 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:46:16.131 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.131 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.131 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.131 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.132 DEBUG:boto:Method: GET 2024-04-12T03:46:16.132 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.132 DEBUG:boto:Data: 2024-04-12T03:46:16.132 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.132 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.132 DEBUG:boto:Params: {} 2024-04-12T03:46:16.132 DEBUG:boto:Token: None 2024-04-12T03:46:16.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20= 2024-04-12T03:46:16.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20=', 'Content-Length': '0'} 2024-04-12T03:46:16.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e5ecaefc965ce2-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.135 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.135 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.135 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.135 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.135 DEBUG:boto:Method: GET 2024-04-12T03:46:16.135 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.135 DEBUG:boto:Data: 2024-04-12T03:46:16.135 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.135 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.136 DEBUG:boto:Params: {} 2024-04-12T03:46:16.136 DEBUG:boto:Token: None 2024-04-12T03:46:16.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20= 2024-04-12T03:46:16.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20=', 'Content-Length': '0'} 2024-04-12T03:46:16.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1522466ea624fe9-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.138 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.138 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.138 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.138 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.138 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.138 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.138 DEBUG:boto:Data: 2024-04-12T03:46:16.139 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.139 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.139 DEBUG:boto:Params: {} 2024-04-12T03:46:16.139 DEBUG:boto:Token: None 2024-04-12T03:46:16.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XI1bPG7qHE/g55OpeMSYiRl+WcE= 2024-04-12T03:46:16.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XI1bPG7qHE/g55OpeMSYiRl+WcE=', 'Content-Length': '0'} 2024-04-12T03:46:16.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f83101d2d5a8da-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.141 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.141 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.141 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.141 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.141 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.141 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.142 DEBUG:boto:Data: 2024-04-12T03:46:16.142 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.142 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.142 DEBUG:boto:Params: {} 2024-04-12T03:46:16.142 DEBUG:boto:Token: None 2024-04-12T03:46:16.142 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.142 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XI1bPG7qHE/g55OpeMSYiRl+WcE= 2024-04-12T03:46:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XI1bPG7qHE/g55OpeMSYiRl+WcE=', 'Content-Length': '0'} 2024-04-12T03:46:16.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db7c5832133cfd5-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:46:16.144 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.144 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.144 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.145 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.145 DEBUG:boto:Method: GET 2024-04-12T03:46:16.145 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.145 DEBUG:boto:Data: 2024-04-12T03:46:16.145 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.145 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.145 DEBUG:boto:Params: {} 2024-04-12T03:46:16.145 DEBUG:boto:Token: None 2024-04-12T03:46:16.145 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20= 2024-04-12T03:46:16.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20=', 'Content-Length': '0'} 2024-04-12T03:46:16.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d22072f35836fb4-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.147 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.148 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.148 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.148 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.148 DEBUG:boto:Method: GET 2024-04-12T03:46:16.148 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.148 DEBUG:boto:Data: 2024-04-12T03:46:16.148 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.148 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.148 DEBUG:boto:Params: {} 2024-04-12T03:46:16.149 DEBUG:boto:Token: None 2024-04-12T03:46:16.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:46:16.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20= 2024-04-12T03:46:16.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fXosD20o6WQquR6KawhrZoZDl20=', 'Content-Length': '0'} 2024-04-12T03:46:16.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d4c14642685291-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.151 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.151 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.151 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.151 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.151 DEBUG:boto:Method: GET 2024-04-12T03:46:16.151 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.151 DEBUG:boto:Data: 2024-04-12T03:46:16.152 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.152 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.152 DEBUG:boto:Params: {} 2024-04-12T03:46:16.152 DEBUG:boto:Token: None 2024-04-12T03:46:16.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe8dc6f42202c7b-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.154 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.155 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.155 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.155 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.155 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.155 DEBUG:boto:Method: GET 2024-04-12T03:46:16.155 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.155 DEBUG:boto:Data: 2024-04-12T03:46:16.155 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.155 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.156 DEBUG:boto:Params: {} 2024-04-12T03:46:16.156 DEBUG:boto:Token: None 2024-04-12T03:46:16.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a70ea200c72bc14-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.202 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.203 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.203 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.204 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.204 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:46:16.204 DEBUG:boto:Data: 2024-04-12T03:46:16.204 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.204 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.204 DEBUG:boto:Params: {} 2024-04-12T03:46:16.204 DEBUG:boto:Token: None 2024-04-12T03:46:16.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76 2024-04-12T03:46:16.205 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWuUdYuSr9MYxQauz+FdW4kLhT8= 2024-04-12T03:46:16.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWuUdYuSr9MYxQauz+FdW4kLhT8=', 'Content-Length': '0'} 2024-04-12T03:46:16.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d978640978071cb-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.207 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.207 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.207 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.207 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:46:16.207 DEBUG:boto:Data: 2024-04-12T03:46:16.207 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.207 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.207 DEBUG:boto:Params: {} 2024-04-12T03:46:16.207 DEBUG:boto:Token: None 2024-04-12T03:46:16.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76 2024-04-12T03:46:16.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bWuUdYuSr9MYxQauz+FdW4kLhT8= 2024-04-12T03:46:16.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bWuUdYuSr9MYxQauz+FdW4kLhT8=', 'Content-Length': '0'} 2024-04-12T03:46:16.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000561292e4628b876-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:46:16.210 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.210 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.210 DEBUG:boto:Method: GET 2024-04-12T03:46:16.210 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:46:16.210 DEBUG:boto:Data: 2024-04-12T03:46:16.210 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.211 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.211 DEBUG:boto:Params: {} 2024-04-12T03:46:16.211 DEBUG:boto:Token: None 2024-04-12T03:46:16.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76 2024-04-12T03:46:16.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJ6xyYEw50ttE6K0CHiZGb/WsCo= 2024-04-12T03:46:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJ6xyYEw50ttE6K0CHiZGb/WsCo=', 'Content-Length': '0'} 2024-04-12T03:46:16.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e497358b7460bc-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.254 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:46:16.255 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:46:16.255 DEBUG:boto:Method: GET 2024-04-12T03:46:16.255 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:46:16.255 DEBUG:boto:Data: 2024-04-12T03:46:16.255 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.255 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.255 DEBUG:boto:Params: {} 2024-04-12T03:46:16.256 DEBUG:boto:Token: None 2024-04-12T03:46:16.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj76 2024-04-12T03:46:16.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJ6xyYEw50ttE6K0CHiZGb/WsCo= 2024-04-12T03:46:16.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJ6xyYEw50ttE6K0CHiZGb/WsCo=', 'Content-Length': '0'} 2024-04-12T03:46:16.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7c97ae9ba43099-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.258 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.258 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.258 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.258 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.258 DEBUG:boto:Method: GET 2024-04-12T03:46:16.258 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.259 DEBUG:boto:Data: 2024-04-12T03:46:16.259 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.259 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.259 DEBUG:boto:Params: {} 2024-04-12T03:46:16.259 DEBUG:boto:Token: None 2024-04-12T03:46:16.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.259 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004718173a60dab448-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.261 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.262 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.262 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.262 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.262 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.262 DEBUG:boto:Method: GET 2024-04-12T03:46:16.262 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:46:16.262 DEBUG:boto:Data: 2024-04-12T03:46:16.263 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.263 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.263 DEBUG:boto:Params: {} 2024-04-12T03:46:16.263 DEBUG:boto:Token: None 2024-04-12T03:46:16.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f68120875514a79-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.266 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:46:16.266 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.267 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.267 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.267 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.267 DEBUG:boto:Method: GET 2024-04-12T03:46:16.267 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.267 DEBUG:boto:Data: 2024-04-12T03:46:16.267 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.267 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.267 DEBUG:boto:Params: {} 2024-04-12T03:46:16.267 DEBUG:boto:Token: None 2024-04-12T03:46:16.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.268 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk= 2024-04-12T03:46:16.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk=', 'Content-Length': '0'} 2024-04-12T03:46:16.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b52ab0ed36e3742c-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.269 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.270 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.270 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.270 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.270 DEBUG:boto:Method: GET 2024-04-12T03:46:16.270 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.270 DEBUG:boto:Data: 2024-04-12T03:46:16.270 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.270 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.270 DEBUG:boto:Params: {} 2024-04-12T03:46:16.271 DEBUG:boto:Token: None 2024-04-12T03:46:16.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk= 2024-04-12T03:46:16.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk=', 'Content-Length': '0'} 2024-04-12T03:46:16.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000210b439250d5e364-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.273 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.273 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.273 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.273 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.273 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.273 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.273 DEBUG:boto:Data: 2024-04-12T03:46:16.274 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.274 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.274 DEBUG:boto:Params: {} 2024-04-12T03:46:16.274 DEBUG:boto:Token: None 2024-04-12T03:46:16.274 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.274 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wL06xqZpPi1oPI7mE5AyEf82O/k= 2024-04-12T03:46:16.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wL06xqZpPi1oPI7mE5AyEf82O/k=', 'Content-Length': '0'} 2024-04-12T03:46:16.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009df64682af489d6-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.276 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.276 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.276 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.276 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.276 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.277 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.277 DEBUG:boto:Data: 2024-04-12T03:46:16.277 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.277 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.277 DEBUG:boto:Params: {} 2024-04-12T03:46:16.277 DEBUG:boto:Token: None 2024-04-12T03:46:16.277 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wL06xqZpPi1oPI7mE5AyEf82O/k= 2024-04-12T03:46:16.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wL06xqZpPi1oPI7mE5AyEf82O/k=', 'Content-Length': '0'} 2024-04-12T03:46:16.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805c916d792f1fb6-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:46:16.279 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.280 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.280 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.280 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.280 DEBUG:boto:Method: GET 2024-04-12T03:46:16.280 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.280 DEBUG:boto:Data: 2024-04-12T03:46:16.280 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.280 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.280 DEBUG:boto:Params: {} 2024-04-12T03:46:16.281 DEBUG:boto:Token: None 2024-04-12T03:46:16.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk= 2024-04-12T03:46:16.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk=', 'Content-Length': '0'} 2024-04-12T03:46:16.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a58c63e0b1c6b30-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.287 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.287 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.287 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.287 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.287 DEBUG:boto:Method: GET 2024-04-12T03:46:16.287 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.287 DEBUG:boto:Data: 2024-04-12T03:46:16.288 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.288 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.288 DEBUG:boto:Params: {} 2024-04-12T03:46:16.288 DEBUG:boto:Token: None 2024-04-12T03:46:16.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:46:16.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk= 2024-04-12T03:46:16.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FyQzpd4CwF+CeMaMMTDyxTAronk=', 'Content-Length': '0'} 2024-04-12T03:46:16.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a62313bdf944c0e-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.290 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.290 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.290 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.291 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.291 DEBUG:boto:Method: GET 2024-04-12T03:46:16.291 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.291 DEBUG:boto:Data: 2024-04-12T03:46:16.291 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.291 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.291 DEBUG:boto:Params: {} 2024-04-12T03:46:16.291 DEBUG:boto:Token: None 2024-04-12T03:46:16.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6f6a7cb3b2cbe40-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.294 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.294 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.294 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.294 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.294 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.295 DEBUG:boto:Method: GET 2024-04-12T03:46:16.295 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.295 DEBUG:boto:Data: 2024-04-12T03:46:16.295 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.295 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.295 DEBUG:boto:Params: {} 2024-04-12T03:46:16.295 DEBUG:boto:Token: None 2024-04-12T03:46:16.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c0e9532d290faa4-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.298 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.298 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.299 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.299 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.299 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:46:16.299 DEBUG:boto:Data: 2024-04-12T03:46:16.299 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.299 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.299 DEBUG:boto:Params: {} 2024-04-12T03:46:16.299 DEBUG:boto:Token: None 2024-04-12T03:46:16.299 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85 2024-04-12T03:46:16.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6P9mjMkGw1ffdocbfu1+0N4EUCI= 2024-04-12T03:46:16.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6P9mjMkGw1ffdocbfu1+0N4EUCI=', 'Content-Length': '0'} 2024-04-12T03:46:16.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030859ca465c82a1e-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.301 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.301 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.302 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.302 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:46:16.302 DEBUG:boto:Data: 2024-04-12T03:46:16.302 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.302 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.302 DEBUG:boto:Params: {} 2024-04-12T03:46:16.302 DEBUG:boto:Token: None 2024-04-12T03:46:16.302 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85 2024-04-12T03:46:16.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6P9mjMkGw1ffdocbfu1+0N4EUCI= 2024-04-12T03:46:16.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6P9mjMkGw1ffdocbfu1+0N4EUCI=', 'Content-Length': '0'} 2024-04-12T03:46:16.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073eabcdba080a917-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:46:16.304 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.304 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.305 DEBUG:boto:Method: GET 2024-04-12T03:46:16.305 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:46:16.305 DEBUG:boto:Data: 2024-04-12T03:46:16.305 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.305 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.305 DEBUG:boto:Params: {} 2024-04-12T03:46:16.305 DEBUG:boto:Token: None 2024-04-12T03:46:16.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85 2024-04-12T03:46:16.305 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KU4GMvmCyo34M2XcWqXmf6+7aAQ= 2024-04-12T03:46:16.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KU4GMvmCyo34M2XcWqXmf6+7aAQ=', 'Content-Length': '0'} 2024-04-12T03:46:16.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9cb8176b839b1e-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.307 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:46:16.307 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:46:16.307 DEBUG:boto:Method: GET 2024-04-12T03:46:16.307 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:46:16.308 DEBUG:boto:Data: 2024-04-12T03:46:16.308 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.308 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.308 DEBUG:boto:Params: {} 2024-04-12T03:46:16.308 DEBUG:boto:Token: None 2024-04-12T03:46:16.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj85 2024-04-12T03:46:16.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KU4GMvmCyo34M2XcWqXmf6+7aAQ= 2024-04-12T03:46:16.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KU4GMvmCyo34M2XcWqXmf6+7aAQ=', 'Content-Length': '0'} 2024-04-12T03:46:16.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085ca13081428ed16-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.310 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.311 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.311 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.311 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.311 DEBUG:boto:Method: GET 2024-04-12T03:46:16.311 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.311 DEBUG:boto:Data: 2024-04-12T03:46:16.311 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.311 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.311 DEBUG:boto:Params: {} 2024-04-12T03:46:16.312 DEBUG:boto:Token: None 2024-04-12T03:46:16.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002489bbb50adf90b-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.358 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.359 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.359 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.359 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.359 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.359 DEBUG:boto:Method: GET 2024-04-12T03:46:16.359 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:46:16.360 DEBUG:boto:Data: 2024-04-12T03:46:16.360 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.360 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.360 DEBUG:boto:Params: {} 2024-04-12T03:46:16.360 DEBUG:boto:Token: None 2024-04-12T03:46:16.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cbe557f54c5a68c-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.363 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:46:16.364 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.364 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.364 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.364 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.364 DEBUG:boto:Method: GET 2024-04-12T03:46:16.364 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.364 DEBUG:boto:Data: 2024-04-12T03:46:16.364 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.365 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.365 DEBUG:boto:Params: {} 2024-04-12T03:46:16.365 DEBUG:boto:Token: None 2024-04-12T03:46:16.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.365 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw= 2024-04-12T03:46:16.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw=', 'Content-Length': '0'} 2024-04-12T03:46:16.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002480b0a5e7986913-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.367 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.367 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.368 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.368 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.368 DEBUG:boto:Method: GET 2024-04-12T03:46:16.368 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.368 DEBUG:boto:Data: 2024-04-12T03:46:16.368 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.368 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.369 DEBUG:boto:Params: {} 2024-04-12T03:46:16.369 DEBUG:boto:Token: None 2024-04-12T03:46:16.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw= 2024-04-12T03:46:16.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw=', 'Content-Length': '0'} 2024-04-12T03:46:16.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002394ba3a644a2a8e-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.414 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.414 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.415 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.415 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.415 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.415 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.415 DEBUG:boto:Data: 2024-04-12T03:46:16.415 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.415 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.415 DEBUG:boto:Params: {} 2024-04-12T03:46:16.415 DEBUG:boto:Token: None 2024-04-12T03:46:16.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cIRGCKE4P0fGpYjgRcmjGDuZ+7c= 2024-04-12T03:46:16.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cIRGCKE4P0fGpYjgRcmjGDuZ+7c=', 'Content-Length': '0'} 2024-04-12T03:46:16.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec6df2629afa309-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.418 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.418 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.418 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.418 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.418 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.418 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.418 DEBUG:boto:Data: 2024-04-12T03:46:16.419 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.419 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.419 DEBUG:boto:Params: {} 2024-04-12T03:46:16.419 DEBUG:boto:Token: None 2024-04-12T03:46:16.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cIRGCKE4P0fGpYjgRcmjGDuZ+7c= 2024-04-12T03:46:16.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cIRGCKE4P0fGpYjgRcmjGDuZ+7c=', 'Content-Length': '0'} 2024-04-12T03:46:16.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa1bec6776115d8-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:46:16.421 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.421 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.422 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.422 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.422 DEBUG:boto:Method: GET 2024-04-12T03:46:16.422 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.422 DEBUG:boto:Data: 2024-04-12T03:46:16.422 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.422 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.422 DEBUG:boto:Params: {} 2024-04-12T03:46:16.422 DEBUG:boto:Token: None 2024-04-12T03:46:16.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.423 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw= 2024-04-12T03:46:16.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw=', 'Content-Length': '0'} 2024-04-12T03:46:16.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc617f3118488b5a-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.425 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.425 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.425 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.425 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.425 DEBUG:boto:Method: GET 2024-04-12T03:46:16.425 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.425 DEBUG:boto:Data: 2024-04-12T03:46:16.425 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.426 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.426 DEBUG:boto:Params: {} 2024-04-12T03:46:16.426 DEBUG:boto:Token: None 2024-04-12T03:46:16.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:46:16.426 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw= 2024-04-12T03:46:16.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+UuAEAAie0OXd6j4C/A/Af8glw=', 'Content-Length': '0'} 2024-04-12T03:46:16.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3652da3419cf307-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.428 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.429 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.429 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.429 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.429 DEBUG:boto:Method: GET 2024-04-12T03:46:16.429 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.429 DEBUG:boto:Data: 2024-04-12T03:46:16.429 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.430 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.430 DEBUG:boto:Params: {} 2024-04-12T03:46:16.430 DEBUG:boto:Token: None 2024-04-12T03:46:16.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89347f2d86741d2-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.433 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.433 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.433 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.433 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.434 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.434 DEBUG:boto:Method: GET 2024-04-12T03:46:16.434 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.434 DEBUG:boto:Data: 2024-04-12T03:46:16.434 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.434 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.435 DEBUG:boto:Params: {} 2024-04-12T03:46:16.435 DEBUG:boto:Token: None 2024-04-12T03:46:16.435 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebf41e46a012980a-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.438 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.438 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.438 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.439 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.439 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:46:16.439 DEBUG:boto:Data: 2024-04-12T03:46:16.439 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.439 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.439 DEBUG:boto:Params: {} 2024-04-12T03:46:16.439 DEBUG:boto:Token: None 2024-04-12T03:46:16.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95 2024-04-12T03:46:16.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DCp7I9K7ivj/ZtGt0+mBK4+OPu4= 2024-04-12T03:46:16.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DCp7I9K7ivj/ZtGt0+mBK4+OPu4=', 'Content-Length': '0'} 2024-04-12T03:46:16.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5c2cd171959071-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.441 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.441 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.442 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.442 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:46:16.442 DEBUG:boto:Data: 2024-04-12T03:46:16.442 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.442 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.442 DEBUG:boto:Params: {} 2024-04-12T03:46:16.443 DEBUG:boto:Token: None 2024-04-12T03:46:16.443 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95 2024-04-12T03:46:16.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DCp7I9K7ivj/ZtGt0+mBK4+OPu4= 2024-04-12T03:46:16.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DCp7I9K7ivj/ZtGt0+mBK4+OPu4=', 'Content-Length': '0'} 2024-04-12T03:46:16.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd5b53707823728-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:46:16.445 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.445 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.445 DEBUG:boto:Method: GET 2024-04-12T03:46:16.445 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:46:16.445 DEBUG:boto:Data: 2024-04-12T03:46:16.445 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.445 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.445 DEBUG:boto:Params: {} 2024-04-12T03:46:16.446 DEBUG:boto:Token: None 2024-04-12T03:46:16.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95 2024-04-12T03:46:16.446 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+qCbr1Cz+3tnqdVR5lD6UcHEn+I= 2024-04-12T03:46:16.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+qCbr1Cz+3tnqdVR5lD6UcHEn+I=', 'Content-Length': '0'} 2024-04-12T03:46:16.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076755adfcb6ea680-006618ae88-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.448 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:46:16.448 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:46:16.448 DEBUG:boto:Method: GET 2024-04-12T03:46:16.448 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:46:16.448 DEBUG:boto:Data: 2024-04-12T03:46:16.448 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.448 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.449 DEBUG:boto:Params: {} 2024-04-12T03:46:16.449 DEBUG:boto:Token: None 2024-04-12T03:46:16.449 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj95 2024-04-12T03:46:16.449 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+qCbr1Cz+3tnqdVR5lD6UcHEn+I= 2024-04-12T03:46:16.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+qCbr1Cz+3tnqdVR5lD6UcHEn+I=', 'Content-Length': '0'} 2024-04-12T03:46:16.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680d20c5cb923c8f-006618ae88-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.451 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.451 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.451 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.451 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.452 DEBUG:boto:Method: GET 2024-04-12T03:46:16.452 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.452 DEBUG:boto:Data: 2024-04-12T03:46:16.452 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:46:16.452 DEBUG:boto:Port: 8001 2024-04-12T03:46:16.452 DEBUG:boto:Params: {} 2024-04-12T03:46:16.452 DEBUG:boto:Token: None 2024-04-12T03:46:16.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069fcd934063e19fd-006618ae88-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.455 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.455 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.455 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.455 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.455 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.455 DEBUG:boto:Method: GET 2024-04-12T03:46:16.455 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:46:16.456 DEBUG:boto:Data: 2024-04-12T03:46:16.456 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.456 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.456 DEBUG:boto:Params: {} 2024-04-12T03:46:16.456 DEBUG:boto:Token: None 2024-04-12T03:46:16.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk= 2024-04-12T03:46:16.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jooKm5aOusfS/OHw98b+WdRpGHk=', 'Content-Length': '0'} 2024-04-12T03:46:16.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc22219859398ee-006618ae88-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.459 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:16.459 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:46:16.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:16.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:16.460 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:46:16.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:16.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:16.557 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:46:16.557 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:46:16.564 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2024-04-12T03:46:16.611 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.612 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.612 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.612 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:16.612 DEBUG:boto:Data: 2024-04-12T03:46:16.612 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.612 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.612 DEBUG:boto:Params: {} 2024-04-12T03:46:16.612 DEBUG:boto:Token: None 2024-04-12T03:46:16.613 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk= 2024-04-12T03:46:16.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk=', 'Content-Length': '0'} 2024-04-12T03:46:16.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000621a83256cc78490-006618ae88-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.616 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:46:16.616 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:46:16.616 DEBUG:boto:Method: PUT 2024-04-12T03:46:16.616 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:46:16.616 DEBUG:boto:Data: 2024-04-12T03:46:16.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:46:16.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.617 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.617 DEBUG:boto:Params: {} 2024-04-12T03:46:16.617 DEBUG:boto:Token: None 2024-04-12T03:46:16.617 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj23 2024-04-12T03:46:16.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPf1zrUBaKH/uLeIAhbMS6Nr/zU= 2024-04-12T03:46:16.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPf1zrUBaKH/uLeIAhbMS6Nr/zU='} 2024-04-12T03:46:16.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df4be24460bbead5-006618ae88-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.663 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.663 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.663 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.663 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:16.663 DEBUG:boto:Data: 2024-04-12T03:46:16.663 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.664 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.664 DEBUG:boto:Params: {} 2024-04-12T03:46:16.664 DEBUG:boto:Token: None 2024-04-12T03:46:16.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk= 2024-04-12T03:46:16.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk=', 'Content-Length': '0'} 2024-04-12T03:46:16.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcefedad23fdefc4-006618ae88-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.667 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:46:16.667 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:46:16.667 DEBUG:boto:Method: PUT 2024-04-12T03:46:16.667 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:46:16.668 DEBUG:boto:Data: 2024-04-12T03:46:16.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:46:16.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.668 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.668 DEBUG:boto:Params: {} 2024-04-12T03:46:16.668 DEBUG:boto:Token: None 2024-04-12T03:46:16.668 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj46 2024-04-12T03:46:16.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iJd/rkOxQ6AAB/8zfCjPGFcwjFE= 2024-04-12T03:46:16.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iJd/rkOxQ6AAB/8zfCjPGFcwjFE='} 2024-04-12T03:46:16.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020f86cb2a76f6836-006618ae88-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.674 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.674 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.674 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.674 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:16.674 DEBUG:boto:Data: 2024-04-12T03:46:16.674 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.674 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.674 DEBUG:boto:Params: {} 2024-04-12T03:46:16.675 DEBUG:boto:Token: None 2024-04-12T03:46:16.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk= 2024-04-12T03:46:16.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk=', 'Content-Length': '0'} 2024-04-12T03:46:16.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036b8c985608a4c84-006618ae88-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.678 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:46:16.678 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:46:16.678 DEBUG:boto:Method: PUT 2024-04-12T03:46:16.678 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:46:16.678 DEBUG:boto:Data: 2024-04-12T03:46:16.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:46:16.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.678 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.678 DEBUG:boto:Params: {} 2024-04-12T03:46:16.679 DEBUG:boto:Token: None 2024-04-12T03:46:16.679 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj69 2024-04-12T03:46:16.679 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Unnj117UwrqSO/gTwip76+1z2hs= 2024-04-12T03:46:16.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Unnj117UwrqSO/gTwip76+1z2hs='} 2024-04-12T03:46:16.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bde73679b5b8fd3f-006618ae88-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.723 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.723 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.723 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.723 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:16.723 DEBUG:boto:Data: 2024-04-12T03:46:16.723 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.723 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.723 DEBUG:boto:Params: {} 2024-04-12T03:46:16.724 DEBUG:boto:Token: None 2024-04-12T03:46:16.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk= 2024-04-12T03:46:16.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk=', 'Content-Length': '0'} 2024-04-12T03:46:16.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bfc3b60a955ea0-006618ae88-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.727 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:46:16.727 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:46:16.727 DEBUG:boto:Method: PUT 2024-04-12T03:46:16.727 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:46:16.727 DEBUG:boto:Data: 2024-04-12T03:46:16.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:46:16.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.728 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.728 DEBUG:boto:Params: {} 2024-04-12T03:46:16.728 DEBUG:boto:Token: None 2024-04-12T03:46:16.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj92 2024-04-12T03:46:16.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yinsXb/P8q621X8+zMSUJLINjlY= 2024-04-12T03:46:16.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yinsXb/P8q621X8+zMSUJLINjlY='} 2024-04-12T03:46:16.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022bef4284223b27f-006618ae88-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.733 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:46:16.733 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:46:16.734 DEBUG:boto:Method: HEAD 2024-04-12T03:46:16.734 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:46:16.734 DEBUG:boto:Data: 2024-04-12T03:46:16.734 DEBUG:boto:Headers: {} 2024-04-12T03:46:16.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.734 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.734 DEBUG:boto:Params: {} 2024-04-12T03:46:16.734 DEBUG:boto:Token: None 2024-04-12T03:46:16.734 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/ 2024-04-12T03:46:16.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk= 2024-04-12T03:46:16.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VK72rjZkXin0izSTv9h7FzXvPRk=', 'Content-Length': '0'} 2024-04-12T03:46:16.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003ad7653d3b1340e-006618ae88-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.737 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:46:16.737 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:46:16.738 DEBUG:boto:Method: PUT 2024-04-12T03:46:16.738 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:46:16.738 DEBUG:boto:Data: 2024-04-12T03:46:16.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:46:16.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:46:16.738 DEBUG:boto:Port: 8000 2024-04-12T03:46:16.738 DEBUG:boto:Params: {} 2024-04-12T03:46:16.738 DEBUG:boto:Token: None 2024-04-12T03:46:16.738 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:46:16 GMT /dgbpxg-61/obj115 2024-04-12T03:46:16.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k9cqlluGxTb8Z4T+KnrcJplm43w= 2024-04-12T03:46:16.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k9cqlluGxTb8Z4T+KnrcJplm43w='} 2024-04-12T03:46:16.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0c93904525b2f73-006618ae88-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:16.783 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:46:16.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:16.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:16.783 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:16.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:16.839 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:16.875+0000 7fb508818780 1 waiting to reach latest gen 3, on 2.. 2024-04-12T03:46:21.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:21.877+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:21.877 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:21.877 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:26.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:26.879+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:26.879 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:26.879 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:28.811 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:46:28.812+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:46:31.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:31.881+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:31.880 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:31.880 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:36.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:36.883+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:36.882 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:36.882 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:41.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:41.884+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:41.884 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:41.884 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:46.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:46.886+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:46.887 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:46.887 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:51.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:51.888+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:51.887 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:51.888 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:46:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:46:56.890+0000 7fb508818780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:56.889 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:46:56.889 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:47:01.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:01.892+0000 7fb508818780 1 bucket sync caught up with source: 2024-04-12T03:47:01.891 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:47:01.891 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.87.6, , 00000000002.110.6, , , , , , 00000000001.73.6, , 00000000001.99.6, , , , ] 2024-04-12T03:47:01.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:01.892+0000 7fb508818780 0 bucket checkpoint complete 2024-04-12T03:47:01.908 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:47:01.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:01.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:01.909 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:01.972 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:01.972 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:02.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:02.013+0000 7f09e7de2780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:02.011 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:47:02.011 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000002.442.6, , , 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:47:07.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:07.015+0000 7f09e7de2780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:07.013 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000001.147.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:47:07.013 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000002.442.6, , , 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:47:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:12.016+0000 7f09e7de2780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:12.014 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000001.2091.6, 00000000001.1925.6, 00000000002.151.6, 00000000001.434.6, , , 00000000002.425.6, 00000000004.2236.6, 00000000003.1483.6] 2024-04-12T03:47:12.014 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000002.442.6, , , 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:47:17.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:17.017+0000 7f09e7de2780 1 bucket sync caught up with source: 2024-04-12T03:47:17.016 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000002.442.6, , , 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:47:17.016 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000002.442.6, , , 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:47:17.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:17.017+0000 7f09e7de2780 0 bucket checkpoint complete 2024-04-12T03:47:17.032 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:47:17.032 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.033 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.033 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.033 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:17.033 DEBUG:boto:Data: 2024-04-12T03:47:17.033 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.033 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.033 DEBUG:boto:Params: {} 2024-04-12T03:47:17.034 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:47:17.034 DEBUG:boto:Token: None 2024-04-12T03:47:17.034 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uPI08fOeNMKpGWBEfccvFcjTGhM= 2024-04-12T03:47:17.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uPI08fOeNMKpGWBEfccvFcjTGhM=', 'Content-Length': '0'} 2024-04-12T03:47:17.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0360a10f6c91262-006618aec5-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.066 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.066 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.066 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.066 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:17.066 DEBUG:boto:Data: 2024-04-12T03:47:17.066 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.066 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.066 DEBUG:boto:Params: {} 2024-04-12T03:47:17.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:47:17.067 DEBUG:boto:Token: None 2024-04-12T03:47:17.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uPI08fOeNMKpGWBEfccvFcjTGhM= 2024-04-12T03:47:17.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uPI08fOeNMKpGWBEfccvFcjTGhM=', 'Content-Length': '0'} 2024-04-12T03:47:17.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a1c5e2d50664726-006618aec5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:47:17.086 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.086 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.086 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.086 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.086 DEBUG:boto:Method: GET 2024-04-12T03:47:17.086 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.086 DEBUG:boto:Data: 2024-04-12T03:47:17.087 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.087 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.087 DEBUG:boto:Params: {} 2024-04-12T03:47:17.087 DEBUG:boto:Token: None 2024-04-12T03:47:17.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/?versions 2024-04-12T03:47:17.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA= 2024-04-12T03:47:17.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA=', 'Content-Length': '0'} 2024-04-12T03:47:17.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f23e526bea34e1d5-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.090 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:47:17.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:47:17.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:47:17.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:47:17.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:47:17.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:47:17.094 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.094 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.094 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.094 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.094 DEBUG:boto:Method: GET 2024-04-12T03:47:17.094 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.094 DEBUG:boto:Data: 2024-04-12T03:47:17.095 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.095 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.095 DEBUG:boto:Params: {} 2024-04-12T03:47:17.095 DEBUG:boto:Token: None 2024-04-12T03:47:17.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/?versions 2024-04-12T03:47:17.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA= 2024-04-12T03:47:17.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA=', 'Content-Length': '0'} 2024-04-12T03:47:17.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d37697d4d001665-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.098 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:47:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:47:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:47:17.101 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.101 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.101 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.101 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.102 DEBUG:boto:Method: GET 2024-04-12T03:47:17.102 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.102 DEBUG:boto:Data: 2024-04-12T03:47:17.102 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.102 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.102 DEBUG:boto:Params: {} 2024-04-12T03:47:17.102 DEBUG:boto:Token: None 2024-04-12T03:47:17.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/?versions 2024-04-12T03:47:17.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA= 2024-04-12T03:47:17.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA=', 'Content-Length': '0'} 2024-04-12T03:47:17.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f03cf164a12b10e-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.105 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.106 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.106 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.106 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.107 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.107 DEBUG:boto:Method: GET 2024-04-12T03:47:17.107 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:47:17.107 DEBUG:boto:Data: 2024-04-12T03:47:17.107 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.107 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.107 DEBUG:boto:Params: {} 2024-04-12T03:47:17.107 DEBUG:boto:Token: None 2024-04-12T03:47:17.107 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/?versions 2024-04-12T03:47:17.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA= 2024-04-12T03:47:17.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bnBtsEuPLTJ8toZpZfNFh18xfEA=', 'Content-Length': '0'} 2024-04-12T03:47:17.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d1de7cdf84ff95-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.111 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:47:17.112 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.112 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.112 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.112 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.113 DEBUG:boto:Method: GET 2024-04-12T03:47:17.113 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.113 DEBUG:boto:Data: 2024-04-12T03:47:17.113 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.113 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.113 DEBUG:boto:Params: {} 2024-04-12T03:47:17.113 DEBUG:boto:Token: None 2024-04-12T03:47:17.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4= 2024-04-12T03:47:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4=', 'Content-Length': '0'} 2024-04-12T03:47:17.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9be8845f0d2a612-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.116 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.116 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.116 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.116 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.116 DEBUG:boto:Method: GET 2024-04-12T03:47:17.116 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.117 DEBUG:boto:Data: 2024-04-12T03:47:17.117 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.117 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.117 DEBUG:boto:Params: {} 2024-04-12T03:47:17.117 DEBUG:boto:Token: None 2024-04-12T03:47:17.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4= 2024-04-12T03:47:17.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4=', 'Content-Length': '0'} 2024-04-12T03:47:17.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a57274f164efd78e-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.119 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.119 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.120 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.120 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.120 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.120 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.120 DEBUG:boto:Data: 2024-04-12T03:47:17.120 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.120 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.120 DEBUG:boto:Params: {} 2024-04-12T03:47:17.120 DEBUG:boto:Token: None 2024-04-12T03:47:17.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TozEnqKCDAT1bEuZ4Hl+OsubqQc= 2024-04-12T03:47:17.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TozEnqKCDAT1bEuZ4Hl+OsubqQc=', 'Content-Length': '0'} 2024-04-12T03:47:17.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006628e298f3c77c42-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.123 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.123 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.123 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.123 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.123 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.123 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.123 DEBUG:boto:Data: 2024-04-12T03:47:17.123 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.124 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.124 DEBUG:boto:Params: {} 2024-04-12T03:47:17.124 DEBUG:boto:Token: None 2024-04-12T03:47:17.124 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TozEnqKCDAT1bEuZ4Hl+OsubqQc= 2024-04-12T03:47:17.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TozEnqKCDAT1bEuZ4Hl+OsubqQc=', 'Content-Length': '0'} 2024-04-12T03:47:17.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a4cad7416fa110-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:47:17.126 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.126 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.126 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.126 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.126 DEBUG:boto:Method: GET 2024-04-12T03:47:17.127 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.127 DEBUG:boto:Data: 2024-04-12T03:47:17.127 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.127 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.127 DEBUG:boto:Params: {} 2024-04-12T03:47:17.127 DEBUG:boto:Token: None 2024-04-12T03:47:17.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4= 2024-04-12T03:47:17.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4=', 'Content-Length': '0'} 2024-04-12T03:47:17.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f2abd03e115dc5-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.129 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.129 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.130 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.130 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.130 DEBUG:boto:Method: GET 2024-04-12T03:47:17.130 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.130 DEBUG:boto:Data: 2024-04-12T03:47:17.130 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.130 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.130 DEBUG:boto:Params: {} 2024-04-12T03:47:17.130 DEBUG:boto:Token: None 2024-04-12T03:47:17.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:47:17.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4= 2024-04-12T03:47:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jSUrFX3r6+VbcuYjGTpKFf820v4=', 'Content-Length': '0'} 2024-04-12T03:47:17.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df28d675e78fd580-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.178 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.179 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.179 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.179 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.179 DEBUG:boto:Method: GET 2024-04-12T03:47:17.179 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.179 DEBUG:boto:Data: 2024-04-12T03:47:17.179 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.179 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.179 DEBUG:boto:Params: {} 2024-04-12T03:47:17.179 DEBUG:boto:Token: None 2024-04-12T03:47:17.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d430b6f5c3b82f40-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.182 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.183 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.183 DEBUG:boto:Method: GET 2024-04-12T03:47:17.183 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.183 DEBUG:boto:Data: 2024-04-12T03:47:17.184 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.184 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.184 DEBUG:boto:Params: {} 2024-04-12T03:47:17.184 DEBUG:boto:Token: None 2024-04-12T03:47:17.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006173e57579ecdf05-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.188 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.188 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.188 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.188 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.188 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:47:17.188 DEBUG:boto:Data: 2024-04-12T03:47:17.188 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.189 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.189 DEBUG:boto:Params: {} 2024-04-12T03:47:17.189 DEBUG:boto:Token: None 2024-04-12T03:47:17.189 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11 2024-04-12T03:47:17.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oZB37thS//z3OJTzvUESi2dkPcE= 2024-04-12T03:47:17.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oZB37thS//z3OJTzvUESi2dkPcE=', 'Content-Length': '0'} 2024-04-12T03:47:17.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a55367b178509fd-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.191 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.191 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.191 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.191 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:47:17.191 DEBUG:boto:Data: 2024-04-12T03:47:17.191 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.192 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.192 DEBUG:boto:Params: {} 2024-04-12T03:47:17.192 DEBUG:boto:Token: None 2024-04-12T03:47:17.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11 2024-04-12T03:47:17.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oZB37thS//z3OJTzvUESi2dkPcE= 2024-04-12T03:47:17.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oZB37thS//z3OJTzvUESi2dkPcE=', 'Content-Length': '0'} 2024-04-12T03:47:17.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88270bb5567f745-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:47:17.194 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.194 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.194 DEBUG:boto:Method: GET 2024-04-12T03:47:17.194 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:47:17.195 DEBUG:boto:Data: 2024-04-12T03:47:17.195 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.195 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.195 DEBUG:boto:Params: {} 2024-04-12T03:47:17.195 DEBUG:boto:Token: None 2024-04-12T03:47:17.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11 2024-04-12T03:47:17.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fmmAR00obCYUMMUDnl9IaH5yzpo= 2024-04-12T03:47:17.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fmmAR00obCYUMMUDnl9IaH5yzpo=', 'Content-Length': '0'} 2024-04-12T03:47:17.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5d88b5d9ff0661-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.242 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:47:17.242 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:47:17.242 DEBUG:boto:Method: GET 2024-04-12T03:47:17.243 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:47:17.243 DEBUG:boto:Data: 2024-04-12T03:47:17.243 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.243 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.243 DEBUG:boto:Params: {} 2024-04-12T03:47:17.243 DEBUG:boto:Token: None 2024-04-12T03:47:17.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj11 2024-04-12T03:47:17.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fmmAR00obCYUMMUDnl9IaH5yzpo= 2024-04-12T03:47:17.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fmmAR00obCYUMMUDnl9IaH5yzpo=', 'Content-Length': '0'} 2024-04-12T03:47:17.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c57a86a2c859d6b-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.245 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.246 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.246 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.246 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.246 DEBUG:boto:Method: GET 2024-04-12T03:47:17.246 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.246 DEBUG:boto:Data: 2024-04-12T03:47:17.246 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.246 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.246 DEBUG:boto:Params: {} 2024-04-12T03:47:17.246 DEBUG:boto:Token: None 2024-04-12T03:47:17.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826a046c72eadc87-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.249 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.250 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.250 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.250 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.250 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.250 DEBUG:boto:Method: GET 2024-04-12T03:47:17.250 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:47:17.250 DEBUG:boto:Data: 2024-04-12T03:47:17.250 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.250 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.251 DEBUG:boto:Params: {} 2024-04-12T03:47:17.251 DEBUG:boto:Token: None 2024-04-12T03:47:17.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060adf8a6f888bf09-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.254 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:47:17.255 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.255 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.255 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.255 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.255 DEBUG:boto:Method: GET 2024-04-12T03:47:17.255 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.255 DEBUG:boto:Data: 2024-04-12T03:47:17.255 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.256 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.256 DEBUG:boto:Params: {} 2024-04-12T03:47:17.256 DEBUG:boto:Token: None 2024-04-12T03:47:17.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI= 2024-04-12T03:47:17.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI=', 'Content-Length': '0'} 2024-04-12T03:47:17.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c797c40c5df561-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.258 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.259 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.259 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.259 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.259 DEBUG:boto:Method: GET 2024-04-12T03:47:17.259 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.259 DEBUG:boto:Data: 2024-04-12T03:47:17.259 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.259 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.259 DEBUG:boto:Params: {} 2024-04-12T03:47:17.259 DEBUG:boto:Token: None 2024-04-12T03:47:17.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI= 2024-04-12T03:47:17.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI=', 'Content-Length': '0'} 2024-04-12T03:47:17.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042cc5eb7e38c92a7-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.302 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.302 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.303 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.303 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.303 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.303 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.303 DEBUG:boto:Data: 2024-04-12T03:47:17.303 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.303 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.303 DEBUG:boto:Params: {} 2024-04-12T03:47:17.303 DEBUG:boto:Token: None 2024-04-12T03:47:17.304 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nkeuOLBiHZClYqVqWhMfJxhA9bs= 2024-04-12T03:47:17.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nkeuOLBiHZClYqVqWhMfJxhA9bs=', 'Content-Length': '0'} 2024-04-12T03:47:17.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740f6d64f1720d7c-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.306 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.306 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.306 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.306 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.306 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.306 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.306 DEBUG:boto:Data: 2024-04-12T03:47:17.306 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.307 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.307 DEBUG:boto:Params: {} 2024-04-12T03:47:17.307 DEBUG:boto:Token: None 2024-04-12T03:47:17.307 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nkeuOLBiHZClYqVqWhMfJxhA9bs= 2024-04-12T03:47:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nkeuOLBiHZClYqVqWhMfJxhA9bs=', 'Content-Length': '0'} 2024-04-12T03:47:17.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2a2a8c75f789e9-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:47:17.309 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.309 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.309 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.310 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.310 DEBUG:boto:Method: GET 2024-04-12T03:47:17.310 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.310 DEBUG:boto:Data: 2024-04-12T03:47:17.310 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.310 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.310 DEBUG:boto:Params: {} 2024-04-12T03:47:17.310 DEBUG:boto:Token: None 2024-04-12T03:47:17.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI= 2024-04-12T03:47:17.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI=', 'Content-Length': '0'} 2024-04-12T03:47:17.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f79f1c4fff91a79-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.313 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.313 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.313 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.313 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.313 DEBUG:boto:Method: GET 2024-04-12T03:47:17.313 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.313 DEBUG:boto:Data: 2024-04-12T03:47:17.314 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.314 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.314 DEBUG:boto:Params: {} 2024-04-12T03:47:17.314 DEBUG:boto:Token: None 2024-04-12T03:47:17.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:47:17.314 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI= 2024-04-12T03:47:17.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LxWCIa3NNK/Z2Qcby32QNDOX1ZI=', 'Content-Length': '0'} 2024-04-12T03:47:17.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720e27e1b681a8ea-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.316 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.317 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.317 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.317 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.317 DEBUG:boto:Method: GET 2024-04-12T03:47:17.317 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.317 DEBUG:boto:Data: 2024-04-12T03:47:17.317 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.317 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.317 DEBUG:boto:Params: {} 2024-04-12T03:47:17.317 DEBUG:boto:Token: None 2024-04-12T03:47:17.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.318 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8cdf05ccda50b5-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.320 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.320 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.321 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.321 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.321 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.321 DEBUG:boto:Method: GET 2024-04-12T03:47:17.321 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.321 DEBUG:boto:Data: 2024-04-12T03:47:17.321 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.321 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.321 DEBUG:boto:Params: {} 2024-04-12T03:47:17.322 DEBUG:boto:Token: None 2024-04-12T03:47:17.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c61dc90f0282270a-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.325 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.325 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.325 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.326 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.326 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:47:17.326 DEBUG:boto:Data: 2024-04-12T03:47:17.326 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.326 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.326 DEBUG:boto:Params: {} 2024-04-12T03:47:17.326 DEBUG:boto:Token: None 2024-04-12T03:47:17.326 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115 2024-04-12T03:47:17.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEyGY2I+6j+EbDbeXyoP86CkPAI= 2024-04-12T03:47:17.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEyGY2I+6j+EbDbeXyoP86CkPAI=', 'Content-Length': '0'} 2024-04-12T03:47:17.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb1170e87ff228f-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.328 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.328 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.328 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.329 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:47:17.329 DEBUG:boto:Data: 2024-04-12T03:47:17.329 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.329 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.329 DEBUG:boto:Params: {} 2024-04-12T03:47:17.329 DEBUG:boto:Token: None 2024-04-12T03:47:17.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115 2024-04-12T03:47:17.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SEyGY2I+6j+EbDbeXyoP86CkPAI= 2024-04-12T03:47:17.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SEyGY2I+6j+EbDbeXyoP86CkPAI=', 'Content-Length': '0'} 2024-04-12T03:47:17.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0d1dd848133723-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:47:17.331 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.331 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.332 DEBUG:boto:Method: GET 2024-04-12T03:47:17.332 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:47:17.332 DEBUG:boto:Data: 2024-04-12T03:47:17.332 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.332 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.332 DEBUG:boto:Params: {} 2024-04-12T03:47:17.332 DEBUG:boto:Token: None 2024-04-12T03:47:17.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115 2024-04-12T03:47:17.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pyrRGajoyq7/3iEgam0HYLyH3Cs= 2024-04-12T03:47:17.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pyrRGajoyq7/3iEgam0HYLyH3Cs=', 'Content-Length': '0'} 2024-04-12T03:47:17.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb8653769196a1f-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.335 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:47:17.335 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:47:17.335 DEBUG:boto:Method: GET 2024-04-12T03:47:17.335 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:47:17.335 DEBUG:boto:Data: 2024-04-12T03:47:17.335 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.335 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.335 DEBUG:boto:Params: {} 2024-04-12T03:47:17.335 DEBUG:boto:Token: None 2024-04-12T03:47:17.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj115 2024-04-12T03:47:17.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pyrRGajoyq7/3iEgam0HYLyH3Cs= 2024-04-12T03:47:17.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pyrRGajoyq7/3iEgam0HYLyH3Cs=', 'Content-Length': '0'} 2024-04-12T03:47:17.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003622e63d01e87f75-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.338 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.338 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.338 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.338 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.338 DEBUG:boto:Method: GET 2024-04-12T03:47:17.338 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.338 DEBUG:boto:Data: 2024-04-12T03:47:17.338 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.338 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.339 DEBUG:boto:Params: {} 2024-04-12T03:47:17.339 DEBUG:boto:Token: None 2024-04-12T03:47:17.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001972a8d5078b709-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.341 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.341 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.342 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.342 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.342 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.342 DEBUG:boto:Method: GET 2024-04-12T03:47:17.342 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:47:17.342 DEBUG:boto:Data: 2024-04-12T03:47:17.342 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.342 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.342 DEBUG:boto:Params: {} 2024-04-12T03:47:17.342 DEBUG:boto:Token: None 2024-04-12T03:47:17.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fa8275e12d8648-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.346 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:47:17.346 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.346 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.347 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.347 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.347 DEBUG:boto:Method: GET 2024-04-12T03:47:17.347 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.347 DEBUG:boto:Data: 2024-04-12T03:47:17.347 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.347 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.347 DEBUG:boto:Params: {} 2024-04-12T03:47:17.347 DEBUG:boto:Token: None 2024-04-12T03:47:17.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc= 2024-04-12T03:47:17.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc=', 'Content-Length': '0'} 2024-04-12T03:47:17.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043605a83482c7e92-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.350 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.350 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.350 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.350 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.350 DEBUG:boto:Method: GET 2024-04-12T03:47:17.350 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.350 DEBUG:boto:Data: 2024-04-12T03:47:17.350 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.351 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.351 DEBUG:boto:Params: {} 2024-04-12T03:47:17.351 DEBUG:boto:Token: None 2024-04-12T03:47:17.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.351 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc= 2024-04-12T03:47:17.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc=', 'Content-Length': '0'} 2024-04-12T03:47:17.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bb602b98636541-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.353 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.353 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.353 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.353 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.354 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.354 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.354 DEBUG:boto:Data: 2024-04-12T03:47:17.354 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.354 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.354 DEBUG:boto:Params: {} 2024-04-12T03:47:17.354 DEBUG:boto:Token: None 2024-04-12T03:47:17.354 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.354 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:urJFFXKHOBp65lMcdcKTP2whqCI= 2024-04-12T03:47:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:urJFFXKHOBp65lMcdcKTP2whqCI=', 'Content-Length': '0'} 2024-04-12T03:47:17.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953bcdc23d966657-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.356 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.356 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.356 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.357 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.357 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.357 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.357 DEBUG:boto:Data: 2024-04-12T03:47:17.357 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.357 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.357 DEBUG:boto:Params: {} 2024-04-12T03:47:17.357 DEBUG:boto:Token: None 2024-04-12T03:47:17.357 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.358 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:urJFFXKHOBp65lMcdcKTP2whqCI= 2024-04-12T03:47:17.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:urJFFXKHOBp65lMcdcKTP2whqCI=', 'Content-Length': '0'} 2024-04-12T03:47:17.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f8fb092a6f2c02-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:47:17.360 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.360 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.360 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.360 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.360 DEBUG:boto:Method: GET 2024-04-12T03:47:17.360 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.360 DEBUG:boto:Data: 2024-04-12T03:47:17.360 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.360 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.360 DEBUG:boto:Params: {} 2024-04-12T03:47:17.361 DEBUG:boto:Token: None 2024-04-12T03:47:17.361 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.361 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc= 2024-04-12T03:47:17.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc=', 'Content-Length': '0'} 2024-04-12T03:47:17.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f19d74bd6ddc18-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.363 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.363 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.363 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.363 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.363 DEBUG:boto:Method: GET 2024-04-12T03:47:17.363 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.363 DEBUG:boto:Data: 2024-04-12T03:47:17.363 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.364 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.364 DEBUG:boto:Params: {} 2024-04-12T03:47:17.364 DEBUG:boto:Token: None 2024-04-12T03:47:17.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:47:17.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc= 2024-04-12T03:47:17.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j43pc+mLb7QxnZmTB5104U6orjc=', 'Content-Length': '0'} 2024-04-12T03:47:17.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a56916849ab52f-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.410 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.413 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.413 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.413 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.413 DEBUG:boto:Method: GET 2024-04-12T03:47:17.414 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.414 DEBUG:boto:Data: 2024-04-12T03:47:17.414 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.414 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.414 DEBUG:boto:Params: {} 2024-04-12T03:47:17.414 DEBUG:boto:Token: None 2024-04-12T03:47:17.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.415 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007458d29d1574f7f7-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.417 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.417 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.418 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.418 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.418 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.418 DEBUG:boto:Method: GET 2024-04-12T03:47:17.418 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.418 DEBUG:boto:Data: 2024-04-12T03:47:17.418 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.418 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.418 DEBUG:boto:Params: {} 2024-04-12T03:47:17.418 DEBUG:boto:Token: None 2024-04-12T03:47:17.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ea2470925783e0-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.422 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.422 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.422 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.422 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.423 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:47:17.423 DEBUG:boto:Data: 2024-04-12T03:47:17.423 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.423 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.423 DEBUG:boto:Params: {} 2024-04-12T03:47:17.423 DEBUG:boto:Token: None 2024-04-12T03:47:17.423 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17 2024-04-12T03:47:17.423 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ijPfjSKF8Ar4mOzMuq1g3G0MzLA= 2024-04-12T03:47:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ijPfjSKF8Ar4mOzMuq1g3G0MzLA=', 'Content-Length': '0'} 2024-04-12T03:47:17.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000362990ea6d2c9e23-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.425 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.426 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.426 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.426 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:47:17.426 DEBUG:boto:Data: 2024-04-12T03:47:17.426 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.426 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.426 DEBUG:boto:Params: {} 2024-04-12T03:47:17.426 DEBUG:boto:Token: None 2024-04-12T03:47:17.426 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17 2024-04-12T03:47:17.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ijPfjSKF8Ar4mOzMuq1g3G0MzLA= 2024-04-12T03:47:17.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ijPfjSKF8Ar4mOzMuq1g3G0MzLA=', 'Content-Length': '0'} 2024-04-12T03:47:17.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012539a7d1438f394-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:47:17.429 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.429 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.429 DEBUG:boto:Method: GET 2024-04-12T03:47:17.429 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:47:17.429 DEBUG:boto:Data: 2024-04-12T03:47:17.429 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.429 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.429 DEBUG:boto:Params: {} 2024-04-12T03:47:17.429 DEBUG:boto:Token: None 2024-04-12T03:47:17.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17 2024-04-12T03:47:17.430 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GNM1TuiMniPAnZEIcCc/e5RuiS4= 2024-04-12T03:47:17.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GNM1TuiMniPAnZEIcCc/e5RuiS4=', 'Content-Length': '0'} 2024-04-12T03:47:17.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12fd50a31da75d3-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.431 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:47:17.432 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:47:17.432 DEBUG:boto:Method: GET 2024-04-12T03:47:17.432 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:47:17.432 DEBUG:boto:Data: 2024-04-12T03:47:17.432 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.432 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.432 DEBUG:boto:Params: {} 2024-04-12T03:47:17.432 DEBUG:boto:Token: None 2024-04-12T03:47:17.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj17 2024-04-12T03:47:17.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GNM1TuiMniPAnZEIcCc/e5RuiS4= 2024-04-12T03:47:17.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GNM1TuiMniPAnZEIcCc/e5RuiS4=', 'Content-Length': '0'} 2024-04-12T03:47:17.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e0239591017867-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.435 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.435 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.435 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.435 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.435 DEBUG:boto:Method: GET 2024-04-12T03:47:17.435 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.435 DEBUG:boto:Data: 2024-04-12T03:47:17.435 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.436 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.436 DEBUG:boto:Params: {} 2024-04-12T03:47:17.436 DEBUG:boto:Token: None 2024-04-12T03:47:17.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.436 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170eaf07add0099d-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.478 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.479 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.479 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.479 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.479 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.479 DEBUG:boto:Method: GET 2024-04-12T03:47:17.479 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:47:17.479 DEBUG:boto:Data: 2024-04-12T03:47:17.479 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.479 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.480 DEBUG:boto:Params: {} 2024-04-12T03:47:17.480 DEBUG:boto:Token: None 2024-04-12T03:47:17.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000770d26eea4bb5072-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.483 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:47:17.483 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.483 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.483 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.484 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.484 DEBUG:boto:Method: GET 2024-04-12T03:47:17.484 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.484 DEBUG:boto:Data: 2024-04-12T03:47:17.484 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.484 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.484 DEBUG:boto:Params: {} 2024-04-12T03:47:17.484 DEBUG:boto:Token: None 2024-04-12T03:47:17.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.485 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE= 2024-04-12T03:47:17.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE=', 'Content-Length': '0'} 2024-04-12T03:47:17.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c634e97a60758e-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.487 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.487 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.487 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.487 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.487 DEBUG:boto:Method: GET 2024-04-12T03:47:17.487 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.487 DEBUG:boto:Data: 2024-04-12T03:47:17.487 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.487 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.487 DEBUG:boto:Params: {} 2024-04-12T03:47:17.488 DEBUG:boto:Token: None 2024-04-12T03:47:17.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE= 2024-04-12T03:47:17.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE=', 'Content-Length': '0'} 2024-04-12T03:47:17.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcfc35bcb06b1544-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.490 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.490 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.490 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.490 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.490 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.490 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.490 DEBUG:boto:Data: 2024-04-12T03:47:17.490 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.491 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.491 DEBUG:boto:Params: {} 2024-04-12T03:47:17.491 DEBUG:boto:Token: None 2024-04-12T03:47:17.491 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XUO9lg5kdpmESu2uiJyh4wTXfl8= 2024-04-12T03:47:17.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XUO9lg5kdpmESu2uiJyh4wTXfl8=', 'Content-Length': '0'} 2024-04-12T03:47:17.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000742224e78ac2e0c8-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.493 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.493 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.493 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.493 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.493 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.493 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.493 DEBUG:boto:Data: 2024-04-12T03:47:17.493 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.494 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.494 DEBUG:boto:Params: {} 2024-04-12T03:47:17.494 DEBUG:boto:Token: None 2024-04-12T03:47:17.494 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.494 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XUO9lg5kdpmESu2uiJyh4wTXfl8= 2024-04-12T03:47:17.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XUO9lg5kdpmESu2uiJyh4wTXfl8=', 'Content-Length': '0'} 2024-04-12T03:47:17.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca94c6e630cc90c4-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:47:17.496 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.496 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.496 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.496 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.496 DEBUG:boto:Method: GET 2024-04-12T03:47:17.497 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.497 DEBUG:boto:Data: 2024-04-12T03:47:17.497 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.497 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.497 DEBUG:boto:Params: {} 2024-04-12T03:47:17.497 DEBUG:boto:Token: None 2024-04-12T03:47:17.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.497 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE= 2024-04-12T03:47:17.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE=', 'Content-Length': '0'} 2024-04-12T03:47:17.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9e381c1a76439a-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.499 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.500 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.500 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.500 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.500 DEBUG:boto:Method: GET 2024-04-12T03:47:17.500 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.500 DEBUG:boto:Data: 2024-04-12T03:47:17.500 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.500 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.500 DEBUG:boto:Params: {} 2024-04-12T03:47:17.500 DEBUG:boto:Token: None 2024-04-12T03:47:17.501 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:47:17.501 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE= 2024-04-12T03:47:17.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WVcmlyrgGMpwOYLKTdYxJ/3rCE=', 'Content-Length': '0'} 2024-04-12T03:47:17.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84a805c653b3a76-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.503 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.503 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.503 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.503 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.503 DEBUG:boto:Method: GET 2024-04-12T03:47:17.503 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.503 DEBUG:boto:Data: 2024-04-12T03:47:17.503 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.503 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.504 DEBUG:boto:Params: {} 2024-04-12T03:47:17.504 DEBUG:boto:Token: None 2024-04-12T03:47:17.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcdce61f219f7285-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.554 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.555 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.555 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.555 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.555 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.555 DEBUG:boto:Method: GET 2024-04-12T03:47:17.555 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.555 DEBUG:boto:Data: 2024-04-12T03:47:17.555 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.555 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.556 DEBUG:boto:Params: {} 2024-04-12T03:47:17.556 DEBUG:boto:Token: None 2024-04-12T03:47:17.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccdd309067ef792-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.559 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.560 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.560 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.560 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.560 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:47:17.560 DEBUG:boto:Data: 2024-04-12T03:47:17.560 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.560 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.560 DEBUG:boto:Params: {} 2024-04-12T03:47:17.560 DEBUG:boto:Token: None 2024-04-12T03:47:17.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19 2024-04-12T03:47:17.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UO5OL+Yb2U/DZJJ7+q0N1PH+xhI= 2024-04-12T03:47:17.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UO5OL+Yb2U/DZJJ7+q0N1PH+xhI=', 'Content-Length': '0'} 2024-04-12T03:47:17.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401768ae8156d647-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.562 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.563 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.563 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.563 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:47:17.563 DEBUG:boto:Data: 2024-04-12T03:47:17.563 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.563 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.563 DEBUG:boto:Params: {} 2024-04-12T03:47:17.563 DEBUG:boto:Token: None 2024-04-12T03:47:17.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19 2024-04-12T03:47:17.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UO5OL+Yb2U/DZJJ7+q0N1PH+xhI= 2024-04-12T03:47:17.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UO5OL+Yb2U/DZJJ7+q0N1PH+xhI=', 'Content-Length': '0'} 2024-04-12T03:47:17.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda384043dd5eaa2-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:47:17.566 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.566 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.566 DEBUG:boto:Method: GET 2024-04-12T03:47:17.566 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:47:17.566 DEBUG:boto:Data: 2024-04-12T03:47:17.566 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.566 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.567 DEBUG:boto:Params: {} 2024-04-12T03:47:17.567 DEBUG:boto:Token: None 2024-04-12T03:47:17.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19 2024-04-12T03:47:17.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jA+SZYIz9Ef6E4A/OjxzUR+jLF8= 2024-04-12T03:47:17.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jA+SZYIz9Ef6E4A/OjxzUR+jLF8=', 'Content-Length': '0'} 2024-04-12T03:47:17.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d458cc98d260d987-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.569 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:47:17.569 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:47:17.569 DEBUG:boto:Method: GET 2024-04-12T03:47:17.569 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:47:17.569 DEBUG:boto:Data: 2024-04-12T03:47:17.569 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.569 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.569 DEBUG:boto:Params: {} 2024-04-12T03:47:17.570 DEBUG:boto:Token: None 2024-04-12T03:47:17.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj19 2024-04-12T03:47:17.570 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jA+SZYIz9Ef6E4A/OjxzUR+jLF8= 2024-04-12T03:47:17.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jA+SZYIz9Ef6E4A/OjxzUR+jLF8=', 'Content-Length': '0'} 2024-04-12T03:47:17.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f617c87fce7d4390-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.614 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.614 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.615 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.615 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.615 DEBUG:boto:Method: GET 2024-04-12T03:47:17.615 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.615 DEBUG:boto:Data: 2024-04-12T03:47:17.615 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.615 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.615 DEBUG:boto:Params: {} 2024-04-12T03:47:17.615 DEBUG:boto:Token: None 2024-04-12T03:47:17.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c424c7ce84e3c6df-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.618 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.618 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.619 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.619 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.619 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.619 DEBUG:boto:Method: GET 2024-04-12T03:47:17.619 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:47:17.619 DEBUG:boto:Data: 2024-04-12T03:47:17.619 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.619 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.619 DEBUG:boto:Params: {} 2024-04-12T03:47:17.619 DEBUG:boto:Token: None 2024-04-12T03:47:17.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db581f797faf0d1-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.623 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:47:17.623 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.623 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.623 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.623 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.623 DEBUG:boto:Method: GET 2024-04-12T03:47:17.624 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.624 DEBUG:boto:Data: 2024-04-12T03:47:17.624 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.624 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.624 DEBUG:boto:Params: {} 2024-04-12T03:47:17.624 DEBUG:boto:Token: None 2024-04-12T03:47:17.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g= 2024-04-12T03:47:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g=', 'Content-Length': '0'} 2024-04-12T03:47:17.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4ff6b4385e7f1a-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.626 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.626 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.626 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.627 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.627 DEBUG:boto:Method: GET 2024-04-12T03:47:17.627 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.627 DEBUG:boto:Data: 2024-04-12T03:47:17.627 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.627 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.627 DEBUG:boto:Params: {} 2024-04-12T03:47:17.627 DEBUG:boto:Token: None 2024-04-12T03:47:17.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g= 2024-04-12T03:47:17.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g=', 'Content-Length': '0'} 2024-04-12T03:47:17.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3666b20a4705448-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.630 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.630 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.630 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.630 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.630 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.630 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.630 DEBUG:boto:Data: 2024-04-12T03:47:17.630 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.631 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.631 DEBUG:boto:Params: {} 2024-04-12T03:47:17.631 DEBUG:boto:Token: None 2024-04-12T03:47:17.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.631 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/RHn1edPAXiA8bU0ke8qA/yOs20= 2024-04-12T03:47:17.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/RHn1edPAXiA8bU0ke8qA/yOs20=', 'Content-Length': '0'} 2024-04-12T03:47:17.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492a2494b6e7e211-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.633 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.633 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.633 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.633 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.633 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.633 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.634 DEBUG:boto:Data: 2024-04-12T03:47:17.634 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.634 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.634 DEBUG:boto:Params: {} 2024-04-12T03:47:17.634 DEBUG:boto:Token: None 2024-04-12T03:47:17.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.634 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/RHn1edPAXiA8bU0ke8qA/yOs20= 2024-04-12T03:47:17.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/RHn1edPAXiA8bU0ke8qA/yOs20=', 'Content-Length': '0'} 2024-04-12T03:47:17.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fec1c596ce7c10-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:47:17.636 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.636 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.636 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.636 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.637 DEBUG:boto:Method: GET 2024-04-12T03:47:17.637 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.637 DEBUG:boto:Data: 2024-04-12T03:47:17.637 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.637 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.637 DEBUG:boto:Params: {} 2024-04-12T03:47:17.637 DEBUG:boto:Token: None 2024-04-12T03:47:17.637 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g= 2024-04-12T03:47:17.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g=', 'Content-Length': '0'} 2024-04-12T03:47:17.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025100987eb83c0fa-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.640 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.640 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.640 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.640 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.640 DEBUG:boto:Method: GET 2024-04-12T03:47:17.640 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.640 DEBUG:boto:Data: 2024-04-12T03:47:17.640 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.640 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.640 DEBUG:boto:Params: {} 2024-04-12T03:47:17.641 DEBUG:boto:Token: None 2024-04-12T03:47:17.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:47:17.641 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g= 2024-04-12T03:47:17.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OzozfXvbHK2S7u1L+Q2Y57i3m2g=', 'Content-Length': '0'} 2024-04-12T03:47:17.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693d532c4d4b42e8-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.643 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.643 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.643 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.643 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.643 DEBUG:boto:Method: GET 2024-04-12T03:47:17.643 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.643 DEBUG:boto:Data: 2024-04-12T03:47:17.643 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.643 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.643 DEBUG:boto:Params: {} 2024-04-12T03:47:17.644 DEBUG:boto:Token: None 2024-04-12T03:47:17.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069c436b5fab8a757-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.646 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.647 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.647 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.647 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.647 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.647 DEBUG:boto:Method: GET 2024-04-12T03:47:17.647 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.647 DEBUG:boto:Data: 2024-04-12T03:47:17.647 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.647 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.648 DEBUG:boto:Params: {} 2024-04-12T03:47:17.648 DEBUG:boto:Token: None 2024-04-12T03:47:17.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464eff0f3f5bb60c-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.694 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.695 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.695 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.695 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.695 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:47:17.695 DEBUG:boto:Data: 2024-04-12T03:47:17.695 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.695 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.695 DEBUG:boto:Params: {} 2024-04-12T03:47:17.696 DEBUG:boto:Token: None 2024-04-12T03:47:17.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22 2024-04-12T03:47:17.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TgWVrCXFq3YKPw4zrkREF2yWwXc= 2024-04-12T03:47:17.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TgWVrCXFq3YKPw4zrkREF2yWwXc=', 'Content-Length': '0'} 2024-04-12T03:47:17.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add8e3143ce79f4a-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.698 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.698 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.698 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.698 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:47:17.698 DEBUG:boto:Data: 2024-04-12T03:47:17.698 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.698 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.698 DEBUG:boto:Params: {} 2024-04-12T03:47:17.698 DEBUG:boto:Token: None 2024-04-12T03:47:17.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22 2024-04-12T03:47:17.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TgWVrCXFq3YKPw4zrkREF2yWwXc= 2024-04-12T03:47:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TgWVrCXFq3YKPw4zrkREF2yWwXc=', 'Content-Length': '0'} 2024-04-12T03:47:17.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002abeeeb10f56414e-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:47:17.701 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.701 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.701 DEBUG:boto:Method: GET 2024-04-12T03:47:17.701 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:47:17.701 DEBUG:boto:Data: 2024-04-12T03:47:17.701 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.702 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.702 DEBUG:boto:Params: {} 2024-04-12T03:47:17.702 DEBUG:boto:Token: None 2024-04-12T03:47:17.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22 2024-04-12T03:47:17.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1SbmpkVdl/lFO1SYQzhlq5bFYsw= 2024-04-12T03:47:17.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SbmpkVdl/lFO1SYQzhlq5bFYsw=', 'Content-Length': '0'} 2024-04-12T03:47:17.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103cadbdbe728394-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.704 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:47:17.704 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:47:17.704 DEBUG:boto:Method: GET 2024-04-12T03:47:17.704 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:47:17.704 DEBUG:boto:Data: 2024-04-12T03:47:17.704 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.705 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.705 DEBUG:boto:Params: {} 2024-04-12T03:47:17.705 DEBUG:boto:Token: None 2024-04-12T03:47:17.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj22 2024-04-12T03:47:17.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1SbmpkVdl/lFO1SYQzhlq5bFYsw= 2024-04-12T03:47:17.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SbmpkVdl/lFO1SYQzhlq5bFYsw=', 'Content-Length': '0'} 2024-04-12T03:47:17.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301a8e1c7e665e60-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.707 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.707 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.707 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.707 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.708 DEBUG:boto:Method: GET 2024-04-12T03:47:17.708 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.708 DEBUG:boto:Data: 2024-04-12T03:47:17.708 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.708 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.708 DEBUG:boto:Params: {} 2024-04-12T03:47:17.708 DEBUG:boto:Token: None 2024-04-12T03:47:17.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad531bdda180be0-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.711 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.711 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.712 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.712 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.712 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.712 DEBUG:boto:Method: GET 2024-04-12T03:47:17.712 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:47:17.712 DEBUG:boto:Data: 2024-04-12T03:47:17.712 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.712 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.712 DEBUG:boto:Params: {} 2024-04-12T03:47:17.713 DEBUG:boto:Token: None 2024-04-12T03:47:17.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.713 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d03fb0e2ca0c62b-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.716 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:47:17.716 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.716 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.716 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.717 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.717 DEBUG:boto:Method: GET 2024-04-12T03:47:17.717 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.717 DEBUG:boto:Data: 2024-04-12T03:47:17.717 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.717 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.717 DEBUG:boto:Params: {} 2024-04-12T03:47:17.717 DEBUG:boto:Token: None 2024-04-12T03:47:17.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk= 2024-04-12T03:47:17.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk=', 'Content-Length': '0'} 2024-04-12T03:47:17.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1bf21e8ef9d6ffc-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.766 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.766 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.766 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.767 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.767 DEBUG:boto:Method: GET 2024-04-12T03:47:17.767 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.767 DEBUG:boto:Data: 2024-04-12T03:47:17.767 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.767 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.767 DEBUG:boto:Params: {} 2024-04-12T03:47:17.767 DEBUG:boto:Token: None 2024-04-12T03:47:17.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk= 2024-04-12T03:47:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk=', 'Content-Length': '0'} 2024-04-12T03:47:17.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7efb29fb3b3fa0d-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.770 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.770 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.770 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.770 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.770 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.770 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.770 DEBUG:boto:Data: 2024-04-12T03:47:17.770 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.770 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.770 DEBUG:boto:Params: {} 2024-04-12T03:47:17.771 DEBUG:boto:Token: None 2024-04-12T03:47:17.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EdU/lY2Vc+sT/wPFY3GDaGFOLXE= 2024-04-12T03:47:17.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EdU/lY2Vc+sT/wPFY3GDaGFOLXE=', 'Content-Length': '0'} 2024-04-12T03:47:17.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c326d62ec8d70d9a-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.773 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.773 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.773 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.773 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.773 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.773 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.773 DEBUG:boto:Data: 2024-04-12T03:47:17.774 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.774 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.774 DEBUG:boto:Params: {} 2024-04-12T03:47:17.774 DEBUG:boto:Token: None 2024-04-12T03:47:17.774 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.774 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EdU/lY2Vc+sT/wPFY3GDaGFOLXE= 2024-04-12T03:47:17.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EdU/lY2Vc+sT/wPFY3GDaGFOLXE=', 'Content-Length': '0'} 2024-04-12T03:47:17.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067fe6cd1413c1a53-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:47:17.776 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.776 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.776 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.777 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.777 DEBUG:boto:Method: GET 2024-04-12T03:47:17.777 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.777 DEBUG:boto:Data: 2024-04-12T03:47:17.777 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.777 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.777 DEBUG:boto:Params: {} 2024-04-12T03:47:17.777 DEBUG:boto:Token: None 2024-04-12T03:47:17.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.778 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk= 2024-04-12T03:47:17.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk=', 'Content-Length': '0'} 2024-04-12T03:47:17.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc82441ba684d2a5-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.779 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.780 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.780 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.780 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.780 DEBUG:boto:Method: GET 2024-04-12T03:47:17.780 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.780 DEBUG:boto:Data: 2024-04-12T03:47:17.780 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.780 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.780 DEBUG:boto:Params: {} 2024-04-12T03:47:17.780 DEBUG:boto:Token: None 2024-04-12T03:47:17.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:47:17.781 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk= 2024-04-12T03:47:17.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yv2y7TTeyYJkER0UvHSLWRoibAk=', 'Content-Length': '0'} 2024-04-12T03:47:17.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000058acd0fa10e7c5-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.783 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.783 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.783 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.783 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.783 DEBUG:boto:Method: GET 2024-04-12T03:47:17.783 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.783 DEBUG:boto:Data: 2024-04-12T03:47:17.783 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.784 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.784 DEBUG:boto:Params: {} 2024-04-12T03:47:17.784 DEBUG:boto:Token: None 2024-04-12T03:47:17.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b66757289b7bb37e-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.786 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.787 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.787 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.787 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.787 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.787 DEBUG:boto:Method: GET 2024-04-12T03:47:17.787 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.787 DEBUG:boto:Data: 2024-04-12T03:47:17.787 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.788 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.788 DEBUG:boto:Params: {} 2024-04-12T03:47:17.788 DEBUG:boto:Token: None 2024-04-12T03:47:17.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de36555d89673b98-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.838 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.839 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.839 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.839 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.839 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:47:17.839 DEBUG:boto:Data: 2024-04-12T03:47:17.839 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.839 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.839 DEBUG:boto:Params: {} 2024-04-12T03:47:17.840 DEBUG:boto:Token: None 2024-04-12T03:47:17.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23 2024-04-12T03:47:17.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e355i5bRWImxUqC9knrik74QiXg= 2024-04-12T03:47:17.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e355i5bRWImxUqC9knrik74QiXg=', 'Content-Length': '0'} 2024-04-12T03:47:17.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178a9b903bac4311-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.842 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.842 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.842 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.842 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:47:17.842 DEBUG:boto:Data: 2024-04-12T03:47:17.842 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.843 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.843 DEBUG:boto:Params: {} 2024-04-12T03:47:17.843 DEBUG:boto:Token: None 2024-04-12T03:47:17.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23 2024-04-12T03:47:17.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e355i5bRWImxUqC9knrik74QiXg= 2024-04-12T03:47:17.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e355i5bRWImxUqC9knrik74QiXg=', 'Content-Length': '0'} 2024-04-12T03:47:17.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fb543e74c4097e-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:47:17.845 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.845 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.845 DEBUG:boto:Method: GET 2024-04-12T03:47:17.845 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:47:17.845 DEBUG:boto:Data: 2024-04-12T03:47:17.845 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.846 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.846 DEBUG:boto:Params: {} 2024-04-12T03:47:17.846 DEBUG:boto:Token: None 2024-04-12T03:47:17.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23 2024-04-12T03:47:17.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:43oUXILxeCjmMTRdlytkSB7WF7w= 2024-04-12T03:47:17.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:43oUXILxeCjmMTRdlytkSB7WF7w=', 'Content-Length': '0'} 2024-04-12T03:47:17.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f6f0da2ea2d9be-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.848 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:47:17.848 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:47:17.848 DEBUG:boto:Method: GET 2024-04-12T03:47:17.848 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:47:17.848 DEBUG:boto:Data: 2024-04-12T03:47:17.849 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.849 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.849 DEBUG:boto:Params: {} 2024-04-12T03:47:17.849 DEBUG:boto:Token: None 2024-04-12T03:47:17.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj23 2024-04-12T03:47:17.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:43oUXILxeCjmMTRdlytkSB7WF7w= 2024-04-12T03:47:17.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:43oUXILxeCjmMTRdlytkSB7WF7w=', 'Content-Length': '0'} 2024-04-12T03:47:17.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ac88196586c4d6-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.851 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.851 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.851 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.851 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.852 DEBUG:boto:Method: GET 2024-04-12T03:47:17.852 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.852 DEBUG:boto:Data: 2024-04-12T03:47:17.852 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.852 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.852 DEBUG:boto:Params: {} 2024-04-12T03:47:17.852 DEBUG:boto:Token: None 2024-04-12T03:47:17.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088fd51e8ba25fec3-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.854 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.855 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.855 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.855 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.855 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.855 DEBUG:boto:Method: GET 2024-04-12T03:47:17.855 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:47:17.855 DEBUG:boto:Data: 2024-04-12T03:47:17.855 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.855 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.856 DEBUG:boto:Params: {} 2024-04-12T03:47:17.856 DEBUG:boto:Token: None 2024-04-12T03:47:17.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df8881b177d6c795-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.859 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:47:17.859 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.859 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.859 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.860 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.860 DEBUG:boto:Method: GET 2024-04-12T03:47:17.860 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.860 DEBUG:boto:Data: 2024-04-12T03:47:17.860 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.860 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.860 DEBUG:boto:Params: {} 2024-04-12T03:47:17.860 DEBUG:boto:Token: None 2024-04-12T03:47:17.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY= 2024-04-12T03:47:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY=', 'Content-Length': '0'} 2024-04-12T03:47:17.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21259161ce76a53-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.906 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.906 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.906 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.907 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.907 DEBUG:boto:Method: GET 2024-04-12T03:47:17.907 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.907 DEBUG:boto:Data: 2024-04-12T03:47:17.907 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.907 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.907 DEBUG:boto:Params: {} 2024-04-12T03:47:17.907 DEBUG:boto:Token: None 2024-04-12T03:47:17.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY= 2024-04-12T03:47:17.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY=', 'Content-Length': '0'} 2024-04-12T03:47:17.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76864e8cfd3e434-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.909 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.910 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.910 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.910 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.910 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.910 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.910 DEBUG:boto:Data: 2024-04-12T03:47:17.910 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.910 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.910 DEBUG:boto:Params: {} 2024-04-12T03:47:17.911 DEBUG:boto:Token: None 2024-04-12T03:47:17.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E2lLRX0X9ns1BTq/LOlKmAYR/bk= 2024-04-12T03:47:17.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E2lLRX0X9ns1BTq/LOlKmAYR/bk=', 'Content-Length': '0'} 2024-04-12T03:47:17.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0fd2516b32b0554-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.913 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.913 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.913 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.913 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.913 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.913 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.913 DEBUG:boto:Data: 2024-04-12T03:47:17.913 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.913 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.913 DEBUG:boto:Params: {} 2024-04-12T03:47:17.914 DEBUG:boto:Token: None 2024-04-12T03:47:17.914 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.914 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E2lLRX0X9ns1BTq/LOlKmAYR/bk= 2024-04-12T03:47:17.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E2lLRX0X9ns1BTq/LOlKmAYR/bk=', 'Content-Length': '0'} 2024-04-12T03:47:17.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e568464dac21b859-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:47:17.916 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.916 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.916 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.916 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.916 DEBUG:boto:Method: GET 2024-04-12T03:47:17.916 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.916 DEBUG:boto:Data: 2024-04-12T03:47:17.917 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.917 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.917 DEBUG:boto:Params: {} 2024-04-12T03:47:17.917 DEBUG:boto:Token: None 2024-04-12T03:47:17.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY= 2024-04-12T03:47:17.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY=', 'Content-Length': '0'} 2024-04-12T03:47:17.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008968c57e555ae8e8-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.919 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.919 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.919 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.919 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.919 DEBUG:boto:Method: GET 2024-04-12T03:47:17.919 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.919 DEBUG:boto:Data: 2024-04-12T03:47:17.920 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.920 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.920 DEBUG:boto:Params: {} 2024-04-12T03:47:17.920 DEBUG:boto:Token: None 2024-04-12T03:47:17.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:47:17.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY= 2024-04-12T03:47:17.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zcpIe6SK2fD7imG2U6exoJ5/7mY=', 'Content-Length': '0'} 2024-04-12T03:47:17.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446112f970c6755b-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.922 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.922 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.922 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.922 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.923 DEBUG:boto:Method: GET 2024-04-12T03:47:17.923 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.923 DEBUG:boto:Data: 2024-04-12T03:47:17.923 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.923 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.923 DEBUG:boto:Params: {} 2024-04-12T03:47:17.923 DEBUG:boto:Token: None 2024-04-12T03:47:17.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7910bfe29132a7d-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.926 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.926 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.926 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.926 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.926 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.926 DEBUG:boto:Method: GET 2024-04-12T03:47:17.927 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.927 DEBUG:boto:Data: 2024-04-12T03:47:17.927 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.927 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.927 DEBUG:boto:Params: {} 2024-04-12T03:47:17.927 DEBUG:boto:Token: None 2024-04-12T03:47:17.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bcea58b2885849-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.930 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.931 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.931 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.931 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.931 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:47:17.931 DEBUG:boto:Data: 2024-04-12T03:47:17.931 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.931 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.931 DEBUG:boto:Params: {} 2024-04-12T03:47:17.931 DEBUG:boto:Token: None 2024-04-12T03:47:17.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33 2024-04-12T03:47:17.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Xv+naOjDO/4bgJTgNeeE6gF0XE= 2024-04-12T03:47:17.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Xv+naOjDO/4bgJTgNeeE6gF0XE=', 'Content-Length': '0'} 2024-04-12T03:47:17.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd07b9509812d380-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.933 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.933 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.933 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.934 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:47:17.934 DEBUG:boto:Data: 2024-04-12T03:47:17.934 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.934 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.934 DEBUG:boto:Params: {} 2024-04-12T03:47:17.934 DEBUG:boto:Token: None 2024-04-12T03:47:17.934 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33 2024-04-12T03:47:17.934 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Xv+naOjDO/4bgJTgNeeE6gF0XE= 2024-04-12T03:47:17.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Xv+naOjDO/4bgJTgNeeE6gF0XE=', 'Content-Length': '0'} 2024-04-12T03:47:17.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008131f7ebea517ca2-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:47:17.936 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.936 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.936 DEBUG:boto:Method: GET 2024-04-12T03:47:17.936 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:47:17.936 DEBUG:boto:Data: 2024-04-12T03:47:17.937 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.937 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.937 DEBUG:boto:Params: {} 2024-04-12T03:47:17.937 DEBUG:boto:Token: None 2024-04-12T03:47:17.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33 2024-04-12T03:47:17.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmwUNimvo4BH+nlEaUhEnHmubS0= 2024-04-12T03:47:17.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmwUNimvo4BH+nlEaUhEnHmubS0=', 'Content-Length': '0'} 2024-04-12T03:47:17.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a43f5667e3911c7-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.939 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:47:17.939 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:47:17.939 DEBUG:boto:Method: GET 2024-04-12T03:47:17.939 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:47:17.939 DEBUG:boto:Data: 2024-04-12T03:47:17.939 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.939 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.939 DEBUG:boto:Params: {} 2024-04-12T03:47:17.940 DEBUG:boto:Token: None 2024-04-12T03:47:17.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj33 2024-04-12T03:47:17.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZmwUNimvo4BH+nlEaUhEnHmubS0= 2024-04-12T03:47:17.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZmwUNimvo4BH+nlEaUhEnHmubS0=', 'Content-Length': '0'} 2024-04-12T03:47:17.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5adbe446cc9cfb-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.982 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.982 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.983 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.983 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.983 DEBUG:boto:Method: GET 2024-04-12T03:47:17.983 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.983 DEBUG:boto:Data: 2024-04-12T03:47:17.983 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.983 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.983 DEBUG:boto:Params: {} 2024-04-12T03:47:17.983 DEBUG:boto:Token: None 2024-04-12T03:47:17.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055456b09abd86cf4-006618aec5-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.986 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.986 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:17.986 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:17.986 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.987 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.987 DEBUG:boto:Method: GET 2024-04-12T03:47:17.987 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:47:17.987 DEBUG:boto:Data: 2024-04-12T03:47:17.987 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.987 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.987 DEBUG:boto:Params: {} 2024-04-12T03:47:17.987 DEBUG:boto:Token: None 2024-04-12T03:47:17.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/ 2024-04-12T03:47:17.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM= 2024-04-12T03:47:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0KQQjoygogSIkUm6j0BxJUwmsrM=', 'Content-Length': '0'} 2024-04-12T03:47:17.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d350a5df5b496d-006618aec5-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.990 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:17.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:47:17.991 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:17.991 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:17.991 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.991 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.991 DEBUG:boto:Method: GET 2024-04-12T03:47:17.991 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.991 DEBUG:boto:Data: 2024-04-12T03:47:17.992 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.992 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.992 DEBUG:boto:Params: {} 2024-04-12T03:47:17.992 DEBUG:boto:Token: None 2024-04-12T03:47:17.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t8wEYHJuTZtiN2vpuvE7NYm8ZNQ= 2024-04-12T03:47:17.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t8wEYHJuTZtiN2vpuvE7NYm8ZNQ=', 'Content-Length': '0'} 2024-04-12T03:47:17.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48f6dfcb6c15a98-006618aec5-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.994 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:17.994 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:17.994 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.994 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.994 DEBUG:boto:Method: GET 2024-04-12T03:47:17.994 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.995 DEBUG:boto:Data: 2024-04-12T03:47:17.995 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:17.995 DEBUG:boto:Port: 8000 2024-04-12T03:47:17.995 DEBUG:boto:Params: {} 2024-04-12T03:47:17.995 DEBUG:boto:Token: None 2024-04-12T03:47:17.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t8wEYHJuTZtiN2vpuvE7NYm8ZNQ= 2024-04-12T03:47:17.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t8wEYHJuTZtiN2vpuvE7NYm8ZNQ=', 'Content-Length': '0'} 2024-04-12T03:47:17.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024718b83a3374730-006618aec5-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:17.997 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:17.997 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:17.997 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.997 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.997 DEBUG:boto:Method: HEAD 2024-04-12T03:47:17.998 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.998 DEBUG:boto:Data: 2024-04-12T03:47:17.998 DEBUG:boto:Headers: {} 2024-04-12T03:47:17.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:17.998 DEBUG:boto:Port: 8001 2024-04-12T03:47:17.998 DEBUG:boto:Params: {} 2024-04-12T03:47:17.998 DEBUG:boto:Token: None 2024-04-12T03:47:17.998 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:17 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:17.998 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2je3Yv1iauHb3IpYJkONMUAoawA= 2024-04-12T03:47:17.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2je3Yv1iauHb3IpYJkONMUAoawA=', 'Content-Length': '0'} 2024-04-12T03:47:18.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2e7ec38e4ef3ec5-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.000 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.000 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.000 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.000 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.001 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.001 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.001 DEBUG:boto:Data: 2024-04-12T03:47:18.001 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.001 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.001 DEBUG:boto:Params: {} 2024-04-12T03:47:18.001 DEBUG:boto:Token: None 2024-04-12T03:47:18.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Icl4+T3If+5t9W3uIsnoy1ZBWyM= 2024-04-12T03:47:18.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Icl4+T3If+5t9W3uIsnoy1ZBWyM=', 'Content-Length': '0'} 2024-04-12T03:47:18.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004318839ff071f605-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:47:18.003 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.003 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.004 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.004 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.004 DEBUG:boto:Method: GET 2024-04-12T03:47:18.004 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.004 DEBUG:boto:Data: 2024-04-12T03:47:18.004 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.004 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.004 DEBUG:boto:Params: {} 2024-04-12T03:47:18.004 DEBUG:boto:Token: None 2024-04-12T03:47:18.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.005 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cdscq4drmTfdmhN9LeErz6rCcRQ= 2024-04-12T03:47:18.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cdscq4drmTfdmhN9LeErz6rCcRQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac05d4a0b5805588-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.007 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.008 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.008 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.008 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.008 DEBUG:boto:Method: GET 2024-04-12T03:47:18.008 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.008 DEBUG:boto:Data: 2024-04-12T03:47:18.008 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.008 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.008 DEBUG:boto:Params: {} 2024-04-12T03:47:18.008 DEBUG:boto:Token: None 2024-04-12T03:47:18.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:47:18.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cdscq4drmTfdmhN9LeErz6rCcRQ= 2024-04-12T03:47:18.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cdscq4drmTfdmhN9LeErz6rCcRQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611c592b93d4b4ed-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.054 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.054 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.055 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.055 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.055 DEBUG:boto:Method: GET 2024-04-12T03:47:18.055 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.055 DEBUG:boto:Data: 2024-04-12T03:47:18.055 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.055 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.055 DEBUG:boto:Params: {} 2024-04-12T03:47:18.055 DEBUG:boto:Token: None 2024-04-12T03:47:18.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b983424bce115a-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.058 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.058 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.058 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.058 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.059 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.059 DEBUG:boto:Method: GET 2024-04-12T03:47:18.059 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.059 DEBUG:boto:Data: 2024-04-12T03:47:18.059 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.059 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.059 DEBUG:boto:Params: {} 2024-04-12T03:47:18.059 DEBUG:boto:Token: None 2024-04-12T03:47:18.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f91c0fe15e40cd57-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.063 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.063 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.063 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.063 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.063 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:47:18.063 DEBUG:boto:Data: 2024-04-12T03:47:18.064 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.064 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.064 DEBUG:boto:Params: {} 2024-04-12T03:47:18.064 DEBUG:boto:Token: None 2024-04-12T03:47:18.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34 2024-04-12T03:47:18.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XuK6EFWSiHEJbDSe2ga53nqYpdU= 2024-04-12T03:47:18.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XuK6EFWSiHEJbDSe2ga53nqYpdU=', 'Content-Length': '0'} 2024-04-12T03:47:18.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003870c71386794353-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.066 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.066 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.066 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.066 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:47:18.066 DEBUG:boto:Data: 2024-04-12T03:47:18.066 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.067 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.067 DEBUG:boto:Params: {} 2024-04-12T03:47:18.067 DEBUG:boto:Token: None 2024-04-12T03:47:18.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34 2024-04-12T03:47:18.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XuK6EFWSiHEJbDSe2ga53nqYpdU= 2024-04-12T03:47:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XuK6EFWSiHEJbDSe2ga53nqYpdU=', 'Content-Length': '0'} 2024-04-12T03:47:18.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb23168facb626f-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:47:18.069 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.069 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.069 DEBUG:boto:Method: GET 2024-04-12T03:47:18.069 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:47:18.069 DEBUG:boto:Data: 2024-04-12T03:47:18.070 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.070 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.070 DEBUG:boto:Params: {} 2024-04-12T03:47:18.070 DEBUG:boto:Token: None 2024-04-12T03:47:18.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34 2024-04-12T03:47:18.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n6L4RP8m+7IkpRrQQlep2d983dw= 2024-04-12T03:47:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n6L4RP8m+7IkpRrQQlep2d983dw=', 'Content-Length': '0'} 2024-04-12T03:47:18.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a323f3748e00910-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.072 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:47:18.072 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:47:18.072 DEBUG:boto:Method: GET 2024-04-12T03:47:18.072 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:47:18.072 DEBUG:boto:Data: 2024-04-12T03:47:18.072 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.073 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.073 DEBUG:boto:Params: {} 2024-04-12T03:47:18.073 DEBUG:boto:Token: None 2024-04-12T03:47:18.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj34 2024-04-12T03:47:18.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n6L4RP8m+7IkpRrQQlep2d983dw= 2024-04-12T03:47:18.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n6L4RP8m+7IkpRrQQlep2d983dw=', 'Content-Length': '0'} 2024-04-12T03:47:18.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55782db53453f36-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.075 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.075 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.075 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.075 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.075 DEBUG:boto:Method: GET 2024-04-12T03:47:18.075 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.076 DEBUG:boto:Data: 2024-04-12T03:47:18.076 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.076 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.076 DEBUG:boto:Params: {} 2024-04-12T03:47:18.076 DEBUG:boto:Token: None 2024-04-12T03:47:18.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ac6d0cd5a51d09b-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.078 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.079 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.079 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.079 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.079 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.079 DEBUG:boto:Method: GET 2024-04-12T03:47:18.079 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:47:18.079 DEBUG:boto:Data: 2024-04-12T03:47:18.079 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.079 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.080 DEBUG:boto:Params: {} 2024-04-12T03:47:18.080 DEBUG:boto:Token: None 2024-04-12T03:47:18.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5b7de5e1e28c14-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.126 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:47:18.127 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.127 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.127 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.127 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.127 DEBUG:boto:Method: GET 2024-04-12T03:47:18.127 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.128 DEBUG:boto:Data: 2024-04-12T03:47:18.128 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.128 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.128 DEBUG:boto:Params: {} 2024-04-12T03:47:18.128 DEBUG:boto:Token: None 2024-04-12T03:47:18.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY= 2024-04-12T03:47:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY=', 'Content-Length': '0'} 2024-04-12T03:47:18.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071f72f726ebe95a-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.131 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.131 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.131 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.131 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.131 DEBUG:boto:Method: GET 2024-04-12T03:47:18.131 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.131 DEBUG:boto:Data: 2024-04-12T03:47:18.131 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.131 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.131 DEBUG:boto:Params: {} 2024-04-12T03:47:18.132 DEBUG:boto:Token: None 2024-04-12T03:47:18.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY= 2024-04-12T03:47:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY=', 'Content-Length': '0'} 2024-04-12T03:47:18.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc33189e8c7549c-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.134 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.134 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.134 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.134 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.134 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.134 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.134 DEBUG:boto:Data: 2024-04-12T03:47:18.134 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.135 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.135 DEBUG:boto:Params: {} 2024-04-12T03:47:18.135 DEBUG:boto:Token: None 2024-04-12T03:47:18.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VWCWcY1HXBF2rtuwSsOpIqj+25g= 2024-04-12T03:47:18.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VWCWcY1HXBF2rtuwSsOpIqj+25g=', 'Content-Length': '0'} 2024-04-12T03:47:18.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005069933bc4dd9c6c-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.137 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.137 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.137 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.137 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.137 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.137 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.137 DEBUG:boto:Data: 2024-04-12T03:47:18.137 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.137 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.137 DEBUG:boto:Params: {} 2024-04-12T03:47:18.138 DEBUG:boto:Token: None 2024-04-12T03:47:18.138 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VWCWcY1HXBF2rtuwSsOpIqj+25g= 2024-04-12T03:47:18.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VWCWcY1HXBF2rtuwSsOpIqj+25g=', 'Content-Length': '0'} 2024-04-12T03:47:18.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cb7b116ccce628-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:47:18.140 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.140 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.140 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.140 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.140 DEBUG:boto:Method: GET 2024-04-12T03:47:18.140 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.140 DEBUG:boto:Data: 2024-04-12T03:47:18.140 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.141 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.141 DEBUG:boto:Params: {} 2024-04-12T03:47:18.141 DEBUG:boto:Token: None 2024-04-12T03:47:18.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY= 2024-04-12T03:47:18.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY=', 'Content-Length': '0'} 2024-04-12T03:47:18.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32d9c65a23af169-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.190 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.190 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.190 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.191 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.191 DEBUG:boto:Method: GET 2024-04-12T03:47:18.191 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.191 DEBUG:boto:Data: 2024-04-12T03:47:18.191 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.191 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.191 DEBUG:boto:Params: {} 2024-04-12T03:47:18.191 DEBUG:boto:Token: None 2024-04-12T03:47:18.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:47:18.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY= 2024-04-12T03:47:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yPbF5L6dVaWebHJqNc/keQCmyaY=', 'Content-Length': '0'} 2024-04-12T03:47:18.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916d69cf86c82531-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.193 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.194 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.194 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.194 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.194 DEBUG:boto:Method: GET 2024-04-12T03:47:18.194 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.194 DEBUG:boto:Data: 2024-04-12T03:47:18.194 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.194 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.194 DEBUG:boto:Params: {} 2024-04-12T03:47:18.195 DEBUG:boto:Token: None 2024-04-12T03:47:18.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcaf5f56887e455b-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.197 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.197 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.198 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.198 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.198 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.198 DEBUG:boto:Method: GET 2024-04-12T03:47:18.198 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.198 DEBUG:boto:Data: 2024-04-12T03:47:18.198 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.198 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.198 DEBUG:boto:Params: {} 2024-04-12T03:47:18.198 DEBUG:boto:Token: None 2024-04-12T03:47:18.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ed744651262184-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.202 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.202 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.202 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.202 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.202 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:47:18.202 DEBUG:boto:Data: 2024-04-12T03:47:18.203 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.203 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.203 DEBUG:boto:Params: {} 2024-04-12T03:47:18.203 DEBUG:boto:Token: None 2024-04-12T03:47:18.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38 2024-04-12T03:47:18.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QdDPDdpoXyuxgnCJQOxyR8Ipo7s= 2024-04-12T03:47:18.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdDPDdpoXyuxgnCJQOxyR8Ipo7s=', 'Content-Length': '0'} 2024-04-12T03:47:18.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005598bb1665e3f666-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.205 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.205 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.205 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.205 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:47:18.205 DEBUG:boto:Data: 2024-04-12T03:47:18.206 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.206 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.206 DEBUG:boto:Params: {} 2024-04-12T03:47:18.206 DEBUG:boto:Token: None 2024-04-12T03:47:18.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38 2024-04-12T03:47:18.206 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QdDPDdpoXyuxgnCJQOxyR8Ipo7s= 2024-04-12T03:47:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdDPDdpoXyuxgnCJQOxyR8Ipo7s=', 'Content-Length': '0'} 2024-04-12T03:47:18.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74fc5f18f5d39a6-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:47:18.208 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.208 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.208 DEBUG:boto:Method: GET 2024-04-12T03:47:18.208 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:47:18.208 DEBUG:boto:Data: 2024-04-12T03:47:18.208 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.209 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.209 DEBUG:boto:Params: {} 2024-04-12T03:47:18.209 DEBUG:boto:Token: None 2024-04-12T03:47:18.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38 2024-04-12T03:47:18.209 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0FujmRS4/j/k9dliE//Le4l49vg= 2024-04-12T03:47:18.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0FujmRS4/j/k9dliE//Le4l49vg=', 'Content-Length': '0'} 2024-04-12T03:47:18.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4032bd7685d5cc0-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.211 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:47:18.211 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:47:18.211 DEBUG:boto:Method: GET 2024-04-12T03:47:18.211 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:47:18.211 DEBUG:boto:Data: 2024-04-12T03:47:18.211 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.211 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.211 DEBUG:boto:Params: {} 2024-04-12T03:47:18.212 DEBUG:boto:Token: None 2024-04-12T03:47:18.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj38 2024-04-12T03:47:18.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0FujmRS4/j/k9dliE//Le4l49vg= 2024-04-12T03:47:18.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0FujmRS4/j/k9dliE//Le4l49vg=', 'Content-Length': '0'} 2024-04-12T03:47:18.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1206edd4c1e2767-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.214 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.214 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.214 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.214 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.214 DEBUG:boto:Method: GET 2024-04-12T03:47:18.214 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.214 DEBUG:boto:Data: 2024-04-12T03:47:18.214 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.214 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.214 DEBUG:boto:Params: {} 2024-04-12T03:47:18.215 DEBUG:boto:Token: None 2024-04-12T03:47:18.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f8d979493bd3f5-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.217 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.217 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.218 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.218 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.218 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.218 DEBUG:boto:Method: GET 2024-04-12T03:47:18.218 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:47:18.218 DEBUG:boto:Data: 2024-04-12T03:47:18.218 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.218 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.218 DEBUG:boto:Params: {} 2024-04-12T03:47:18.218 DEBUG:boto:Token: None 2024-04-12T03:47:18.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000854d7b00ae5923eb-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.222 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:47:18.222 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.222 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.222 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.222 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.223 DEBUG:boto:Method: GET 2024-04-12T03:47:18.223 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.223 DEBUG:boto:Data: 2024-04-12T03:47:18.223 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.223 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.223 DEBUG:boto:Params: {} 2024-04-12T03:47:18.223 DEBUG:boto:Token: None 2024-04-12T03:47:18.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc= 2024-04-12T03:47:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc=', 'Content-Length': '0'} 2024-04-12T03:47:18.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00bcdaa6daa748d-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.225 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.225 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.225 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.226 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.226 DEBUG:boto:Method: GET 2024-04-12T03:47:18.226 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.226 DEBUG:boto:Data: 2024-04-12T03:47:18.226 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.226 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.226 DEBUG:boto:Params: {} 2024-04-12T03:47:18.226 DEBUG:boto:Token: None 2024-04-12T03:47:18.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc= 2024-04-12T03:47:18.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc=', 'Content-Length': '0'} 2024-04-12T03:47:18.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88efdfa85506b00-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.228 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.228 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.228 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.229 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.229 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.229 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.229 DEBUG:boto:Data: 2024-04-12T03:47:18.229 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.229 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.229 DEBUG:boto:Params: {} 2024-04-12T03:47:18.229 DEBUG:boto:Token: None 2024-04-12T03:47:18.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNFILFQW56p6U8c6F0W6YG28bjI= 2024-04-12T03:47:18.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNFILFQW56p6U8c6F0W6YG28bjI=', 'Content-Length': '0'} 2024-04-12T03:47:18.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f421e2b3e9fc31-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.233 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.234 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.234 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.234 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.234 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.234 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.234 DEBUG:boto:Data: 2024-04-12T03:47:18.234 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.234 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.234 DEBUG:boto:Params: {} 2024-04-12T03:47:18.235 DEBUG:boto:Token: None 2024-04-12T03:47:18.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JNFILFQW56p6U8c6F0W6YG28bjI= 2024-04-12T03:47:18.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JNFILFQW56p6U8c6F0W6YG28bjI=', 'Content-Length': '0'} 2024-04-12T03:47:18.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009642832cf85c57ad-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:47:18.237 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.237 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.237 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.237 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.237 DEBUG:boto:Method: GET 2024-04-12T03:47:18.237 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.237 DEBUG:boto:Data: 2024-04-12T03:47:18.237 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.238 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.238 DEBUG:boto:Params: {} 2024-04-12T03:47:18.238 DEBUG:boto:Token: None 2024-04-12T03:47:18.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc= 2024-04-12T03:47:18.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc=', 'Content-Length': '0'} 2024-04-12T03:47:18.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fea07cbdba60421-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.286 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.287 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.287 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.287 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.287 DEBUG:boto:Method: GET 2024-04-12T03:47:18.287 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.287 DEBUG:boto:Data: 2024-04-12T03:47:18.287 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.287 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.287 DEBUG:boto:Params: {} 2024-04-12T03:47:18.288 DEBUG:boto:Token: None 2024-04-12T03:47:18.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:47:18.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc= 2024-04-12T03:47:18.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4aJdVMu6q4IdzRhj75aBxSjjpoc=', 'Content-Length': '0'} 2024-04-12T03:47:18.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004446f7950d361cb3-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.290 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.290 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.290 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.290 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.290 DEBUG:boto:Method: GET 2024-04-12T03:47:18.291 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.291 DEBUG:boto:Data: 2024-04-12T03:47:18.291 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.291 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.291 DEBUG:boto:Params: {} 2024-04-12T03:47:18.291 DEBUG:boto:Token: None 2024-04-12T03:47:18.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006159f1f62ad27f35-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.294 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.294 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.294 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.294 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.294 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.295 DEBUG:boto:Method: GET 2024-04-12T03:47:18.295 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.295 DEBUG:boto:Data: 2024-04-12T03:47:18.295 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.295 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.295 DEBUG:boto:Params: {} 2024-04-12T03:47:18.295 DEBUG:boto:Token: None 2024-04-12T03:47:18.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0446cb1d768f163-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.298 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.299 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.299 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.299 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.299 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:47:18.299 DEBUG:boto:Data: 2024-04-12T03:47:18.299 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.299 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.299 DEBUG:boto:Params: {} 2024-04-12T03:47:18.299 DEBUG:boto:Token: None 2024-04-12T03:47:18.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44 2024-04-12T03:47:18.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eS5+TPfW4h+lYIv1Wk0MvKDYOcU= 2024-04-12T03:47:18.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eS5+TPfW4h+lYIv1Wk0MvKDYOcU=', 'Content-Length': '0'} 2024-04-12T03:47:18.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e6691e6c64fc29b-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.301 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.302 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.302 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.302 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:47:18.302 DEBUG:boto:Data: 2024-04-12T03:47:18.302 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.302 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.302 DEBUG:boto:Params: {} 2024-04-12T03:47:18.302 DEBUG:boto:Token: None 2024-04-12T03:47:18.302 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44 2024-04-12T03:47:18.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eS5+TPfW4h+lYIv1Wk0MvKDYOcU= 2024-04-12T03:47:18.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eS5+TPfW4h+lYIv1Wk0MvKDYOcU=', 'Content-Length': '0'} 2024-04-12T03:47:18.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9708ae411c4c91-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:47:18.305 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.305 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.305 DEBUG:boto:Method: GET 2024-04-12T03:47:18.305 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:47:18.305 DEBUG:boto:Data: 2024-04-12T03:47:18.305 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.305 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.305 DEBUG:boto:Params: {} 2024-04-12T03:47:18.306 DEBUG:boto:Token: None 2024-04-12T03:47:18.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44 2024-04-12T03:47:18.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ieirqDIFHWQJdRuUACMoMbgEmRQ= 2024-04-12T03:47:18.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ieirqDIFHWQJdRuUACMoMbgEmRQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750cc53ddea17e3a-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.308 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:47:18.308 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:47:18.308 DEBUG:boto:Method: GET 2024-04-12T03:47:18.308 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:47:18.308 DEBUG:boto:Data: 2024-04-12T03:47:18.309 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.309 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.309 DEBUG:boto:Params: {} 2024-04-12T03:47:18.309 DEBUG:boto:Token: None 2024-04-12T03:47:18.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj44 2024-04-12T03:47:18.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ieirqDIFHWQJdRuUACMoMbgEmRQ= 2024-04-12T03:47:18.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ieirqDIFHWQJdRuUACMoMbgEmRQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b341151e9ea17b5f-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.311 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.311 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.312 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.312 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.312 DEBUG:boto:Method: GET 2024-04-12T03:47:18.312 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.312 DEBUG:boto:Data: 2024-04-12T03:47:18.312 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.312 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.312 DEBUG:boto:Params: {} 2024-04-12T03:47:18.312 DEBUG:boto:Token: None 2024-04-12T03:47:18.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b8728093e8fe423-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.315 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.315 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.315 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.315 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.315 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.315 DEBUG:boto:Method: GET 2024-04-12T03:47:18.316 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:47:18.316 DEBUG:boto:Data: 2024-04-12T03:47:18.316 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.316 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.316 DEBUG:boto:Params: {} 2024-04-12T03:47:18.316 DEBUG:boto:Token: None 2024-04-12T03:47:18.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6f4298853ffa4f-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.362 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:47:18.363 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.363 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.363 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.363 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.363 DEBUG:boto:Method: GET 2024-04-12T03:47:18.363 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.364 DEBUG:boto:Data: 2024-04-12T03:47:18.364 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.364 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.364 DEBUG:boto:Params: {} 2024-04-12T03:47:18.364 DEBUG:boto:Token: None 2024-04-12T03:47:18.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.364 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w= 2024-04-12T03:47:18.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w=', 'Content-Length': '0'} 2024-04-12T03:47:18.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a9fc301d32e49a-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.367 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.367 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.367 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.367 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.367 DEBUG:boto:Method: GET 2024-04-12T03:47:18.367 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.367 DEBUG:boto:Data: 2024-04-12T03:47:18.367 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.367 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.367 DEBUG:boto:Params: {} 2024-04-12T03:47:18.368 DEBUG:boto:Token: None 2024-04-12T03:47:18.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w= 2024-04-12T03:47:18.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w=', 'Content-Length': '0'} 2024-04-12T03:47:18.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0f0ee3c6fac060-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.370 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.370 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.370 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.370 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.370 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.371 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.371 DEBUG:boto:Data: 2024-04-12T03:47:18.371 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.371 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.371 DEBUG:boto:Params: {} 2024-04-12T03:47:18.371 DEBUG:boto:Token: None 2024-04-12T03:47:18.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iPycBypFr9K/xINefsR2/1ulBo0= 2024-04-12T03:47:18.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iPycBypFr9K/xINefsR2/1ulBo0=', 'Content-Length': '0'} 2024-04-12T03:47:18.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d69e1458f19a3b-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.373 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.374 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.374 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.374 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.374 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.374 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.374 DEBUG:boto:Data: 2024-04-12T03:47:18.374 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.374 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.374 DEBUG:boto:Params: {} 2024-04-12T03:47:18.375 DEBUG:boto:Token: None 2024-04-12T03:47:18.375 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iPycBypFr9K/xINefsR2/1ulBo0= 2024-04-12T03:47:18.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iPycBypFr9K/xINefsR2/1ulBo0=', 'Content-Length': '0'} 2024-04-12T03:47:18.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c2429efa3eb547c-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:47:18.378 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.378 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.378 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.378 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.378 DEBUG:boto:Method: GET 2024-04-12T03:47:18.378 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.378 DEBUG:boto:Data: 2024-04-12T03:47:18.379 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.379 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.379 DEBUG:boto:Params: {} 2024-04-12T03:47:18.379 DEBUG:boto:Token: None 2024-04-12T03:47:18.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w= 2024-04-12T03:47:18.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w=', 'Content-Length': '0'} 2024-04-12T03:47:18.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec50898505aa93fa-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.381 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.381 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.381 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.381 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.381 DEBUG:boto:Method: GET 2024-04-12T03:47:18.382 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.382 DEBUG:boto:Data: 2024-04-12T03:47:18.382 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.382 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.382 DEBUG:boto:Params: {} 2024-04-12T03:47:18.382 DEBUG:boto:Token: None 2024-04-12T03:47:18.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:47:18.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w= 2024-04-12T03:47:18.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OmprUOm1nHCrKQqjhfFJrdaUx0w=', 'Content-Length': '0'} 2024-04-12T03:47:18.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c598c176494e0cc-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.384 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.385 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.385 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.385 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.385 DEBUG:boto:Method: GET 2024-04-12T03:47:18.385 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.385 DEBUG:boto:Data: 2024-04-12T03:47:18.385 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.385 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.385 DEBUG:boto:Params: {} 2024-04-12T03:47:18.386 DEBUG:boto:Token: None 2024-04-12T03:47:18.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d5079792787966-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.388 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.388 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.389 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.389 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.389 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.389 DEBUG:boto:Method: GET 2024-04-12T03:47:18.389 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.389 DEBUG:boto:Data: 2024-04-12T03:47:18.389 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.390 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.390 DEBUG:boto:Params: {} 2024-04-12T03:47:18.390 DEBUG:boto:Token: None 2024-04-12T03:47:18.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3a75fbd14973e5-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.393 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.393 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.393 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.394 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.394 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:47:18.394 DEBUG:boto:Data: 2024-04-12T03:47:18.394 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.394 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.394 DEBUG:boto:Params: {} 2024-04-12T03:47:18.394 DEBUG:boto:Token: None 2024-04-12T03:47:18.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46 2024-04-12T03:47:18.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zEN/q/myCLu072TANgJ5tYQaic= 2024-04-12T03:47:18.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zEN/q/myCLu072TANgJ5tYQaic=', 'Content-Length': '0'} 2024-04-12T03:47:18.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059adabb04b10551a-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.396 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.396 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.396 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.396 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:47:18.397 DEBUG:boto:Data: 2024-04-12T03:47:18.397 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.397 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.397 DEBUG:boto:Params: {} 2024-04-12T03:47:18.397 DEBUG:boto:Token: None 2024-04-12T03:47:18.397 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46 2024-04-12T03:47:18.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zEN/q/myCLu072TANgJ5tYQaic= 2024-04-12T03:47:18.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zEN/q/myCLu072TANgJ5tYQaic=', 'Content-Length': '0'} 2024-04-12T03:47:18.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065114cbb11458ef7-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:47:18.399 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.400 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.400 DEBUG:boto:Method: GET 2024-04-12T03:47:18.400 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:47:18.400 DEBUG:boto:Data: 2024-04-12T03:47:18.400 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.400 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.400 DEBUG:boto:Params: {} 2024-04-12T03:47:18.400 DEBUG:boto:Token: None 2024-04-12T03:47:18.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46 2024-04-12T03:47:18.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xDC8bDEXqd7zGVTw+jbghjLB0zA= 2024-04-12T03:47:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xDC8bDEXqd7zGVTw+jbghjLB0zA=', 'Content-Length': '0'} 2024-04-12T03:47:18.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357c88902fe2256e-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.402 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:47:18.402 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:47:18.402 DEBUG:boto:Method: GET 2024-04-12T03:47:18.403 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:47:18.403 DEBUG:boto:Data: 2024-04-12T03:47:18.403 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.403 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.403 DEBUG:boto:Params: {} 2024-04-12T03:47:18.403 DEBUG:boto:Token: None 2024-04-12T03:47:18.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj46 2024-04-12T03:47:18.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xDC8bDEXqd7zGVTw+jbghjLB0zA= 2024-04-12T03:47:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xDC8bDEXqd7zGVTw+jbghjLB0zA=', 'Content-Length': '0'} 2024-04-12T03:47:18.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000549d55b778590aee-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.406 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.406 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.406 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.406 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.406 DEBUG:boto:Method: GET 2024-04-12T03:47:18.406 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.406 DEBUG:boto:Data: 2024-04-12T03:47:18.406 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.406 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.407 DEBUG:boto:Params: {} 2024-04-12T03:47:18.407 DEBUG:boto:Token: None 2024-04-12T03:47:18.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005847e4134e7f4949-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.409 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.409 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.409 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.409 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.410 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.410 DEBUG:boto:Method: GET 2024-04-12T03:47:18.410 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:47:18.410 DEBUG:boto:Data: 2024-04-12T03:47:18.410 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.410 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.410 DEBUG:boto:Params: {} 2024-04-12T03:47:18.411 DEBUG:boto:Token: None 2024-04-12T03:47:18.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055e4a99cd6e6406-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.414 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:47:18.414 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.414 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.414 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.415 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.415 DEBUG:boto:Method: GET 2024-04-12T03:47:18.415 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.415 DEBUG:boto:Data: 2024-04-12T03:47:18.415 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.415 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.415 DEBUG:boto:Params: {} 2024-04-12T03:47:18.415 DEBUG:boto:Token: None 2024-04-12T03:47:18.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k= 2024-04-12T03:47:18.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k=', 'Content-Length': '0'} 2024-04-12T03:47:18.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e4c6c40dda2a0d-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.417 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.418 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.418 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.418 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.418 DEBUG:boto:Method: GET 2024-04-12T03:47:18.418 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.418 DEBUG:boto:Data: 2024-04-12T03:47:18.418 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.418 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.419 DEBUG:boto:Params: {} 2024-04-12T03:47:18.419 DEBUG:boto:Token: None 2024-04-12T03:47:18.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k= 2024-04-12T03:47:18.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k=', 'Content-Length': '0'} 2024-04-12T03:47:18.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e313dc5ccf0151-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.421 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.421 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.421 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.421 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.421 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.421 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.422 DEBUG:boto:Data: 2024-04-12T03:47:18.422 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.422 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.422 DEBUG:boto:Params: {} 2024-04-12T03:47:18.422 DEBUG:boto:Token: None 2024-04-12T03:47:18.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eMnVMuWTSksn3KD4Afi1u6fCPEg= 2024-04-12T03:47:18.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eMnVMuWTSksn3KD4Afi1u6fCPEg=', 'Content-Length': '0'} 2024-04-12T03:47:18.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc67c61f9076ea6-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.424 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.424 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.424 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.424 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.424 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.424 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.424 DEBUG:boto:Data: 2024-04-12T03:47:18.425 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.425 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.425 DEBUG:boto:Params: {} 2024-04-12T03:47:18.425 DEBUG:boto:Token: None 2024-04-12T03:47:18.425 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eMnVMuWTSksn3KD4Afi1u6fCPEg= 2024-04-12T03:47:18.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eMnVMuWTSksn3KD4Afi1u6fCPEg=', 'Content-Length': '0'} 2024-04-12T03:47:18.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b1dcea2978f4e9-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:47:18.427 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.427 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.427 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.427 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.427 DEBUG:boto:Method: GET 2024-04-12T03:47:18.428 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.428 DEBUG:boto:Data: 2024-04-12T03:47:18.428 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.428 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.428 DEBUG:boto:Params: {} 2024-04-12T03:47:18.428 DEBUG:boto:Token: None 2024-04-12T03:47:18.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k= 2024-04-12T03:47:18.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k=', 'Content-Length': '0'} 2024-04-12T03:47:18.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072c7781f40229595-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.430 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.430 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.430 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.430 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.430 DEBUG:boto:Method: GET 2024-04-12T03:47:18.431 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.431 DEBUG:boto:Data: 2024-04-12T03:47:18.431 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.431 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.431 DEBUG:boto:Params: {} 2024-04-12T03:47:18.431 DEBUG:boto:Token: None 2024-04-12T03:47:18.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:47:18.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k= 2024-04-12T03:47:18.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9V+JOhSTbFYR4hvOhdl0VVBQ06k=', 'Content-Length': '0'} 2024-04-12T03:47:18.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a9a068121ee109-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.474 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.474 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.475 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.475 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.475 DEBUG:boto:Method: GET 2024-04-12T03:47:18.475 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.475 DEBUG:boto:Data: 2024-04-12T03:47:18.475 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.475 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.475 DEBUG:boto:Params: {} 2024-04-12T03:47:18.476 DEBUG:boto:Token: None 2024-04-12T03:47:18.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f8c03c064a10b4-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.478 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.479 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.479 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.479 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.479 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.479 DEBUG:boto:Method: GET 2024-04-12T03:47:18.479 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.479 DEBUG:boto:Data: 2024-04-12T03:47:18.479 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.479 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.479 DEBUG:boto:Params: {} 2024-04-12T03:47:18.480 DEBUG:boto:Token: None 2024-04-12T03:47:18.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e03752c51fce0e-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.482 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.483 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.483 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.483 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.483 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:47:18.483 DEBUG:boto:Data: 2024-04-12T03:47:18.483 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.483 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.484 DEBUG:boto:Params: {} 2024-04-12T03:47:18.484 DEBUG:boto:Token: None 2024-04-12T03:47:18.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51 2024-04-12T03:47:18.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N4jkqwq4LPAIWh5nhAAmoXK+HM8= 2024-04-12T03:47:18.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N4jkqwq4LPAIWh5nhAAmoXK+HM8=', 'Content-Length': '0'} 2024-04-12T03:47:18.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a430889e4c07912-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.486 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.486 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.486 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.486 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:47:18.486 DEBUG:boto:Data: 2024-04-12T03:47:18.486 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.486 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.486 DEBUG:boto:Params: {} 2024-04-12T03:47:18.487 DEBUG:boto:Token: None 2024-04-12T03:47:18.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51 2024-04-12T03:47:18.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N4jkqwq4LPAIWh5nhAAmoXK+HM8= 2024-04-12T03:47:18.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N4jkqwq4LPAIWh5nhAAmoXK+HM8=', 'Content-Length': '0'} 2024-04-12T03:47:18.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9036014d563622-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:47:18.489 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.489 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.489 DEBUG:boto:Method: GET 2024-04-12T03:47:18.489 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:47:18.489 DEBUG:boto:Data: 2024-04-12T03:47:18.489 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.489 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.490 DEBUG:boto:Params: {} 2024-04-12T03:47:18.490 DEBUG:boto:Token: None 2024-04-12T03:47:18.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51 2024-04-12T03:47:18.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uk+aSPjqvp8yvNDKCpgvSrWq81Q= 2024-04-12T03:47:18.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uk+aSPjqvp8yvNDKCpgvSrWq81Q=', 'Content-Length': '0'} 2024-04-12T03:47:18.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1cabc0422d42f4e-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.492 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:47:18.492 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:47:18.492 DEBUG:boto:Method: GET 2024-04-12T03:47:18.492 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:47:18.492 DEBUG:boto:Data: 2024-04-12T03:47:18.492 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.492 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.493 DEBUG:boto:Params: {} 2024-04-12T03:47:18.493 DEBUG:boto:Token: None 2024-04-12T03:47:18.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj51 2024-04-12T03:47:18.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uk+aSPjqvp8yvNDKCpgvSrWq81Q= 2024-04-12T03:47:18.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uk+aSPjqvp8yvNDKCpgvSrWq81Q=', 'Content-Length': '0'} 2024-04-12T03:47:18.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003166d44de65dfe84-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.495 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.495 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.495 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.495 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.495 DEBUG:boto:Method: GET 2024-04-12T03:47:18.495 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.495 DEBUG:boto:Data: 2024-04-12T03:47:18.496 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.496 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.496 DEBUG:boto:Params: {} 2024-04-12T03:47:18.496 DEBUG:boto:Token: None 2024-04-12T03:47:18.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039758b81c25e97d0-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.498 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.498 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.499 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.499 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.499 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.499 DEBUG:boto:Method: GET 2024-04-12T03:47:18.499 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:47:18.499 DEBUG:boto:Data: 2024-04-12T03:47:18.499 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.499 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.499 DEBUG:boto:Params: {} 2024-04-12T03:47:18.500 DEBUG:boto:Token: None 2024-04-12T03:47:18.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000320d5dc5ceda23df-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.546 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:47:18.547 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.547 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.547 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.547 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.547 DEBUG:boto:Method: GET 2024-04-12T03:47:18.547 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.547 DEBUG:boto:Data: 2024-04-12T03:47:18.548 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.548 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.548 DEBUG:boto:Params: {} 2024-04-12T03:47:18.548 DEBUG:boto:Token: None 2024-04-12T03:47:18.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4= 2024-04-12T03:47:18.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4=', 'Content-Length': '0'} 2024-04-12T03:47:18.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f212e24fa41380-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.550 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.550 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.551 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.551 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.551 DEBUG:boto:Method: GET 2024-04-12T03:47:18.551 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.551 DEBUG:boto:Data: 2024-04-12T03:47:18.551 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.551 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.551 DEBUG:boto:Params: {} 2024-04-12T03:47:18.551 DEBUG:boto:Token: None 2024-04-12T03:47:18.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4= 2024-04-12T03:47:18.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4=', 'Content-Length': '0'} 2024-04-12T03:47:18.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58354ee6c7003f0-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.554 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.554 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.554 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.554 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.554 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.554 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.554 DEBUG:boto:Data: 2024-04-12T03:47:18.554 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.554 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.555 DEBUG:boto:Params: {} 2024-04-12T03:47:18.555 DEBUG:boto:Token: None 2024-04-12T03:47:18.555 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s2GTdPX8ff1jjt68QwmwvTZmUR4= 2024-04-12T03:47:18.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s2GTdPX8ff1jjt68QwmwvTZmUR4=', 'Content-Length': '0'} 2024-04-12T03:47:18.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcb98e1b8173c15-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.557 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.557 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.557 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.557 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.557 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.557 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.557 DEBUG:boto:Data: 2024-04-12T03:47:18.557 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.558 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.558 DEBUG:boto:Params: {} 2024-04-12T03:47:18.558 DEBUG:boto:Token: None 2024-04-12T03:47:18.558 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.558 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s2GTdPX8ff1jjt68QwmwvTZmUR4= 2024-04-12T03:47:18.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s2GTdPX8ff1jjt68QwmwvTZmUR4=', 'Content-Length': '0'} 2024-04-12T03:47:18.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8167538947c856f-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:47:18.560 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.560 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.560 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.560 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.560 DEBUG:boto:Method: GET 2024-04-12T03:47:18.560 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.560 DEBUG:boto:Data: 2024-04-12T03:47:18.560 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.561 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.561 DEBUG:boto:Params: {} 2024-04-12T03:47:18.561 DEBUG:boto:Token: None 2024-04-12T03:47:18.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4= 2024-04-12T03:47:18.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4=', 'Content-Length': '0'} 2024-04-12T03:47:18.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a50a4a59f924fb1-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.563 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.563 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.563 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.563 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.563 DEBUG:boto:Method: GET 2024-04-12T03:47:18.563 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.563 DEBUG:boto:Data: 2024-04-12T03:47:18.563 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.564 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.564 DEBUG:boto:Params: {} 2024-04-12T03:47:18.564 DEBUG:boto:Token: None 2024-04-12T03:47:18.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:47:18.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4= 2024-04-12T03:47:18.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uv06QGRu/r8Yn0Ew7AV/0YJf3G4=', 'Content-Length': '0'} 2024-04-12T03:47:18.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096bc7735037cfa7a-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.567 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.567 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.567 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.567 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.567 DEBUG:boto:Method: GET 2024-04-12T03:47:18.567 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.567 DEBUG:boto:Data: 2024-04-12T03:47:18.567 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.567 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.567 DEBUG:boto:Params: {} 2024-04-12T03:47:18.568 DEBUG:boto:Token: None 2024-04-12T03:47:18.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001444f6b0262bb550-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.614 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.615 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.615 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.615 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.615 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.615 DEBUG:boto:Method: GET 2024-04-12T03:47:18.616 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.616 DEBUG:boto:Data: 2024-04-12T03:47:18.616 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.616 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.616 DEBUG:boto:Params: {} 2024-04-12T03:47:18.616 DEBUG:boto:Token: None 2024-04-12T03:47:18.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449c554dd4d89ddf-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.620 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.620 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.620 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.620 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.620 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:47:18.621 DEBUG:boto:Data: 2024-04-12T03:47:18.621 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.621 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.621 DEBUG:boto:Params: {} 2024-04-12T03:47:18.621 DEBUG:boto:Token: None 2024-04-12T03:47:18.621 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55 2024-04-12T03:47:18.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VFcnmgJnxYMk5PERfmL2nRuh5pk= 2024-04-12T03:47:18.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VFcnmgJnxYMk5PERfmL2nRuh5pk=', 'Content-Length': '0'} 2024-04-12T03:47:18.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e276585205872bc-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.623 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.623 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.623 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.623 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:47:18.623 DEBUG:boto:Data: 2024-04-12T03:47:18.623 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.624 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.624 DEBUG:boto:Params: {} 2024-04-12T03:47:18.624 DEBUG:boto:Token: None 2024-04-12T03:47:18.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55 2024-04-12T03:47:18.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VFcnmgJnxYMk5PERfmL2nRuh5pk= 2024-04-12T03:47:18.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VFcnmgJnxYMk5PERfmL2nRuh5pk=', 'Content-Length': '0'} 2024-04-12T03:47:18.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002894fc28f81a8bc6-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:47:18.626 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.626 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.627 DEBUG:boto:Method: GET 2024-04-12T03:47:18.627 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:47:18.627 DEBUG:boto:Data: 2024-04-12T03:47:18.627 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.627 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.627 DEBUG:boto:Params: {} 2024-04-12T03:47:18.627 DEBUG:boto:Token: None 2024-04-12T03:47:18.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55 2024-04-12T03:47:18.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lIhLhRzuin+jdqqLCMwCH8dJ1VA= 2024-04-12T03:47:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lIhLhRzuin+jdqqLCMwCH8dJ1VA=', 'Content-Length': '0'} 2024-04-12T03:47:18.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883846382c354b74-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.629 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:47:18.629 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:47:18.629 DEBUG:boto:Method: GET 2024-04-12T03:47:18.629 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:47:18.630 DEBUG:boto:Data: 2024-04-12T03:47:18.630 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.630 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.630 DEBUG:boto:Params: {} 2024-04-12T03:47:18.630 DEBUG:boto:Token: None 2024-04-12T03:47:18.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj55 2024-04-12T03:47:18.630 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lIhLhRzuin+jdqqLCMwCH8dJ1VA= 2024-04-12T03:47:18.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lIhLhRzuin+jdqqLCMwCH8dJ1VA=', 'Content-Length': '0'} 2024-04-12T03:47:18.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870489e159e05412-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.632 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.632 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.632 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.632 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.632 DEBUG:boto:Method: GET 2024-04-12T03:47:18.632 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.633 DEBUG:boto:Data: 2024-04-12T03:47:18.633 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.633 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.633 DEBUG:boto:Params: {} 2024-04-12T03:47:18.633 DEBUG:boto:Token: None 2024-04-12T03:47:18.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbff7850926e0f09-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.636 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.636 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.636 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.636 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.636 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.636 DEBUG:boto:Method: GET 2024-04-12T03:47:18.637 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:47:18.637 DEBUG:boto:Data: 2024-04-12T03:47:18.637 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.637 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.637 DEBUG:boto:Params: {} 2024-04-12T03:47:18.637 DEBUG:boto:Token: None 2024-04-12T03:47:18.637 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066351c8838f58864-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.640 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:47:18.641 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.641 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.641 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.641 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.641 DEBUG:boto:Method: GET 2024-04-12T03:47:18.641 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.641 DEBUG:boto:Data: 2024-04-12T03:47:18.641 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.642 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.642 DEBUG:boto:Params: {} 2024-04-12T03:47:18.642 DEBUG:boto:Token: None 2024-04-12T03:47:18.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE= 2024-04-12T03:47:18.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE=', 'Content-Length': '0'} 2024-04-12T03:47:18.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021255b1432683513-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.686 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.687 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.687 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.687 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.687 DEBUG:boto:Method: GET 2024-04-12T03:47:18.687 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.687 DEBUG:boto:Data: 2024-04-12T03:47:18.687 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.687 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.687 DEBUG:boto:Params: {} 2024-04-12T03:47:18.687 DEBUG:boto:Token: None 2024-04-12T03:47:18.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE= 2024-04-12T03:47:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE=', 'Content-Length': '0'} 2024-04-12T03:47:18.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acbe3c64c7e7a0ed-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.690 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.690 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.690 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.690 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.690 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.690 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.691 DEBUG:boto:Data: 2024-04-12T03:47:18.691 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.691 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.691 DEBUG:boto:Params: {} 2024-04-12T03:47:18.691 DEBUG:boto:Token: None 2024-04-12T03:47:18.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kZiZOUq1W/+Rxj5NXKDKmTmYsHQ= 2024-04-12T03:47:18.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kZiZOUq1W/+Rxj5NXKDKmTmYsHQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f1d84cad984778-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.693 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.693 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.694 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.694 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.694 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.694 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.694 DEBUG:boto:Data: 2024-04-12T03:47:18.694 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.694 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.694 DEBUG:boto:Params: {} 2024-04-12T03:47:18.694 DEBUG:boto:Token: None 2024-04-12T03:47:18.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kZiZOUq1W/+Rxj5NXKDKmTmYsHQ= 2024-04-12T03:47:18.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kZiZOUq1W/+Rxj5NXKDKmTmYsHQ=', 'Content-Length': '0'} 2024-04-12T03:47:18.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d02b29c242fd146-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:47:18.697 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.697 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.697 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.697 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.697 DEBUG:boto:Method: GET 2024-04-12T03:47:18.697 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.697 DEBUG:boto:Data: 2024-04-12T03:47:18.697 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.698 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.698 DEBUG:boto:Params: {} 2024-04-12T03:47:18.698 DEBUG:boto:Token: None 2024-04-12T03:47:18.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE= 2024-04-12T03:47:18.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE=', 'Content-Length': '0'} 2024-04-12T03:47:18.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec6571203e621aa-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.700 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.700 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.700 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.700 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.700 DEBUG:boto:Method: GET 2024-04-12T03:47:18.700 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.700 DEBUG:boto:Data: 2024-04-12T03:47:18.701 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.701 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.701 DEBUG:boto:Params: {} 2024-04-12T03:47:18.701 DEBUG:boto:Token: None 2024-04-12T03:47:18.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:47:18.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE= 2024-04-12T03:47:18.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZYgNmrKd7eAQHo5G2Mp+juBYxE=', 'Content-Length': '0'} 2024-04-12T03:47:18.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5c8b9fc43399f7-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.703 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.703 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.703 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.704 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.704 DEBUG:boto:Method: GET 2024-04-12T03:47:18.704 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.704 DEBUG:boto:Data: 2024-04-12T03:47:18.704 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.704 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.704 DEBUG:boto:Params: {} 2024-04-12T03:47:18.704 DEBUG:boto:Token: None 2024-04-12T03:47:18.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031c639d8d2e77a14-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.707 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.707 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.707 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.708 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.708 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.708 DEBUG:boto:Method: GET 2024-04-12T03:47:18.708 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.708 DEBUG:boto:Data: 2024-04-12T03:47:18.708 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.708 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.708 DEBUG:boto:Params: {} 2024-04-12T03:47:18.708 DEBUG:boto:Token: None 2024-04-12T03:47:18.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e9c10d515d1833-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.758 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.759 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.759 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.759 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.759 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:47:18.759 DEBUG:boto:Data: 2024-04-12T03:47:18.759 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.759 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.759 DEBUG:boto:Params: {} 2024-04-12T03:47:18.760 DEBUG:boto:Token: None 2024-04-12T03:47:18.760 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57 2024-04-12T03:47:18.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPwPdEuaTqSLT91ANrXRf1+Gp6g= 2024-04-12T03:47:18.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPwPdEuaTqSLT91ANrXRf1+Gp6g=', 'Content-Length': '0'} 2024-04-12T03:47:18.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af627d8918b825b0-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.762 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.762 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.762 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.762 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:47:18.762 DEBUG:boto:Data: 2024-04-12T03:47:18.762 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.763 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.763 DEBUG:boto:Params: {} 2024-04-12T03:47:18.763 DEBUG:boto:Token: None 2024-04-12T03:47:18.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57 2024-04-12T03:47:18.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPwPdEuaTqSLT91ANrXRf1+Gp6g= 2024-04-12T03:47:18.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPwPdEuaTqSLT91ANrXRf1+Gp6g=', 'Content-Length': '0'} 2024-04-12T03:47:18.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e88d9f8bb2ec4b-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:47:18.765 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.765 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.765 DEBUG:boto:Method: GET 2024-04-12T03:47:18.766 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:47:18.766 DEBUG:boto:Data: 2024-04-12T03:47:18.766 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.766 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.766 DEBUG:boto:Params: {} 2024-04-12T03:47:18.766 DEBUG:boto:Token: None 2024-04-12T03:47:18.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57 2024-04-12T03:47:18.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vDf1ADer8gczKy5YatXTW8SUQ+A= 2024-04-12T03:47:18.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vDf1ADer8gczKy5YatXTW8SUQ+A=', 'Content-Length': '0'} 2024-04-12T03:47:18.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e42ff6c481faec-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.768 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:47:18.769 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:47:18.769 DEBUG:boto:Method: GET 2024-04-12T03:47:18.769 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:47:18.769 DEBUG:boto:Data: 2024-04-12T03:47:18.769 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.769 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.769 DEBUG:boto:Params: {} 2024-04-12T03:47:18.769 DEBUG:boto:Token: None 2024-04-12T03:47:18.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj57 2024-04-12T03:47:18.769 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vDf1ADer8gczKy5YatXTW8SUQ+A= 2024-04-12T03:47:18.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vDf1ADer8gczKy5YatXTW8SUQ+A=', 'Content-Length': '0'} 2024-04-12T03:47:18.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282f15adb5b4b92a-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.772 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.772 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.772 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.772 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.772 DEBUG:boto:Method: GET 2024-04-12T03:47:18.772 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.772 DEBUG:boto:Data: 2024-04-12T03:47:18.772 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.772 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.773 DEBUG:boto:Params: {} 2024-04-12T03:47:18.773 DEBUG:boto:Token: None 2024-04-12T03:47:18.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51d26410503bce0-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.775 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.776 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.776 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.776 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.776 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.776 DEBUG:boto:Method: GET 2024-04-12T03:47:18.776 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:47:18.776 DEBUG:boto:Data: 2024-04-12T03:47:18.776 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.777 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.777 DEBUG:boto:Params: {} 2024-04-12T03:47:18.777 DEBUG:boto:Token: None 2024-04-12T03:47:18.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.777 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ceb2cac53d7e0bc-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.780 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:47:18.781 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.781 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.781 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.781 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.781 DEBUG:boto:Method: GET 2024-04-12T03:47:18.781 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.781 DEBUG:boto:Data: 2024-04-12T03:47:18.781 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.781 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.782 DEBUG:boto:Params: {} 2024-04-12T03:47:18.782 DEBUG:boto:Token: None 2024-04-12T03:47:18.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM= 2024-04-12T03:47:18.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM=', 'Content-Length': '0'} 2024-04-12T03:47:18.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db4f70b20ba51ae5-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.784 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.784 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.784 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.784 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.784 DEBUG:boto:Method: GET 2024-04-12T03:47:18.785 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.785 DEBUG:boto:Data: 2024-04-12T03:47:18.785 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.785 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.785 DEBUG:boto:Params: {} 2024-04-12T03:47:18.785 DEBUG:boto:Token: None 2024-04-12T03:47:18.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM= 2024-04-12T03:47:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM=', 'Content-Length': '0'} 2024-04-12T03:47:18.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e1e39ac2c78394-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.830 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.830 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.831 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.831 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.831 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.831 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.831 DEBUG:boto:Data: 2024-04-12T03:47:18.831 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.831 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.831 DEBUG:boto:Params: {} 2024-04-12T03:47:18.831 DEBUG:boto:Token: None 2024-04-12T03:47:18.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zc6Z5XYNrWMkUz09IWWqa2xkg20= 2024-04-12T03:47:18.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zc6Z5XYNrWMkUz09IWWqa2xkg20=', 'Content-Length': '0'} 2024-04-12T03:47:18.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdaea8f934f85af8-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.833 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.834 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.834 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.834 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.834 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.834 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.834 DEBUG:boto:Data: 2024-04-12T03:47:18.834 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.834 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.834 DEBUG:boto:Params: {} 2024-04-12T03:47:18.835 DEBUG:boto:Token: None 2024-04-12T03:47:18.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zc6Z5XYNrWMkUz09IWWqa2xkg20= 2024-04-12T03:47:18.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zc6Z5XYNrWMkUz09IWWqa2xkg20=', 'Content-Length': '0'} 2024-04-12T03:47:18.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ff32a33aafc115-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:47:18.837 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.837 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.837 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.837 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.837 DEBUG:boto:Method: GET 2024-04-12T03:47:18.837 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.837 DEBUG:boto:Data: 2024-04-12T03:47:18.837 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.838 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.838 DEBUG:boto:Params: {} 2024-04-12T03:47:18.838 DEBUG:boto:Token: None 2024-04-12T03:47:18.838 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.838 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM= 2024-04-12T03:47:18.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM=', 'Content-Length': '0'} 2024-04-12T03:47:18.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd37ec76d02e0ead-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.840 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.840 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.840 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.840 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.840 DEBUG:boto:Method: GET 2024-04-12T03:47:18.841 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.841 DEBUG:boto:Data: 2024-04-12T03:47:18.841 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.841 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.841 DEBUG:boto:Params: {} 2024-04-12T03:47:18.841 DEBUG:boto:Token: None 2024-04-12T03:47:18.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:47:18.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM= 2024-04-12T03:47:18.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pXSgbfTELPcGTQpMxVnt3Z9l+fM=', 'Content-Length': '0'} 2024-04-12T03:47:18.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be07b7d66150b79d-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.843 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.843 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.843 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.843 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.844 DEBUG:boto:Method: GET 2024-04-12T03:47:18.844 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.844 DEBUG:boto:Data: 2024-04-12T03:47:18.844 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.844 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.844 DEBUG:boto:Params: {} 2024-04-12T03:47:18.844 DEBUG:boto:Token: None 2024-04-12T03:47:18.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed950553ef07eb3-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.847 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.847 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.847 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.847 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.847 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.847 DEBUG:boto:Method: GET 2024-04-12T03:47:18.848 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.848 DEBUG:boto:Data: 2024-04-12T03:47:18.848 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.848 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.848 DEBUG:boto:Params: {} 2024-04-12T03:47:18.848 DEBUG:boto:Token: None 2024-04-12T03:47:18.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861f8adee9309694-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.854 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.855 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.855 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.855 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.855 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:47:18.855 DEBUG:boto:Data: 2024-04-12T03:47:18.855 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.855 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.855 DEBUG:boto:Params: {} 2024-04-12T03:47:18.856 DEBUG:boto:Token: None 2024-04-12T03:47:18.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68 2024-04-12T03:47:18.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+GGk3IbAXZHjC8mV0Mg8sjl4JZE= 2024-04-12T03:47:18.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+GGk3IbAXZHjC8mV0Mg8sjl4JZE=', 'Content-Length': '0'} 2024-04-12T03:47:18.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc841150b3150b6-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.859 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.859 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.859 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.859 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:47:18.859 DEBUG:boto:Data: 2024-04-12T03:47:18.859 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.859 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.859 DEBUG:boto:Params: {} 2024-04-12T03:47:18.860 DEBUG:boto:Token: None 2024-04-12T03:47:18.860 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68 2024-04-12T03:47:18.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+GGk3IbAXZHjC8mV0Mg8sjl4JZE= 2024-04-12T03:47:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+GGk3IbAXZHjC8mV0Mg8sjl4JZE=', 'Content-Length': '0'} 2024-04-12T03:47:18.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac13dae1e6c7d09-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:47:18.862 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.862 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.862 DEBUG:boto:Method: GET 2024-04-12T03:47:18.862 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:47:18.862 DEBUG:boto:Data: 2024-04-12T03:47:18.862 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.863 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.863 DEBUG:boto:Params: {} 2024-04-12T03:47:18.863 DEBUG:boto:Token: None 2024-04-12T03:47:18.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68 2024-04-12T03:47:18.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/DW5aUwEwKjFGrE/ReRZ/PQ9jo= 2024-04-12T03:47:18.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/DW5aUwEwKjFGrE/ReRZ/PQ9jo=', 'Content-Length': '0'} 2024-04-12T03:47:18.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f33897d044168d8-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.906 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:47:18.907 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:47:18.907 DEBUG:boto:Method: GET 2024-04-12T03:47:18.907 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:47:18.907 DEBUG:boto:Data: 2024-04-12T03:47:18.907 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.907 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.907 DEBUG:boto:Params: {} 2024-04-12T03:47:18.908 DEBUG:boto:Token: None 2024-04-12T03:47:18.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj68 2024-04-12T03:47:18.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/DW5aUwEwKjFGrE/ReRZ/PQ9jo= 2024-04-12T03:47:18.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/DW5aUwEwKjFGrE/ReRZ/PQ9jo=', 'Content-Length': '0'} 2024-04-12T03:47:18.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22051748709f3ab-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.910 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.910 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.910 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.910 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.911 DEBUG:boto:Method: GET 2024-04-12T03:47:18.911 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.911 DEBUG:boto:Data: 2024-04-12T03:47:18.911 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.911 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.911 DEBUG:boto:Params: {} 2024-04-12T03:47:18.911 DEBUG:boto:Token: None 2024-04-12T03:47:18.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a863387372a4d0f-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.914 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.914 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.915 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.915 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.915 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.915 DEBUG:boto:Method: GET 2024-04-12T03:47:18.915 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:47:18.915 DEBUG:boto:Data: 2024-04-12T03:47:18.915 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.915 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.916 DEBUG:boto:Params: {} 2024-04-12T03:47:18.916 DEBUG:boto:Token: None 2024-04-12T03:47:18.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000248860e75902e08a-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.919 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:47:18.919 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.919 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.920 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.920 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.920 DEBUG:boto:Method: GET 2024-04-12T03:47:18.920 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.920 DEBUG:boto:Data: 2024-04-12T03:47:18.920 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.920 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.920 DEBUG:boto:Params: {} 2024-04-12T03:47:18.921 DEBUG:boto:Token: None 2024-04-12T03:47:18.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y= 2024-04-12T03:47:18.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y=', 'Content-Length': '0'} 2024-04-12T03:47:18.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db5bee02736185b7-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.923 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.923 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.923 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.923 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.924 DEBUG:boto:Method: GET 2024-04-12T03:47:18.924 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.924 DEBUG:boto:Data: 2024-04-12T03:47:18.924 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.924 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.924 DEBUG:boto:Params: {} 2024-04-12T03:47:18.924 DEBUG:boto:Token: None 2024-04-12T03:47:18.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y= 2024-04-12T03:47:18.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y=', 'Content-Length': '0'} 2024-04-12T03:47:18.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8a8714432af8c3-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.927 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.927 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.927 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.927 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.927 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.927 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.927 DEBUG:boto:Data: 2024-04-12T03:47:18.928 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.928 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.928 DEBUG:boto:Params: {} 2024-04-12T03:47:18.928 DEBUG:boto:Token: None 2024-04-12T03:47:18.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:miHQc88uGIrtgtaFfTv+VvU6yqM= 2024-04-12T03:47:18.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:miHQc88uGIrtgtaFfTv+VvU6yqM=', 'Content-Length': '0'} 2024-04-12T03:47:18.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0117e9dd8e04fd-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.930 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.930 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.930 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.930 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.931 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.931 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.931 DEBUG:boto:Data: 2024-04-12T03:47:18.931 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.931 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.931 DEBUG:boto:Params: {} 2024-04-12T03:47:18.931 DEBUG:boto:Token: None 2024-04-12T03:47:18.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:miHQc88uGIrtgtaFfTv+VvU6yqM= 2024-04-12T03:47:18.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:miHQc88uGIrtgtaFfTv+VvU6yqM=', 'Content-Length': '0'} 2024-04-12T03:47:18.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093c64aa92f8a3e8-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:47:18.933 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.933 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.934 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.934 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.934 DEBUG:boto:Method: GET 2024-04-12T03:47:18.934 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.934 DEBUG:boto:Data: 2024-04-12T03:47:18.934 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.934 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.934 DEBUG:boto:Params: {} 2024-04-12T03:47:18.935 DEBUG:boto:Token: None 2024-04-12T03:47:18.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y= 2024-04-12T03:47:18.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y=', 'Content-Length': '0'} 2024-04-12T03:47:18.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cce371bb9f2711-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.937 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.937 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.937 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.937 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.937 DEBUG:boto:Method: GET 2024-04-12T03:47:18.937 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.937 DEBUG:boto:Data: 2024-04-12T03:47:18.937 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.938 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.938 DEBUG:boto:Params: {} 2024-04-12T03:47:18.938 DEBUG:boto:Token: None 2024-04-12T03:47:18.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:47:18.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y= 2024-04-12T03:47:18.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3WUDKV7zehgjMGRUifeqGTxa58Y=', 'Content-Length': '0'} 2024-04-12T03:47:18.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6793e6e0a358f4-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.940 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.940 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.941 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.941 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.941 DEBUG:boto:Method: GET 2024-04-12T03:47:18.941 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.941 DEBUG:boto:Data: 2024-04-12T03:47:18.941 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.941 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.941 DEBUG:boto:Params: {} 2024-04-12T03:47:18.942 DEBUG:boto:Token: None 2024-04-12T03:47:18.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d1bd2a1a6d52eee-006618aec6-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.944 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.944 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:18.944 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:18.945 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.945 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.945 DEBUG:boto:Method: GET 2024-04-12T03:47:18.945 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:47:18.945 DEBUG:boto:Data: 2024-04-12T03:47:18.945 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.945 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.945 DEBUG:boto:Params: {} 2024-04-12T03:47:18.945 DEBUG:boto:Token: None 2024-04-12T03:47:18.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/ 2024-04-12T03:47:18.946 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8= 2024-04-12T03:47:18.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1nFG/wumspdeujJr+f8nkA+KvP8=', 'Content-Length': '0'} 2024-04-12T03:47:18.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98653379d6d25cd-006618aec6-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.949 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:18.949 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.949 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.949 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.949 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:47:18.949 DEBUG:boto:Data: 2024-04-12T03:47:18.950 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.950 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.950 DEBUG:boto:Params: {} 2024-04-12T03:47:18.950 DEBUG:boto:Token: None 2024-04-12T03:47:18.950 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69 2024-04-12T03:47:18.950 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bTh6yMirmNUH2quXlmZmQ0o9d/4= 2024-04-12T03:47:18.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bTh6yMirmNUH2quXlmZmQ0o9d/4=', 'Content-Length': '0'} 2024-04-12T03:47:18.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e10662075157e2-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.952 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.952 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.952 DEBUG:boto:Method: HEAD 2024-04-12T03:47:18.952 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:47:18.953 DEBUG:boto:Data: 2024-04-12T03:47:18.953 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.953 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.953 DEBUG:boto:Params: {} 2024-04-12T03:47:18.953 DEBUG:boto:Token: None 2024-04-12T03:47:18.953 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69 2024-04-12T03:47:18.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bTh6yMirmNUH2quXlmZmQ0o9d/4= 2024-04-12T03:47:18.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bTh6yMirmNUH2quXlmZmQ0o9d/4=', 'Content-Length': '0'} 2024-04-12T03:47:18.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c12a34786f21dcb-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:47:18.956 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.956 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.956 DEBUG:boto:Method: GET 2024-04-12T03:47:18.956 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:47:18.956 DEBUG:boto:Data: 2024-04-12T03:47:18.956 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:18.956 DEBUG:boto:Port: 8001 2024-04-12T03:47:18.956 DEBUG:boto:Params: {} 2024-04-12T03:47:18.956 DEBUG:boto:Token: None 2024-04-12T03:47:18.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69 2024-04-12T03:47:18.957 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jxUfQI30/A8LVMeYA68CJKX6HA8= 2024-04-12T03:47:18.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jxUfQI30/A8LVMeYA68CJKX6HA8=', 'Content-Length': '0'} 2024-04-12T03:47:18.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28e049ea96e2bd6-006618aec6-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:18.959 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:47:18.959 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:47:18.959 DEBUG:boto:Method: GET 2024-04-12T03:47:18.959 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:47:18.959 DEBUG:boto:Data: 2024-04-12T03:47:18.960 DEBUG:boto:Headers: {} 2024-04-12T03:47:18.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:18.960 DEBUG:boto:Port: 8000 2024-04-12T03:47:18.960 DEBUG:boto:Params: {} 2024-04-12T03:47:18.960 DEBUG:boto:Token: None 2024-04-12T03:47:18.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:18 GMT /dgbpxg-61/obj69 2024-04-12T03:47:18.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jxUfQI30/A8LVMeYA68CJKX6HA8= 2024-04-12T03:47:18.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jxUfQI30/A8LVMeYA68CJKX6HA8=', 'Content-Length': '0'} 2024-04-12T03:47:18.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044476ef14a6cb14b-006618aec6-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.002 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.003 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.003 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.003 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.003 DEBUG:boto:Method: GET 2024-04-12T03:47:19.003 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.003 DEBUG:boto:Data: 2024-04-12T03:47:19.003 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.003 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.003 DEBUG:boto:Params: {} 2024-04-12T03:47:19.004 DEBUG:boto:Token: None 2024-04-12T03:47:19.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc90b29dcd407ee-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.006 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.006 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.006 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.007 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.007 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.007 DEBUG:boto:Method: GET 2024-04-12T03:47:19.007 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:47:19.007 DEBUG:boto:Data: 2024-04-12T03:47:19.007 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.007 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.007 DEBUG:boto:Params: {} 2024-04-12T03:47:19.007 DEBUG:boto:Token: None 2024-04-12T03:47:19.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c631412c814667-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.011 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:47:19.011 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.011 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.011 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.011 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.012 DEBUG:boto:Method: GET 2024-04-12T03:47:19.012 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.012 DEBUG:boto:Data: 2024-04-12T03:47:19.012 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.012 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.012 DEBUG:boto:Params: {} 2024-04-12T03:47:19.012 DEBUG:boto:Token: None 2024-04-12T03:47:19.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs= 2024-04-12T03:47:19.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs=', 'Content-Length': '0'} 2024-04-12T03:47:19.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028556063d90173ed-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.014 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.015 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.015 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.015 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.015 DEBUG:boto:Method: GET 2024-04-12T03:47:19.015 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.015 DEBUG:boto:Data: 2024-04-12T03:47:19.015 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.016 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.016 DEBUG:boto:Params: {} 2024-04-12T03:47:19.016 DEBUG:boto:Token: None 2024-04-12T03:47:19.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs= 2024-04-12T03:47:19.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs=', 'Content-Length': '0'} 2024-04-12T03:47:19.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb240dcd6b7fe93b-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.018 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.018 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.018 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.018 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.018 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.019 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.019 DEBUG:boto:Data: 2024-04-12T03:47:19.019 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.019 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.019 DEBUG:boto:Params: {} 2024-04-12T03:47:19.019 DEBUG:boto:Token: None 2024-04-12T03:47:19.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Dr55e72ekcVJGf1nEGIJu1GDIM= 2024-04-12T03:47:19.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Dr55e72ekcVJGf1nEGIJu1GDIM=', 'Content-Length': '0'} 2024-04-12T03:47:19.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9429e9a246eb9f9-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.021 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.021 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.021 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.022 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.022 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.022 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.022 DEBUG:boto:Data: 2024-04-12T03:47:19.022 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.023 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.023 DEBUG:boto:Params: {} 2024-04-12T03:47:19.023 DEBUG:boto:Token: None 2024-04-12T03:47:19.023 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Dr55e72ekcVJGf1nEGIJu1GDIM= 2024-04-12T03:47:19.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Dr55e72ekcVJGf1nEGIJu1GDIM=', 'Content-Length': '0'} 2024-04-12T03:47:19.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69cc49141eab32f-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:47:19.025 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.025 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.025 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.025 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.025 DEBUG:boto:Method: GET 2024-04-12T03:47:19.025 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.025 DEBUG:boto:Data: 2024-04-12T03:47:19.026 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.026 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.026 DEBUG:boto:Params: {} 2024-04-12T03:47:19.027 DEBUG:boto:Token: None 2024-04-12T03:47:19.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs= 2024-04-12T03:47:19.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs=', 'Content-Length': '0'} 2024-04-12T03:47:19.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cced9b2ad32ec89-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.074 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.075 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.075 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.075 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.075 DEBUG:boto:Method: GET 2024-04-12T03:47:19.075 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.075 DEBUG:boto:Data: 2024-04-12T03:47:19.075 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.075 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.075 DEBUG:boto:Params: {} 2024-04-12T03:47:19.076 DEBUG:boto:Token: None 2024-04-12T03:47:19.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:47:19.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs= 2024-04-12T03:47:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KK4xZxPRTiz4NfoapX4sRvKIjJs=', 'Content-Length': '0'} 2024-04-12T03:47:19.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8057219faff3b0-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.078 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.078 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.078 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.078 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.078 DEBUG:boto:Method: GET 2024-04-12T03:47:19.079 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.079 DEBUG:boto:Data: 2024-04-12T03:47:19.079 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.079 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.079 DEBUG:boto:Params: {} 2024-04-12T03:47:19.079 DEBUG:boto:Token: None 2024-04-12T03:47:19.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4f59d947ebffca-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.082 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.082 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.082 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.082 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.082 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.083 DEBUG:boto:Method: GET 2024-04-12T03:47:19.083 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.083 DEBUG:boto:Data: 2024-04-12T03:47:19.083 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.083 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.083 DEBUG:boto:Params: {} 2024-04-12T03:47:19.083 DEBUG:boto:Token: None 2024-04-12T03:47:19.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013637583ffe2d9b1-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.086 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.087 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.087 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.087 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.087 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:47:19.087 DEBUG:boto:Data: 2024-04-12T03:47:19.087 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.088 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.088 DEBUG:boto:Params: {} 2024-04-12T03:47:19.088 DEBUG:boto:Token: None 2024-04-12T03:47:19.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76 2024-04-12T03:47:19.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kFgG47P6kq3sGe+/AjWdNq+Z9tY= 2024-04-12T03:47:19.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kFgG47P6kq3sGe+/AjWdNq+Z9tY=', 'Content-Length': '0'} 2024-04-12T03:47:19.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b530ad1848b04cb-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.091 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.091 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.091 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.091 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:47:19.091 DEBUG:boto:Data: 2024-04-12T03:47:19.091 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.091 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.091 DEBUG:boto:Params: {} 2024-04-12T03:47:19.092 DEBUG:boto:Token: None 2024-04-12T03:47:19.092 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76 2024-04-12T03:47:19.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kFgG47P6kq3sGe+/AjWdNq+Z9tY= 2024-04-12T03:47:19.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kFgG47P6kq3sGe+/AjWdNq+Z9tY=', 'Content-Length': '0'} 2024-04-12T03:47:19.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028d7ad9eac30f881-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:47:19.094 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.094 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.094 DEBUG:boto:Method: GET 2024-04-12T03:47:19.094 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:47:19.094 DEBUG:boto:Data: 2024-04-12T03:47:19.094 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.095 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.095 DEBUG:boto:Params: {} 2024-04-12T03:47:19.095 DEBUG:boto:Token: None 2024-04-12T03:47:19.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76 2024-04-12T03:47:19.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7awXqEy46OGMJQmFZUo3PM5tvd8= 2024-04-12T03:47:19.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7awXqEy46OGMJQmFZUo3PM5tvd8=', 'Content-Length': '0'} 2024-04-12T03:47:19.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e11d85c32e0f544-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.097 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:47:19.097 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:47:19.097 DEBUG:boto:Method: GET 2024-04-12T03:47:19.097 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:47:19.097 DEBUG:boto:Data: 2024-04-12T03:47:19.097 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.097 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.098 DEBUG:boto:Params: {} 2024-04-12T03:47:19.098 DEBUG:boto:Token: None 2024-04-12T03:47:19.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj76 2024-04-12T03:47:19.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7awXqEy46OGMJQmFZUo3PM5tvd8= 2024-04-12T03:47:19.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7awXqEy46OGMJQmFZUo3PM5tvd8=', 'Content-Length': '0'} 2024-04-12T03:47:19.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830ddf61fb540a4c-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.100 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.100 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.101 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.101 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.101 DEBUG:boto:Method: GET 2024-04-12T03:47:19.101 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.101 DEBUG:boto:Data: 2024-04-12T03:47:19.101 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.101 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.101 DEBUG:boto:Params: {} 2024-04-12T03:47:19.101 DEBUG:boto:Token: None 2024-04-12T03:47:19.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003576ff55d90f8aff-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.104 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.104 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.104 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.104 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.105 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.105 DEBUG:boto:Method: GET 2024-04-12T03:47:19.105 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:47:19.105 DEBUG:boto:Data: 2024-04-12T03:47:19.105 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.105 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.105 DEBUG:boto:Params: {} 2024-04-12T03:47:19.105 DEBUG:boto:Token: None 2024-04-12T03:47:19.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c60a0d1e63bba86-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.108 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:47:19.109 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.109 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.109 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.109 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.109 DEBUG:boto:Method: GET 2024-04-12T03:47:19.109 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.109 DEBUG:boto:Data: 2024-04-12T03:47:19.109 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.110 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.110 DEBUG:boto:Params: {} 2024-04-12T03:47:19.110 DEBUG:boto:Token: None 2024-04-12T03:47:19.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc= 2024-04-12T03:47:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc=', 'Content-Length': '0'} 2024-04-12T03:47:19.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069577860b5e3e1b0-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.112 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.112 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.113 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.113 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.113 DEBUG:boto:Method: GET 2024-04-12T03:47:19.113 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.113 DEBUG:boto:Data: 2024-04-12T03:47:19.113 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.113 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.113 DEBUG:boto:Params: {} 2024-04-12T03:47:19.114 DEBUG:boto:Token: None 2024-04-12T03:47:19.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc= 2024-04-12T03:47:19.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc=', 'Content-Length': '0'} 2024-04-12T03:47:19.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d083d9eebea5b275-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.116 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.116 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.116 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.116 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.116 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.116 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.117 DEBUG:boto:Data: 2024-04-12T03:47:19.117 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.117 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.117 DEBUG:boto:Params: {} 2024-04-12T03:47:19.117 DEBUG:boto:Token: None 2024-04-12T03:47:19.117 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SdsCeiT4gBpm3G1O7ppQCtBegFg= 2024-04-12T03:47:19.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SdsCeiT4gBpm3G1O7ppQCtBegFg=', 'Content-Length': '0'} 2024-04-12T03:47:19.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e1fcf0cfc2c4a8-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.119 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.119 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.119 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.120 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.120 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.120 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.120 DEBUG:boto:Data: 2024-04-12T03:47:19.120 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.120 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.120 DEBUG:boto:Params: {} 2024-04-12T03:47:19.120 DEBUG:boto:Token: None 2024-04-12T03:47:19.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SdsCeiT4gBpm3G1O7ppQCtBegFg= 2024-04-12T03:47:19.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SdsCeiT4gBpm3G1O7ppQCtBegFg=', 'Content-Length': '0'} 2024-04-12T03:47:19.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d7d5b25abe2317a-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:47:19.122 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.123 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.123 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.123 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.123 DEBUG:boto:Method: GET 2024-04-12T03:47:19.123 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.123 DEBUG:boto:Data: 2024-04-12T03:47:19.123 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.123 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.123 DEBUG:boto:Params: {} 2024-04-12T03:47:19.123 DEBUG:boto:Token: None 2024-04-12T03:47:19.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc= 2024-04-12T03:47:19.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc=', 'Content-Length': '0'} 2024-04-12T03:47:19.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb5fc20eca230d3-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.166 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.167 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.167 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.167 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.167 DEBUG:boto:Method: GET 2024-04-12T03:47:19.167 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.167 DEBUG:boto:Data: 2024-04-12T03:47:19.167 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.168 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.168 DEBUG:boto:Params: {} 2024-04-12T03:47:19.168 DEBUG:boto:Token: None 2024-04-12T03:47:19.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:47:19.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc= 2024-04-12T03:47:19.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TJKbBIBUgDP4lxFzQEYPUF4atWc=', 'Content-Length': '0'} 2024-04-12T03:47:19.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08d1ee9bfa07685-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.170 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.170 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.171 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.171 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.171 DEBUG:boto:Method: GET 2024-04-12T03:47:19.171 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.171 DEBUG:boto:Data: 2024-04-12T03:47:19.171 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.171 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.171 DEBUG:boto:Params: {} 2024-04-12T03:47:19.171 DEBUG:boto:Token: None 2024-04-12T03:47:19.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f75f23f530ead557-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.174 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.174 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.174 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.174 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.175 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.175 DEBUG:boto:Method: GET 2024-04-12T03:47:19.175 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.175 DEBUG:boto:Data: 2024-04-12T03:47:19.175 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.175 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.175 DEBUG:boto:Params: {} 2024-04-12T03:47:19.175 DEBUG:boto:Token: None 2024-04-12T03:47:19.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002338762c92cb95c9-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.178 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.179 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.179 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.179 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.179 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:47:19.179 DEBUG:boto:Data: 2024-04-12T03:47:19.179 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.179 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.180 DEBUG:boto:Params: {} 2024-04-12T03:47:19.180 DEBUG:boto:Token: None 2024-04-12T03:47:19.180 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85 2024-04-12T03:47:19.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gtZTMrRRjQIG3XLfQfpQ/4iRTME= 2024-04-12T03:47:19.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gtZTMrRRjQIG3XLfQfpQ/4iRTME=', 'Content-Length': '0'} 2024-04-12T03:47:19.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d3fb13f2c4259f-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.182 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.182 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.182 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.182 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:47:19.182 DEBUG:boto:Data: 2024-04-12T03:47:19.183 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.183 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.183 DEBUG:boto:Params: {} 2024-04-12T03:47:19.183 DEBUG:boto:Token: None 2024-04-12T03:47:19.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85 2024-04-12T03:47:19.183 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gtZTMrRRjQIG3XLfQfpQ/4iRTME= 2024-04-12T03:47:19.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gtZTMrRRjQIG3XLfQfpQ/4iRTME=', 'Content-Length': '0'} 2024-04-12T03:47:19.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce62efb64971626-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:47:19.185 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.185 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.185 DEBUG:boto:Method: GET 2024-04-12T03:47:19.185 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:47:19.185 DEBUG:boto:Data: 2024-04-12T03:47:19.185 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.185 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.186 DEBUG:boto:Params: {} 2024-04-12T03:47:19.186 DEBUG:boto:Token: None 2024-04-12T03:47:19.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85 2024-04-12T03:47:19.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:byUq464XwnorXjLOx0Hu3J+hbEc= 2024-04-12T03:47:19.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:byUq464XwnorXjLOx0Hu3J+hbEc=', 'Content-Length': '0'} 2024-04-12T03:47:19.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ced9d1bfc14f0484-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.188 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:47:19.188 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:47:19.188 DEBUG:boto:Method: GET 2024-04-12T03:47:19.189 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:47:19.189 DEBUG:boto:Data: 2024-04-12T03:47:19.189 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.189 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.189 DEBUG:boto:Params: {} 2024-04-12T03:47:19.189 DEBUG:boto:Token: None 2024-04-12T03:47:19.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj85 2024-04-12T03:47:19.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:byUq464XwnorXjLOx0Hu3J+hbEc= 2024-04-12T03:47:19.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:byUq464XwnorXjLOx0Hu3J+hbEc=', 'Content-Length': '0'} 2024-04-12T03:47:19.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f86213a97c6d3c68-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.234 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.235 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.235 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.235 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.235 DEBUG:boto:Method: GET 2024-04-12T03:47:19.235 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.235 DEBUG:boto:Data: 2024-04-12T03:47:19.235 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.235 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.235 DEBUG:boto:Params: {} 2024-04-12T03:47:19.236 DEBUG:boto:Token: None 2024-04-12T03:47:19.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c1fa27391644e6-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.238 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.239 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.239 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.239 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.239 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.239 DEBUG:boto:Method: GET 2024-04-12T03:47:19.239 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:47:19.239 DEBUG:boto:Data: 2024-04-12T03:47:19.239 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.240 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.240 DEBUG:boto:Params: {} 2024-04-12T03:47:19.240 DEBUG:boto:Token: None 2024-04-12T03:47:19.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116d1a17dde42676-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.243 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:47:19.244 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.244 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.244 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.244 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.244 DEBUG:boto:Method: GET 2024-04-12T03:47:19.244 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.244 DEBUG:boto:Data: 2024-04-12T03:47:19.244 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.245 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.245 DEBUG:boto:Params: {} 2024-04-12T03:47:19.245 DEBUG:boto:Token: None 2024-04-12T03:47:19.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s= 2024-04-12T03:47:19.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s=', 'Content-Length': '0'} 2024-04-12T03:47:19.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043fd86d5634944e7-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.247 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.247 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.247 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.248 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.248 DEBUG:boto:Method: GET 2024-04-12T03:47:19.248 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.248 DEBUG:boto:Data: 2024-04-12T03:47:19.248 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.248 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.248 DEBUG:boto:Params: {} 2024-04-12T03:47:19.248 DEBUG:boto:Token: None 2024-04-12T03:47:19.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s= 2024-04-12T03:47:19.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s=', 'Content-Length': '0'} 2024-04-12T03:47:19.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccdf7fce1704be2e-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.250 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.251 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.251 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.251 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.251 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.251 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.251 DEBUG:boto:Data: 2024-04-12T03:47:19.251 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.251 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.251 DEBUG:boto:Params: {} 2024-04-12T03:47:19.251 DEBUG:boto:Token: None 2024-04-12T03:47:19.252 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+rD1hWv8v7OBROl3dZXaZBUUkA= 2024-04-12T03:47:19.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+rD1hWv8v7OBROl3dZXaZBUUkA=', 'Content-Length': '0'} 2024-04-12T03:47:19.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd54cbae5efa525-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.253 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.254 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.254 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.254 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.254 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.254 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.254 DEBUG:boto:Data: 2024-04-12T03:47:19.254 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.254 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.254 DEBUG:boto:Params: {} 2024-04-12T03:47:19.255 DEBUG:boto:Token: None 2024-04-12T03:47:19.255 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.255 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+rD1hWv8v7OBROl3dZXaZBUUkA= 2024-04-12T03:47:19.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+rD1hWv8v7OBROl3dZXaZBUUkA=', 'Content-Length': '0'} 2024-04-12T03:47:19.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064cbfe32060cc016-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:47:19.257 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.257 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.257 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.257 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.257 DEBUG:boto:Method: GET 2024-04-12T03:47:19.257 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.257 DEBUG:boto:Data: 2024-04-12T03:47:19.257 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.257 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.258 DEBUG:boto:Params: {} 2024-04-12T03:47:19.258 DEBUG:boto:Token: None 2024-04-12T03:47:19.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.258 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s= 2024-04-12T03:47:19.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s=', 'Content-Length': '0'} 2024-04-12T03:47:19.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7c94e988383d77-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.260 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.260 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.260 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.260 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.260 DEBUG:boto:Method: GET 2024-04-12T03:47:19.260 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.260 DEBUG:boto:Data: 2024-04-12T03:47:19.260 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.261 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.261 DEBUG:boto:Params: {} 2024-04-12T03:47:19.261 DEBUG:boto:Token: None 2024-04-12T03:47:19.261 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:47:19.261 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s= 2024-04-12T03:47:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M8eh4+QLjiyLBohdT9W+zJ5Ca/s=', 'Content-Length': '0'} 2024-04-12T03:47:19.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf0f9146d1dab867-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.306 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.307 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.307 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.307 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.307 DEBUG:boto:Method: GET 2024-04-12T03:47:19.307 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.307 DEBUG:boto:Data: 2024-04-12T03:47:19.307 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.307 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.307 DEBUG:boto:Params: {} 2024-04-12T03:47:19.308 DEBUG:boto:Token: None 2024-04-12T03:47:19.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c747abf3492595-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.310 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.310 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.311 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.311 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.311 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.311 DEBUG:boto:Method: GET 2024-04-12T03:47:19.311 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.311 DEBUG:boto:Data: 2024-04-12T03:47:19.311 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.311 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.311 DEBUG:boto:Params: {} 2024-04-12T03:47:19.311 DEBUG:boto:Token: None 2024-04-12T03:47:19.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001948718161066f84-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.314 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.315 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.315 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.315 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.315 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:47:19.315 DEBUG:boto:Data: 2024-04-12T03:47:19.315 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.315 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.315 DEBUG:boto:Params: {} 2024-04-12T03:47:19.316 DEBUG:boto:Token: None 2024-04-12T03:47:19.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92 2024-04-12T03:47:19.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zAuQiRbXc9TSKIjcapfcz9e1R9s= 2024-04-12T03:47:19.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zAuQiRbXc9TSKIjcapfcz9e1R9s=', 'Content-Length': '0'} 2024-04-12T03:47:19.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440b5bdcfb4bf146-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.318 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.318 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.318 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.318 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:47:19.318 DEBUG:boto:Data: 2024-04-12T03:47:19.318 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.318 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.318 DEBUG:boto:Params: {} 2024-04-12T03:47:19.319 DEBUG:boto:Token: None 2024-04-12T03:47:19.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92 2024-04-12T03:47:19.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zAuQiRbXc9TSKIjcapfcz9e1R9s= 2024-04-12T03:47:19.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zAuQiRbXc9TSKIjcapfcz9e1R9s=', 'Content-Length': '0'} 2024-04-12T03:47:19.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411e773069cd4d43-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:47:19.321 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.321 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.321 DEBUG:boto:Method: GET 2024-04-12T03:47:19.322 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:47:19.322 DEBUG:boto:Data: 2024-04-12T03:47:19.322 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.322 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.322 DEBUG:boto:Params: {} 2024-04-12T03:47:19.322 DEBUG:boto:Token: None 2024-04-12T03:47:19.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92 2024-04-12T03:47:19.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mlq/kGl1ZIh0deI446/jGG3/x0g= 2024-04-12T03:47:19.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mlq/kGl1ZIh0deI446/jGG3/x0g=', 'Content-Length': '0'} 2024-04-12T03:47:19.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295723c08a7c84d8-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.324 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:47:19.324 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:47:19.324 DEBUG:boto:Method: GET 2024-04-12T03:47:19.324 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:47:19.325 DEBUG:boto:Data: 2024-04-12T03:47:19.325 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.325 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.325 DEBUG:boto:Params: {} 2024-04-12T03:47:19.325 DEBUG:boto:Token: None 2024-04-12T03:47:19.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj92 2024-04-12T03:47:19.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mlq/kGl1ZIh0deI446/jGG3/x0g= 2024-04-12T03:47:19.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mlq/kGl1ZIh0deI446/jGG3/x0g=', 'Content-Length': '0'} 2024-04-12T03:47:19.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a211bf1fedf11108-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.327 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.327 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.328 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.328 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.328 DEBUG:boto:Method: GET 2024-04-12T03:47:19.328 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.328 DEBUG:boto:Data: 2024-04-12T03:47:19.328 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.328 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.328 DEBUG:boto:Params: {} 2024-04-12T03:47:19.328 DEBUG:boto:Token: None 2024-04-12T03:47:19.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000135101d6095b5f52-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.331 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.331 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.331 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.331 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.331 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.331 DEBUG:boto:Method: GET 2024-04-12T03:47:19.331 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:47:19.332 DEBUG:boto:Data: 2024-04-12T03:47:19.332 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.332 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.332 DEBUG:boto:Params: {} 2024-04-12T03:47:19.332 DEBUG:boto:Token: None 2024-04-12T03:47:19.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e70661bb0d6852-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.378 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:47:19.379 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.379 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.379 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.379 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.379 DEBUG:boto:Method: GET 2024-04-12T03:47:19.380 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.380 DEBUG:boto:Data: 2024-04-12T03:47:19.380 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.380 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.380 DEBUG:boto:Params: {} 2024-04-12T03:47:19.380 DEBUG:boto:Token: None 2024-04-12T03:47:19.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc= 2024-04-12T03:47:19.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc=', 'Content-Length': '0'} 2024-04-12T03:47:19.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b713bb38049d2d6-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.383 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.383 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.383 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.383 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.383 DEBUG:boto:Method: GET 2024-04-12T03:47:19.383 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.383 DEBUG:boto:Data: 2024-04-12T03:47:19.383 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.383 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.384 DEBUG:boto:Params: {} 2024-04-12T03:47:19.384 DEBUG:boto:Token: None 2024-04-12T03:47:19.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc= 2024-04-12T03:47:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc=', 'Content-Length': '0'} 2024-04-12T03:47:19.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080508adece5a4ef9-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.386 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.386 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.386 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.387 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.387 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.387 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.387 DEBUG:boto:Data: 2024-04-12T03:47:19.387 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.387 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.387 DEBUG:boto:Params: {} 2024-04-12T03:47:19.387 DEBUG:boto:Token: None 2024-04-12T03:47:19.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KmmWv4R7fptoGNMY7AbewCIjls0= 2024-04-12T03:47:19.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmWv4R7fptoGNMY7AbewCIjls0=', 'Content-Length': '0'} 2024-04-12T03:47:19.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fdc42c56bee993-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.389 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.390 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.390 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.390 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.390 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.390 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.390 DEBUG:boto:Data: 2024-04-12T03:47:19.390 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.390 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.390 DEBUG:boto:Params: {} 2024-04-12T03:47:19.391 DEBUG:boto:Token: None 2024-04-12T03:47:19.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KmmWv4R7fptoGNMY7AbewCIjls0= 2024-04-12T03:47:19.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmWv4R7fptoGNMY7AbewCIjls0=', 'Content-Length': '0'} 2024-04-12T03:47:19.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18165136622a357-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:47:19.393 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.393 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.393 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.393 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.393 DEBUG:boto:Method: GET 2024-04-12T03:47:19.393 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.393 DEBUG:boto:Data: 2024-04-12T03:47:19.393 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.394 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.394 DEBUG:boto:Params: {} 2024-04-12T03:47:19.394 DEBUG:boto:Token: None 2024-04-12T03:47:19.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.394 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc= 2024-04-12T03:47:19.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc=', 'Content-Length': '0'} 2024-04-12T03:47:19.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f549b97fa4a4ff-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.438 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.439 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.439 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.439 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.439 DEBUG:boto:Method: GET 2024-04-12T03:47:19.439 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.439 DEBUG:boto:Data: 2024-04-12T03:47:19.439 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.439 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.439 DEBUG:boto:Params: {} 2024-04-12T03:47:19.440 DEBUG:boto:Token: None 2024-04-12T03:47:19.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:47:19.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc= 2024-04-12T03:47:19.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IE5jy5GnihCytlY790tcrpJU/xc=', 'Content-Length': '0'} 2024-04-12T03:47:19.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cacb53c82aa8f050-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.442 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.442 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.442 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.442 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.442 DEBUG:boto:Method: GET 2024-04-12T03:47:19.443 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.443 DEBUG:boto:Data: 2024-04-12T03:47:19.443 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.443 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.443 DEBUG:boto:Params: {} 2024-04-12T03:47:19.443 DEBUG:boto:Token: None 2024-04-12T03:47:19.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f13a8aa8b01347-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.445 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.446 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.446 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.446 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.446 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.446 DEBUG:boto:Method: GET 2024-04-12T03:47:19.446 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.446 DEBUG:boto:Data: 2024-04-12T03:47:19.446 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.447 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.447 DEBUG:boto:Params: {} 2024-04-12T03:47:19.447 DEBUG:boto:Token: None 2024-04-12T03:47:19.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1eb393e2c4cffca-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.450 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.450 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.450 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.450 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.451 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:47:19.451 DEBUG:boto:Data: 2024-04-12T03:47:19.451 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.451 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.451 DEBUG:boto:Params: {} 2024-04-12T03:47:19.451 DEBUG:boto:Token: None 2024-04-12T03:47:19.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95 2024-04-12T03:47:19.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CoUbGMHbBBLiJSfHGH08EDZ1hF8= 2024-04-12T03:47:19.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CoUbGMHbBBLiJSfHGH08EDZ1hF8=', 'Content-Length': '0'} 2024-04-12T03:47:19.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002af4d4fa33c22d9f-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.453 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.453 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.453 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.453 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:47:19.453 DEBUG:boto:Data: 2024-04-12T03:47:19.453 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.454 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.454 DEBUG:boto:Params: {} 2024-04-12T03:47:19.454 DEBUG:boto:Token: None 2024-04-12T03:47:19.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95 2024-04-12T03:47:19.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CoUbGMHbBBLiJSfHGH08EDZ1hF8= 2024-04-12T03:47:19.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CoUbGMHbBBLiJSfHGH08EDZ1hF8=', 'Content-Length': '0'} 2024-04-12T03:47:19.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0bfe8992bc089d-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:47:19.456 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.456 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.456 DEBUG:boto:Method: GET 2024-04-12T03:47:19.456 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:47:19.456 DEBUG:boto:Data: 2024-04-12T03:47:19.456 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.457 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.457 DEBUG:boto:Params: {} 2024-04-12T03:47:19.457 DEBUG:boto:Token: None 2024-04-12T03:47:19.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95 2024-04-12T03:47:19.457 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cFyWO5+6AideGjyB0UniQ98doM8= 2024-04-12T03:47:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cFyWO5+6AideGjyB0UniQ98doM8=', 'Content-Length': '0'} 2024-04-12T03:47:19.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8a8978f867789c-006618aec7-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.459 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:47:19.459 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:47:19.459 DEBUG:boto:Method: GET 2024-04-12T03:47:19.459 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:47:19.459 DEBUG:boto:Data: 2024-04-12T03:47:19.459 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.459 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.459 DEBUG:boto:Params: {} 2024-04-12T03:47:19.460 DEBUG:boto:Token: None 2024-04-12T03:47:19.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj95 2024-04-12T03:47:19.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cFyWO5+6AideGjyB0UniQ98doM8= 2024-04-12T03:47:19.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cFyWO5+6AideGjyB0UniQ98doM8=', 'Content-Length': '0'} 2024-04-12T03:47:19.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede9921c4b4c97e9-006618aec7-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.462 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.462 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.462 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.462 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.462 DEBUG:boto:Method: GET 2024-04-12T03:47:19.462 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.462 DEBUG:boto:Data: 2024-04-12T03:47:19.462 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:47:19.463 DEBUG:boto:Port: 8001 2024-04-12T03:47:19.463 DEBUG:boto:Params: {} 2024-04-12T03:47:19.463 DEBUG:boto:Token: None 2024-04-12T03:47:19.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e116a05fa60da8-006618aec7-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.465 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.466 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.466 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.466 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.466 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.466 DEBUG:boto:Method: GET 2024-04-12T03:47:19.466 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:47:19.466 DEBUG:boto:Data: 2024-04-12T03:47:19.466 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.466 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.467 DEBUG:boto:Params: {} 2024-04-12T03:47:19.467 DEBUG:boto:Token: None 2024-04-12T03:47:19.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k= 2024-04-12T03:47:19.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bh1U/SBnETJj2cUpLWDC69yix6k=', 'Content-Length': '0'} 2024-04-12T03:47:19.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6036a21bc8c56e-006618aec7-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.469 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:19.470 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:47:19.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:19.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:19.471 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:47:19.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:19.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:19.567 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:47:19.567 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:47:19.575 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2024-04-12T03:47:19.627 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.627 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.627 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.628 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:19.628 DEBUG:boto:Data: 2024-04-12T03:47:19.628 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.628 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.628 DEBUG:boto:Params: {} 2024-04-12T03:47:19.628 DEBUG:boto:Token: None 2024-04-12T03:47:19.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w= 2024-04-12T03:47:19.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w=', 'Content-Length': '0'} 2024-04-12T03:47:19.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005786cb2346eb4323-006618aec7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.632 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:47:19.632 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:47:19.632 DEBUG:boto:Method: PUT 2024-04-12T03:47:19.632 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:47:19.632 DEBUG:boto:Data: 2024-04-12T03:47:19.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:47:19.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.633 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.633 DEBUG:boto:Params: {} 2024-04-12T03:47:19.633 DEBUG:boto:Token: None 2024-04-12T03:47:19.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj29 2024-04-12T03:47:19.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K+alrXRSbeyRYCOO0LMdy/4JilQ= 2024-04-12T03:47:19.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K+alrXRSbeyRYCOO0LMdy/4JilQ='} 2024-04-12T03:47:19.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ccb2f09abcfd134-006618aec7-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.638 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.638 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.638 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.639 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:19.639 DEBUG:boto:Data: 2024-04-12T03:47:19.639 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.639 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.639 DEBUG:boto:Params: {} 2024-04-12T03:47:19.639 DEBUG:boto:Token: None 2024-04-12T03:47:19.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w= 2024-04-12T03:47:19.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w=', 'Content-Length': '0'} 2024-04-12T03:47:19.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e5ab4fb30d563fc-006618aec7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.644 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:47:19.644 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:47:19.644 DEBUG:boto:Method: PUT 2024-04-12T03:47:19.644 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:47:19.644 DEBUG:boto:Data: 2024-04-12T03:47:19.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:47:19.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.645 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.645 DEBUG:boto:Params: {} 2024-04-12T03:47:19.645 DEBUG:boto:Token: None 2024-04-12T03:47:19.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj58 2024-04-12T03:47:19.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H30L5JChy5SZl3vC4lNzVrGp2Jg= 2024-04-12T03:47:19.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H30L5JChy5SZl3vC4lNzVrGp2Jg='} 2024-04-12T03:47:19.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a478a6f9485464de-006618aec7-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.687 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.687 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.687 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.687 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:19.687 DEBUG:boto:Data: 2024-04-12T03:47:19.687 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.688 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.688 DEBUG:boto:Params: {} 2024-04-12T03:47:19.688 DEBUG:boto:Token: None 2024-04-12T03:47:19.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w= 2024-04-12T03:47:19.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w=', 'Content-Length': '0'} 2024-04-12T03:47:19.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9b092b7e8214c6f-006618aec7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.692 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:47:19.692 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:47:19.692 DEBUG:boto:Method: PUT 2024-04-12T03:47:19.692 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:47:19.692 DEBUG:boto:Data: 2024-04-12T03:47:19.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:47:19.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.693 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.693 DEBUG:boto:Params: {} 2024-04-12T03:47:19.693 DEBUG:boto:Token: None 2024-04-12T03:47:19.693 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj87 2024-04-12T03:47:19.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dDqkNEJydetdIe5b2su2zMTy7hU= 2024-04-12T03:47:19.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dDqkNEJydetdIe5b2su2zMTy7hU='} 2024-04-12T03:47:19.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050a66995f823a0ae-006618aec7-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.698 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.698 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.698 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.698 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:19.699 DEBUG:boto:Data: 2024-04-12T03:47:19.699 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.699 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.699 DEBUG:boto:Params: {} 2024-04-12T03:47:19.699 DEBUG:boto:Token: None 2024-04-12T03:47:19.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w= 2024-04-12T03:47:19.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w=', 'Content-Length': '0'} 2024-04-12T03:47:19.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf2aa12032c9583c-006618aec7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.703 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:47:19.703 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:47:19.703 DEBUG:boto:Method: PUT 2024-04-12T03:47:19.703 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:47:19.703 DEBUG:boto:Data: 2024-04-12T03:47:19.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:47:19.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.703 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.703 DEBUG:boto:Params: {} 2024-04-12T03:47:19.704 DEBUG:boto:Token: None 2024-04-12T03:47:19.704 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj116 2024-04-12T03:47:19.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Le1AtywOOEYu+CbY8ueoOPE4aSI= 2024-04-12T03:47:19.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Le1AtywOOEYu+CbY8ueoOPE4aSI='} 2024-04-12T03:47:19.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a4dbf8b837e6f83-006618aec7-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.751 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:47:19.751 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:47:19.751 DEBUG:boto:Method: HEAD 2024-04-12T03:47:19.751 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:47:19.751 DEBUG:boto:Data: 2024-04-12T03:47:19.751 DEBUG:boto:Headers: {} 2024-04-12T03:47:19.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.751 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.752 DEBUG:boto:Params: {} 2024-04-12T03:47:19.752 DEBUG:boto:Token: None 2024-04-12T03:47:19.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/ 2024-04-12T03:47:19.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w= 2024-04-12T03:47:19.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZ0m2dVptXeie7HIdrG0tHUgV9w=', 'Content-Length': '0'} 2024-04-12T03:47:19.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000319ee662a7d6050-006618aec7-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.755 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:47:19.755 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:47:19.755 DEBUG:boto:Method: PUT 2024-04-12T03:47:19.756 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:47:19.756 DEBUG:boto:Data: 2024-04-12T03:47:19.756 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:47:19.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:47:19.756 DEBUG:boto:Port: 8000 2024-04-12T03:47:19.756 DEBUG:boto:Params: {} 2024-04-12T03:47:19.756 DEBUG:boto:Token: None 2024-04-12T03:47:19.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:47:19 GMT /dgbpxg-61/obj145 2024-04-12T03:47:19.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S7KD0ijtKBn4rttbNOQal7L06JI= 2024-04-12T03:47:19.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S7KD0ijtKBn4rttbNOQal7L06JI='} 2024-04-12T03:47:19.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001137b743fce318f7-006618aec7-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:19.761 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:47:19.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:19.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:19.762 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:19.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:19.817 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:19.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:19.854+0000 7f8b63b1d780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T03:47:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:24.854+0000 7f8b63b1d780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T03:47:29.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:29.855+0000 7f8b63b1d780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T03:47:34.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:34.855+0000 7f8b63b1d780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T03:47:39.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:39.858+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:39.856 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:39.856 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:44.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:44.861+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:44.858 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:44.858 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:49.863+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:49.860 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:49.860 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:54.865+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:54.862 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:54.862 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:59.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:47:59.867+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:59.864 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:47:59.864 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:04.869+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:04.866 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:04.866 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:09.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:09.870+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:09.868 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:09.868 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:14.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:14.872+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:14.870 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:14.870 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:19.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:19.875+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:19.872 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:19.872 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:24.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:24.878+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:24.874 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:24.874 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:29.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:29.880+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:29.876 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:29.876 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:34.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:34.882+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:34.878 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:34.878 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:39.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:39.884+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:39.881 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:39.881 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:44.886+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:44.882 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:44.882 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:49.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:49.887+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:49.884 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:49.884 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:54.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:54.889+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:54.886 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, , , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:54.886 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:59.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:48:59.891+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:59.888 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:48:59.888 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:04.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:04.895+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:04.891 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:04.891 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:09.897+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:09.893 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:09.893 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:14.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:14.899+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:14.894 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:14.895 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:19.901+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:19.897 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:19.897 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:24.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:24.904+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:24.899 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:24.899 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:29.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:29.906+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:29.901 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:29.901 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:34.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:34.907+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:34.903 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:34.903 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:39.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:39.909+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:39.905 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:39.905 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:44.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:44.913+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:44.908 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:44.908 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:49.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:49.915+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:49.910 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:49.910 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:54.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:54.917+0000 7f8b63b1d780 1 waiting for incremental sync to catch up: 2024-04-12T03:49:54.912 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , , 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:54.912 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:59.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:59.919+0000 7f8b63b1d780 1 bucket sync caught up with source: 2024-04-12T03:49:59.914 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:59.914 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.63.6, 00000000001.95.6, , , , , 00000000001.92.6, , , , , , , , , , , 00000000002.64.6, 00000000001.75.6, , , , , , , , , , ] 2024-04-12T03:49:59.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:49:59.919+0000 7f8b63b1d780 0 bucket checkpoint complete 2024-04-12T03:49:59.932 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:49:59.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:59.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:59.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:59.992 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:59.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:00.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:00.036+0000 7f706bd2b780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:00.031 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:00.031 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000003.155.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:05.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:05.038+0000 7f706bd2b780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:05.033 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:05.033 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000003.155.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:10.040+0000 7f706bd2b780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:10.034 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000002.151.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:10.034 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000003.155.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:15.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:15.041+0000 7f706bd2b780 1 bucket sync caught up with source: 2024-04-12T03:50:15.035 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000003.155.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:15.035 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1072.6, 00000000002.2097.6, 00000000001.1925.6, 00000000003.155.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000005.2240.6, 00000000004.1491.6] 2024-04-12T03:50:15.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:15.041+0000 7f706bd2b780 0 bucket checkpoint complete 2024-04-12T03:50:15.056 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:50:15.056 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.056 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.057 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.057 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:15.057 DEBUG:boto:Data: 2024-04-12T03:50:15.057 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.057 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.057 DEBUG:boto:Params: {} 2024-04-12T03:50:15.058 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:50:15.058 DEBUG:boto:Token: None 2024-04-12T03:50:15.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ry+5TA6ZHqhuIOcxKMEApjupzlA= 2024-04-12T03:50:15.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ry+5TA6ZHqhuIOcxKMEApjupzlA=', 'Content-Length': '0'} 2024-04-12T03:50:15.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000001e659616c56121-006618af77-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.077 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.077 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.077 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.077 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:15.077 DEBUG:boto:Data: 2024-04-12T03:50:15.077 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.077 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.078 DEBUG:boto:Params: {} 2024-04-12T03:50:15.078 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:50:15.078 DEBUG:boto:Token: None 2024-04-12T03:50:15.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ry+5TA6ZHqhuIOcxKMEApjupzlA= 2024-04-12T03:50:15.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ry+5TA6ZHqhuIOcxKMEApjupzlA=', 'Content-Length': '0'} 2024-04-12T03:50:15.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce2b8710deb621e1-006618af77-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:50:15.110 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.110 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.110 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.111 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.111 DEBUG:boto:Method: GET 2024-04-12T03:50:15.111 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.111 DEBUG:boto:Data: 2024-04-12T03:50:15.111 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.111 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.111 DEBUG:boto:Params: {} 2024-04-12T03:50:15.111 DEBUG:boto:Token: None 2024-04-12T03:50:15.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/?versions 2024-04-12T03:50:15.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM= 2024-04-12T03:50:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM=', 'Content-Length': '0'} 2024-04-12T03:50:15.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9075a22bf6a819-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.115 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:50:15.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:50:15.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:50:15.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:50:15.119 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.119 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.119 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.119 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.120 DEBUG:boto:Method: GET 2024-04-12T03:50:15.120 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.120 DEBUG:boto:Data: 2024-04-12T03:50:15.120 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.120 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.120 DEBUG:boto:Params: {} 2024-04-12T03:50:15.120 DEBUG:boto:Token: None 2024-04-12T03:50:15.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/?versions 2024-04-12T03:50:15.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM= 2024-04-12T03:50:15.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM=', 'Content-Length': '0'} 2024-04-12T03:50:15.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061afd9feb963eb82-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.124 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:50:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:50:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:50:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:50:15.128 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.128 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.128 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.128 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.129 DEBUG:boto:Method: GET 2024-04-12T03:50:15.129 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.129 DEBUG:boto:Data: 2024-04-12T03:50:15.129 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.129 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.129 DEBUG:boto:Params: {} 2024-04-12T03:50:15.129 DEBUG:boto:Token: None 2024-04-12T03:50:15.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/?versions 2024-04-12T03:50:15.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM= 2024-04-12T03:50:15.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM=', 'Content-Length': '0'} 2024-04-12T03:50:15.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b014c985893811d-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.133 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.134 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.134 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.134 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.135 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.135 DEBUG:boto:Method: GET 2024-04-12T03:50:15.135 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:50:15.135 DEBUG:boto:Data: 2024-04-12T03:50:15.135 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.135 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.135 DEBUG:boto:Params: {} 2024-04-12T03:50:15.135 DEBUG:boto:Token: None 2024-04-12T03:50:15.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/?versions 2024-04-12T03:50:15.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM= 2024-04-12T03:50:15.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3hOjUqdYQbwmEkvQV9wmIkhurpM=', 'Content-Length': '0'} 2024-04-12T03:50:15.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7f17a483d861c8-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.140 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:50:15.142 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.142 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.142 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.142 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.142 DEBUG:boto:Method: GET 2024-04-12T03:50:15.142 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.142 DEBUG:boto:Data: 2024-04-12T03:50:15.142 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.142 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.143 DEBUG:boto:Params: {} 2024-04-12T03:50:15.143 DEBUG:boto:Token: None 2024-04-12T03:50:15.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I= 2024-04-12T03:50:15.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I=', 'Content-Length': '0'} 2024-04-12T03:50:15.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5d24350673e5ad-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.145 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.145 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.145 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.145 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.145 DEBUG:boto:Method: GET 2024-04-12T03:50:15.145 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.145 DEBUG:boto:Data: 2024-04-12T03:50:15.145 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.146 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.146 DEBUG:boto:Params: {} 2024-04-12T03:50:15.146 DEBUG:boto:Token: None 2024-04-12T03:50:15.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I= 2024-04-12T03:50:15.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I=', 'Content-Length': '0'} 2024-04-12T03:50:15.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246c51b7f5216fd8-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.148 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.148 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.148 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.149 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.149 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.149 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.149 DEBUG:boto:Data: 2024-04-12T03:50:15.149 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.149 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.149 DEBUG:boto:Params: {} 2024-04-12T03:50:15.149 DEBUG:boto:Token: None 2024-04-12T03:50:15.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:marcCCkoY5i8zrEIwbxqaoB8JSU= 2024-04-12T03:50:15.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:marcCCkoY5i8zrEIwbxqaoB8JSU=', 'Content-Length': '0'} 2024-04-12T03:50:15.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f5061fe6f8eb07-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.151 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.151 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.152 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.152 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.152 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.152 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.152 DEBUG:boto:Data: 2024-04-12T03:50:15.152 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.152 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.152 DEBUG:boto:Params: {} 2024-04-12T03:50:15.152 DEBUG:boto:Token: None 2024-04-12T03:50:15.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:marcCCkoY5i8zrEIwbxqaoB8JSU= 2024-04-12T03:50:15.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:marcCCkoY5i8zrEIwbxqaoB8JSU=', 'Content-Length': '0'} 2024-04-12T03:50:15.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c1dacb3effc6a0-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:50:15.155 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.155 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.155 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.155 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.155 DEBUG:boto:Method: GET 2024-04-12T03:50:15.155 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.155 DEBUG:boto:Data: 2024-04-12T03:50:15.155 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.156 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.156 DEBUG:boto:Params: {} 2024-04-12T03:50:15.156 DEBUG:boto:Token: None 2024-04-12T03:50:15.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I= 2024-04-12T03:50:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I=', 'Content-Length': '0'} 2024-04-12T03:50:15.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a07d8f2ea4b685e-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.158 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.158 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.158 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.158 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.158 DEBUG:boto:Method: GET 2024-04-12T03:50:15.158 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.159 DEBUG:boto:Data: 2024-04-12T03:50:15.159 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.159 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.159 DEBUG:boto:Params: {} 2024-04-12T03:50:15.159 DEBUG:boto:Token: None 2024-04-12T03:50:15.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:50:15.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I= 2024-04-12T03:50:15.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YkzbQPBqlY4WOhzPCX99CAkfK4I=', 'Content-Length': '0'} 2024-04-12T03:50:15.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f55f834acdd5086-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.202 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.202 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.203 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.203 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.203 DEBUG:boto:Method: GET 2024-04-12T03:50:15.203 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.203 DEBUG:boto:Data: 2024-04-12T03:50:15.203 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.203 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.203 DEBUG:boto:Params: {} 2024-04-12T03:50:15.203 DEBUG:boto:Token: None 2024-04-12T03:50:15.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fc12f38890233b3-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.206 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.206 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.207 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.207 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.207 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.207 DEBUG:boto:Method: GET 2024-04-12T03:50:15.207 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.207 DEBUG:boto:Data: 2024-04-12T03:50:15.207 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.207 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.207 DEBUG:boto:Params: {} 2024-04-12T03:50:15.207 DEBUG:boto:Token: None 2024-04-12T03:50:15.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002574ae4388e793ba-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.211 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.212 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.212 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.212 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.212 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:50:15.212 DEBUG:boto:Data: 2024-04-12T03:50:15.212 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.212 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.212 DEBUG:boto:Params: {} 2024-04-12T03:50:15.213 DEBUG:boto:Token: None 2024-04-12T03:50:15.213 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11 2024-04-12T03:50:15.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7p0A5pR8d7z15ccNdrrqZytz0P4= 2024-04-12T03:50:15.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7p0A5pR8d7z15ccNdrrqZytz0P4=', 'Content-Length': '0'} 2024-04-12T03:50:15.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f6b5525619b2bfe-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.215 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.215 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.215 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.215 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:50:15.215 DEBUG:boto:Data: 2024-04-12T03:50:15.215 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.215 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.216 DEBUG:boto:Params: {} 2024-04-12T03:50:15.216 DEBUG:boto:Token: None 2024-04-12T03:50:15.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11 2024-04-12T03:50:15.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7p0A5pR8d7z15ccNdrrqZytz0P4= 2024-04-12T03:50:15.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7p0A5pR8d7z15ccNdrrqZytz0P4=', 'Content-Length': '0'} 2024-04-12T03:50:15.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93b02b13d7266ff-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:50:15.218 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.218 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.218 DEBUG:boto:Method: GET 2024-04-12T03:50:15.218 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:50:15.218 DEBUG:boto:Data: 2024-04-12T03:50:15.218 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.219 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.219 DEBUG:boto:Params: {} 2024-04-12T03:50:15.219 DEBUG:boto:Token: None 2024-04-12T03:50:15.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11 2024-04-12T03:50:15.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qrWGdK0fHh0deyGyW52eCBLpId0= 2024-04-12T03:50:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qrWGdK0fHh0deyGyW52eCBLpId0=', 'Content-Length': '0'} 2024-04-12T03:50:15.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080cfced292e9d004-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.262 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:50:15.262 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:50:15.263 DEBUG:boto:Method: GET 2024-04-12T03:50:15.263 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:50:15.263 DEBUG:boto:Data: 2024-04-12T03:50:15.263 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.263 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.263 DEBUG:boto:Params: {} 2024-04-12T03:50:15.263 DEBUG:boto:Token: None 2024-04-12T03:50:15.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj11 2024-04-12T03:50:15.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qrWGdK0fHh0deyGyW52eCBLpId0= 2024-04-12T03:50:15.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qrWGdK0fHh0deyGyW52eCBLpId0=', 'Content-Length': '0'} 2024-04-12T03:50:15.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e48cea382cacd11-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.266 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.266 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.266 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.266 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.266 DEBUG:boto:Method: GET 2024-04-12T03:50:15.266 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.266 DEBUG:boto:Data: 2024-04-12T03:50:15.266 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.266 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.266 DEBUG:boto:Params: {} 2024-04-12T03:50:15.267 DEBUG:boto:Token: None 2024-04-12T03:50:15.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1899dbe9d8c02c-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.270 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.270 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.270 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.270 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.270 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.271 DEBUG:boto:Method: GET 2024-04-12T03:50:15.271 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:50:15.271 DEBUG:boto:Data: 2024-04-12T03:50:15.271 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.271 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.271 DEBUG:boto:Params: {} 2024-04-12T03:50:15.271 DEBUG:boto:Token: None 2024-04-12T03:50:15.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.271 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba50ffdb4e659e6-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.275 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:50:15.275 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.275 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.275 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.275 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.276 DEBUG:boto:Method: GET 2024-04-12T03:50:15.276 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.276 DEBUG:boto:Data: 2024-04-12T03:50:15.276 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.276 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.276 DEBUG:boto:Params: {} 2024-04-12T03:50:15.276 DEBUG:boto:Token: None 2024-04-12T03:50:15.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw= 2024-04-12T03:50:15.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw=', 'Content-Length': '0'} 2024-04-12T03:50:15.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a42e9cb34fbc872-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.278 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.279 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.279 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.279 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.279 DEBUG:boto:Method: GET 2024-04-12T03:50:15.279 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.279 DEBUG:boto:Data: 2024-04-12T03:50:15.279 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.279 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.279 DEBUG:boto:Params: {} 2024-04-12T03:50:15.280 DEBUG:boto:Token: None 2024-04-12T03:50:15.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw= 2024-04-12T03:50:15.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw=', 'Content-Length': '0'} 2024-04-12T03:50:15.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a36676aaac9452-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.322 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.323 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.323 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.323 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.323 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.323 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.323 DEBUG:boto:Data: 2024-04-12T03:50:15.324 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.324 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.324 DEBUG:boto:Params: {} 2024-04-12T03:50:15.324 DEBUG:boto:Token: None 2024-04-12T03:50:15.325 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:64wKAPvrArGk+ZKhEVINXxA7F7I= 2024-04-12T03:50:15.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:64wKAPvrArGk+ZKhEVINXxA7F7I=', 'Content-Length': '0'} 2024-04-12T03:50:15.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f56cd451a3f116-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.327 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.328 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.328 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.328 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.328 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.328 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.328 DEBUG:boto:Data: 2024-04-12T03:50:15.329 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.329 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.329 DEBUG:boto:Params: {} 2024-04-12T03:50:15.329 DEBUG:boto:Token: None 2024-04-12T03:50:15.330 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.330 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:64wKAPvrArGk+ZKhEVINXxA7F7I= 2024-04-12T03:50:15.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:64wKAPvrArGk+ZKhEVINXxA7F7I=', 'Content-Length': '0'} 2024-04-12T03:50:15.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c106cf3e4a96ea-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:50:15.332 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.332 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.332 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.332 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.333 DEBUG:boto:Method: GET 2024-04-12T03:50:15.333 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.333 DEBUG:boto:Data: 2024-04-12T03:50:15.333 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.333 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.333 DEBUG:boto:Params: {} 2024-04-12T03:50:15.333 DEBUG:boto:Token: None 2024-04-12T03:50:15.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw= 2024-04-12T03:50:15.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw=', 'Content-Length': '0'} 2024-04-12T03:50:15.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629eb7163992dbc2-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.336 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.336 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.336 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.336 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.336 DEBUG:boto:Method: GET 2024-04-12T03:50:15.336 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.336 DEBUG:boto:Data: 2024-04-12T03:50:15.336 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.337 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.337 DEBUG:boto:Params: {} 2024-04-12T03:50:15.337 DEBUG:boto:Token: None 2024-04-12T03:50:15.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:50:15.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw= 2024-04-12T03:50:15.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rufiDV8bUrZxQiPz+YwLlFBtgYw=', 'Content-Length': '0'} 2024-04-12T03:50:15.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d489a4afdf35724-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.339 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.339 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.339 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.339 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.339 DEBUG:boto:Method: GET 2024-04-12T03:50:15.340 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.340 DEBUG:boto:Data: 2024-04-12T03:50:15.340 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.340 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.340 DEBUG:boto:Params: {} 2024-04-12T03:50:15.340 DEBUG:boto:Token: None 2024-04-12T03:50:15.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081a76c87a6f10633-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.343 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.343 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.343 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.343 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.343 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.344 DEBUG:boto:Method: GET 2024-04-12T03:50:15.344 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.344 DEBUG:boto:Data: 2024-04-12T03:50:15.344 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.344 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.344 DEBUG:boto:Params: {} 2024-04-12T03:50:15.344 DEBUG:boto:Token: None 2024-04-12T03:50:15.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008424df3110e0ea40-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.348 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.348 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.348 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.348 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.348 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:50:15.348 DEBUG:boto:Data: 2024-04-12T03:50:15.348 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.349 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.349 DEBUG:boto:Params: {} 2024-04-12T03:50:15.349 DEBUG:boto:Token: None 2024-04-12T03:50:15.349 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115 2024-04-12T03:50:15.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T+1u9gWz8PK0+StgrJtEjJRoAB0= 2024-04-12T03:50:15.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T+1u9gWz8PK0+StgrJtEjJRoAB0=', 'Content-Length': '0'} 2024-04-12T03:50:15.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18e6f541627e3c7-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.351 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.351 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.351 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.351 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:50:15.351 DEBUG:boto:Data: 2024-04-12T03:50:15.351 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.351 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.351 DEBUG:boto:Params: {} 2024-04-12T03:50:15.352 DEBUG:boto:Token: None 2024-04-12T03:50:15.352 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115 2024-04-12T03:50:15.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T+1u9gWz8PK0+StgrJtEjJRoAB0= 2024-04-12T03:50:15.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T+1u9gWz8PK0+StgrJtEjJRoAB0=', 'Content-Length': '0'} 2024-04-12T03:50:15.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50f87a648eb3722-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:50:15.354 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.354 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.354 DEBUG:boto:Method: GET 2024-04-12T03:50:15.354 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:50:15.354 DEBUG:boto:Data: 2024-04-12T03:50:15.354 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.354 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.354 DEBUG:boto:Params: {} 2024-04-12T03:50:15.355 DEBUG:boto:Token: None 2024-04-12T03:50:15.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115 2024-04-12T03:50:15.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bojo9pCoCLFAakOzzYWpWMV5SyU= 2024-04-12T03:50:15.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bojo9pCoCLFAakOzzYWpWMV5SyU=', 'Content-Length': '0'} 2024-04-12T03:50:15.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cca621a62b65787-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.398 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:50:15.398 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:50:15.398 DEBUG:boto:Method: GET 2024-04-12T03:50:15.399 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:50:15.399 DEBUG:boto:Data: 2024-04-12T03:50:15.399 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.399 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.399 DEBUG:boto:Params: {} 2024-04-12T03:50:15.399 DEBUG:boto:Token: None 2024-04-12T03:50:15.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj115 2024-04-12T03:50:15.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bojo9pCoCLFAakOzzYWpWMV5SyU= 2024-04-12T03:50:15.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bojo9pCoCLFAakOzzYWpWMV5SyU=', 'Content-Length': '0'} 2024-04-12T03:50:15.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081649113dd513a66-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.402 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.402 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.402 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.403 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.403 DEBUG:boto:Method: GET 2024-04-12T03:50:15.403 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.403 DEBUG:boto:Data: 2024-04-12T03:50:15.403 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.403 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.403 DEBUG:boto:Params: {} 2024-04-12T03:50:15.403 DEBUG:boto:Token: None 2024-04-12T03:50:15.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001896c3cdc1c75e6b-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.406 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.406 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.406 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.406 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.406 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.406 DEBUG:boto:Method: GET 2024-04-12T03:50:15.406 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:50:15.407 DEBUG:boto:Data: 2024-04-12T03:50:15.407 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.407 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.407 DEBUG:boto:Params: {} 2024-04-12T03:50:15.407 DEBUG:boto:Token: None 2024-04-12T03:50:15.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000769795d4597eced0-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.411 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:50:15.411 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.411 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.411 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.411 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.411 DEBUG:boto:Method: GET 2024-04-12T03:50:15.411 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.412 DEBUG:boto:Data: 2024-04-12T03:50:15.412 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.412 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.412 DEBUG:boto:Params: {} 2024-04-12T03:50:15.412 DEBUG:boto:Token: None 2024-04-12T03:50:15.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw= 2024-04-12T03:50:15.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw=', 'Content-Length': '0'} 2024-04-12T03:50:15.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050923c7c9f3a8aba-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.414 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.415 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.415 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.415 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.415 DEBUG:boto:Method: GET 2024-04-12T03:50:15.415 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.415 DEBUG:boto:Data: 2024-04-12T03:50:15.415 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.415 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.415 DEBUG:boto:Params: {} 2024-04-12T03:50:15.416 DEBUG:boto:Token: None 2024-04-12T03:50:15.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw= 2024-04-12T03:50:15.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw=', 'Content-Length': '0'} 2024-04-12T03:50:15.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e562e4a5e9beefd-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.418 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.418 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.418 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.418 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.418 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.418 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.418 DEBUG:boto:Data: 2024-04-12T03:50:15.418 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.418 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.419 DEBUG:boto:Params: {} 2024-04-12T03:50:15.419 DEBUG:boto:Token: None 2024-04-12T03:50:15.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V5hG/AHVRuomMWcr3T0kltiZXdQ= 2024-04-12T03:50:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V5hG/AHVRuomMWcr3T0kltiZXdQ=', 'Content-Length': '0'} 2024-04-12T03:50:15.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039329d756f35626e-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.421 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.421 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.421 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.421 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.421 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.421 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.421 DEBUG:boto:Data: 2024-04-12T03:50:15.421 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.422 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.422 DEBUG:boto:Params: {} 2024-04-12T03:50:15.422 DEBUG:boto:Token: None 2024-04-12T03:50:15.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V5hG/AHVRuomMWcr3T0kltiZXdQ= 2024-04-12T03:50:15.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V5hG/AHVRuomMWcr3T0kltiZXdQ=', 'Content-Length': '0'} 2024-04-12T03:50:15.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093bae1c95591c3d2-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:50:15.424 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.424 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.424 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.424 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.424 DEBUG:boto:Method: GET 2024-04-12T03:50:15.424 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.424 DEBUG:boto:Data: 2024-04-12T03:50:15.424 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.425 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.425 DEBUG:boto:Params: {} 2024-04-12T03:50:15.425 DEBUG:boto:Token: None 2024-04-12T03:50:15.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw= 2024-04-12T03:50:15.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw=', 'Content-Length': '0'} 2024-04-12T03:50:15.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8faadf0ff7c09b7-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.427 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.427 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.427 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.427 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.428 DEBUG:boto:Method: GET 2024-04-12T03:50:15.428 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.428 DEBUG:boto:Data: 2024-04-12T03:50:15.428 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.428 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.428 DEBUG:boto:Params: {} 2024-04-12T03:50:15.428 DEBUG:boto:Token: None 2024-04-12T03:50:15.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:50:15.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw= 2024-04-12T03:50:15.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:did2khz6vFXIBAQW050+WzIf+Vw=', 'Content-Length': '0'} 2024-04-12T03:50:15.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b73a71b5dff34c-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.430 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.430 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.430 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.430 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.431 DEBUG:boto:Method: GET 2024-04-12T03:50:15.431 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.431 DEBUG:boto:Data: 2024-04-12T03:50:15.431 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.431 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.431 DEBUG:boto:Params: {} 2024-04-12T03:50:15.431 DEBUG:boto:Token: None 2024-04-12T03:50:15.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2e377577bff4a0-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.474 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.475 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.475 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.475 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.475 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.475 DEBUG:boto:Method: GET 2024-04-12T03:50:15.475 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.475 DEBUG:boto:Data: 2024-04-12T03:50:15.475 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.476 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.476 DEBUG:boto:Params: {} 2024-04-12T03:50:15.476 DEBUG:boto:Token: None 2024-04-12T03:50:15.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005795fb1310d836d-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.480 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.480 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.480 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.480 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.480 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:50:15.480 DEBUG:boto:Data: 2024-04-12T03:50:15.480 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.481 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.481 DEBUG:boto:Params: {} 2024-04-12T03:50:15.481 DEBUG:boto:Token: None 2024-04-12T03:50:15.481 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116 2024-04-12T03:50:15.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AFx3Asbl6RZ6GzgZ85jBkLsKZUk= 2024-04-12T03:50:15.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AFx3Asbl6RZ6GzgZ85jBkLsKZUk=', 'Content-Length': '0'} 2024-04-12T03:50:15.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1709173e5328ec-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.483 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.483 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.483 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.483 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:50:15.483 DEBUG:boto:Data: 2024-04-12T03:50:15.483 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.484 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.484 DEBUG:boto:Params: {} 2024-04-12T03:50:15.484 DEBUG:boto:Token: None 2024-04-12T03:50:15.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116 2024-04-12T03:50:15.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AFx3Asbl6RZ6GzgZ85jBkLsKZUk= 2024-04-12T03:50:15.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AFx3Asbl6RZ6GzgZ85jBkLsKZUk=', 'Content-Length': '0'} 2024-04-12T03:50:15.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41350298f9dc21b-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:50:15.486 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.486 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.486 DEBUG:boto:Method: GET 2024-04-12T03:50:15.487 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:50:15.487 DEBUG:boto:Data: 2024-04-12T03:50:15.487 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.487 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.487 DEBUG:boto:Params: {} 2024-04-12T03:50:15.487 DEBUG:boto:Token: None 2024-04-12T03:50:15.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116 2024-04-12T03:50:15.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vDJKxg62UqbFvATff8PbOZ2tOgw= 2024-04-12T03:50:15.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vDJKxg62UqbFvATff8PbOZ2tOgw=', 'Content-Length': '0'} 2024-04-12T03:50:15.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f6d024e5db23eab-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.489 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:50:15.489 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:50:15.489 DEBUG:boto:Method: GET 2024-04-12T03:50:15.489 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:50:15.490 DEBUG:boto:Data: 2024-04-12T03:50:15.490 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.490 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.490 DEBUG:boto:Params: {} 2024-04-12T03:50:15.490 DEBUG:boto:Token: None 2024-04-12T03:50:15.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj116 2024-04-12T03:50:15.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vDJKxg62UqbFvATff8PbOZ2tOgw= 2024-04-12T03:50:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vDJKxg62UqbFvATff8PbOZ2tOgw=', 'Content-Length': '0'} 2024-04-12T03:50:15.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ccc7b6d76451557-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.492 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.492 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.492 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.492 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.493 DEBUG:boto:Method: GET 2024-04-12T03:50:15.493 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.493 DEBUG:boto:Data: 2024-04-12T03:50:15.493 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.493 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.493 DEBUG:boto:Params: {} 2024-04-12T03:50:15.493 DEBUG:boto:Token: None 2024-04-12T03:50:15.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da4f205f9873ec2b-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.496 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.496 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.496 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.496 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.496 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.497 DEBUG:boto:Method: GET 2024-04-12T03:50:15.497 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:50:15.497 DEBUG:boto:Data: 2024-04-12T03:50:15.497 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.497 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.497 DEBUG:boto:Params: {} 2024-04-12T03:50:15.497 DEBUG:boto:Token: None 2024-04-12T03:50:15.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.497 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1039544006db46a-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.500 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:50:15.501 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.501 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.501 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.501 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.501 DEBUG:boto:Method: GET 2024-04-12T03:50:15.501 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.502 DEBUG:boto:Data: 2024-04-12T03:50:15.502 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.502 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.502 DEBUG:boto:Params: {} 2024-04-12T03:50:15.502 DEBUG:boto:Token: None 2024-04-12T03:50:15.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.502 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM= 2024-04-12T03:50:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM=', 'Content-Length': '0'} 2024-04-12T03:50:15.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daab892cfcce329-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.546 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.547 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.547 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.547 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.547 DEBUG:boto:Method: GET 2024-04-12T03:50:15.547 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.547 DEBUG:boto:Data: 2024-04-12T03:50:15.547 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.547 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.547 DEBUG:boto:Params: {} 2024-04-12T03:50:15.547 DEBUG:boto:Token: None 2024-04-12T03:50:15.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM= 2024-04-12T03:50:15.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM=', 'Content-Length': '0'} 2024-04-12T03:50:15.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591a5d343f5230ab-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.550 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.550 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.550 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.550 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.550 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.550 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.550 DEBUG:boto:Data: 2024-04-12T03:50:15.551 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.551 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.551 DEBUG:boto:Params: {} 2024-04-12T03:50:15.551 DEBUG:boto:Token: None 2024-04-12T03:50:15.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ke8LNPA6yUtVVD+Xra4CorMTpzo= 2024-04-12T03:50:15.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ke8LNPA6yUtVVD+Xra4CorMTpzo=', 'Content-Length': '0'} 2024-04-12T03:50:15.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55ce79e7764dd21-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.553 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.553 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.553 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.553 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.553 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.553 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.553 DEBUG:boto:Data: 2024-04-12T03:50:15.553 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.554 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.554 DEBUG:boto:Params: {} 2024-04-12T03:50:15.554 DEBUG:boto:Token: None 2024-04-12T03:50:15.554 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.554 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ke8LNPA6yUtVVD+Xra4CorMTpzo= 2024-04-12T03:50:15.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ke8LNPA6yUtVVD+Xra4CorMTpzo=', 'Content-Length': '0'} 2024-04-12T03:50:15.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e1300944af2380-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:50:15.556 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.556 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.556 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.556 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.556 DEBUG:boto:Method: GET 2024-04-12T03:50:15.556 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.557 DEBUG:boto:Data: 2024-04-12T03:50:15.557 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.557 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.557 DEBUG:boto:Params: {} 2024-04-12T03:50:15.557 DEBUG:boto:Token: None 2024-04-12T03:50:15.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.557 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM= 2024-04-12T03:50:15.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM=', 'Content-Length': '0'} 2024-04-12T03:50:15.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7a507d908bb1949-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.560 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.560 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.560 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.560 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.560 DEBUG:boto:Method: GET 2024-04-12T03:50:15.560 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.560 DEBUG:boto:Data: 2024-04-12T03:50:15.560 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.560 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.560 DEBUG:boto:Params: {} 2024-04-12T03:50:15.561 DEBUG:boto:Token: None 2024-04-12T03:50:15.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:50:15.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM= 2024-04-12T03:50:15.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PbHP1+0TKefg3vAjUD/jUgRFpkM=', 'Content-Length': '0'} 2024-04-12T03:50:15.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009779d5172144c490-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.563 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.563 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.563 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.564 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.564 DEBUG:boto:Method: GET 2024-04-12T03:50:15.564 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.564 DEBUG:boto:Data: 2024-04-12T03:50:15.564 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.564 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.564 DEBUG:boto:Params: {} 2024-04-12T03:50:15.564 DEBUG:boto:Token: None 2024-04-12T03:50:15.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620bea6a7067ad73-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.567 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.567 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.568 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.568 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.568 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.568 DEBUG:boto:Method: GET 2024-04-12T03:50:15.568 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.568 DEBUG:boto:Data: 2024-04-12T03:50:15.568 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.568 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.568 DEBUG:boto:Params: {} 2024-04-12T03:50:15.569 DEBUG:boto:Token: None 2024-04-12T03:50:15.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.569 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069142bb6323478a2-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.614 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.615 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.615 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.615 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.615 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:50:15.615 DEBUG:boto:Data: 2024-04-12T03:50:15.615 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.615 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.616 DEBUG:boto:Params: {} 2024-04-12T03:50:15.616 DEBUG:boto:Token: None 2024-04-12T03:50:15.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145 2024-04-12T03:50:15.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1BQThlW5XOS57fDbqukm6CJN6pY= 2024-04-12T03:50:15.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1BQThlW5XOS57fDbqukm6CJN6pY=', 'Content-Length': '0'} 2024-04-12T03:50:15.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000409648ae92a396c9-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.618 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.618 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.618 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.618 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:50:15.618 DEBUG:boto:Data: 2024-04-12T03:50:15.618 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.618 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.618 DEBUG:boto:Params: {} 2024-04-12T03:50:15.619 DEBUG:boto:Token: None 2024-04-12T03:50:15.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145 2024-04-12T03:50:15.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1BQThlW5XOS57fDbqukm6CJN6pY= 2024-04-12T03:50:15.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1BQThlW5XOS57fDbqukm6CJN6pY=', 'Content-Length': '0'} 2024-04-12T03:50:15.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005673b4de33018317-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:50:15.621 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.621 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.621 DEBUG:boto:Method: GET 2024-04-12T03:50:15.621 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:50:15.621 DEBUG:boto:Data: 2024-04-12T03:50:15.621 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.621 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.621 DEBUG:boto:Params: {} 2024-04-12T03:50:15.622 DEBUG:boto:Token: None 2024-04-12T03:50:15.622 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145 2024-04-12T03:50:15.622 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fdqkO8kHAyJor0G5AOmHyYOF0iw= 2024-04-12T03:50:15.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fdqkO8kHAyJor0G5AOmHyYOF0iw=', 'Content-Length': '0'} 2024-04-12T03:50:15.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e59b111cb8a180b-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.624 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:50:15.624 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:50:15.624 DEBUG:boto:Method: GET 2024-04-12T03:50:15.624 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:50:15.624 DEBUG:boto:Data: 2024-04-12T03:50:15.624 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.625 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.625 DEBUG:boto:Params: {} 2024-04-12T03:50:15.625 DEBUG:boto:Token: None 2024-04-12T03:50:15.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj145 2024-04-12T03:50:15.625 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fdqkO8kHAyJor0G5AOmHyYOF0iw= 2024-04-12T03:50:15.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fdqkO8kHAyJor0G5AOmHyYOF0iw=', 'Content-Length': '0'} 2024-04-12T03:50:15.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b02c0679741e7f4-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.627 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.627 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.627 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.627 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.627 DEBUG:boto:Method: GET 2024-04-12T03:50:15.627 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.627 DEBUG:boto:Data: 2024-04-12T03:50:15.628 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.628 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.628 DEBUG:boto:Params: {} 2024-04-12T03:50:15.628 DEBUG:boto:Token: None 2024-04-12T03:50:15.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e51d85aa59a358-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.632 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.632 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.632 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.632 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.632 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.632 DEBUG:boto:Method: GET 2024-04-12T03:50:15.633 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:50:15.633 DEBUG:boto:Data: 2024-04-12T03:50:15.633 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.633 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.633 DEBUG:boto:Params: {} 2024-04-12T03:50:15.633 DEBUG:boto:Token: None 2024-04-12T03:50:15.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5a0c961a796a42-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.637 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:50:15.637 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.637 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.637 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.638 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.638 DEBUG:boto:Method: GET 2024-04-12T03:50:15.638 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.638 DEBUG:boto:Data: 2024-04-12T03:50:15.638 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.638 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.638 DEBUG:boto:Params: {} 2024-04-12T03:50:15.638 DEBUG:boto:Token: None 2024-04-12T03:50:15.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4= 2024-04-12T03:50:15.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4=', 'Content-Length': '0'} 2024-04-12T03:50:15.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9142a00d15678ea-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.682 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.683 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.683 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.683 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.683 DEBUG:boto:Method: GET 2024-04-12T03:50:15.683 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.683 DEBUG:boto:Data: 2024-04-12T03:50:15.683 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.683 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.684 DEBUG:boto:Params: {} 2024-04-12T03:50:15.684 DEBUG:boto:Token: None 2024-04-12T03:50:15.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4= 2024-04-12T03:50:15.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4=', 'Content-Length': '0'} 2024-04-12T03:50:15.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002abc4e3b36777d2f-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.687 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.687 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.687 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.687 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.687 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.687 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.687 DEBUG:boto:Data: 2024-04-12T03:50:15.687 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.687 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.687 DEBUG:boto:Params: {} 2024-04-12T03:50:15.688 DEBUG:boto:Token: None 2024-04-12T03:50:15.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:++3SnyBYgE9edfV/2fM7+t7m/+E= 2024-04-12T03:50:15.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:++3SnyBYgE9edfV/2fM7+t7m/+E=', 'Content-Length': '0'} 2024-04-12T03:50:15.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a330f06f37ef48-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.690 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.690 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.690 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.690 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.690 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.691 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.691 DEBUG:boto:Data: 2024-04-12T03:50:15.691 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.691 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.691 DEBUG:boto:Params: {} 2024-04-12T03:50:15.691 DEBUG:boto:Token: None 2024-04-12T03:50:15.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:++3SnyBYgE9edfV/2fM7+t7m/+E= 2024-04-12T03:50:15.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:++3SnyBYgE9edfV/2fM7+t7m/+E=', 'Content-Length': '0'} 2024-04-12T03:50:15.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8858e4cae705a50-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:50:15.693 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.693 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.694 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.694 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.694 DEBUG:boto:Method: GET 2024-04-12T03:50:15.694 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.694 DEBUG:boto:Data: 2024-04-12T03:50:15.694 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.694 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.694 DEBUG:boto:Params: {} 2024-04-12T03:50:15.694 DEBUG:boto:Token: None 2024-04-12T03:50:15.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4= 2024-04-12T03:50:15.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4=', 'Content-Length': '0'} 2024-04-12T03:50:15.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a382a0199bb208c7-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.697 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.697 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.697 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.697 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.697 DEBUG:boto:Method: GET 2024-04-12T03:50:15.697 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.698 DEBUG:boto:Data: 2024-04-12T03:50:15.698 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.698 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.698 DEBUG:boto:Params: {} 2024-04-12T03:50:15.698 DEBUG:boto:Token: None 2024-04-12T03:50:15.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:50:15.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4= 2024-04-12T03:50:15.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98GhMvsD36juT4fn/mkuL8ez2Q4=', 'Content-Length': '0'} 2024-04-12T03:50:15.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de392eef7f8f9429-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.742 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.742 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.743 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.743 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.743 DEBUG:boto:Method: GET 2024-04-12T03:50:15.743 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.743 DEBUG:boto:Data: 2024-04-12T03:50:15.743 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.743 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.743 DEBUG:boto:Params: {} 2024-04-12T03:50:15.743 DEBUG:boto:Token: None 2024-04-12T03:50:15.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003782f92d915f0be3-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.746 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.746 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.746 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.747 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.747 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.747 DEBUG:boto:Method: GET 2024-04-12T03:50:15.747 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.747 DEBUG:boto:Data: 2024-04-12T03:50:15.747 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.747 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.747 DEBUG:boto:Params: {} 2024-04-12T03:50:15.747 DEBUG:boto:Token: None 2024-04-12T03:50:15.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063cc4696612c965-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.751 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.751 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.751 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.751 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.752 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:50:15.752 DEBUG:boto:Data: 2024-04-12T03:50:15.752 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.752 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.752 DEBUG:boto:Params: {} 2024-04-12T03:50:15.752 DEBUG:boto:Token: None 2024-04-12T03:50:15.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17 2024-04-12T03:50:15.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88SyLjY6TZ+3CFdlmOnk5wZxJ1Y= 2024-04-12T03:50:15.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88SyLjY6TZ+3CFdlmOnk5wZxJ1Y=', 'Content-Length': '0'} 2024-04-12T03:50:15.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b4b762f614ac577-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.754 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.754 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.754 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.754 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:50:15.755 DEBUG:boto:Data: 2024-04-12T03:50:15.755 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.755 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.755 DEBUG:boto:Params: {} 2024-04-12T03:50:15.755 DEBUG:boto:Token: None 2024-04-12T03:50:15.755 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17 2024-04-12T03:50:15.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88SyLjY6TZ+3CFdlmOnk5wZxJ1Y= 2024-04-12T03:50:15.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88SyLjY6TZ+3CFdlmOnk5wZxJ1Y=', 'Content-Length': '0'} 2024-04-12T03:50:15.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969068b36b54845e-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:50:15.757 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.757 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.757 DEBUG:boto:Method: GET 2024-04-12T03:50:15.757 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:50:15.758 DEBUG:boto:Data: 2024-04-12T03:50:15.758 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.758 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.758 DEBUG:boto:Params: {} 2024-04-12T03:50:15.758 DEBUG:boto:Token: None 2024-04-12T03:50:15.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17 2024-04-12T03:50:15.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+kcdmatl1tSUEdFUzxts8HQrzmg= 2024-04-12T03:50:15.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+kcdmatl1tSUEdFUzxts8HQrzmg=', 'Content-Length': '0'} 2024-04-12T03:50:15.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad0cf99fa9b310a-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.760 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:50:15.760 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:50:15.760 DEBUG:boto:Method: GET 2024-04-12T03:50:15.760 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:50:15.760 DEBUG:boto:Data: 2024-04-12T03:50:15.760 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.761 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.761 DEBUG:boto:Params: {} 2024-04-12T03:50:15.761 DEBUG:boto:Token: None 2024-04-12T03:50:15.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj17 2024-04-12T03:50:15.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+kcdmatl1tSUEdFUzxts8HQrzmg= 2024-04-12T03:50:15.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+kcdmatl1tSUEdFUzxts8HQrzmg=', 'Content-Length': '0'} 2024-04-12T03:50:15.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006337886918dcd75d-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.763 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.763 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.763 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.763 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.763 DEBUG:boto:Method: GET 2024-04-12T03:50:15.763 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.764 DEBUG:boto:Data: 2024-04-12T03:50:15.764 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.764 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.764 DEBUG:boto:Params: {} 2024-04-12T03:50:15.764 DEBUG:boto:Token: None 2024-04-12T03:50:15.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b498eff4770d477-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.766 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.767 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.767 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.767 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.767 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.767 DEBUG:boto:Method: GET 2024-04-12T03:50:15.767 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:50:15.767 DEBUG:boto:Data: 2024-04-12T03:50:15.767 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.768 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.768 DEBUG:boto:Params: {} 2024-04-12T03:50:15.768 DEBUG:boto:Token: None 2024-04-12T03:50:15.768 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.768 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f72177d7100378c6-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.818 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:50:15.819 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.819 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.819 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.819 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.819 DEBUG:boto:Method: GET 2024-04-12T03:50:15.819 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.819 DEBUG:boto:Data: 2024-04-12T03:50:15.820 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.820 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.820 DEBUG:boto:Params: {} 2024-04-12T03:50:15.820 DEBUG:boto:Token: None 2024-04-12T03:50:15.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg= 2024-04-12T03:50:15.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg=', 'Content-Length': '0'} 2024-04-12T03:50:15.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009022d96cff57ffe5-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.822 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.822 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.823 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.823 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.823 DEBUG:boto:Method: GET 2024-04-12T03:50:15.823 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.823 DEBUG:boto:Data: 2024-04-12T03:50:15.823 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.823 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.823 DEBUG:boto:Params: {} 2024-04-12T03:50:15.823 DEBUG:boto:Token: None 2024-04-12T03:50:15.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg= 2024-04-12T03:50:15.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg=', 'Content-Length': '0'} 2024-04-12T03:50:15.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879cec3f0042dddb-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.825 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.825 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.826 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.826 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.826 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.826 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.826 DEBUG:boto:Data: 2024-04-12T03:50:15.826 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.826 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.826 DEBUG:boto:Params: {} 2024-04-12T03:50:15.826 DEBUG:boto:Token: None 2024-04-12T03:50:15.827 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K0SQ1MWiEUaMHUE84dNvY6EdLBU= 2024-04-12T03:50:15.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K0SQ1MWiEUaMHUE84dNvY6EdLBU=', 'Content-Length': '0'} 2024-04-12T03:50:15.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048cc697d2512c666-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.828 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.829 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.829 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.829 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.829 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.829 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.829 DEBUG:boto:Data: 2024-04-12T03:50:15.829 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.829 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.829 DEBUG:boto:Params: {} 2024-04-12T03:50:15.829 DEBUG:boto:Token: None 2024-04-12T03:50:15.830 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K0SQ1MWiEUaMHUE84dNvY6EdLBU= 2024-04-12T03:50:15.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K0SQ1MWiEUaMHUE84dNvY6EdLBU=', 'Content-Length': '0'} 2024-04-12T03:50:15.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba91c18e0430699b-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:50:15.832 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.832 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.832 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.832 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.832 DEBUG:boto:Method: GET 2024-04-12T03:50:15.832 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.832 DEBUG:boto:Data: 2024-04-12T03:50:15.832 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.833 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.833 DEBUG:boto:Params: {} 2024-04-12T03:50:15.833 DEBUG:boto:Token: None 2024-04-12T03:50:15.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.833 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg= 2024-04-12T03:50:15.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg=', 'Content-Length': '0'} 2024-04-12T03:50:15.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d679ae4b436070cf-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.835 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.835 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.835 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.835 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.835 DEBUG:boto:Method: GET 2024-04-12T03:50:15.835 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.835 DEBUG:boto:Data: 2024-04-12T03:50:15.835 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.836 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.836 DEBUG:boto:Params: {} 2024-04-12T03:50:15.836 DEBUG:boto:Token: None 2024-04-12T03:50:15.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:50:15.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg= 2024-04-12T03:50:15.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xmCkZirn6oWDFLua7N3R9JHVSg=', 'Content-Length': '0'} 2024-04-12T03:50:15.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df16134cc84eabe-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.838 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.838 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.839 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.839 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.839 DEBUG:boto:Method: GET 2024-04-12T03:50:15.839 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.839 DEBUG:boto:Data: 2024-04-12T03:50:15.839 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.839 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.839 DEBUG:boto:Params: {} 2024-04-12T03:50:15.839 DEBUG:boto:Token: None 2024-04-12T03:50:15.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc94a9a935da374-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.842 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.842 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.842 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.842 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.843 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.843 DEBUG:boto:Method: GET 2024-04-12T03:50:15.843 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.843 DEBUG:boto:Data: 2024-04-12T03:50:15.843 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.843 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.843 DEBUG:boto:Params: {} 2024-04-12T03:50:15.843 DEBUG:boto:Token: None 2024-04-12T03:50:15.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717fcbad342c23ab-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.850 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.850 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.850 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.850 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.850 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:50:15.850 DEBUG:boto:Data: 2024-04-12T03:50:15.851 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.851 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.851 DEBUG:boto:Params: {} 2024-04-12T03:50:15.851 DEBUG:boto:Token: None 2024-04-12T03:50:15.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19 2024-04-12T03:50:15.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wagT1teScLtYADf3FAN+13ZolLg= 2024-04-12T03:50:15.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wagT1teScLtYADf3FAN+13ZolLg=', 'Content-Length': '0'} 2024-04-12T03:50:15.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3f648b66c6befc-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.853 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.853 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.853 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.853 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:50:15.853 DEBUG:boto:Data: 2024-04-12T03:50:15.854 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.854 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.854 DEBUG:boto:Params: {} 2024-04-12T03:50:15.854 DEBUG:boto:Token: None 2024-04-12T03:50:15.854 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19 2024-04-12T03:50:15.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wagT1teScLtYADf3FAN+13ZolLg= 2024-04-12T03:50:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wagT1teScLtYADf3FAN+13ZolLg=', 'Content-Length': '0'} 2024-04-12T03:50:15.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006601e076726936a1-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:50:15.856 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.856 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.856 DEBUG:boto:Method: GET 2024-04-12T03:50:15.856 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:50:15.857 DEBUG:boto:Data: 2024-04-12T03:50:15.857 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.857 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.857 DEBUG:boto:Params: {} 2024-04-12T03:50:15.857 DEBUG:boto:Token: None 2024-04-12T03:50:15.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19 2024-04-12T03:50:15.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kIYfUkOsUsqUYJpheNJTe1mqCIM= 2024-04-12T03:50:15.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kIYfUkOsUsqUYJpheNJTe1mqCIM=', 'Content-Length': '0'} 2024-04-12T03:50:15.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1c57b22b148044-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.859 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:50:15.859 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:50:15.859 DEBUG:boto:Method: GET 2024-04-12T03:50:15.859 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:50:15.859 DEBUG:boto:Data: 2024-04-12T03:50:15.859 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.860 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.860 DEBUG:boto:Params: {} 2024-04-12T03:50:15.860 DEBUG:boto:Token: None 2024-04-12T03:50:15.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj19 2024-04-12T03:50:15.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kIYfUkOsUsqUYJpheNJTe1mqCIM= 2024-04-12T03:50:15.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kIYfUkOsUsqUYJpheNJTe1mqCIM=', 'Content-Length': '0'} 2024-04-12T03:50:15.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c2ac1ed6599ebe-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.862 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.862 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.862 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.862 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.862 DEBUG:boto:Method: GET 2024-04-12T03:50:15.862 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.863 DEBUG:boto:Data: 2024-04-12T03:50:15.863 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.863 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.863 DEBUG:boto:Params: {} 2024-04-12T03:50:15.863 DEBUG:boto:Token: None 2024-04-12T03:50:15.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058086f09c34397c9-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.910 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.911 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.911 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.911 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.911 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.911 DEBUG:boto:Method: GET 2024-04-12T03:50:15.911 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:50:15.911 DEBUG:boto:Data: 2024-04-12T03:50:15.912 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.912 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.912 DEBUG:boto:Params: {} 2024-04-12T03:50:15.912 DEBUG:boto:Token: None 2024-04-12T03:50:15.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c0ad44ec5c74141-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.915 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:50:15.916 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.916 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.916 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.916 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.916 DEBUG:boto:Method: GET 2024-04-12T03:50:15.916 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.916 DEBUG:boto:Data: 2024-04-12T03:50:15.916 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.917 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.917 DEBUG:boto:Params: {} 2024-04-12T03:50:15.917 DEBUG:boto:Token: None 2024-04-12T03:50:15.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg= 2024-04-12T03:50:15.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg=', 'Content-Length': '0'} 2024-04-12T03:50:15.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326a76840d6c852a-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.919 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.919 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.919 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.919 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.919 DEBUG:boto:Method: GET 2024-04-12T03:50:15.919 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.920 DEBUG:boto:Data: 2024-04-12T03:50:15.920 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.920 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.920 DEBUG:boto:Params: {} 2024-04-12T03:50:15.920 DEBUG:boto:Token: None 2024-04-12T03:50:15.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg= 2024-04-12T03:50:15.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg=', 'Content-Length': '0'} 2024-04-12T03:50:15.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83c11c79e79980b-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.966 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.966 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.967 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.967 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.967 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.967 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.967 DEBUG:boto:Data: 2024-04-12T03:50:15.967 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.967 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.967 DEBUG:boto:Params: {} 2024-04-12T03:50:15.967 DEBUG:boto:Token: None 2024-04-12T03:50:15.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2W8dSE5tKGHV83JoK07zGbO1jtk= 2024-04-12T03:50:15.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2W8dSE5tKGHV83JoK07zGbO1jtk=', 'Content-Length': '0'} 2024-04-12T03:50:15.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e87d34a54662b1-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.970 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.970 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.970 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.970 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.970 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.970 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.971 DEBUG:boto:Data: 2024-04-12T03:50:15.971 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.971 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.971 DEBUG:boto:Params: {} 2024-04-12T03:50:15.971 DEBUG:boto:Token: None 2024-04-12T03:50:15.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2W8dSE5tKGHV83JoK07zGbO1jtk= 2024-04-12T03:50:15.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2W8dSE5tKGHV83JoK07zGbO1jtk=', 'Content-Length': '0'} 2024-04-12T03:50:15.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447f5cb0f940ab05-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:50:15.973 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.973 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.973 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.973 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.973 DEBUG:boto:Method: GET 2024-04-12T03:50:15.974 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.974 DEBUG:boto:Data: 2024-04-12T03:50:15.974 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.974 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.974 DEBUG:boto:Params: {} 2024-04-12T03:50:15.974 DEBUG:boto:Token: None 2024-04-12T03:50:15.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg= 2024-04-12T03:50:15.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg=', 'Content-Length': '0'} 2024-04-12T03:50:15.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014af8ad4cec97770-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.976 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.977 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.977 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.977 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.977 DEBUG:boto:Method: GET 2024-04-12T03:50:15.977 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.977 DEBUG:boto:Data: 2024-04-12T03:50:15.977 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.977 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.977 DEBUG:boto:Params: {} 2024-04-12T03:50:15.978 DEBUG:boto:Token: None 2024-04-12T03:50:15.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:50:15.978 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg= 2024-04-12T03:50:15.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HiYwadpvO3hC0Cs9pJRwxdf2sWg=', 'Content-Length': '0'} 2024-04-12T03:50:15.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb4af5317f4e44e-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.980 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.980 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.980 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.980 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.980 DEBUG:boto:Method: GET 2024-04-12T03:50:15.980 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.980 DEBUG:boto:Data: 2024-04-12T03:50:15.980 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.981 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.981 DEBUG:boto:Params: {} 2024-04-12T03:50:15.981 DEBUG:boto:Token: None 2024-04-12T03:50:15.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa3a131fec95a30c-006618af77-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.983 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.984 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:15.984 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:15.984 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.984 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.984 DEBUG:boto:Method: GET 2024-04-12T03:50:15.984 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:50:15.984 DEBUG:boto:Data: 2024-04-12T03:50:15.984 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.984 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.984 DEBUG:boto:Params: {} 2024-04-12T03:50:15.985 DEBUG:boto:Token: None 2024-04-12T03:50:15.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/ 2024-04-12T03:50:15.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI= 2024-04-12T03:50:15.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d+p9JLEnYUXq3IctmdzIwRem6TI=', 'Content-Length': '0'} 2024-04-12T03:50:15.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096660ba2b1762bee-006618af77-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.988 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:15.988 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.988 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.988 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.988 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:50:15.989 DEBUG:boto:Data: 2024-04-12T03:50:15.989 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.989 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.989 DEBUG:boto:Params: {} 2024-04-12T03:50:15.989 DEBUG:boto:Token: None 2024-04-12T03:50:15.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22 2024-04-12T03:50:15.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rB0iADgeBuj0+oCgfcGVC4xE6a0= 2024-04-12T03:50:15.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rB0iADgeBuj0+oCgfcGVC4xE6a0=', 'Content-Length': '0'} 2024-04-12T03:50:15.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9715271d7f59a7b-006618af77-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.991 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.991 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.991 DEBUG:boto:Method: HEAD 2024-04-12T03:50:15.991 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:50:15.991 DEBUG:boto:Data: 2024-04-12T03:50:15.992 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:15.992 DEBUG:boto:Port: 8000 2024-04-12T03:50:15.992 DEBUG:boto:Params: {} 2024-04-12T03:50:15.992 DEBUG:boto:Token: None 2024-04-12T03:50:15.992 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22 2024-04-12T03:50:15.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rB0iADgeBuj0+oCgfcGVC4xE6a0= 2024-04-12T03:50:15.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rB0iADgeBuj0+oCgfcGVC4xE6a0=', 'Content-Length': '0'} 2024-04-12T03:50:15.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fcc1d7624169735-006618af77-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:15.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:50:15.994 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:15.994 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:15.994 DEBUG:boto:Method: GET 2024-04-12T03:50:15.995 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:50:15.995 DEBUG:boto:Data: 2024-04-12T03:50:15.995 DEBUG:boto:Headers: {} 2024-04-12T03:50:15.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:15.995 DEBUG:boto:Port: 8001 2024-04-12T03:50:15.995 DEBUG:boto:Params: {} 2024-04-12T03:50:15.995 DEBUG:boto:Token: None 2024-04-12T03:50:15.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:15 GMT /dgbpxg-61/obj22 2024-04-12T03:50:15.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NXzkLSZfC8Dr04vUB8NnCr8gQZI= 2024-04-12T03:50:15.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:15 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NXzkLSZfC8Dr04vUB8NnCr8gQZI=', 'Content-Length': '0'} 2024-04-12T03:50:15.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b2367e4b683f16-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.042 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:50:16.042 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:50:16.043 DEBUG:boto:Method: GET 2024-04-12T03:50:16.043 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:50:16.043 DEBUG:boto:Data: 2024-04-12T03:50:16.043 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.043 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.043 DEBUG:boto:Params: {} 2024-04-12T03:50:16.043 DEBUG:boto:Token: None 2024-04-12T03:50:16.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj22 2024-04-12T03:50:16.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jsT2MWO8mfZ2pU5k76FaN/2dKWc= 2024-04-12T03:50:16.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jsT2MWO8mfZ2pU5k76FaN/2dKWc=', 'Content-Length': '0'} 2024-04-12T03:50:16.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ddc631c288f5b6-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.046 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.046 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.046 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.046 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.046 DEBUG:boto:Method: GET 2024-04-12T03:50:16.046 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.046 DEBUG:boto:Data: 2024-04-12T03:50:16.046 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.047 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.047 DEBUG:boto:Params: {} 2024-04-12T03:50:16.047 DEBUG:boto:Token: None 2024-04-12T03:50:16.047 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.047 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401d0e06e694a9bc-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.049 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.050 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.050 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.050 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.050 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.050 DEBUG:boto:Method: GET 2024-04-12T03:50:16.050 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:50:16.050 DEBUG:boto:Data: 2024-04-12T03:50:16.051 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.051 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.051 DEBUG:boto:Params: {} 2024-04-12T03:50:16.051 DEBUG:boto:Token: None 2024-04-12T03:50:16.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.053 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea288b206f24ff4-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.057 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:50:16.057 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.058 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.058 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.058 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.058 DEBUG:boto:Method: GET 2024-04-12T03:50:16.058 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.058 DEBUG:boto:Data: 2024-04-12T03:50:16.058 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.058 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.058 DEBUG:boto:Params: {} 2024-04-12T03:50:16.059 DEBUG:boto:Token: None 2024-04-12T03:50:16.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc= 2024-04-12T03:50:16.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc=', 'Content-Length': '0'} 2024-04-12T03:50:16.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40e2efdae2909d6-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.061 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.061 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.061 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.061 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.061 DEBUG:boto:Method: GET 2024-04-12T03:50:16.061 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.062 DEBUG:boto:Data: 2024-04-12T03:50:16.062 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.062 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.062 DEBUG:boto:Params: {} 2024-04-12T03:50:16.062 DEBUG:boto:Token: None 2024-04-12T03:50:16.062 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc= 2024-04-12T03:50:16.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc=', 'Content-Length': '0'} 2024-04-12T03:50:16.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da622d4e5d93b4af-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.064 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.064 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.064 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.065 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.065 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.065 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.065 DEBUG:boto:Data: 2024-04-12T03:50:16.065 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.065 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.065 DEBUG:boto:Params: {} 2024-04-12T03:50:16.065 DEBUG:boto:Token: None 2024-04-12T03:50:16.065 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SG4mGRL115ImgjXLYcGU5Xren9s= 2024-04-12T03:50:16.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SG4mGRL115ImgjXLYcGU5Xren9s=', 'Content-Length': '0'} 2024-04-12T03:50:16.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008042b1c4b4413f67-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.068 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.068 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.068 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.068 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.068 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.068 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.068 DEBUG:boto:Data: 2024-04-12T03:50:16.068 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.068 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.068 DEBUG:boto:Params: {} 2024-04-12T03:50:16.069 DEBUG:boto:Token: None 2024-04-12T03:50:16.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SG4mGRL115ImgjXLYcGU5Xren9s= 2024-04-12T03:50:16.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SG4mGRL115ImgjXLYcGU5Xren9s=', 'Content-Length': '0'} 2024-04-12T03:50:16.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a751a26423b4d5b8-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:50:16.071 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.071 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.071 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.071 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.071 DEBUG:boto:Method: GET 2024-04-12T03:50:16.071 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.072 DEBUG:boto:Data: 2024-04-12T03:50:16.072 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.072 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.072 DEBUG:boto:Params: {} 2024-04-12T03:50:16.072 DEBUG:boto:Token: None 2024-04-12T03:50:16.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc= 2024-04-12T03:50:16.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc=', 'Content-Length': '0'} 2024-04-12T03:50:16.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f28fa6b59fdb6c6-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.074 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.074 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.074 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.074 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.075 DEBUG:boto:Method: GET 2024-04-12T03:50:16.075 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.075 DEBUG:boto:Data: 2024-04-12T03:50:16.075 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.075 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.075 DEBUG:boto:Params: {} 2024-04-12T03:50:16.075 DEBUG:boto:Token: None 2024-04-12T03:50:16.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:50:16.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc= 2024-04-12T03:50:16.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rmYneAV4SdNP/6iUX37Io75GQkc=', 'Content-Length': '0'} 2024-04-12T03:50:16.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009271c1298abc7440-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.118 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.119 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.119 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.119 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.119 DEBUG:boto:Method: GET 2024-04-12T03:50:16.119 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.119 DEBUG:boto:Data: 2024-04-12T03:50:16.119 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.119 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.119 DEBUG:boto:Params: {} 2024-04-12T03:50:16.119 DEBUG:boto:Token: None 2024-04-12T03:50:16.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba55c739c1d7a22-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.122 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.123 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.123 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.123 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.123 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.123 DEBUG:boto:Method: GET 2024-04-12T03:50:16.123 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.123 DEBUG:boto:Data: 2024-04-12T03:50:16.123 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.124 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.124 DEBUG:boto:Params: {} 2024-04-12T03:50:16.124 DEBUG:boto:Token: None 2024-04-12T03:50:16.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d5af74f28ffd14-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.127 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.128 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.128 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.128 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.128 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:50:16.128 DEBUG:boto:Data: 2024-04-12T03:50:16.128 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.128 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.128 DEBUG:boto:Params: {} 2024-04-12T03:50:16.129 DEBUG:boto:Token: None 2024-04-12T03:50:16.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23 2024-04-12T03:50:16.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YRVH8sJrUV+CAUs0n3rzDrZu4Eo= 2024-04-12T03:50:16.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YRVH8sJrUV+CAUs0n3rzDrZu4Eo=', 'Content-Length': '0'} 2024-04-12T03:50:16.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cc842df6c51799-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.131 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.131 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.131 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.131 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:50:16.131 DEBUG:boto:Data: 2024-04-12T03:50:16.131 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.131 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.131 DEBUG:boto:Params: {} 2024-04-12T03:50:16.132 DEBUG:boto:Token: None 2024-04-12T03:50:16.132 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23 2024-04-12T03:50:16.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YRVH8sJrUV+CAUs0n3rzDrZu4Eo= 2024-04-12T03:50:16.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YRVH8sJrUV+CAUs0n3rzDrZu4Eo=', 'Content-Length': '0'} 2024-04-12T03:50:16.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7755a47753b4e9-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:50:16.134 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.134 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.134 DEBUG:boto:Method: GET 2024-04-12T03:50:16.134 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:50:16.134 DEBUG:boto:Data: 2024-04-12T03:50:16.134 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.134 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.135 DEBUG:boto:Params: {} 2024-04-12T03:50:16.135 DEBUG:boto:Token: None 2024-04-12T03:50:16.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23 2024-04-12T03:50:16.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QIB3Y0JO0enbf+IOqdExdCjC8hQ= 2024-04-12T03:50:16.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QIB3Y0JO0enbf+IOqdExdCjC8hQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000848ebfcb5d6b6a2-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.178 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:50:16.179 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:50:16.179 DEBUG:boto:Method: GET 2024-04-12T03:50:16.179 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:50:16.179 DEBUG:boto:Data: 2024-04-12T03:50:16.179 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.179 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.180 DEBUG:boto:Params: {} 2024-04-12T03:50:16.180 DEBUG:boto:Token: None 2024-04-12T03:50:16.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj23 2024-04-12T03:50:16.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QIB3Y0JO0enbf+IOqdExdCjC8hQ= 2024-04-12T03:50:16.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QIB3Y0JO0enbf+IOqdExdCjC8hQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ce49c85363a6eb-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.183 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.183 DEBUG:boto:Method: GET 2024-04-12T03:50:16.184 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.184 DEBUG:boto:Data: 2024-04-12T03:50:16.184 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.184 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.184 DEBUG:boto:Params: {} 2024-04-12T03:50:16.185 DEBUG:boto:Token: None 2024-04-12T03:50:16.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.185 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08ac6a1ebc5c7a3-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.188 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.188 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.188 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.189 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.189 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.189 DEBUG:boto:Method: GET 2024-04-12T03:50:16.189 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:50:16.189 DEBUG:boto:Data: 2024-04-12T03:50:16.189 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.190 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.190 DEBUG:boto:Params: {} 2024-04-12T03:50:16.190 DEBUG:boto:Token: None 2024-04-12T03:50:16.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2eb3b5ff76504a5-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.234 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:50:16.235 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.235 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.235 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.235 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.235 DEBUG:boto:Method: GET 2024-04-12T03:50:16.235 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.236 DEBUG:boto:Data: 2024-04-12T03:50:16.236 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.236 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.236 DEBUG:boto:Params: {} 2024-04-12T03:50:16.236 DEBUG:boto:Token: None 2024-04-12T03:50:16.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.237 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU= 2024-04-12T03:50:16.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU=', 'Content-Length': '0'} 2024-04-12T03:50:16.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eddad75dbd7adfa3-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.239 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.239 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.239 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.239 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.239 DEBUG:boto:Method: GET 2024-04-12T03:50:16.240 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.240 DEBUG:boto:Data: 2024-04-12T03:50:16.240 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.240 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.240 DEBUG:boto:Params: {} 2024-04-12T03:50:16.241 DEBUG:boto:Token: None 2024-04-12T03:50:16.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU= 2024-04-12T03:50:16.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU=', 'Content-Length': '0'} 2024-04-12T03:50:16.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003936a0187db78e90-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.243 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.243 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.243 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.244 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.244 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.245 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.245 DEBUG:boto:Data: 2024-04-12T03:50:16.245 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.245 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.246 DEBUG:boto:Params: {} 2024-04-12T03:50:16.246 DEBUG:boto:Token: None 2024-04-12T03:50:16.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6zdLG1m2bK8rKB2zs46GVCnaj8g= 2024-04-12T03:50:16.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6zdLG1m2bK8rKB2zs46GVCnaj8g=', 'Content-Length': '0'} 2024-04-12T03:50:16.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082fa6f9f0384f2da-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.249 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.249 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.249 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.249 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.249 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.249 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.249 DEBUG:boto:Data: 2024-04-12T03:50:16.249 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.249 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.250 DEBUG:boto:Params: {} 2024-04-12T03:50:16.250 DEBUG:boto:Token: None 2024-04-12T03:50:16.250 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.250 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6zdLG1m2bK8rKB2zs46GVCnaj8g= 2024-04-12T03:50:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6zdLG1m2bK8rKB2zs46GVCnaj8g=', 'Content-Length': '0'} 2024-04-12T03:50:16.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055ebf9e097184d4-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:50:16.252 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.252 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.252 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.252 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.253 DEBUG:boto:Method: GET 2024-04-12T03:50:16.253 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.253 DEBUG:boto:Data: 2024-04-12T03:50:16.253 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.253 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.253 DEBUG:boto:Params: {} 2024-04-12T03:50:16.253 DEBUG:boto:Token: None 2024-04-12T03:50:16.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU= 2024-04-12T03:50:16.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU=', 'Content-Length': '0'} 2024-04-12T03:50:16.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1738a822b49f6e8-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.258 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.258 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.259 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.259 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.259 DEBUG:boto:Method: GET 2024-04-12T03:50:16.259 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.259 DEBUG:boto:Data: 2024-04-12T03:50:16.259 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.259 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.259 DEBUG:boto:Params: {} 2024-04-12T03:50:16.259 DEBUG:boto:Token: None 2024-04-12T03:50:16.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:50:16.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU= 2024-04-12T03:50:16.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MS7Kq4N8pKL9wozy8wap12gf/JU=', 'Content-Length': '0'} 2024-04-12T03:50:16.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb7ce02e18d46d0a-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.262 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.262 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.262 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.262 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.262 DEBUG:boto:Method: GET 2024-04-12T03:50:16.262 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.262 DEBUG:boto:Data: 2024-04-12T03:50:16.263 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.263 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.263 DEBUG:boto:Params: {} 2024-04-12T03:50:16.263 DEBUG:boto:Token: None 2024-04-12T03:50:16.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d7557fab0251cd-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.310 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.311 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.311 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.311 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.311 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.311 DEBUG:boto:Method: GET 2024-04-12T03:50:16.311 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.311 DEBUG:boto:Data: 2024-04-12T03:50:16.311 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.312 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.312 DEBUG:boto:Params: {} 2024-04-12T03:50:16.312 DEBUG:boto:Token: None 2024-04-12T03:50:16.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001da7380ba404b789-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.315 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.316 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.316 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.316 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.316 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:50:16.316 DEBUG:boto:Data: 2024-04-12T03:50:16.316 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.316 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.316 DEBUG:boto:Params: {} 2024-04-12T03:50:16.317 DEBUG:boto:Token: None 2024-04-12T03:50:16.317 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29 2024-04-12T03:50:16.317 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k5GARUv1eTe444c+bItRTQeEjXU= 2024-04-12T03:50:16.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k5GARUv1eTe444c+bItRTQeEjXU=', 'Content-Length': '0'} 2024-04-12T03:50:16.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a942c130962a6453-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.319 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.319 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.319 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.319 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:50:16.319 DEBUG:boto:Data: 2024-04-12T03:50:16.319 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.319 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.320 DEBUG:boto:Params: {} 2024-04-12T03:50:16.320 DEBUG:boto:Token: None 2024-04-12T03:50:16.320 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29 2024-04-12T03:50:16.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k5GARUv1eTe444c+bItRTQeEjXU= 2024-04-12T03:50:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k5GARUv1eTe444c+bItRTQeEjXU=', 'Content-Length': '0'} 2024-04-12T03:50:16.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e8cbf289a865c5-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:50:16.322 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.322 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.322 DEBUG:boto:Method: GET 2024-04-12T03:50:16.323 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:50:16.323 DEBUG:boto:Data: 2024-04-12T03:50:16.323 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.323 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.323 DEBUG:boto:Params: {} 2024-04-12T03:50:16.323 DEBUG:boto:Token: None 2024-04-12T03:50:16.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29 2024-04-12T03:50:16.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZFfofaWOw2bjh5K3/4PZKii71QE= 2024-04-12T03:50:16.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZFfofaWOw2bjh5K3/4PZKii71QE=', 'Content-Length': '0'} 2024-04-12T03:50:16.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8fd7c30afd5c41b-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.325 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:50:16.326 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:50:16.326 DEBUG:boto:Method: GET 2024-04-12T03:50:16.326 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:50:16.326 DEBUG:boto:Data: 2024-04-12T03:50:16.326 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.326 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.326 DEBUG:boto:Params: {} 2024-04-12T03:50:16.326 DEBUG:boto:Token: None 2024-04-12T03:50:16.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj29 2024-04-12T03:50:16.327 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZFfofaWOw2bjh5K3/4PZKii71QE= 2024-04-12T03:50:16.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZFfofaWOw2bjh5K3/4PZKii71QE=', 'Content-Length': '0'} 2024-04-12T03:50:16.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004859199601da782a-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.374 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.374 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.374 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.375 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.375 DEBUG:boto:Method: GET 2024-04-12T03:50:16.375 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.375 DEBUG:boto:Data: 2024-04-12T03:50:16.375 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.375 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.375 DEBUG:boto:Params: {} 2024-04-12T03:50:16.375 DEBUG:boto:Token: None 2024-04-12T03:50:16.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519cd9fa5ae38c38-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.378 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.378 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.378 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.378 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.379 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.379 DEBUG:boto:Method: GET 2024-04-12T03:50:16.379 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:50:16.379 DEBUG:boto:Data: 2024-04-12T03:50:16.379 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.379 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.379 DEBUG:boto:Params: {} 2024-04-12T03:50:16.379 DEBUG:boto:Token: None 2024-04-12T03:50:16.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bfd95bddaede148-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.383 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:50:16.384 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.384 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.384 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.384 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.384 DEBUG:boto:Method: GET 2024-04-12T03:50:16.384 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.384 DEBUG:boto:Data: 2024-04-12T03:50:16.384 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.385 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.385 DEBUG:boto:Params: {} 2024-04-12T03:50:16.385 DEBUG:boto:Token: None 2024-04-12T03:50:16.385 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.385 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ= 2024-04-12T03:50:16.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f66ac78fdfb1e4-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.387 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.387 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.387 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.387 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.387 DEBUG:boto:Method: GET 2024-04-12T03:50:16.387 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.388 DEBUG:boto:Data: 2024-04-12T03:50:16.388 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.388 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.388 DEBUG:boto:Params: {} 2024-04-12T03:50:16.388 DEBUG:boto:Token: None 2024-04-12T03:50:16.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ= 2024-04-12T03:50:16.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003fcf9621abd8211-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.390 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.391 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.391 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.391 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.391 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.391 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.391 DEBUG:boto:Data: 2024-04-12T03:50:16.391 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.391 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.391 DEBUG:boto:Params: {} 2024-04-12T03:50:16.392 DEBUG:boto:Token: None 2024-04-12T03:50:16.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3rfd9LfxiuaqZHF6jX+rCNsjOA= 2024-04-12T03:50:16.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3rfd9LfxiuaqZHF6jX+rCNsjOA=', 'Content-Length': '0'} 2024-04-12T03:50:16.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fb0ca1300ab3a5-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.394 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.394 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.394 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.394 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.394 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.394 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.394 DEBUG:boto:Data: 2024-04-12T03:50:16.394 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.395 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.395 DEBUG:boto:Params: {} 2024-04-12T03:50:16.395 DEBUG:boto:Token: None 2024-04-12T03:50:16.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P3rfd9LfxiuaqZHF6jX+rCNsjOA= 2024-04-12T03:50:16.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P3rfd9LfxiuaqZHF6jX+rCNsjOA=', 'Content-Length': '0'} 2024-04-12T03:50:16.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9030ff64c112fd7-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:50:16.397 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.397 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.397 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.397 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.397 DEBUG:boto:Method: GET 2024-04-12T03:50:16.397 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.398 DEBUG:boto:Data: 2024-04-12T03:50:16.398 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.398 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.398 DEBUG:boto:Params: {} 2024-04-12T03:50:16.398 DEBUG:boto:Token: None 2024-04-12T03:50:16.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ= 2024-04-12T03:50:16.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e24e916f9f37c66-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.442 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.442 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.443 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.443 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.443 DEBUG:boto:Method: GET 2024-04-12T03:50:16.443 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.443 DEBUG:boto:Data: 2024-04-12T03:50:16.443 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.443 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.443 DEBUG:boto:Params: {} 2024-04-12T03:50:16.443 DEBUG:boto:Token: None 2024-04-12T03:50:16.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:50:16.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ= 2024-04-12T03:50:16.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LQBT4I5Q6r6Q5n4gc4wJOc5yjDQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e4f4011f0b37498-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.446 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.446 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.446 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.446 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.446 DEBUG:boto:Method: GET 2024-04-12T03:50:16.447 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.447 DEBUG:boto:Data: 2024-04-12T03:50:16.447 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.447 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.447 DEBUG:boto:Params: {} 2024-04-12T03:50:16.447 DEBUG:boto:Token: None 2024-04-12T03:50:16.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000484c2b64603a494e-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.449 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.450 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.450 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.450 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.450 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.450 DEBUG:boto:Method: GET 2024-04-12T03:50:16.450 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.451 DEBUG:boto:Data: 2024-04-12T03:50:16.451 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.451 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.451 DEBUG:boto:Params: {} 2024-04-12T03:50:16.451 DEBUG:boto:Token: None 2024-04-12T03:50:16.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073142e9697ae3949-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.454 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.455 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.455 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.455 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.455 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:50:16.455 DEBUG:boto:Data: 2024-04-12T03:50:16.455 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.455 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.456 DEBUG:boto:Params: {} 2024-04-12T03:50:16.456 DEBUG:boto:Token: None 2024-04-12T03:50:16.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33 2024-04-12T03:50:16.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CT+uz42/Yy5W/8CnT2A9thnd0is= 2024-04-12T03:50:16.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CT+uz42/Yy5W/8CnT2A9thnd0is=', 'Content-Length': '0'} 2024-04-12T03:50:16.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2895978247fad21-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.458 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.458 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.458 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.458 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:50:16.458 DEBUG:boto:Data: 2024-04-12T03:50:16.458 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.458 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.459 DEBUG:boto:Params: {} 2024-04-12T03:50:16.461 DEBUG:boto:Token: None 2024-04-12T03:50:16.462 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33 2024-04-12T03:50:16.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CT+uz42/Yy5W/8CnT2A9thnd0is= 2024-04-12T03:50:16.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CT+uz42/Yy5W/8CnT2A9thnd0is=', 'Content-Length': '0'} 2024-04-12T03:50:16.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c330a5ee5104d18-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:50:16.464 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.464 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.464 DEBUG:boto:Method: GET 2024-04-12T03:50:16.464 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:50:16.464 DEBUG:boto:Data: 2024-04-12T03:50:16.464 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.465 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.465 DEBUG:boto:Params: {} 2024-04-12T03:50:16.465 DEBUG:boto:Token: None 2024-04-12T03:50:16.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33 2024-04-12T03:50:16.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d1AXvsd6vROItViBAB4kdiuP84E= 2024-04-12T03:50:16.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d1AXvsd6vROItViBAB4kdiuP84E=', 'Content-Length': '0'} 2024-04-12T03:50:16.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea0ec83d38da3e0-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.467 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:50:16.467 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:50:16.467 DEBUG:boto:Method: GET 2024-04-12T03:50:16.467 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:50:16.467 DEBUG:boto:Data: 2024-04-12T03:50:16.467 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.468 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.468 DEBUG:boto:Params: {} 2024-04-12T03:50:16.468 DEBUG:boto:Token: None 2024-04-12T03:50:16.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj33 2024-04-12T03:50:16.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d1AXvsd6vROItViBAB4kdiuP84E= 2024-04-12T03:50:16.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d1AXvsd6vROItViBAB4kdiuP84E=', 'Content-Length': '0'} 2024-04-12T03:50:16.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ba54eccd559fa5-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.470 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.470 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.470 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.470 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.470 DEBUG:boto:Method: GET 2024-04-12T03:50:16.471 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.471 DEBUG:boto:Data: 2024-04-12T03:50:16.471 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.471 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.471 DEBUG:boto:Params: {} 2024-04-12T03:50:16.471 DEBUG:boto:Token: None 2024-04-12T03:50:16.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023afaaa03517da26-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.474 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.474 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.474 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.474 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.474 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.475 DEBUG:boto:Method: GET 2024-04-12T03:50:16.475 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:50:16.475 DEBUG:boto:Data: 2024-04-12T03:50:16.475 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.475 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.475 DEBUG:boto:Params: {} 2024-04-12T03:50:16.475 DEBUG:boto:Token: None 2024-04-12T03:50:16.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000382c560d6cda8551-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.478 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:50:16.479 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.479 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.479 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.479 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.479 DEBUG:boto:Method: GET 2024-04-12T03:50:16.479 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.479 DEBUG:boto:Data: 2024-04-12T03:50:16.479 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.480 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.480 DEBUG:boto:Params: {} 2024-04-12T03:50:16.480 DEBUG:boto:Token: None 2024-04-12T03:50:16.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo= 2024-04-12T03:50:16.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo=', 'Content-Length': '0'} 2024-04-12T03:50:16.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226eded7301515c3-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.482 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.482 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.482 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.482 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.482 DEBUG:boto:Method: GET 2024-04-12T03:50:16.483 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.483 DEBUG:boto:Data: 2024-04-12T03:50:16.483 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.483 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.483 DEBUG:boto:Params: {} 2024-04-12T03:50:16.483 DEBUG:boto:Token: None 2024-04-12T03:50:16.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo= 2024-04-12T03:50:16.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo=', 'Content-Length': '0'} 2024-04-12T03:50:16.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dee0449f2ff7bbd-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.485 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.485 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.485 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.486 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.486 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.486 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.486 DEBUG:boto:Data: 2024-04-12T03:50:16.486 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.486 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.486 DEBUG:boto:Params: {} 2024-04-12T03:50:16.487 DEBUG:boto:Token: None 2024-04-12T03:50:16.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1oUGuDNWGH6TcaTtqC5gYa0GrmA= 2024-04-12T03:50:16.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1oUGuDNWGH6TcaTtqC5gYa0GrmA=', 'Content-Length': '0'} 2024-04-12T03:50:16.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9480d46ada4a81-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.489 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.489 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.489 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.489 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.489 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.489 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.489 DEBUG:boto:Data: 2024-04-12T03:50:16.489 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.489 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.489 DEBUG:boto:Params: {} 2024-04-12T03:50:16.490 DEBUG:boto:Token: None 2024-04-12T03:50:16.490 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1oUGuDNWGH6TcaTtqC5gYa0GrmA= 2024-04-12T03:50:16.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1oUGuDNWGH6TcaTtqC5gYa0GrmA=', 'Content-Length': '0'} 2024-04-12T03:50:16.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b6cfa4291cae24-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:50:16.492 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.492 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.492 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.492 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.492 DEBUG:boto:Method: GET 2024-04-12T03:50:16.492 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.493 DEBUG:boto:Data: 2024-04-12T03:50:16.493 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.493 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.493 DEBUG:boto:Params: {} 2024-04-12T03:50:16.493 DEBUG:boto:Token: None 2024-04-12T03:50:16.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo= 2024-04-12T03:50:16.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo=', 'Content-Length': '0'} 2024-04-12T03:50:16.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b40b1965bbfc1b-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.495 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.495 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.496 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.496 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.496 DEBUG:boto:Method: GET 2024-04-12T03:50:16.496 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.496 DEBUG:boto:Data: 2024-04-12T03:50:16.496 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.496 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.496 DEBUG:boto:Params: {} 2024-04-12T03:50:16.496 DEBUG:boto:Token: None 2024-04-12T03:50:16.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:50:16.497 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo= 2024-04-12T03:50:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A5A87Vqhwtb2ESgQwZX41RSXYHo=', 'Content-Length': '0'} 2024-04-12T03:50:16.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9be3365654ddc9-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.499 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.499 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.499 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.499 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.499 DEBUG:boto:Method: GET 2024-04-12T03:50:16.499 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.499 DEBUG:boto:Data: 2024-04-12T03:50:16.499 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.500 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.500 DEBUG:boto:Params: {} 2024-04-12T03:50:16.500 DEBUG:boto:Token: None 2024-04-12T03:50:16.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000327538b911e9c787-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.550 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.551 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.551 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.551 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.551 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.551 DEBUG:boto:Method: GET 2024-04-12T03:50:16.551 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.551 DEBUG:boto:Data: 2024-04-12T03:50:16.551 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.552 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.552 DEBUG:boto:Params: {} 2024-04-12T03:50:16.552 DEBUG:boto:Token: None 2024-04-12T03:50:16.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf740579b65ffc6-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.555 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.556 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.556 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.556 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.556 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:50:16.556 DEBUG:boto:Data: 2024-04-12T03:50:16.556 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.556 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.557 DEBUG:boto:Params: {} 2024-04-12T03:50:16.557 DEBUG:boto:Token: None 2024-04-12T03:50:16.557 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34 2024-04-12T03:50:16.557 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AkoMFsVxALo9AX1WF4F+IoxBi30= 2024-04-12T03:50:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AkoMFsVxALo9AX1WF4F+IoxBi30=', 'Content-Length': '0'} 2024-04-12T03:50:16.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003894d77f054d050b-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.559 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.559 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.559 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.559 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:50:16.559 DEBUG:boto:Data: 2024-04-12T03:50:16.559 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.559 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.559 DEBUG:boto:Params: {} 2024-04-12T03:50:16.559 DEBUG:boto:Token: None 2024-04-12T03:50:16.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34 2024-04-12T03:50:16.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AkoMFsVxALo9AX1WF4F+IoxBi30= 2024-04-12T03:50:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AkoMFsVxALo9AX1WF4F+IoxBi30=', 'Content-Length': '0'} 2024-04-12T03:50:16.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001776382846e94eec-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:50:16.561 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.562 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.562 DEBUG:boto:Method: GET 2024-04-12T03:50:16.562 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:50:16.562 DEBUG:boto:Data: 2024-04-12T03:50:16.562 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.562 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.562 DEBUG:boto:Params: {} 2024-04-12T03:50:16.562 DEBUG:boto:Token: None 2024-04-12T03:50:16.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34 2024-04-12T03:50:16.563 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIj0sMZoAtnRyx5YN4rpVmkkmCg= 2024-04-12T03:50:16.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIj0sMZoAtnRyx5YN4rpVmkkmCg=', 'Content-Length': '0'} 2024-04-12T03:50:16.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481f5b5ee4d35709-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.564 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:50:16.564 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:50:16.564 DEBUG:boto:Method: GET 2024-04-12T03:50:16.565 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:50:16.565 DEBUG:boto:Data: 2024-04-12T03:50:16.565 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.565 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.565 DEBUG:boto:Params: {} 2024-04-12T03:50:16.565 DEBUG:boto:Token: None 2024-04-12T03:50:16.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj34 2024-04-12T03:50:16.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nIj0sMZoAtnRyx5YN4rpVmkkmCg= 2024-04-12T03:50:16.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nIj0sMZoAtnRyx5YN4rpVmkkmCg=', 'Content-Length': '0'} 2024-04-12T03:50:16.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d20c3816fb8989-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.614 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.614 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.615 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.615 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.615 DEBUG:boto:Method: GET 2024-04-12T03:50:16.615 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.615 DEBUG:boto:Data: 2024-04-12T03:50:16.615 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.615 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.615 DEBUG:boto:Params: {} 2024-04-12T03:50:16.615 DEBUG:boto:Token: None 2024-04-12T03:50:16.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.616 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ca6728894ee79a-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.618 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.619 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.619 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.619 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.619 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.619 DEBUG:boto:Method: GET 2024-04-12T03:50:16.619 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:50:16.619 DEBUG:boto:Data: 2024-04-12T03:50:16.619 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.620 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.620 DEBUG:boto:Params: {} 2024-04-12T03:50:16.620 DEBUG:boto:Token: None 2024-04-12T03:50:16.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4ca2be036f4acb-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.624 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:50:16.625 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.625 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.625 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.625 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.625 DEBUG:boto:Method: GET 2024-04-12T03:50:16.626 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.626 DEBUG:boto:Data: 2024-04-12T03:50:16.626 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.626 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.626 DEBUG:boto:Params: {} 2024-04-12T03:50:16.626 DEBUG:boto:Token: None 2024-04-12T03:50:16.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24= 2024-04-12T03:50:16.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24=', 'Content-Length': '0'} 2024-04-12T03:50:16.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861bd558023a088a-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.629 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.629 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.629 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.629 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.629 DEBUG:boto:Method: GET 2024-04-12T03:50:16.629 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.629 DEBUG:boto:Data: 2024-04-12T03:50:16.629 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.629 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.629 DEBUG:boto:Params: {} 2024-04-12T03:50:16.630 DEBUG:boto:Token: None 2024-04-12T03:50:16.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.630 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24= 2024-04-12T03:50:16.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24=', 'Content-Length': '0'} 2024-04-12T03:50:16.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bccf511aaa7cdd7-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.632 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.632 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.633 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.633 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.633 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.633 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.633 DEBUG:boto:Data: 2024-04-12T03:50:16.633 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.633 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.633 DEBUG:boto:Params: {} 2024-04-12T03:50:16.633 DEBUG:boto:Token: None 2024-04-12T03:50:16.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.634 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kVjXRzxwlA0ymmUy/rO3eaPdRXg= 2024-04-12T03:50:16.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kVjXRzxwlA0ymmUy/rO3eaPdRXg=', 'Content-Length': '0'} 2024-04-12T03:50:16.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c834c3f0df9fec2-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.635 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.635 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.635 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.636 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.636 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.636 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.636 DEBUG:boto:Data: 2024-04-12T03:50:16.636 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.636 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.636 DEBUG:boto:Params: {} 2024-04-12T03:50:16.636 DEBUG:boto:Token: None 2024-04-12T03:50:16.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kVjXRzxwlA0ymmUy/rO3eaPdRXg= 2024-04-12T03:50:16.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kVjXRzxwlA0ymmUy/rO3eaPdRXg=', 'Content-Length': '0'} 2024-04-12T03:50:16.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b6584c77e66985-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:50:16.638 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.638 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.639 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.639 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.639 DEBUG:boto:Method: GET 2024-04-12T03:50:16.639 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.639 DEBUG:boto:Data: 2024-04-12T03:50:16.639 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.639 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.639 DEBUG:boto:Params: {} 2024-04-12T03:50:16.639 DEBUG:boto:Token: None 2024-04-12T03:50:16.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24= 2024-04-12T03:50:16.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24=', 'Content-Length': '0'} 2024-04-12T03:50:16.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fffcf27c709154-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.642 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.642 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.642 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.642 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.642 DEBUG:boto:Method: GET 2024-04-12T03:50:16.642 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.642 DEBUG:boto:Data: 2024-04-12T03:50:16.642 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.643 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.643 DEBUG:boto:Params: {} 2024-04-12T03:50:16.643 DEBUG:boto:Token: None 2024-04-12T03:50:16.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:50:16.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24= 2024-04-12T03:50:16.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iyhG9gotlq22Kh2gayrpntOKv24=', 'Content-Length': '0'} 2024-04-12T03:50:16.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3acec7e4d049217-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.645 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.645 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.645 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.645 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.645 DEBUG:boto:Method: GET 2024-04-12T03:50:16.646 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.646 DEBUG:boto:Data: 2024-04-12T03:50:16.646 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.646 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.646 DEBUG:boto:Params: {} 2024-04-12T03:50:16.646 DEBUG:boto:Token: None 2024-04-12T03:50:16.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.646 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6cd3f34beaedaf5-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.648 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.649 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.649 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.649 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.649 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.649 DEBUG:boto:Method: GET 2024-04-12T03:50:16.649 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.649 DEBUG:boto:Data: 2024-04-12T03:50:16.650 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.650 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.650 DEBUG:boto:Params: {} 2024-04-12T03:50:16.650 DEBUG:boto:Token: None 2024-04-12T03:50:16.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.650 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c94f1039a6b5b4-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.653 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.654 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.654 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.654 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.654 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:50:16.654 DEBUG:boto:Data: 2024-04-12T03:50:16.654 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.655 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.655 DEBUG:boto:Params: {} 2024-04-12T03:50:16.655 DEBUG:boto:Token: None 2024-04-12T03:50:16.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38 2024-04-12T03:50:16.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eYtKIm1Oge51ccSmDGEqw+ZR8WI= 2024-04-12T03:50:16.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eYtKIm1Oge51ccSmDGEqw+ZR8WI=', 'Content-Length': '0'} 2024-04-12T03:50:16.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1409685c9b4d7bd-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.657 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.657 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.657 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.657 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:50:16.657 DEBUG:boto:Data: 2024-04-12T03:50:16.657 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.657 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.657 DEBUG:boto:Params: {} 2024-04-12T03:50:16.658 DEBUG:boto:Token: None 2024-04-12T03:50:16.658 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38 2024-04-12T03:50:16.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eYtKIm1Oge51ccSmDGEqw+ZR8WI= 2024-04-12T03:50:16.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eYtKIm1Oge51ccSmDGEqw+ZR8WI=', 'Content-Length': '0'} 2024-04-12T03:50:16.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002285acf062237414-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:50:16.660 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.660 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.660 DEBUG:boto:Method: GET 2024-04-12T03:50:16.660 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:50:16.660 DEBUG:boto:Data: 2024-04-12T03:50:16.660 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.660 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.660 DEBUG:boto:Params: {} 2024-04-12T03:50:16.661 DEBUG:boto:Token: None 2024-04-12T03:50:16.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38 2024-04-12T03:50:16.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sHdu+G1wovh97Hjl/BFwA/7Ihus= 2024-04-12T03:50:16.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sHdu+G1wovh97Hjl/BFwA/7Ihus=', 'Content-Length': '0'} 2024-04-12T03:50:16.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004376b43ebe8d615c-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.666 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:50:16.666 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:50:16.666 DEBUG:boto:Method: GET 2024-04-12T03:50:16.666 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:50:16.666 DEBUG:boto:Data: 2024-04-12T03:50:16.666 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.666 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.666 DEBUG:boto:Params: {} 2024-04-12T03:50:16.667 DEBUG:boto:Token: None 2024-04-12T03:50:16.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj38 2024-04-12T03:50:16.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sHdu+G1wovh97Hjl/BFwA/7Ihus= 2024-04-12T03:50:16.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sHdu+G1wovh97Hjl/BFwA/7Ihus=', 'Content-Length': '0'} 2024-04-12T03:50:16.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969d0a16e59115ae-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.669 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.669 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.669 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.669 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.669 DEBUG:boto:Method: GET 2024-04-12T03:50:16.669 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.669 DEBUG:boto:Data: 2024-04-12T03:50:16.670 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.670 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.670 DEBUG:boto:Params: {} 2024-04-12T03:50:16.670 DEBUG:boto:Token: None 2024-04-12T03:50:16.670 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.670 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012994e0bd3410832-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.714 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.715 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.715 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.715 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.715 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.715 DEBUG:boto:Method: GET 2024-04-12T03:50:16.715 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:50:16.715 DEBUG:boto:Data: 2024-04-12T03:50:16.715 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.716 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.716 DEBUG:boto:Params: {} 2024-04-12T03:50:16.716 DEBUG:boto:Token: None 2024-04-12T03:50:16.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000057c3b13282413c4-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.720 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:50:16.720 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.720 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.720 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.720 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.720 DEBUG:boto:Method: GET 2024-04-12T03:50:16.721 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.721 DEBUG:boto:Data: 2024-04-12T03:50:16.721 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.721 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.721 DEBUG:boto:Params: {} 2024-04-12T03:50:16.721 DEBUG:boto:Token: None 2024-04-12T03:50:16.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ= 2024-04-12T03:50:16.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a9ab7ad7ccee5a-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.724 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.724 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.724 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.724 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.724 DEBUG:boto:Method: GET 2024-04-12T03:50:16.724 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.724 DEBUG:boto:Data: 2024-04-12T03:50:16.724 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.724 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.725 DEBUG:boto:Params: {} 2024-04-12T03:50:16.725 DEBUG:boto:Token: None 2024-04-12T03:50:16.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ= 2024-04-12T03:50:16.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950bc21099b6eeb1-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.770 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.771 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.771 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.771 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.771 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.771 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.771 DEBUG:boto:Data: 2024-04-12T03:50:16.771 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.771 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.772 DEBUG:boto:Params: {} 2024-04-12T03:50:16.772 DEBUG:boto:Token: None 2024-04-12T03:50:16.772 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uz34bTbk0iasxEh6cUQGqZRCUG8= 2024-04-12T03:50:16.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uz34bTbk0iasxEh6cUQGqZRCUG8=', 'Content-Length': '0'} 2024-04-12T03:50:16.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b149f1bc1615d762-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.774 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.774 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.774 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.774 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.774 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.774 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.775 DEBUG:boto:Data: 2024-04-12T03:50:16.775 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.775 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.775 DEBUG:boto:Params: {} 2024-04-12T03:50:16.775 DEBUG:boto:Token: None 2024-04-12T03:50:16.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uz34bTbk0iasxEh6cUQGqZRCUG8= 2024-04-12T03:50:16.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uz34bTbk0iasxEh6cUQGqZRCUG8=', 'Content-Length': '0'} 2024-04-12T03:50:16.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7f017e987de173-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:50:16.777 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.777 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.777 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.777 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.778 DEBUG:boto:Method: GET 2024-04-12T03:50:16.778 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.778 DEBUG:boto:Data: 2024-04-12T03:50:16.778 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.778 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.778 DEBUG:boto:Params: {} 2024-04-12T03:50:16.778 DEBUG:boto:Token: None 2024-04-12T03:50:16.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ= 2024-04-12T03:50:16.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f8224c29487dc7-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.781 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.781 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.781 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.781 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.781 DEBUG:boto:Method: GET 2024-04-12T03:50:16.781 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.781 DEBUG:boto:Data: 2024-04-12T03:50:16.782 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.782 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.782 DEBUG:boto:Params: {} 2024-04-12T03:50:16.782 DEBUG:boto:Token: None 2024-04-12T03:50:16.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:50:16.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ= 2024-04-12T03:50:16.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DNcJSYKlpKYduqKG5ekKGPJIwQ=', 'Content-Length': '0'} 2024-04-12T03:50:16.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001713721b9f0367c4-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.784 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.784 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.784 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.784 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.784 DEBUG:boto:Method: GET 2024-04-12T03:50:16.785 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.785 DEBUG:boto:Data: 2024-04-12T03:50:16.785 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.785 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.785 DEBUG:boto:Params: {} 2024-04-12T03:50:16.785 DEBUG:boto:Token: None 2024-04-12T03:50:16.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db13f364895439c1-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.787 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.788 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.788 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.788 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.788 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.788 DEBUG:boto:Method: GET 2024-04-12T03:50:16.788 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.788 DEBUG:boto:Data: 2024-04-12T03:50:16.788 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.789 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.789 DEBUG:boto:Params: {} 2024-04-12T03:50:16.789 DEBUG:boto:Token: None 2024-04-12T03:50:16.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8cef48fd335e5d-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.792 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.793 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.793 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.793 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.793 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:50:16.793 DEBUG:boto:Data: 2024-04-12T03:50:16.793 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.793 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.794 DEBUG:boto:Params: {} 2024-04-12T03:50:16.794 DEBUG:boto:Token: None 2024-04-12T03:50:16.794 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44 2024-04-12T03:50:16.794 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OyYKQ1zmUAvhsnIjAqP8wt2HNjE= 2024-04-12T03:50:16.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OyYKQ1zmUAvhsnIjAqP8wt2HNjE=', 'Content-Length': '0'} 2024-04-12T03:50:16.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79fd97d80837407-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.796 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.796 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.796 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.796 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:50:16.796 DEBUG:boto:Data: 2024-04-12T03:50:16.796 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.797 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.797 DEBUG:boto:Params: {} 2024-04-12T03:50:16.797 DEBUG:boto:Token: None 2024-04-12T03:50:16.797 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44 2024-04-12T03:50:16.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OyYKQ1zmUAvhsnIjAqP8wt2HNjE= 2024-04-12T03:50:16.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OyYKQ1zmUAvhsnIjAqP8wt2HNjE=', 'Content-Length': '0'} 2024-04-12T03:50:16.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007946197ded509d0f-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:50:16.799 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.799 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.799 DEBUG:boto:Method: GET 2024-04-12T03:50:16.799 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:50:16.799 DEBUG:boto:Data: 2024-04-12T03:50:16.799 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.800 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.800 DEBUG:boto:Params: {} 2024-04-12T03:50:16.800 DEBUG:boto:Token: None 2024-04-12T03:50:16.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44 2024-04-12T03:50:16.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qToe8L/+yxruwTQy0B/XjohCc3k= 2024-04-12T03:50:16.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qToe8L/+yxruwTQy0B/XjohCc3k=', 'Content-Length': '0'} 2024-04-12T03:50:16.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef4c8d475899bfc-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.802 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:50:16.802 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:50:16.802 DEBUG:boto:Method: GET 2024-04-12T03:50:16.802 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:50:16.802 DEBUG:boto:Data: 2024-04-12T03:50:16.802 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.802 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.803 DEBUG:boto:Params: {} 2024-04-12T03:50:16.803 DEBUG:boto:Token: None 2024-04-12T03:50:16.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj44 2024-04-12T03:50:16.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qToe8L/+yxruwTQy0B/XjohCc3k= 2024-04-12T03:50:16.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qToe8L/+yxruwTQy0B/XjohCc3k=', 'Content-Length': '0'} 2024-04-12T03:50:16.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a5f59d4eac1413-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.846 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.847 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.847 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.847 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.847 DEBUG:boto:Method: GET 2024-04-12T03:50:16.847 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.847 DEBUG:boto:Data: 2024-04-12T03:50:16.847 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.847 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.847 DEBUG:boto:Params: {} 2024-04-12T03:50:16.848 DEBUG:boto:Token: None 2024-04-12T03:50:16.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008acaeee9f6ed278c-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.850 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.851 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.851 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.851 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.851 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.851 DEBUG:boto:Method: GET 2024-04-12T03:50:16.851 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:50:16.852 DEBUG:boto:Data: 2024-04-12T03:50:16.852 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.852 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.852 DEBUG:boto:Params: {} 2024-04-12T03:50:16.852 DEBUG:boto:Token: None 2024-04-12T03:50:16.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086709a89fba39f55-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.855 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:50:16.856 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.856 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.856 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.856 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.856 DEBUG:boto:Method: GET 2024-04-12T03:50:16.856 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.857 DEBUG:boto:Data: 2024-04-12T03:50:16.857 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.857 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.857 DEBUG:boto:Params: {} 2024-04-12T03:50:16.857 DEBUG:boto:Token: None 2024-04-12T03:50:16.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4= 2024-04-12T03:50:16.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4=', 'Content-Length': '0'} 2024-04-12T03:50:16.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000694de70d4dc22503-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.906 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.907 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.907 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.907 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.907 DEBUG:boto:Method: GET 2024-04-12T03:50:16.907 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.907 DEBUG:boto:Data: 2024-04-12T03:50:16.907 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.907 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.907 DEBUG:boto:Params: {} 2024-04-12T03:50:16.908 DEBUG:boto:Token: None 2024-04-12T03:50:16.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4= 2024-04-12T03:50:16.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4=', 'Content-Length': '0'} 2024-04-12T03:50:16.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005362fe25c2ac451d-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.910 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.910 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.910 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.910 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.910 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.911 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.911 DEBUG:boto:Data: 2024-04-12T03:50:16.911 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.911 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.911 DEBUG:boto:Params: {} 2024-04-12T03:50:16.911 DEBUG:boto:Token: None 2024-04-12T03:50:16.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zuJUPc1OkyxrV5nKbLfkuH7mObM= 2024-04-12T03:50:16.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zuJUPc1OkyxrV5nKbLfkuH7mObM=', 'Content-Length': '0'} 2024-04-12T03:50:16.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed76231a18203939-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.913 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.913 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.913 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.913 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.913 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.913 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.914 DEBUG:boto:Data: 2024-04-12T03:50:16.914 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.914 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.914 DEBUG:boto:Params: {} 2024-04-12T03:50:16.914 DEBUG:boto:Token: None 2024-04-12T03:50:16.914 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.914 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zuJUPc1OkyxrV5nKbLfkuH7mObM= 2024-04-12T03:50:16.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zuJUPc1OkyxrV5nKbLfkuH7mObM=', 'Content-Length': '0'} 2024-04-12T03:50:16.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e0d014e376ea133-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:50:16.916 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.916 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.916 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.916 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.916 DEBUG:boto:Method: GET 2024-04-12T03:50:16.917 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.917 DEBUG:boto:Data: 2024-04-12T03:50:16.917 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.917 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.917 DEBUG:boto:Params: {} 2024-04-12T03:50:16.917 DEBUG:boto:Token: None 2024-04-12T03:50:16.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4= 2024-04-12T03:50:16.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4=', 'Content-Length': '0'} 2024-04-12T03:50:16.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b34d5f466dfd14-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.919 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.919 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.919 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.919 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.920 DEBUG:boto:Method: GET 2024-04-12T03:50:16.920 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.920 DEBUG:boto:Data: 2024-04-12T03:50:16.920 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.920 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.920 DEBUG:boto:Params: {} 2024-04-12T03:50:16.920 DEBUG:boto:Token: None 2024-04-12T03:50:16.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:50:16.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4= 2024-04-12T03:50:16.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mO8WKyA8Uuk5fR6oYyQeASZuxh4=', 'Content-Length': '0'} 2024-04-12T03:50:16.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003332a1dbb5771568-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.922 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.922 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.922 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.923 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.923 DEBUG:boto:Method: GET 2024-04-12T03:50:16.923 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.923 DEBUG:boto:Data: 2024-04-12T03:50:16.923 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.923 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.923 DEBUG:boto:Params: {} 2024-04-12T03:50:16.923 DEBUG:boto:Token: None 2024-04-12T03:50:16.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b1495a73c890ae1-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.925 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.926 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.926 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.926 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.926 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.926 DEBUG:boto:Method: GET 2024-04-12T03:50:16.926 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.926 DEBUG:boto:Data: 2024-04-12T03:50:16.926 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.927 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.927 DEBUG:boto:Params: {} 2024-04-12T03:50:16.927 DEBUG:boto:Token: None 2024-04-12T03:50:16.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f64658582f8ea1d5-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.931 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.931 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.931 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.931 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.931 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:50:16.931 DEBUG:boto:Data: 2024-04-12T03:50:16.931 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.932 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.932 DEBUG:boto:Params: {} 2024-04-12T03:50:16.932 DEBUG:boto:Token: None 2024-04-12T03:50:16.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46 2024-04-12T03:50:16.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LoRu/QMZsPt0168k72GQmqaNWEI= 2024-04-12T03:50:16.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LoRu/QMZsPt0168k72GQmqaNWEI=', 'Content-Length': '0'} 2024-04-12T03:50:16.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140e752a3222f402-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.934 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.934 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.934 DEBUG:boto:Method: HEAD 2024-04-12T03:50:16.934 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:50:16.934 DEBUG:boto:Data: 2024-04-12T03:50:16.934 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.935 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.935 DEBUG:boto:Params: {} 2024-04-12T03:50:16.935 DEBUG:boto:Token: None 2024-04-12T03:50:16.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46 2024-04-12T03:50:16.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LoRu/QMZsPt0168k72GQmqaNWEI= 2024-04-12T03:50:16.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LoRu/QMZsPt0168k72GQmqaNWEI=', 'Content-Length': '0'} 2024-04-12T03:50:16.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662649d21ed97db9-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:50:16.937 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.937 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.937 DEBUG:boto:Method: GET 2024-04-12T03:50:16.937 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:50:16.937 DEBUG:boto:Data: 2024-04-12T03:50:16.937 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.937 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.937 DEBUG:boto:Params: {} 2024-04-12T03:50:16.938 DEBUG:boto:Token: None 2024-04-12T03:50:16.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46 2024-04-12T03:50:16.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rTMnCCvFk+d7ESEeXD3fijRlHIo= 2024-04-12T03:50:16.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rTMnCCvFk+d7ESEeXD3fijRlHIo=', 'Content-Length': '0'} 2024-04-12T03:50:16.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d353f3e00c138801-006618af78-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.940 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:50:16.940 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:50:16.940 DEBUG:boto:Method: GET 2024-04-12T03:50:16.940 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:50:16.940 DEBUG:boto:Data: 2024-04-12T03:50:16.940 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.940 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.940 DEBUG:boto:Params: {} 2024-04-12T03:50:16.940 DEBUG:boto:Token: None 2024-04-12T03:50:16.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/obj46 2024-04-12T03:50:16.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rTMnCCvFk+d7ESEeXD3fijRlHIo= 2024-04-12T03:50:16.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rTMnCCvFk+d7ESEeXD3fijRlHIo=', 'Content-Length': '0'} 2024-04-12T03:50:16.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dabebf24d7114bf-006618af78-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.943 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.943 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.943 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.943 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.943 DEBUG:boto:Method: GET 2024-04-12T03:50:16.943 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.943 DEBUG:boto:Data: 2024-04-12T03:50:16.943 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:16.944 DEBUG:boto:Port: 8001 2024-04-12T03:50:16.944 DEBUG:boto:Params: {} 2024-04-12T03:50:16.944 DEBUG:boto:Token: None 2024-04-12T03:50:16.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4f175ff3d32df9-006618af78-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.946 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.946 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:16.946 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:16.947 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.947 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.947 DEBUG:boto:Method: GET 2024-04-12T03:50:16.947 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:50:16.947 DEBUG:boto:Data: 2024-04-12T03:50:16.947 DEBUG:boto:Headers: {} 2024-04-12T03:50:16.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:16.947 DEBUG:boto:Port: 8000 2024-04-12T03:50:16.947 DEBUG:boto:Params: {} 2024-04-12T03:50:16.947 DEBUG:boto:Token: None 2024-04-12T03:50:16.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:16 GMT /dgbpxg-61/ 2024-04-12T03:50:16.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U= 2024-04-12T03:50:16.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:16 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:z6MyQ37/wuisiTTsUeKik9pZM8U=', 'Content-Length': '0'} 2024-04-12T03:50:16.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc303cb09502ec63-006618af78-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:16.998 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:16.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:50:16.999 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:16.999 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:16.999 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:16.999 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:16.999 DEBUG:boto:Method: GET 2024-04-12T03:50:16.999 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:16.999 DEBUG:boto:Data: 2024-04-12T03:50:16.999 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.000 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.000 DEBUG:boto:Params: {} 2024-04-12T03:50:17.000 DEBUG:boto:Token: None 2024-04-12T03:50:17.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg= 2024-04-12T03:50:17.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg=', 'Content-Length': '0'} 2024-04-12T03:50:17.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009acbf30a1c550adc-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.002 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.002 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.002 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.002 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.003 DEBUG:boto:Method: GET 2024-04-12T03:50:17.003 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.003 DEBUG:boto:Data: 2024-04-12T03:50:17.003 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.003 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.003 DEBUG:boto:Params: {} 2024-04-12T03:50:17.003 DEBUG:boto:Token: None 2024-04-12T03:50:17.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg= 2024-04-12T03:50:17.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg=', 'Content-Length': '0'} 2024-04-12T03:50:17.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8d4efd38449ec20-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.005 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.006 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.006 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.006 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.006 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.006 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.006 DEBUG:boto:Data: 2024-04-12T03:50:17.006 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.006 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.006 DEBUG:boto:Params: {} 2024-04-12T03:50:17.007 DEBUG:boto:Token: None 2024-04-12T03:50:17.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C1kQgWyJBFaMYCAdAm/EUbnlAwg= 2024-04-12T03:50:17.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C1kQgWyJBFaMYCAdAm/EUbnlAwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcd631a6315f5cc-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.008 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.009 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.009 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.009 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.009 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.009 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.009 DEBUG:boto:Data: 2024-04-12T03:50:17.009 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.009 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.009 DEBUG:boto:Params: {} 2024-04-12T03:50:17.009 DEBUG:boto:Token: None 2024-04-12T03:50:17.010 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.010 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C1kQgWyJBFaMYCAdAm/EUbnlAwg= 2024-04-12T03:50:17.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C1kQgWyJBFaMYCAdAm/EUbnlAwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a17a0ca0250b65-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:50:17.012 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.012 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.012 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.012 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.012 DEBUG:boto:Method: GET 2024-04-12T03:50:17.012 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.012 DEBUG:boto:Data: 2024-04-12T03:50:17.012 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.013 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.013 DEBUG:boto:Params: {} 2024-04-12T03:50:17.013 DEBUG:boto:Token: None 2024-04-12T03:50:17.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg= 2024-04-12T03:50:17.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg=', 'Content-Length': '0'} 2024-04-12T03:50:17.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007569b349874e306c-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.015 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.015 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.015 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.015 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.015 DEBUG:boto:Method: GET 2024-04-12T03:50:17.015 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.015 DEBUG:boto:Data: 2024-04-12T03:50:17.016 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.016 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.016 DEBUG:boto:Params: {} 2024-04-12T03:50:17.016 DEBUG:boto:Token: None 2024-04-12T03:50:17.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:50:17.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg= 2024-04-12T03:50:17.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2nydUPo1eQy6Is3vdecsAKPQoDg=', 'Content-Length': '0'} 2024-04-12T03:50:17.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f2a4bc58738f51-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.018 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.018 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.018 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.018 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.019 DEBUG:boto:Method: GET 2024-04-12T03:50:17.019 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.019 DEBUG:boto:Data: 2024-04-12T03:50:17.019 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.019 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.019 DEBUG:boto:Params: {} 2024-04-12T03:50:17.019 DEBUG:boto:Token: None 2024-04-12T03:50:17.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b588828d5701b8-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.062 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.063 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.063 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.063 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.063 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.063 DEBUG:boto:Method: GET 2024-04-12T03:50:17.063 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.063 DEBUG:boto:Data: 2024-04-12T03:50:17.063 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.064 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.064 DEBUG:boto:Params: {} 2024-04-12T03:50:17.064 DEBUG:boto:Token: None 2024-04-12T03:50:17.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001529cfa003553b9b-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.068 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.068 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.069 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.069 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.069 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:50:17.069 DEBUG:boto:Data: 2024-04-12T03:50:17.069 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.069 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.069 DEBUG:boto:Params: {} 2024-04-12T03:50:17.069 DEBUG:boto:Token: None 2024-04-12T03:50:17.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51 2024-04-12T03:50:17.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsSmSWDDD/jhcujtlv4nCNJdm9w= 2024-04-12T03:50:17.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsSmSWDDD/jhcujtlv4nCNJdm9w=', 'Content-Length': '0'} 2024-04-12T03:50:17.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ecd54e1d9cb86c1-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.071 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.071 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.072 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.072 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:50:17.072 DEBUG:boto:Data: 2024-04-12T03:50:17.072 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.072 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.072 DEBUG:boto:Params: {} 2024-04-12T03:50:17.072 DEBUG:boto:Token: None 2024-04-12T03:50:17.072 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51 2024-04-12T03:50:17.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JsSmSWDDD/jhcujtlv4nCNJdm9w= 2024-04-12T03:50:17.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JsSmSWDDD/jhcujtlv4nCNJdm9w=', 'Content-Length': '0'} 2024-04-12T03:50:17.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000563b480c43029829-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:50:17.078 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.078 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.078 DEBUG:boto:Method: GET 2024-04-12T03:50:17.078 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:50:17.078 DEBUG:boto:Data: 2024-04-12T03:50:17.078 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.078 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.078 DEBUG:boto:Params: {} 2024-04-12T03:50:17.079 DEBUG:boto:Token: None 2024-04-12T03:50:17.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51 2024-04-12T03:50:17.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TenBVl3WZT1PIbdCpEbGhvjnS18= 2024-04-12T03:50:17.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TenBVl3WZT1PIbdCpEbGhvjnS18=', 'Content-Length': '0'} 2024-04-12T03:50:17.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1a4c2c54958b4f-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.080 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:50:17.081 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:50:17.081 DEBUG:boto:Method: GET 2024-04-12T03:50:17.081 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:50:17.081 DEBUG:boto:Data: 2024-04-12T03:50:17.081 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.081 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.081 DEBUG:boto:Params: {} 2024-04-12T03:50:17.081 DEBUG:boto:Token: None 2024-04-12T03:50:17.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj51 2024-04-12T03:50:17.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TenBVl3WZT1PIbdCpEbGhvjnS18= 2024-04-12T03:50:17.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TenBVl3WZT1PIbdCpEbGhvjnS18=', 'Content-Length': '0'} 2024-04-12T03:50:17.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045daef6039c89fff-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.084 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.084 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.084 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.084 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.084 DEBUG:boto:Method: GET 2024-04-12T03:50:17.084 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.084 DEBUG:boto:Data: 2024-04-12T03:50:17.084 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.085 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.085 DEBUG:boto:Params: {} 2024-04-12T03:50:17.085 DEBUG:boto:Token: None 2024-04-12T03:50:17.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076edfb452860f4d3-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.088 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.088 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.088 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.088 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.088 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.089 DEBUG:boto:Method: GET 2024-04-12T03:50:17.089 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:50:17.089 DEBUG:boto:Data: 2024-04-12T03:50:17.089 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.089 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.089 DEBUG:boto:Params: {} 2024-04-12T03:50:17.089 DEBUG:boto:Token: None 2024-04-12T03:50:17.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee596fcaaa4f3ba-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.093 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:50:17.093 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.093 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.093 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.093 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.093 DEBUG:boto:Method: GET 2024-04-12T03:50:17.094 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.094 DEBUG:boto:Data: 2024-04-12T03:50:17.094 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.094 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.094 DEBUG:boto:Params: {} 2024-04-12T03:50:17.094 DEBUG:boto:Token: None 2024-04-12T03:50:17.094 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE= 2024-04-12T03:50:17.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE=', 'Content-Length': '0'} 2024-04-12T03:50:17.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dba41fb479bd9e0-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.096 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.096 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.096 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.097 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.097 DEBUG:boto:Method: GET 2024-04-12T03:50:17.097 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.097 DEBUG:boto:Data: 2024-04-12T03:50:17.097 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.097 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.097 DEBUG:boto:Params: {} 2024-04-12T03:50:17.097 DEBUG:boto:Token: None 2024-04-12T03:50:17.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE= 2024-04-12T03:50:17.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE=', 'Content-Length': '0'} 2024-04-12T03:50:17.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000443a7ed227cf6b48-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.142 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.143 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.143 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.143 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.143 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.143 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.143 DEBUG:boto:Data: 2024-04-12T03:50:17.143 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.143 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.143 DEBUG:boto:Params: {} 2024-04-12T03:50:17.144 DEBUG:boto:Token: None 2024-04-12T03:50:17.144 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x6ZsDW9+vy3uD4KdQDDvRr1fj3I= 2024-04-12T03:50:17.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x6ZsDW9+vy3uD4KdQDDvRr1fj3I=', 'Content-Length': '0'} 2024-04-12T03:50:17.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035863bbb2960bc7-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.146 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.146 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.146 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.146 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.146 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.147 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.147 DEBUG:boto:Data: 2024-04-12T03:50:17.147 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.147 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.147 DEBUG:boto:Params: {} 2024-04-12T03:50:17.147 DEBUG:boto:Token: None 2024-04-12T03:50:17.147 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x6ZsDW9+vy3uD4KdQDDvRr1fj3I= 2024-04-12T03:50:17.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x6ZsDW9+vy3uD4KdQDDvRr1fj3I=', 'Content-Length': '0'} 2024-04-12T03:50:17.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0cab4a6151027b-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:50:17.149 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.150 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.150 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.150 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.150 DEBUG:boto:Method: GET 2024-04-12T03:50:17.150 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.150 DEBUG:boto:Data: 2024-04-12T03:50:17.150 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.150 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.150 DEBUG:boto:Params: {} 2024-04-12T03:50:17.151 DEBUG:boto:Token: None 2024-04-12T03:50:17.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE= 2024-04-12T03:50:17.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE=', 'Content-Length': '0'} 2024-04-12T03:50:17.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d194acf4a97e83a3-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.153 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.153 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.153 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.153 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.154 DEBUG:boto:Method: GET 2024-04-12T03:50:17.154 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.154 DEBUG:boto:Data: 2024-04-12T03:50:17.154 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.154 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.154 DEBUG:boto:Params: {} 2024-04-12T03:50:17.154 DEBUG:boto:Token: None 2024-04-12T03:50:17.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:50:17.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE= 2024-04-12T03:50:17.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8GhZThaB+j8Gis27GBDQkIoN8nE=', 'Content-Length': '0'} 2024-04-12T03:50:17.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87ffca9534c8fef-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.156 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.157 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.157 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.157 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.157 DEBUG:boto:Method: GET 2024-04-12T03:50:17.157 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.157 DEBUG:boto:Data: 2024-04-12T03:50:17.157 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.157 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.157 DEBUG:boto:Params: {} 2024-04-12T03:50:17.157 DEBUG:boto:Token: None 2024-04-12T03:50:17.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d762d511d8594e-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.160 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.160 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.160 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.160 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.160 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.161 DEBUG:boto:Method: GET 2024-04-12T03:50:17.161 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.161 DEBUG:boto:Data: 2024-04-12T03:50:17.161 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.161 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.161 DEBUG:boto:Params: {} 2024-04-12T03:50:17.161 DEBUG:boto:Token: None 2024-04-12T03:50:17.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe65db0ba88c4c8d-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.165 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.165 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.165 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.165 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.165 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:50:17.165 DEBUG:boto:Data: 2024-04-12T03:50:17.165 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.166 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.166 DEBUG:boto:Params: {} 2024-04-12T03:50:17.166 DEBUG:boto:Token: None 2024-04-12T03:50:17.166 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55 2024-04-12T03:50:17.166 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6Hn/cbz5qZnt3glNWVujcTrcNk= 2024-04-12T03:50:17.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6Hn/cbz5qZnt3glNWVujcTrcNk=', 'Content-Length': '0'} 2024-04-12T03:50:17.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4ff26236253c36-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.168 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.168 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.168 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.168 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:50:17.168 DEBUG:boto:Data: 2024-04-12T03:50:17.168 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.168 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.168 DEBUG:boto:Params: {} 2024-04-12T03:50:17.169 DEBUG:boto:Token: None 2024-04-12T03:50:17.169 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55 2024-04-12T03:50:17.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6Hn/cbz5qZnt3glNWVujcTrcNk= 2024-04-12T03:50:17.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6Hn/cbz5qZnt3glNWVujcTrcNk=', 'Content-Length': '0'} 2024-04-12T03:50:17.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e76ca4ef7df4582-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:50:17.171 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.171 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.171 DEBUG:boto:Method: GET 2024-04-12T03:50:17.171 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:50:17.171 DEBUG:boto:Data: 2024-04-12T03:50:17.171 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.171 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.171 DEBUG:boto:Params: {} 2024-04-12T03:50:17.171 DEBUG:boto:Token: None 2024-04-12T03:50:17.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55 2024-04-12T03:50:17.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:knSvj8+s6dn3YKy00AUfTPGGXLg= 2024-04-12T03:50:17.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:knSvj8+s6dn3YKy00AUfTPGGXLg=', 'Content-Length': '0'} 2024-04-12T03:50:17.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000160051a885e4c38-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.173 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:50:17.174 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:50:17.174 DEBUG:boto:Method: GET 2024-04-12T03:50:17.174 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:50:17.174 DEBUG:boto:Data: 2024-04-12T03:50:17.174 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.174 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.174 DEBUG:boto:Params: {} 2024-04-12T03:50:17.174 DEBUG:boto:Token: None 2024-04-12T03:50:17.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj55 2024-04-12T03:50:17.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:knSvj8+s6dn3YKy00AUfTPGGXLg= 2024-04-12T03:50:17.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:knSvj8+s6dn3YKy00AUfTPGGXLg=', 'Content-Length': '0'} 2024-04-12T03:50:17.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000257ab850a1ce77f-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.219 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.219 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.219 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.219 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.219 DEBUG:boto:Method: GET 2024-04-12T03:50:17.219 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.219 DEBUG:boto:Data: 2024-04-12T03:50:17.219 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.220 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.220 DEBUG:boto:Params: {} 2024-04-12T03:50:17.220 DEBUG:boto:Token: None 2024-04-12T03:50:17.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a55264a8f70786-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.222 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.224 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.224 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.224 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.224 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.225 DEBUG:boto:Method: GET 2024-04-12T03:50:17.225 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:50:17.226 DEBUG:boto:Data: 2024-04-12T03:50:17.226 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.226 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.226 DEBUG:boto:Params: {} 2024-04-12T03:50:17.226 DEBUG:boto:Token: None 2024-04-12T03:50:17.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000813ddb80033891f-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.230 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:50:17.230 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.230 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.230 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.230 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.231 DEBUG:boto:Method: GET 2024-04-12T03:50:17.231 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.231 DEBUG:boto:Data: 2024-04-12T03:50:17.231 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.231 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.231 DEBUG:boto:Params: {} 2024-04-12T03:50:17.231 DEBUG:boto:Token: None 2024-04-12T03:50:17.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM= 2024-04-12T03:50:17.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM=', 'Content-Length': '0'} 2024-04-12T03:50:17.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000148ac8c0f4d92182-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.274 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.275 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.275 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.275 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.275 DEBUG:boto:Method: GET 2024-04-12T03:50:17.275 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.275 DEBUG:boto:Data: 2024-04-12T03:50:17.275 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.275 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.275 DEBUG:boto:Params: {} 2024-04-12T03:50:17.276 DEBUG:boto:Token: None 2024-04-12T03:50:17.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM= 2024-04-12T03:50:17.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM=', 'Content-Length': '0'} 2024-04-12T03:50:17.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dde28ed5129bd70-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.278 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.278 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.278 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.278 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.278 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.279 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.279 DEBUG:boto:Data: 2024-04-12T03:50:17.279 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.279 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.279 DEBUG:boto:Params: {} 2024-04-12T03:50:17.279 DEBUG:boto:Token: None 2024-04-12T03:50:17.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:60mA0bSIQqpO3VpdBmJLO/DoSFA= 2024-04-12T03:50:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:60mA0bSIQqpO3VpdBmJLO/DoSFA=', 'Content-Length': '0'} 2024-04-12T03:50:17.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8c939ec4307dba-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.281 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.281 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.281 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.282 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.282 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.282 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.282 DEBUG:boto:Data: 2024-04-12T03:50:17.282 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.282 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.282 DEBUG:boto:Params: {} 2024-04-12T03:50:17.282 DEBUG:boto:Token: None 2024-04-12T03:50:17.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:60mA0bSIQqpO3VpdBmJLO/DoSFA= 2024-04-12T03:50:17.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:60mA0bSIQqpO3VpdBmJLO/DoSFA=', 'Content-Length': '0'} 2024-04-12T03:50:17.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ee47597363cfcc-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:50:17.284 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.285 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.285 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.285 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.285 DEBUG:boto:Method: GET 2024-04-12T03:50:17.285 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.285 DEBUG:boto:Data: 2024-04-12T03:50:17.285 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.285 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.285 DEBUG:boto:Params: {} 2024-04-12T03:50:17.285 DEBUG:boto:Token: None 2024-04-12T03:50:17.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM= 2024-04-12T03:50:17.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM=', 'Content-Length': '0'} 2024-04-12T03:50:17.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e08331bc88638e6a-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.287 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.288 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.288 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.288 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.288 DEBUG:boto:Method: GET 2024-04-12T03:50:17.288 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.288 DEBUG:boto:Data: 2024-04-12T03:50:17.288 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.288 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.288 DEBUG:boto:Params: {} 2024-04-12T03:50:17.288 DEBUG:boto:Token: None 2024-04-12T03:50:17.289 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:50:17.289 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM= 2024-04-12T03:50:17.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CYDwnWK+xXRJYny7Jr4HbKYd0VM=', 'Content-Length': '0'} 2024-04-12T03:50:17.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be0d0e0f9e1df0f-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.291 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.291 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.291 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.291 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.291 DEBUG:boto:Method: GET 2024-04-12T03:50:17.291 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.291 DEBUG:boto:Data: 2024-04-12T03:50:17.291 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.291 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.291 DEBUG:boto:Params: {} 2024-04-12T03:50:17.292 DEBUG:boto:Token: None 2024-04-12T03:50:17.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd170a1612f8f2b-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.294 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.295 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.295 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.295 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.295 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.295 DEBUG:boto:Method: GET 2024-04-12T03:50:17.295 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.295 DEBUG:boto:Data: 2024-04-12T03:50:17.295 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.296 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.296 DEBUG:boto:Params: {} 2024-04-12T03:50:17.296 DEBUG:boto:Token: None 2024-04-12T03:50:17.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492fbba914d50b84-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.300 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.300 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.300 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.300 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.300 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:50:17.300 DEBUG:boto:Data: 2024-04-12T03:50:17.300 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.301 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.301 DEBUG:boto:Params: {} 2024-04-12T03:50:17.301 DEBUG:boto:Token: None 2024-04-12T03:50:17.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57 2024-04-12T03:50:17.301 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XS2pVB5EfBNtX8us0CD3qkHbmHI= 2024-04-12T03:50:17.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XS2pVB5EfBNtX8us0CD3qkHbmHI=', 'Content-Length': '0'} 2024-04-12T03:50:17.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b1b25ee9a589ee-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.303 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.303 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.303 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.303 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:50:17.303 DEBUG:boto:Data: 2024-04-12T03:50:17.304 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.304 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.304 DEBUG:boto:Params: {} 2024-04-12T03:50:17.304 DEBUG:boto:Token: None 2024-04-12T03:50:17.304 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57 2024-04-12T03:50:17.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XS2pVB5EfBNtX8us0CD3qkHbmHI= 2024-04-12T03:50:17.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XS2pVB5EfBNtX8us0CD3qkHbmHI=', 'Content-Length': '0'} 2024-04-12T03:50:17.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e8f817ee4dbd0c-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:50:17.306 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.306 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.306 DEBUG:boto:Method: GET 2024-04-12T03:50:17.306 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:50:17.306 DEBUG:boto:Data: 2024-04-12T03:50:17.306 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.307 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.307 DEBUG:boto:Params: {} 2024-04-12T03:50:17.307 DEBUG:boto:Token: None 2024-04-12T03:50:17.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57 2024-04-12T03:50:17.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n/GDPk23WBE8GyLZTpx5Eea6B7k= 2024-04-12T03:50:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n/GDPk23WBE8GyLZTpx5Eea6B7k=', 'Content-Length': '0'} 2024-04-12T03:50:17.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7777b430ccc83c-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.309 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:50:17.309 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:50:17.309 DEBUG:boto:Method: GET 2024-04-12T03:50:17.309 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:50:17.309 DEBUG:boto:Data: 2024-04-12T03:50:17.310 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.310 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.310 DEBUG:boto:Params: {} 2024-04-12T03:50:17.310 DEBUG:boto:Token: None 2024-04-12T03:50:17.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj57 2024-04-12T03:50:17.310 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n/GDPk23WBE8GyLZTpx5Eea6B7k= 2024-04-12T03:50:17.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n/GDPk23WBE8GyLZTpx5Eea6B7k=', 'Content-Length': '0'} 2024-04-12T03:50:17.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243914e42e315e5b-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.354 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.355 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.355 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.355 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.355 DEBUG:boto:Method: GET 2024-04-12T03:50:17.355 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.355 DEBUG:boto:Data: 2024-04-12T03:50:17.355 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.355 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.355 DEBUG:boto:Params: {} 2024-04-12T03:50:17.356 DEBUG:boto:Token: None 2024-04-12T03:50:17.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a7fe60d9bc99e8-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.358 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.359 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.359 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.359 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.359 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.359 DEBUG:boto:Method: GET 2024-04-12T03:50:17.359 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:50:17.359 DEBUG:boto:Data: 2024-04-12T03:50:17.360 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.360 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.360 DEBUG:boto:Params: {} 2024-04-12T03:50:17.360 DEBUG:boto:Token: None 2024-04-12T03:50:17.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcee76b163ae96ea-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.364 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:50:17.364 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.364 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.364 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.365 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.365 DEBUG:boto:Method: GET 2024-04-12T03:50:17.365 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.365 DEBUG:boto:Data: 2024-04-12T03:50:17.365 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.365 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.365 DEBUG:boto:Params: {} 2024-04-12T03:50:17.365 DEBUG:boto:Token: None 2024-04-12T03:50:17.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg= 2024-04-12T03:50:17.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg=', 'Content-Length': '0'} 2024-04-12T03:50:17.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b21f15b8651c74ba-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.410 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.411 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.411 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.411 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.411 DEBUG:boto:Method: GET 2024-04-12T03:50:17.411 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.411 DEBUG:boto:Data: 2024-04-12T03:50:17.411 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.411 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.411 DEBUG:boto:Params: {} 2024-04-12T03:50:17.412 DEBUG:boto:Token: None 2024-04-12T03:50:17.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg= 2024-04-12T03:50:17.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg=', 'Content-Length': '0'} 2024-04-12T03:50:17.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007aabc7f95d004de-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.414 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.414 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.414 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.415 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.415 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.415 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.415 DEBUG:boto:Data: 2024-04-12T03:50:17.415 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.415 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.415 DEBUG:boto:Params: {} 2024-04-12T03:50:17.415 DEBUG:boto:Token: None 2024-04-12T03:50:17.415 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d3MgRf1WruB92/JlozQkgpNlhck= 2024-04-12T03:50:17.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d3MgRf1WruB92/JlozQkgpNlhck=', 'Content-Length': '0'} 2024-04-12T03:50:17.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf6fc7d989bf25b-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.417 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.417 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.417 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.418 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.418 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.418 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.418 DEBUG:boto:Data: 2024-04-12T03:50:17.418 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.418 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.418 DEBUG:boto:Params: {} 2024-04-12T03:50:17.418 DEBUG:boto:Token: None 2024-04-12T03:50:17.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d3MgRf1WruB92/JlozQkgpNlhck= 2024-04-12T03:50:17.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d3MgRf1WruB92/JlozQkgpNlhck=', 'Content-Length': '0'} 2024-04-12T03:50:17.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb50bc7e55e1724-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:50:17.421 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.421 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.421 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.421 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.421 DEBUG:boto:Method: GET 2024-04-12T03:50:17.421 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.421 DEBUG:boto:Data: 2024-04-12T03:50:17.421 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.422 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.422 DEBUG:boto:Params: {} 2024-04-12T03:50:17.422 DEBUG:boto:Token: None 2024-04-12T03:50:17.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg= 2024-04-12T03:50:17.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg=', 'Content-Length': '0'} 2024-04-12T03:50:17.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011df111ad03c16c9-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.424 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.424 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.424 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.424 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.424 DEBUG:boto:Method: GET 2024-04-12T03:50:17.424 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.424 DEBUG:boto:Data: 2024-04-12T03:50:17.424 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.425 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.425 DEBUG:boto:Params: {} 2024-04-12T03:50:17.425 DEBUG:boto:Token: None 2024-04-12T03:50:17.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:50:17.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg= 2024-04-12T03:50:17.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fV79JAbz0DutYH6LfZEJzJyRkyg=', 'Content-Length': '0'} 2024-04-12T03:50:17.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d10ae09cfe3c921-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.427 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.427 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.427 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.427 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.427 DEBUG:boto:Method: GET 2024-04-12T03:50:17.428 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.428 DEBUG:boto:Data: 2024-04-12T03:50:17.428 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.428 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.428 DEBUG:boto:Params: {} 2024-04-12T03:50:17.428 DEBUG:boto:Token: None 2024-04-12T03:50:17.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b611379ef07338e7-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.430 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.431 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.431 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.431 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.431 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.431 DEBUG:boto:Method: GET 2024-04-12T03:50:17.431 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.431 DEBUG:boto:Data: 2024-04-12T03:50:17.431 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.432 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.432 DEBUG:boto:Params: {} 2024-04-12T03:50:17.432 DEBUG:boto:Token: None 2024-04-12T03:50:17.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000206b59154d9e239e-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.435 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.436 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.436 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.436 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.436 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:50:17.436 DEBUG:boto:Data: 2024-04-12T03:50:17.436 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.436 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.437 DEBUG:boto:Params: {} 2024-04-12T03:50:17.437 DEBUG:boto:Token: None 2024-04-12T03:50:17.437 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58 2024-04-12T03:50:17.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6SAbuHedydJNp+bLgHLZZsY2qpI= 2024-04-12T03:50:17.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6SAbuHedydJNp+bLgHLZZsY2qpI=', 'Content-Length': '0'} 2024-04-12T03:50:17.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c312ad11f1f95423-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.439 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.439 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.439 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.439 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:50:17.440 DEBUG:boto:Data: 2024-04-12T03:50:17.440 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.440 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.440 DEBUG:boto:Params: {} 2024-04-12T03:50:17.440 DEBUG:boto:Token: None 2024-04-12T03:50:17.440 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58 2024-04-12T03:50:17.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6SAbuHedydJNp+bLgHLZZsY2qpI= 2024-04-12T03:50:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6SAbuHedydJNp+bLgHLZZsY2qpI=', 'Content-Length': '0'} 2024-04-12T03:50:17.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051757f992c438760-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:50:17.442 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.442 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.442 DEBUG:boto:Method: GET 2024-04-12T03:50:17.442 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:50:17.443 DEBUG:boto:Data: 2024-04-12T03:50:17.443 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.443 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.443 DEBUG:boto:Params: {} 2024-04-12T03:50:17.443 DEBUG:boto:Token: None 2024-04-12T03:50:17.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58 2024-04-12T03:50:17.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrP87Hu/FVTL0LmCSSvDxgOiCeY= 2024-04-12T03:50:17.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrP87Hu/FVTL0LmCSSvDxgOiCeY=', 'Content-Length': '0'} 2024-04-12T03:50:17.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa09d5b02f6d01b-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.489 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:50:17.490 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:50:17.490 DEBUG:boto:Method: GET 2024-04-12T03:50:17.490 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:50:17.490 DEBUG:boto:Data: 2024-04-12T03:50:17.490 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.490 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.490 DEBUG:boto:Params: {} 2024-04-12T03:50:17.491 DEBUG:boto:Token: None 2024-04-12T03:50:17.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj58 2024-04-12T03:50:17.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LrP87Hu/FVTL0LmCSSvDxgOiCeY= 2024-04-12T03:50:17.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LrP87Hu/FVTL0LmCSSvDxgOiCeY=', 'Content-Length': '0'} 2024-04-12T03:50:17.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007364887f4cea71f6-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.493 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.493 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.493 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.493 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.493 DEBUG:boto:Method: GET 2024-04-12T03:50:17.493 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.494 DEBUG:boto:Data: 2024-04-12T03:50:17.494 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.494 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.494 DEBUG:boto:Params: {} 2024-04-12T03:50:17.494 DEBUG:boto:Token: None 2024-04-12T03:50:17.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.494 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b396ffd42e9234ae-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.497 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.497 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.497 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.497 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.497 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.498 DEBUG:boto:Method: GET 2024-04-12T03:50:17.498 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:50:17.498 DEBUG:boto:Data: 2024-04-12T03:50:17.498 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.498 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.498 DEBUG:boto:Params: {} 2024-04-12T03:50:17.498 DEBUG:boto:Token: None 2024-04-12T03:50:17.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b659fbd0c1e185a6-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.502 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:50:17.502 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.502 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.503 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.503 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.503 DEBUG:boto:Method: GET 2024-04-12T03:50:17.503 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.503 DEBUG:boto:Data: 2024-04-12T03:50:17.503 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.503 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.503 DEBUG:boto:Params: {} 2024-04-12T03:50:17.503 DEBUG:boto:Token: None 2024-04-12T03:50:17.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w= 2024-04-12T03:50:17.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w=', 'Content-Length': '0'} 2024-04-12T03:50:17.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc18d8f69637ea87-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.506 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.506 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.506 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.506 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.506 DEBUG:boto:Method: GET 2024-04-12T03:50:17.506 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.506 DEBUG:boto:Data: 2024-04-12T03:50:17.506 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.507 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.507 DEBUG:boto:Params: {} 2024-04-12T03:50:17.507 DEBUG:boto:Token: None 2024-04-12T03:50:17.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w= 2024-04-12T03:50:17.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w=', 'Content-Length': '0'} 2024-04-12T03:50:17.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04177e8534b12a1-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.509 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.509 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.510 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.510 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.511 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.511 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.511 DEBUG:boto:Data: 2024-04-12T03:50:17.511 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.511 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.511 DEBUG:boto:Params: {} 2024-04-12T03:50:17.511 DEBUG:boto:Token: None 2024-04-12T03:50:17.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGEJAuKlUBVJ0VEHOUTtVUrliWM= 2024-04-12T03:50:17.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGEJAuKlUBVJ0VEHOUTtVUrliWM=', 'Content-Length': '0'} 2024-04-12T03:50:17.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68a91470c680b8a-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.514 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.514 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.514 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.514 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.514 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.514 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.514 DEBUG:boto:Data: 2024-04-12T03:50:17.514 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.514 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.515 DEBUG:boto:Params: {} 2024-04-12T03:50:17.515 DEBUG:boto:Token: None 2024-04-12T03:50:17.515 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGEJAuKlUBVJ0VEHOUTtVUrliWM= 2024-04-12T03:50:17.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGEJAuKlUBVJ0VEHOUTtVUrliWM=', 'Content-Length': '0'} 2024-04-12T03:50:17.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce05f25e18c18cfc-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:50:17.517 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.517 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.517 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.517 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.517 DEBUG:boto:Method: GET 2024-04-12T03:50:17.517 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.518 DEBUG:boto:Data: 2024-04-12T03:50:17.518 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.518 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.518 DEBUG:boto:Params: {} 2024-04-12T03:50:17.518 DEBUG:boto:Token: None 2024-04-12T03:50:17.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.518 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w= 2024-04-12T03:50:17.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w=', 'Content-Length': '0'} 2024-04-12T03:50:17.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4954efa9a96d0ec-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.521 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.521 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.521 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.521 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.521 DEBUG:boto:Method: GET 2024-04-12T03:50:17.521 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.522 DEBUG:boto:Data: 2024-04-12T03:50:17.522 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.522 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.522 DEBUG:boto:Params: {} 2024-04-12T03:50:17.522 DEBUG:boto:Token: None 2024-04-12T03:50:17.522 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:50:17.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w= 2024-04-12T03:50:17.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yFEvwZrKJaY745OYCdzx7Yr78w=', 'Content-Length': '0'} 2024-04-12T03:50:17.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e8dc804a268940-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.524 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.524 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.525 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.525 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.525 DEBUG:boto:Method: GET 2024-04-12T03:50:17.525 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.525 DEBUG:boto:Data: 2024-04-12T03:50:17.525 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.525 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.525 DEBUG:boto:Params: {} 2024-04-12T03:50:17.525 DEBUG:boto:Token: None 2024-04-12T03:50:17.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.526 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7af6c73fd3812d8-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.528 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.528 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.528 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.528 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.528 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.529 DEBUG:boto:Method: GET 2024-04-12T03:50:17.529 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.529 DEBUG:boto:Data: 2024-04-12T03:50:17.529 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.529 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.529 DEBUG:boto:Params: {} 2024-04-12T03:50:17.529 DEBUG:boto:Token: None 2024-04-12T03:50:17.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f613800b1c2c311-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.533 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.533 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.533 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.533 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.533 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:50:17.533 DEBUG:boto:Data: 2024-04-12T03:50:17.533 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.534 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.534 DEBUG:boto:Params: {} 2024-04-12T03:50:17.534 DEBUG:boto:Token: None 2024-04-12T03:50:17.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68 2024-04-12T03:50:17.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BhVllv93JTvWG/hocBAPtt41kow= 2024-04-12T03:50:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BhVllv93JTvWG/hocBAPtt41kow=', 'Content-Length': '0'} 2024-04-12T03:50:17.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f039f1ea178e742f-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.536 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.536 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.536 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.536 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:50:17.536 DEBUG:boto:Data: 2024-04-12T03:50:17.536 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.536 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.537 DEBUG:boto:Params: {} 2024-04-12T03:50:17.537 DEBUG:boto:Token: None 2024-04-12T03:50:17.537 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68 2024-04-12T03:50:17.537 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BhVllv93JTvWG/hocBAPtt41kow= 2024-04-12T03:50:17.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BhVllv93JTvWG/hocBAPtt41kow=', 'Content-Length': '0'} 2024-04-12T03:50:17.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a182a70e0b4326e7-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:50:17.539 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.539 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.539 DEBUG:boto:Method: GET 2024-04-12T03:50:17.539 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:50:17.539 DEBUG:boto:Data: 2024-04-12T03:50:17.539 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.539 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.539 DEBUG:boto:Params: {} 2024-04-12T03:50:17.540 DEBUG:boto:Token: None 2024-04-12T03:50:17.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68 2024-04-12T03:50:17.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qZ95P9xS7h6at0WtJbaUvQSiPPg= 2024-04-12T03:50:17.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qZ95P9xS7h6at0WtJbaUvQSiPPg=', 'Content-Length': '0'} 2024-04-12T03:50:17.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001df0a4a54bd0a8b3-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.542 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:50:17.542 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:50:17.542 DEBUG:boto:Method: GET 2024-04-12T03:50:17.542 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:50:17.542 DEBUG:boto:Data: 2024-04-12T03:50:17.542 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.543 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.543 DEBUG:boto:Params: {} 2024-04-12T03:50:17.543 DEBUG:boto:Token: None 2024-04-12T03:50:17.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj68 2024-04-12T03:50:17.543 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qZ95P9xS7h6at0WtJbaUvQSiPPg= 2024-04-12T03:50:17.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qZ95P9xS7h6at0WtJbaUvQSiPPg=', 'Content-Length': '0'} 2024-04-12T03:50:17.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d2ce527e83f1bbb-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.545 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.545 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.545 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.545 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.545 DEBUG:boto:Method: GET 2024-04-12T03:50:17.545 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.545 DEBUG:boto:Data: 2024-04-12T03:50:17.545 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.546 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.546 DEBUG:boto:Params: {} 2024-04-12T03:50:17.546 DEBUG:boto:Token: None 2024-04-12T03:50:17.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.546 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086eae5433339e44c-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.548 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.549 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.549 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.549 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.549 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.549 DEBUG:boto:Method: GET 2024-04-12T03:50:17.549 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:50:17.549 DEBUG:boto:Data: 2024-04-12T03:50:17.549 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.549 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.549 DEBUG:boto:Params: {} 2024-04-12T03:50:17.550 DEBUG:boto:Token: None 2024-04-12T03:50:17.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.550 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea4837112d11605c-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.553 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:50:17.554 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.554 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.554 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.554 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.554 DEBUG:boto:Method: GET 2024-04-12T03:50:17.554 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.554 DEBUG:boto:Data: 2024-04-12T03:50:17.554 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.555 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.555 DEBUG:boto:Params: {} 2024-04-12T03:50:17.555 DEBUG:boto:Token: None 2024-04-12T03:50:17.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU= 2024-04-12T03:50:17.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU=', 'Content-Length': '0'} 2024-04-12T03:50:17.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b383c770532ac8e-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.557 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.557 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.557 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.557 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.557 DEBUG:boto:Method: GET 2024-04-12T03:50:17.557 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.557 DEBUG:boto:Data: 2024-04-12T03:50:17.557 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.558 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.558 DEBUG:boto:Params: {} 2024-04-12T03:50:17.558 DEBUG:boto:Token: None 2024-04-12T03:50:17.558 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.558 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU= 2024-04-12T03:50:17.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU=', 'Content-Length': '0'} 2024-04-12T03:50:17.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140764e612c532ad-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.560 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.560 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.560 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.560 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.560 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.560 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.560 DEBUG:boto:Data: 2024-04-12T03:50:17.561 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.561 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.561 DEBUG:boto:Params: {} 2024-04-12T03:50:17.561 DEBUG:boto:Token: None 2024-04-12T03:50:17.561 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SJ6ZCQXPvofEfaN9quMCg2l+LLI= 2024-04-12T03:50:17.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SJ6ZCQXPvofEfaN9quMCg2l+LLI=', 'Content-Length': '0'} 2024-04-12T03:50:17.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020482b63c31119ab-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.563 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.563 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.563 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.563 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.563 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.563 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.564 DEBUG:boto:Data: 2024-04-12T03:50:17.564 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.564 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.564 DEBUG:boto:Params: {} 2024-04-12T03:50:17.564 DEBUG:boto:Token: None 2024-04-12T03:50:17.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SJ6ZCQXPvofEfaN9quMCg2l+LLI= 2024-04-12T03:50:17.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SJ6ZCQXPvofEfaN9quMCg2l+LLI=', 'Content-Length': '0'} 2024-04-12T03:50:17.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a63e5269f44d4e-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:50:17.566 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.566 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.567 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.567 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.567 DEBUG:boto:Method: GET 2024-04-12T03:50:17.567 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.567 DEBUG:boto:Data: 2024-04-12T03:50:17.567 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.567 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.567 DEBUG:boto:Params: {} 2024-04-12T03:50:17.567 DEBUG:boto:Token: None 2024-04-12T03:50:17.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU= 2024-04-12T03:50:17.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU=', 'Content-Length': '0'} 2024-04-12T03:50:17.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38d6519c19e4ce1-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.569 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.569 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.569 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.570 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.570 DEBUG:boto:Method: GET 2024-04-12T03:50:17.570 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.570 DEBUG:boto:Data: 2024-04-12T03:50:17.570 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.570 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.570 DEBUG:boto:Params: {} 2024-04-12T03:50:17.570 DEBUG:boto:Token: None 2024-04-12T03:50:17.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:50:17.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU= 2024-04-12T03:50:17.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3XrVzv0H6gTTD2rKebBxsPiZtmU=', 'Content-Length': '0'} 2024-04-12T03:50:17.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f7d8ac2f84d9ff8-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.572 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.572 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.573 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.573 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.573 DEBUG:boto:Method: GET 2024-04-12T03:50:17.573 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.573 DEBUG:boto:Data: 2024-04-12T03:50:17.573 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.573 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.573 DEBUG:boto:Params: {} 2024-04-12T03:50:17.574 DEBUG:boto:Token: None 2024-04-12T03:50:17.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.574 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007bec071d99dfee6-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.576 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.577 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.577 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.577 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.577 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.577 DEBUG:boto:Method: GET 2024-04-12T03:50:17.577 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.577 DEBUG:boto:Data: 2024-04-12T03:50:17.577 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.577 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.578 DEBUG:boto:Params: {} 2024-04-12T03:50:17.578 DEBUG:boto:Token: None 2024-04-12T03:50:17.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.578 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a27a0c2b6ba58f9-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.581 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.582 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.582 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.582 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.582 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:50:17.582 DEBUG:boto:Data: 2024-04-12T03:50:17.582 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.582 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.582 DEBUG:boto:Params: {} 2024-04-12T03:50:17.582 DEBUG:boto:Token: None 2024-04-12T03:50:17.583 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69 2024-04-12T03:50:17.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GYQy+CA53kUi6xFzeq4IWeioizk= 2024-04-12T03:50:17.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYQy+CA53kUi6xFzeq4IWeioizk=', 'Content-Length': '0'} 2024-04-12T03:50:17.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab485cbb8a88b9c-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.584 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.584 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.585 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.585 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:50:17.585 DEBUG:boto:Data: 2024-04-12T03:50:17.585 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.585 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.585 DEBUG:boto:Params: {} 2024-04-12T03:50:17.585 DEBUG:boto:Token: None 2024-04-12T03:50:17.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69 2024-04-12T03:50:17.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GYQy+CA53kUi6xFzeq4IWeioizk= 2024-04-12T03:50:17.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYQy+CA53kUi6xFzeq4IWeioizk=', 'Content-Length': '0'} 2024-04-12T03:50:17.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b571982793b84ea-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:50:17.587 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.587 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.588 DEBUG:boto:Method: GET 2024-04-12T03:50:17.588 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:50:17.588 DEBUG:boto:Data: 2024-04-12T03:50:17.588 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.588 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.588 DEBUG:boto:Params: {} 2024-04-12T03:50:17.588 DEBUG:boto:Token: None 2024-04-12T03:50:17.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69 2024-04-12T03:50:17.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2UojrwGpaMcm4EhunblVQEzQrOo= 2024-04-12T03:50:17.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2UojrwGpaMcm4EhunblVQEzQrOo=', 'Content-Length': '0'} 2024-04-12T03:50:17.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3c0aeeacf5e2e9-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.590 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:50:17.590 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:50:17.590 DEBUG:boto:Method: GET 2024-04-12T03:50:17.591 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:50:17.591 DEBUG:boto:Data: 2024-04-12T03:50:17.591 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.591 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.591 DEBUG:boto:Params: {} 2024-04-12T03:50:17.591 DEBUG:boto:Token: None 2024-04-12T03:50:17.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj69 2024-04-12T03:50:17.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2UojrwGpaMcm4EhunblVQEzQrOo= 2024-04-12T03:50:17.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2UojrwGpaMcm4EhunblVQEzQrOo=', 'Content-Length': '0'} 2024-04-12T03:50:17.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f070d1893bdf5b4-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.634 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.634 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.635 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.635 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.635 DEBUG:boto:Method: GET 2024-04-12T03:50:17.635 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.635 DEBUG:boto:Data: 2024-04-12T03:50:17.635 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.635 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.635 DEBUG:boto:Params: {} 2024-04-12T03:50:17.635 DEBUG:boto:Token: None 2024-04-12T03:50:17.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097555776ea6f942b-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.638 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.639 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.639 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.639 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.639 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.639 DEBUG:boto:Method: GET 2024-04-12T03:50:17.639 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:50:17.639 DEBUG:boto:Data: 2024-04-12T03:50:17.639 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.640 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.640 DEBUG:boto:Params: {} 2024-04-12T03:50:17.640 DEBUG:boto:Token: None 2024-04-12T03:50:17.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038ccde0cc0d99451-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.644 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:50:17.645 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.645 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.645 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.645 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.645 DEBUG:boto:Method: GET 2024-04-12T03:50:17.645 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.645 DEBUG:boto:Data: 2024-04-12T03:50:17.645 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.646 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.646 DEBUG:boto:Params: {} 2024-04-12T03:50:17.646 DEBUG:boto:Token: None 2024-04-12T03:50:17.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.646 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg= 2024-04-12T03:50:17.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daf06ea85224ef8-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.690 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.690 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.691 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.691 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.691 DEBUG:boto:Method: GET 2024-04-12T03:50:17.691 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.691 DEBUG:boto:Data: 2024-04-12T03:50:17.691 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.691 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.691 DEBUG:boto:Params: {} 2024-04-12T03:50:17.691 DEBUG:boto:Token: None 2024-04-12T03:50:17.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg= 2024-04-12T03:50:17.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2498d61ccbb916-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.694 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.694 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.694 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.694 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.694 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.694 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.694 DEBUG:boto:Data: 2024-04-12T03:50:17.694 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.695 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.695 DEBUG:boto:Params: {} 2024-04-12T03:50:17.695 DEBUG:boto:Token: None 2024-04-12T03:50:17.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IyF0WncE9mFJDJB0yd4JyJE5VkY= 2024-04-12T03:50:17.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IyF0WncE9mFJDJB0yd4JyJE5VkY=', 'Content-Length': '0'} 2024-04-12T03:50:17.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006766d7d904302208-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.697 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.697 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.697 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.697 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.697 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.697 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.697 DEBUG:boto:Data: 2024-04-12T03:50:17.698 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.698 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.698 DEBUG:boto:Params: {} 2024-04-12T03:50:17.698 DEBUG:boto:Token: None 2024-04-12T03:50:17.698 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IyF0WncE9mFJDJB0yd4JyJE5VkY= 2024-04-12T03:50:17.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IyF0WncE9mFJDJB0yd4JyJE5VkY=', 'Content-Length': '0'} 2024-04-12T03:50:17.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc6b5c0a075d026-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:50:17.700 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.700 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.700 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.700 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.700 DEBUG:boto:Method: GET 2024-04-12T03:50:17.700 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.701 DEBUG:boto:Data: 2024-04-12T03:50:17.701 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.701 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.701 DEBUG:boto:Params: {} 2024-04-12T03:50:17.701 DEBUG:boto:Token: None 2024-04-12T03:50:17.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg= 2024-04-12T03:50:17.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003632878585f537fa-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.703 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.703 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.704 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.704 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.704 DEBUG:boto:Method: GET 2024-04-12T03:50:17.704 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.704 DEBUG:boto:Data: 2024-04-12T03:50:17.704 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.704 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.704 DEBUG:boto:Params: {} 2024-04-12T03:50:17.704 DEBUG:boto:Token: None 2024-04-12T03:50:17.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:50:17.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg= 2024-04-12T03:50:17.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sgb3nPDeDuBaW0NifGzIBm7yNwg=', 'Content-Length': '0'} 2024-04-12T03:50:17.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2b6977d185a4ad-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.706 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.707 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.707 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.707 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.707 DEBUG:boto:Method: GET 2024-04-12T03:50:17.707 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.707 DEBUG:boto:Data: 2024-04-12T03:50:17.707 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.707 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.707 DEBUG:boto:Params: {} 2024-04-12T03:50:17.708 DEBUG:boto:Token: None 2024-04-12T03:50:17.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff9047d4b4a4678-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.710 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.710 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.710 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.711 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.711 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.711 DEBUG:boto:Method: GET 2024-04-12T03:50:17.711 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.711 DEBUG:boto:Data: 2024-04-12T03:50:17.711 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.711 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.711 DEBUG:boto:Params: {} 2024-04-12T03:50:17.711 DEBUG:boto:Token: None 2024-04-12T03:50:17.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a08e77b75086ebc-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.715 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.715 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.716 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.716 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.716 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:50:17.716 DEBUG:boto:Data: 2024-04-12T03:50:17.716 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.716 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.716 DEBUG:boto:Params: {} 2024-04-12T03:50:17.716 DEBUG:boto:Token: None 2024-04-12T03:50:17.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76 2024-04-12T03:50:17.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:45WOGxuqlp+69ko3E4RgJEOozbk= 2024-04-12T03:50:17.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:45WOGxuqlp+69ko3E4RgJEOozbk=', 'Content-Length': '0'} 2024-04-12T03:50:17.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b554d56ed3c04646-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.718 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.718 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.718 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.719 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:50:17.719 DEBUG:boto:Data: 2024-04-12T03:50:17.719 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.719 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.719 DEBUG:boto:Params: {} 2024-04-12T03:50:17.719 DEBUG:boto:Token: None 2024-04-12T03:50:17.719 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76 2024-04-12T03:50:17.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:45WOGxuqlp+69ko3E4RgJEOozbk= 2024-04-12T03:50:17.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:45WOGxuqlp+69ko3E4RgJEOozbk=', 'Content-Length': '0'} 2024-04-12T03:50:17.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83043e690b5f1f5-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:50:17.721 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.721 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.721 DEBUG:boto:Method: GET 2024-04-12T03:50:17.722 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:50:17.722 DEBUG:boto:Data: 2024-04-12T03:50:17.722 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.722 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.722 DEBUG:boto:Params: {} 2024-04-12T03:50:17.722 DEBUG:boto:Token: None 2024-04-12T03:50:17.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76 2024-04-12T03:50:17.722 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ISsHLA7mMD0WEYgkYiL/AWyeIg4= 2024-04-12T03:50:17.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ISsHLA7mMD0WEYgkYiL/AWyeIg4=', 'Content-Length': '0'} 2024-04-12T03:50:17.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45e2571906351a1-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.724 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:50:17.724 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:50:17.724 DEBUG:boto:Method: GET 2024-04-12T03:50:17.725 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:50:17.725 DEBUG:boto:Data: 2024-04-12T03:50:17.725 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.725 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.725 DEBUG:boto:Params: {} 2024-04-12T03:50:17.725 DEBUG:boto:Token: None 2024-04-12T03:50:17.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj76 2024-04-12T03:50:17.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ISsHLA7mMD0WEYgkYiL/AWyeIg4= 2024-04-12T03:50:17.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ISsHLA7mMD0WEYgkYiL/AWyeIg4=', 'Content-Length': '0'} 2024-04-12T03:50:17.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000039a3c46f76e06de-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.770 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.771 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.771 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.771 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.771 DEBUG:boto:Method: GET 2024-04-12T03:50:17.771 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.771 DEBUG:boto:Data: 2024-04-12T03:50:17.771 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.771 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.771 DEBUG:boto:Params: {} 2024-04-12T03:50:17.771 DEBUG:boto:Token: None 2024-04-12T03:50:17.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.772 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5024bc05f70e289-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.774 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.774 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.775 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.775 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.775 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.775 DEBUG:boto:Method: GET 2024-04-12T03:50:17.775 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:50:17.775 DEBUG:boto:Data: 2024-04-12T03:50:17.775 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.775 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.775 DEBUG:boto:Params: {} 2024-04-12T03:50:17.775 DEBUG:boto:Token: None 2024-04-12T03:50:17.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003adff5389f39cd2a-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.779 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:50:17.780 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.780 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.780 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.780 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.780 DEBUG:boto:Method: GET 2024-04-12T03:50:17.780 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.780 DEBUG:boto:Data: 2024-04-12T03:50:17.780 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.780 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.781 DEBUG:boto:Params: {} 2024-04-12T03:50:17.781 DEBUG:boto:Token: None 2024-04-12T03:50:17.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.781 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs= 2024-04-12T03:50:17.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs=', 'Content-Length': '0'} 2024-04-12T03:50:17.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1618aba50329e9c-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.783 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.783 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.783 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.783 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.783 DEBUG:boto:Method: GET 2024-04-12T03:50:17.783 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.783 DEBUG:boto:Data: 2024-04-12T03:50:17.783 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.784 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.784 DEBUG:boto:Params: {} 2024-04-12T03:50:17.784 DEBUG:boto:Token: None 2024-04-12T03:50:17.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs= 2024-04-12T03:50:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs=', 'Content-Length': '0'} 2024-04-12T03:50:17.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9282803321878b0-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.786 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.786 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.786 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.786 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.786 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.786 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.786 DEBUG:boto:Data: 2024-04-12T03:50:17.787 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.787 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.787 DEBUG:boto:Params: {} 2024-04-12T03:50:17.787 DEBUG:boto:Token: None 2024-04-12T03:50:17.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.787 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4h/6R68qCJzCNKMU0FRbTqoTeFQ= 2024-04-12T03:50:17.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4h/6R68qCJzCNKMU0FRbTqoTeFQ=', 'Content-Length': '0'} 2024-04-12T03:50:17.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b491b6be42a514f9-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.789 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.789 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.789 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.789 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.789 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.789 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.789 DEBUG:boto:Data: 2024-04-12T03:50:17.789 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.790 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.790 DEBUG:boto:Params: {} 2024-04-12T03:50:17.790 DEBUG:boto:Token: None 2024-04-12T03:50:17.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4h/6R68qCJzCNKMU0FRbTqoTeFQ= 2024-04-12T03:50:17.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4h/6R68qCJzCNKMU0FRbTqoTeFQ=', 'Content-Length': '0'} 2024-04-12T03:50:17.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f089cd8c0ec8ca-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:50:17.792 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.792 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.792 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.792 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.792 DEBUG:boto:Method: GET 2024-04-12T03:50:17.793 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.793 DEBUG:boto:Data: 2024-04-12T03:50:17.793 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.793 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.793 DEBUG:boto:Params: {} 2024-04-12T03:50:17.793 DEBUG:boto:Token: None 2024-04-12T03:50:17.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs= 2024-04-12T03:50:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs=', 'Content-Length': '0'} 2024-04-12T03:50:17.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6ccd8bfaa56f9e-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.838 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.839 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.839 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.839 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.839 DEBUG:boto:Method: GET 2024-04-12T03:50:17.839 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.839 DEBUG:boto:Data: 2024-04-12T03:50:17.839 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.839 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.839 DEBUG:boto:Params: {} 2024-04-12T03:50:17.840 DEBUG:boto:Token: None 2024-04-12T03:50:17.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:50:17.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs= 2024-04-12T03:50:17.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sm7OrzhQ6tFLwFDkJMqVpOE8gEs=', 'Content-Length': '0'} 2024-04-12T03:50:17.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038caa93a9f2ed99b-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.842 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.842 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.842 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.842 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.843 DEBUG:boto:Method: GET 2024-04-12T03:50:17.843 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.843 DEBUG:boto:Data: 2024-04-12T03:50:17.843 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.843 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.843 DEBUG:boto:Params: {} 2024-04-12T03:50:17.843 DEBUG:boto:Token: None 2024-04-12T03:50:17.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf0379f49e711f0-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.846 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.846 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.846 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.846 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.846 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.846 DEBUG:boto:Method: GET 2024-04-12T03:50:17.846 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.847 DEBUG:boto:Data: 2024-04-12T03:50:17.847 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.847 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.847 DEBUG:boto:Params: {} 2024-04-12T03:50:17.847 DEBUG:boto:Token: None 2024-04-12T03:50:17.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd8261aa7fe46cc-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.850 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.851 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.851 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.851 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.851 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:50:17.851 DEBUG:boto:Data: 2024-04-12T03:50:17.851 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.851 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.851 DEBUG:boto:Params: {} 2024-04-12T03:50:17.852 DEBUG:boto:Token: None 2024-04-12T03:50:17.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85 2024-04-12T03:50:17.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vtKwAGSLhblU3bM1kadLYO4aSTc= 2024-04-12T03:50:17.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vtKwAGSLhblU3bM1kadLYO4aSTc=', 'Content-Length': '0'} 2024-04-12T03:50:17.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b31eceee8ae2e39-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.853 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.854 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.854 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.854 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:50:17.854 DEBUG:boto:Data: 2024-04-12T03:50:17.854 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.854 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.854 DEBUG:boto:Params: {} 2024-04-12T03:50:17.854 DEBUG:boto:Token: None 2024-04-12T03:50:17.854 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85 2024-04-12T03:50:17.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vtKwAGSLhblU3bM1kadLYO4aSTc= 2024-04-12T03:50:17.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vtKwAGSLhblU3bM1kadLYO4aSTc=', 'Content-Length': '0'} 2024-04-12T03:50:17.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39e0ac1cb342e1a-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:50:17.856 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.856 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.857 DEBUG:boto:Method: GET 2024-04-12T03:50:17.857 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:50:17.857 DEBUG:boto:Data: 2024-04-12T03:50:17.857 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.857 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.857 DEBUG:boto:Params: {} 2024-04-12T03:50:17.857 DEBUG:boto:Token: None 2024-04-12T03:50:17.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85 2024-04-12T03:50:17.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wczvYN1qtIIQDnOz7o6uDa+Wapk= 2024-04-12T03:50:17.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wczvYN1qtIIQDnOz7o6uDa+Wapk=', 'Content-Length': '0'} 2024-04-12T03:50:17.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cc3e8f9b0fea33-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.859 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:50:17.859 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:50:17.859 DEBUG:boto:Method: GET 2024-04-12T03:50:17.859 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:50:17.859 DEBUG:boto:Data: 2024-04-12T03:50:17.859 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.860 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.860 DEBUG:boto:Params: {} 2024-04-12T03:50:17.860 DEBUG:boto:Token: None 2024-04-12T03:50:17.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj85 2024-04-12T03:50:17.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wczvYN1qtIIQDnOz7o6uDa+Wapk= 2024-04-12T03:50:17.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wczvYN1qtIIQDnOz7o6uDa+Wapk=', 'Content-Length': '0'} 2024-04-12T03:50:17.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004040028f53d132d5-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.902 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.903 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.903 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.903 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.903 DEBUG:boto:Method: GET 2024-04-12T03:50:17.903 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.903 DEBUG:boto:Data: 2024-04-12T03:50:17.903 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.903 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.903 DEBUG:boto:Params: {} 2024-04-12T03:50:17.904 DEBUG:boto:Token: None 2024-04-12T03:50:17.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003edd246a558fca10-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.906 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.907 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.907 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.907 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.907 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.907 DEBUG:boto:Method: GET 2024-04-12T03:50:17.907 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:50:17.907 DEBUG:boto:Data: 2024-04-12T03:50:17.907 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.908 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.908 DEBUG:boto:Params: {} 2024-04-12T03:50:17.908 DEBUG:boto:Token: None 2024-04-12T03:50:17.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f08e4e825b69d32-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.911 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:50:17.912 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.912 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.912 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.912 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.912 DEBUG:boto:Method: GET 2024-04-12T03:50:17.912 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.912 DEBUG:boto:Data: 2024-04-12T03:50:17.912 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.913 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.913 DEBUG:boto:Params: {} 2024-04-12T03:50:17.913 DEBUG:boto:Token: None 2024-04-12T03:50:17.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.913 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY= 2024-04-12T03:50:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY=', 'Content-Length': '0'} 2024-04-12T03:50:17.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075f0d634553c89e5-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.915 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.915 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.915 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.915 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.915 DEBUG:boto:Method: GET 2024-04-12T03:50:17.915 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.915 DEBUG:boto:Data: 2024-04-12T03:50:17.916 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.916 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.916 DEBUG:boto:Params: {} 2024-04-12T03:50:17.916 DEBUG:boto:Token: None 2024-04-12T03:50:17.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY= 2024-04-12T03:50:17.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY=', 'Content-Length': '0'} 2024-04-12T03:50:17.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac990851b380da49-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.918 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.918 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.918 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.919 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.919 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.919 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.919 DEBUG:boto:Data: 2024-04-12T03:50:17.919 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.919 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.919 DEBUG:boto:Params: {} 2024-04-12T03:50:17.919 DEBUG:boto:Token: None 2024-04-12T03:50:17.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2PlSRXKN2Dx30Yq+DBNMPrh0ko= 2024-04-12T03:50:17.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2PlSRXKN2Dx30Yq+DBNMPrh0ko=', 'Content-Length': '0'} 2024-04-12T03:50:17.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfe82b53e7725f4-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.921 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.922 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.922 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.922 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.922 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.922 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.922 DEBUG:boto:Data: 2024-04-12T03:50:17.922 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.922 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.922 DEBUG:boto:Params: {} 2024-04-12T03:50:17.922 DEBUG:boto:Token: None 2024-04-12T03:50:17.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L2PlSRXKN2Dx30Yq+DBNMPrh0ko= 2024-04-12T03:50:17.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L2PlSRXKN2Dx30Yq+DBNMPrh0ko=', 'Content-Length': '0'} 2024-04-12T03:50:17.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2d5b9d1b00e23d-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:50:17.925 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.925 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.925 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.925 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.925 DEBUG:boto:Method: GET 2024-04-12T03:50:17.925 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.925 DEBUG:boto:Data: 2024-04-12T03:50:17.925 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.925 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.925 DEBUG:boto:Params: {} 2024-04-12T03:50:17.925 DEBUG:boto:Token: None 2024-04-12T03:50:17.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.926 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY= 2024-04-12T03:50:17.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY=', 'Content-Length': '0'} 2024-04-12T03:50:17.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c086cf4fa310f173-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.927 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.928 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.928 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.928 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.928 DEBUG:boto:Method: GET 2024-04-12T03:50:17.928 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.928 DEBUG:boto:Data: 2024-04-12T03:50:17.928 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.928 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.928 DEBUG:boto:Params: {} 2024-04-12T03:50:17.929 DEBUG:boto:Token: None 2024-04-12T03:50:17.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:50:17.929 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY= 2024-04-12T03:50:17.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+5pOrtLzeWz4+FmY9h5Jg5UYdAY=', 'Content-Length': '0'} 2024-04-12T03:50:17.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004030c98085384915-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.932 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.932 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.932 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.932 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.933 DEBUG:boto:Method: GET 2024-04-12T03:50:17.933 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.933 DEBUG:boto:Data: 2024-04-12T03:50:17.933 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.933 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.933 DEBUG:boto:Params: {} 2024-04-12T03:50:17.933 DEBUG:boto:Token: None 2024-04-12T03:50:17.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005edc257492cf1ab8-006618af79-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.936 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.936 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.936 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.936 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.936 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.937 DEBUG:boto:Method: GET 2024-04-12T03:50:17.937 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.937 DEBUG:boto:Data: 2024-04-12T03:50:17.937 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.937 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.937 DEBUG:boto:Params: {} 2024-04-12T03:50:17.937 DEBUG:boto:Token: None 2024-04-12T03:50:17.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfeabe3919353b2b-006618af79-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.982 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.983 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.983 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.983 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.983 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:50:17.983 DEBUG:boto:Data: 2024-04-12T03:50:17.983 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.984 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.984 DEBUG:boto:Params: {} 2024-04-12T03:50:17.984 DEBUG:boto:Token: None 2024-04-12T03:50:17.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87 2024-04-12T03:50:17.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AIIkfxC0PaldGJ2EWq4C3UirAno= 2024-04-12T03:50:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AIIkfxC0PaldGJ2EWq4C3UirAno=', 'Content-Length': '0'} 2024-04-12T03:50:17.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca4a159e62a0dd2-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.986 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.986 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.986 DEBUG:boto:Method: HEAD 2024-04-12T03:50:17.986 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:50:17.986 DEBUG:boto:Data: 2024-04-12T03:50:17.986 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.987 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.987 DEBUG:boto:Params: {} 2024-04-12T03:50:17.987 DEBUG:boto:Token: None 2024-04-12T03:50:17.987 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87 2024-04-12T03:50:17.987 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AIIkfxC0PaldGJ2EWq4C3UirAno= 2024-04-12T03:50:17.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AIIkfxC0PaldGJ2EWq4C3UirAno=', 'Content-Length': '0'} 2024-04-12T03:50:17.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078da964b4a19d888-006618af79-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:50:17.989 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.989 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.989 DEBUG:boto:Method: GET 2024-04-12T03:50:17.990 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:50:17.990 DEBUG:boto:Data: 2024-04-12T03:50:17.990 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.990 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.990 DEBUG:boto:Params: {} 2024-04-12T03:50:17.990 DEBUG:boto:Token: None 2024-04-12T03:50:17.990 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87 2024-04-12T03:50:17.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GPH0gXlPM9qkqzAnk2WRB5Wt6Cw= 2024-04-12T03:50:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GPH0gXlPM9qkqzAnk2WRB5Wt6Cw=', 'Content-Length': '0'} 2024-04-12T03:50:17.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084221025201decc5-006618af79-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.992 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:50:17.992 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:50:17.992 DEBUG:boto:Method: GET 2024-04-12T03:50:17.993 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:50:17.993 DEBUG:boto:Data: 2024-04-12T03:50:17.993 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:17.993 DEBUG:boto:Port: 8000 2024-04-12T03:50:17.993 DEBUG:boto:Params: {} 2024-04-12T03:50:17.993 DEBUG:boto:Token: None 2024-04-12T03:50:17.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/obj87 2024-04-12T03:50:17.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GPH0gXlPM9qkqzAnk2WRB5Wt6Cw= 2024-04-12T03:50:17.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GPH0gXlPM9qkqzAnk2WRB5Wt6Cw=', 'Content-Length': '0'} 2024-04-12T03:50:17.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d9ae029cf99b19-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.995 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.995 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.995 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.996 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.996 DEBUG:boto:Method: GET 2024-04-12T03:50:17.996 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.996 DEBUG:boto:Data: 2024-04-12T03:50:17.996 DEBUG:boto:Headers: {} 2024-04-12T03:50:17.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:17.996 DEBUG:boto:Port: 8001 2024-04-12T03:50:17.996 DEBUG:boto:Params: {} 2024-04-12T03:50:17.996 DEBUG:boto:Token: None 2024-04-12T03:50:17.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:17 GMT /dgbpxg-61/ 2024-04-12T03:50:17.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE= 2024-04-12T03:50:17.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmOHqPXnxFbZkOekFEFFfucdMWE=', 'Content-Length': '0'} 2024-04-12T03:50:17.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a9e05c7aeb8fba-006618af7a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:17.999 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:17.999 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:17.999 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:17.999 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:17.999 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:50:18.000 DEBUG:boto:Method: GET 2024-04-12T03:50:18.000 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:50:18.000 DEBUG:boto:Data: 2024-04-12T03:50:18.000 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.000 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.000 DEBUG:boto:Params: {} 2024-04-12T03:50:18.000 DEBUG:boto:Token: None 2024-04-12T03:50:18.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac6fc9ad0cb2294-006618af7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.004 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:50:18.004 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.004 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.004 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.005 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.005 DEBUG:boto:Method: GET 2024-04-12T03:50:18.005 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.005 DEBUG:boto:Data: 2024-04-12T03:50:18.005 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.005 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.005 DEBUG:boto:Params: {} 2024-04-12T03:50:18.005 DEBUG:boto:Token: None 2024-04-12T03:50:18.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w= 2024-04-12T03:50:18.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w=', 'Content-Length': '0'} 2024-04-12T03:50:18.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ba0b86c7af7b09-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.050 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.050 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.050 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.051 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.051 DEBUG:boto:Method: GET 2024-04-12T03:50:18.051 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.051 DEBUG:boto:Data: 2024-04-12T03:50:18.051 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.051 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.051 DEBUG:boto:Params: {} 2024-04-12T03:50:18.051 DEBUG:boto:Token: None 2024-04-12T03:50:18.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w= 2024-04-12T03:50:18.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w=', 'Content-Length': '0'} 2024-04-12T03:50:18.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae86f59082aead7a-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.053 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.054 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.054 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.054 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.054 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.054 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.054 DEBUG:boto:Data: 2024-04-12T03:50:18.054 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.054 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.054 DEBUG:boto:Params: {} 2024-04-12T03:50:18.055 DEBUG:boto:Token: None 2024-04-12T03:50:18.055 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cfkFh1L1iqbeQaelRw1+2u2xHtg= 2024-04-12T03:50:18.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cfkFh1L1iqbeQaelRw1+2u2xHtg=', 'Content-Length': '0'} 2024-04-12T03:50:18.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000434cc752be3220f9-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.057 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.057 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.057 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.057 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.057 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.058 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.058 DEBUG:boto:Data: 2024-04-12T03:50:18.058 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.058 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.058 DEBUG:boto:Params: {} 2024-04-12T03:50:18.058 DEBUG:boto:Token: None 2024-04-12T03:50:18.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cfkFh1L1iqbeQaelRw1+2u2xHtg= 2024-04-12T03:50:18.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cfkFh1L1iqbeQaelRw1+2u2xHtg=', 'Content-Length': '0'} 2024-04-12T03:50:18.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036fcd5f346922e96-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:50:18.060 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.060 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.060 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.061 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.061 DEBUG:boto:Method: GET 2024-04-12T03:50:18.061 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.061 DEBUG:boto:Data: 2024-04-12T03:50:18.061 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.061 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.061 DEBUG:boto:Params: {} 2024-04-12T03:50:18.061 DEBUG:boto:Token: None 2024-04-12T03:50:18.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w= 2024-04-12T03:50:18.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w=', 'Content-Length': '0'} 2024-04-12T03:50:18.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7abfb837989ab5-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.063 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.063 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.063 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.064 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.064 DEBUG:boto:Method: GET 2024-04-12T03:50:18.064 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.064 DEBUG:boto:Data: 2024-04-12T03:50:18.064 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.064 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.064 DEBUG:boto:Params: {} 2024-04-12T03:50:18.064 DEBUG:boto:Token: None 2024-04-12T03:50:18.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:50:18.065 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w= 2024-04-12T03:50:18.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yWDINTAnlNm9QZIPSVmiufkkz0w=', 'Content-Length': '0'} 2024-04-12T03:50:18.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb08b969676a5837-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.066 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.066 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.066 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.067 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.067 DEBUG:boto:Method: GET 2024-04-12T03:50:18.067 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.067 DEBUG:boto:Data: 2024-04-12T03:50:18.067 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.067 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.067 DEBUG:boto:Params: {} 2024-04-12T03:50:18.067 DEBUG:boto:Token: None 2024-04-12T03:50:18.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3e798ba841d61ba-006618af7a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.070 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.070 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.070 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.070 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.071 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.071 DEBUG:boto:Method: GET 2024-04-12T03:50:18.071 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.071 DEBUG:boto:Data: 2024-04-12T03:50:18.071 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.071 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.071 DEBUG:boto:Params: {} 2024-04-12T03:50:18.071 DEBUG:boto:Token: None 2024-04-12T03:50:18.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c0a8242bbc527f-006618af7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.075 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.075 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.075 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.075 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.075 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:50:18.075 DEBUG:boto:Data: 2024-04-12T03:50:18.076 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.076 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.076 DEBUG:boto:Params: {} 2024-04-12T03:50:18.076 DEBUG:boto:Token: None 2024-04-12T03:50:18.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92 2024-04-12T03:50:18.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fwv0Fsw2ZOESiljdSXV0FbiGdzI= 2024-04-12T03:50:18.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fwv0Fsw2ZOESiljdSXV0FbiGdzI=', 'Content-Length': '0'} 2024-04-12T03:50:18.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b37d3272f4cb1f-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.078 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.078 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.078 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.078 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:50:18.079 DEBUG:boto:Data: 2024-04-12T03:50:18.079 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.079 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.079 DEBUG:boto:Params: {} 2024-04-12T03:50:18.079 DEBUG:boto:Token: None 2024-04-12T03:50:18.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92 2024-04-12T03:50:18.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Fwv0Fsw2ZOESiljdSXV0FbiGdzI= 2024-04-12T03:50:18.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Fwv0Fsw2ZOESiljdSXV0FbiGdzI=', 'Content-Length': '0'} 2024-04-12T03:50:18.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d123ee09d783cc-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:50:18.081 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.081 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.081 DEBUG:boto:Method: GET 2024-04-12T03:50:18.081 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:50:18.081 DEBUG:boto:Data: 2024-04-12T03:50:18.082 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.082 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.082 DEBUG:boto:Params: {} 2024-04-12T03:50:18.082 DEBUG:boto:Token: None 2024-04-12T03:50:18.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92 2024-04-12T03:50:18.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4wQVW//KC+uj1jHYDzRDFRez9do= 2024-04-12T03:50:18.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4wQVW//KC+uj1jHYDzRDFRez9do=', 'Content-Length': '0'} 2024-04-12T03:50:18.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a4e1999d1aefa36-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.084 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:50:18.084 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:50:18.084 DEBUG:boto:Method: GET 2024-04-12T03:50:18.084 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:50:18.084 DEBUG:boto:Data: 2024-04-12T03:50:18.085 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.085 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.085 DEBUG:boto:Params: {} 2024-04-12T03:50:18.085 DEBUG:boto:Token: None 2024-04-12T03:50:18.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj92 2024-04-12T03:50:18.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4wQVW//KC+uj1jHYDzRDFRez9do= 2024-04-12T03:50:18.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4wQVW//KC+uj1jHYDzRDFRez9do=', 'Content-Length': '0'} 2024-04-12T03:50:18.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036070a5138516ac6-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.130 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.131 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.131 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.131 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.131 DEBUG:boto:Method: GET 2024-04-12T03:50:18.131 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.131 DEBUG:boto:Data: 2024-04-12T03:50:18.131 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.131 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.131 DEBUG:boto:Params: {} 2024-04-12T03:50:18.132 DEBUG:boto:Token: None 2024-04-12T03:50:18.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213f8a4620206fa2-006618af7a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.134 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.134 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.135 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.135 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.135 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.135 DEBUG:boto:Method: GET 2024-04-12T03:50:18.135 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:50:18.135 DEBUG:boto:Data: 2024-04-12T03:50:18.135 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.135 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.135 DEBUG:boto:Params: {} 2024-04-12T03:50:18.135 DEBUG:boto:Token: None 2024-04-12T03:50:18.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d84fddb7691afc-006618af7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.139 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:50:18.139 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.140 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.140 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.140 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.140 DEBUG:boto:Method: GET 2024-04-12T03:50:18.140 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.140 DEBUG:boto:Data: 2024-04-12T03:50:18.140 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.140 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.140 DEBUG:boto:Params: {} 2024-04-12T03:50:18.140 DEBUG:boto:Token: None 2024-04-12T03:50:18.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY= 2024-04-12T03:50:18.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY=', 'Content-Length': '0'} 2024-04-12T03:50:18.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659b4400d2af9a1b-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.143 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.143 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.143 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.143 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.143 DEBUG:boto:Method: GET 2024-04-12T03:50:18.143 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.143 DEBUG:boto:Data: 2024-04-12T03:50:18.143 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.143 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.143 DEBUG:boto:Params: {} 2024-04-12T03:50:18.144 DEBUG:boto:Token: None 2024-04-12T03:50:18.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY= 2024-04-12T03:50:18.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY=', 'Content-Length': '0'} 2024-04-12T03:50:18.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f758047c09c08fb0-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.146 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.146 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.146 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.146 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.146 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.146 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.146 DEBUG:boto:Data: 2024-04-12T03:50:18.146 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.147 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.147 DEBUG:boto:Params: {} 2024-04-12T03:50:18.147 DEBUG:boto:Token: None 2024-04-12T03:50:18.147 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bazVJpWKe8z1dPf1hlbdcfBmPWs= 2024-04-12T03:50:18.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bazVJpWKe8z1dPf1hlbdcfBmPWs=', 'Content-Length': '0'} 2024-04-12T03:50:18.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691994df78951b27-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.149 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.149 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.149 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.149 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.149 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.149 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.149 DEBUG:boto:Data: 2024-04-12T03:50:18.149 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.150 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.150 DEBUG:boto:Params: {} 2024-04-12T03:50:18.150 DEBUG:boto:Token: None 2024-04-12T03:50:18.150 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bazVJpWKe8z1dPf1hlbdcfBmPWs= 2024-04-12T03:50:18.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bazVJpWKe8z1dPf1hlbdcfBmPWs=', 'Content-Length': '0'} 2024-04-12T03:50:18.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc2bd04f707a560-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:50:18.152 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.152 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.152 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.153 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.153 DEBUG:boto:Method: GET 2024-04-12T03:50:18.153 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.153 DEBUG:boto:Data: 2024-04-12T03:50:18.153 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.153 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.153 DEBUG:boto:Params: {} 2024-04-12T03:50:18.153 DEBUG:boto:Token: None 2024-04-12T03:50:18.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY= 2024-04-12T03:50:18.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY=', 'Content-Length': '0'} 2024-04-12T03:50:18.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b43dba07b02f849-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.202 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.203 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.203 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.203 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.203 DEBUG:boto:Method: GET 2024-04-12T03:50:18.203 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.203 DEBUG:boto:Data: 2024-04-12T03:50:18.203 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.203 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.203 DEBUG:boto:Params: {} 2024-04-12T03:50:18.204 DEBUG:boto:Token: None 2024-04-12T03:50:18.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:50:18.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY= 2024-04-12T03:50:18.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IN5KBS3EXqHy71GtpXsqzufa0zY=', 'Content-Length': '0'} 2024-04-12T03:50:18.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3ca4ce4fdf19f2-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.206 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.207 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.207 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.207 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.207 DEBUG:boto:Method: GET 2024-04-12T03:50:18.207 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.207 DEBUG:boto:Data: 2024-04-12T03:50:18.207 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.207 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.207 DEBUG:boto:Params: {} 2024-04-12T03:50:18.207 DEBUG:boto:Token: None 2024-04-12T03:50:18.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b033612db9972193-006618af7a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.210 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.210 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.210 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.210 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.211 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.211 DEBUG:boto:Method: GET 2024-04-12T03:50:18.211 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.211 DEBUG:boto:Data: 2024-04-12T03:50:18.211 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.211 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.211 DEBUG:boto:Params: {} 2024-04-12T03:50:18.211 DEBUG:boto:Token: None 2024-04-12T03:50:18.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9fdce1823d7ef25-006618af7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.215 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.215 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.215 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.215 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.215 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:50:18.215 DEBUG:boto:Data: 2024-04-12T03:50:18.215 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.216 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.216 DEBUG:boto:Params: {} 2024-04-12T03:50:18.216 DEBUG:boto:Token: None 2024-04-12T03:50:18.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95 2024-04-12T03:50:18.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H2KbgW5i05W/Bcn6VxU3RPvBqtY= 2024-04-12T03:50:18.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H2KbgW5i05W/Bcn6VxU3RPvBqtY=', 'Content-Length': '0'} 2024-04-12T03:50:18.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989eefe572b630ee-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.218 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.218 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.218 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.218 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:50:18.218 DEBUG:boto:Data: 2024-04-12T03:50:18.218 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.219 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.219 DEBUG:boto:Params: {} 2024-04-12T03:50:18.219 DEBUG:boto:Token: None 2024-04-12T03:50:18.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95 2024-04-12T03:50:18.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H2KbgW5i05W/Bcn6VxU3RPvBqtY= 2024-04-12T03:50:18.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H2KbgW5i05W/Bcn6VxU3RPvBqtY=', 'Content-Length': '0'} 2024-04-12T03:50:18.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083cefb308460c043-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:50:18.221 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.221 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.221 DEBUG:boto:Method: GET 2024-04-12T03:50:18.221 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:50:18.221 DEBUG:boto:Data: 2024-04-12T03:50:18.221 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.222 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.222 DEBUG:boto:Params: {} 2024-04-12T03:50:18.222 DEBUG:boto:Token: None 2024-04-12T03:50:18.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95 2024-04-12T03:50:18.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wOzM0Imm4KWAE1Eu+q/ovym/0Xw= 2024-04-12T03:50:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wOzM0Imm4KWAE1Eu+q/ovym/0Xw=', 'Content-Length': '0'} 2024-04-12T03:50:18.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93b06577f520b1a-006618af7a-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.224 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:50:18.224 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:50:18.224 DEBUG:boto:Method: GET 2024-04-12T03:50:18.224 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:50:18.224 DEBUG:boto:Data: 2024-04-12T03:50:18.225 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.225 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.225 DEBUG:boto:Params: {} 2024-04-12T03:50:18.225 DEBUG:boto:Token: None 2024-04-12T03:50:18.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj95 2024-04-12T03:50:18.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wOzM0Imm4KWAE1Eu+q/ovym/0Xw= 2024-04-12T03:50:18.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wOzM0Imm4KWAE1Eu+q/ovym/0Xw=', 'Content-Length': '0'} 2024-04-12T03:50:18.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c8e453faecc773-006618af7a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.274 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.275 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.275 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.275 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.275 DEBUG:boto:Method: GET 2024-04-12T03:50:18.275 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.275 DEBUG:boto:Data: 2024-04-12T03:50:18.275 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:50:18.275 DEBUG:boto:Port: 8001 2024-04-12T03:50:18.275 DEBUG:boto:Params: {} 2024-04-12T03:50:18.276 DEBUG:boto:Token: None 2024-04-12T03:50:18.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dddbc709988b923-006618af7a-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.278 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.279 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.279 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.279 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.279 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.279 DEBUG:boto:Method: GET 2024-04-12T03:50:18.279 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:50:18.279 DEBUG:boto:Data: 2024-04-12T03:50:18.279 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.280 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.280 DEBUG:boto:Params: {} 2024-04-12T03:50:18.280 DEBUG:boto:Token: None 2024-04-12T03:50:18.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8= 2024-04-12T03:50:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J1ny4v0QkCS0tBWOmTdYdpCvim8=', 'Content-Length': '0'} 2024-04-12T03:50:18.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df4636a17843e4b1-006618af7a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.284 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:18.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:50:18.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:18.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:18.286 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:18.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:18.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:18.386 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:50:18.386 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:50:18.396 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2024-04-12T03:50:18.455 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.455 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.455 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.455 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:18.455 DEBUG:boto:Data: 2024-04-12T03:50:18.455 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.456 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.456 DEBUG:boto:Params: {} 2024-04-12T03:50:18.456 DEBUG:boto:Token: None 2024-04-12T03:50:18.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs= 2024-04-12T03:50:18.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs=', 'Content-Length': '0'} 2024-04-12T03:50:18.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8c85a2c30124537-006618af7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.460 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:50:18.460 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:50:18.460 DEBUG:boto:Method: PUT 2024-04-12T03:50:18.460 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:50:18.460 DEBUG:boto:Data: 2024-04-12T03:50:18.461 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:18.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.461 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.461 DEBUG:boto:Params: {} 2024-04-12T03:50:18.461 DEBUG:boto:Token: None 2024-04-12T03:50:18.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj31 2024-04-12T03:50:18.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7Sf1+qnDig48WSu2983eoGh+/60= 2024-04-12T03:50:18.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7Sf1+qnDig48WSu2983eoGh+/60='} 2024-04-12T03:50:18.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b05594c0c3f17ed-006618af7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.467 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.467 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.467 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.467 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:18.467 DEBUG:boto:Data: 2024-04-12T03:50:18.467 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.467 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.467 DEBUG:boto:Params: {} 2024-04-12T03:50:18.468 DEBUG:boto:Token: None 2024-04-12T03:50:18.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs= 2024-04-12T03:50:18.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs=', 'Content-Length': '0'} 2024-04-12T03:50:18.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9f28b05bd43c231-006618af7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.471 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:50:18.471 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:50:18.471 DEBUG:boto:Method: PUT 2024-04-12T03:50:18.471 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:50:18.471 DEBUG:boto:Data: 2024-04-12T03:50:18.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:18.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.472 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.472 DEBUG:boto:Params: {} 2024-04-12T03:50:18.472 DEBUG:boto:Token: None 2024-04-12T03:50:18.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj62 2024-04-12T03:50:18.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YtAFzOHGB1aBm/AcFmIgHjEu3CM= 2024-04-12T03:50:18.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YtAFzOHGB1aBm/AcFmIgHjEu3CM='} 2024-04-12T03:50:18.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fcd745a13eeb667-006618af7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.516 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.516 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.516 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.516 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:18.516 DEBUG:boto:Data: 2024-04-12T03:50:18.516 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.516 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.517 DEBUG:boto:Params: {} 2024-04-12T03:50:18.517 DEBUG:boto:Token: None 2024-04-12T03:50:18.517 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.517 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs= 2024-04-12T03:50:18.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs=', 'Content-Length': '0'} 2024-04-12T03:50:18.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085df270ae37dfd64-006618af7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.521 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:50:18.521 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:50:18.521 DEBUG:boto:Method: PUT 2024-04-12T03:50:18.521 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:50:18.521 DEBUG:boto:Data: 2024-04-12T03:50:18.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:18.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.522 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.522 DEBUG:boto:Params: {} 2024-04-12T03:50:18.522 DEBUG:boto:Token: None 2024-04-12T03:50:18.522 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj93 2024-04-12T03:50:18.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G/U4SApAqwhT+PE3Uia520zrRy8= 2024-04-12T03:50:18.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G/U4SApAqwhT+PE3Uia520zrRy8='} 2024-04-12T03:50:18.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dea004b3bcacece0-006618af7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.530 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.531 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.531 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.531 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:18.531 DEBUG:boto:Data: 2024-04-12T03:50:18.531 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.531 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.531 DEBUG:boto:Params: {} 2024-04-12T03:50:18.531 DEBUG:boto:Token: None 2024-04-12T03:50:18.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs= 2024-04-12T03:50:18.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs=', 'Content-Length': '0'} 2024-04-12T03:50:18.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039dd87b3967ad309-006618af7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.535 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:50:18.535 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:50:18.535 DEBUG:boto:Method: PUT 2024-04-12T03:50:18.535 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:50:18.535 DEBUG:boto:Data: 2024-04-12T03:50:18.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:18.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.536 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.536 DEBUG:boto:Params: {} 2024-04-12T03:50:18.536 DEBUG:boto:Token: None 2024-04-12T03:50:18.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj124 2024-04-12T03:50:18.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OKy5YUgii8VGuDp//iAi62URmlY= 2024-04-12T03:50:18.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OKy5YUgii8VGuDp//iAi62URmlY='} 2024-04-12T03:50:18.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad6c5b1e93f5b164-006618af7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.579 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:50:18.579 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:50:18.579 DEBUG:boto:Method: HEAD 2024-04-12T03:50:18.579 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:50:18.579 DEBUG:boto:Data: 2024-04-12T03:50:18.579 DEBUG:boto:Headers: {} 2024-04-12T03:50:18.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.579 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.580 DEBUG:boto:Params: {} 2024-04-12T03:50:18.580 DEBUG:boto:Token: None 2024-04-12T03:50:18.580 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/ 2024-04-12T03:50:18.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs= 2024-04-12T03:50:18.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lHV13b/OmTq7SoU9NhkxjVGRpzs=', 'Content-Length': '0'} 2024-04-12T03:50:18.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007795f57ccdf9b00a-006618af7a-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.584 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:50:18.584 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:50:18.584 DEBUG:boto:Method: PUT 2024-04-12T03:50:18.584 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:50:18.584 DEBUG:boto:Data: 2024-04-12T03:50:18.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:18.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:50:18.584 DEBUG:boto:Port: 8000 2024-04-12T03:50:18.584 DEBUG:boto:Params: {} 2024-04-12T03:50:18.584 DEBUG:boto:Token: None 2024-04-12T03:50:18.585 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:18 GMT /dgbpxg-61/obj155 2024-04-12T03:50:18.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d0F158t8B0zvGrhRLxTThP69NRk= 2024-04-12T03:50:18.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d0F158t8B0zvGrhRLxTThP69NRk='} 2024-04-12T03:50:18.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001360eb60b276c0eb-006618af7a-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:18.590 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:50:18.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:18.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:18.590 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:18.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:18.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:18.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:18.687+0000 7f5a9a3ad780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T03:50:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:23.687+0000 7f5a9a3ad780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T03:50:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:28.689+0000 7f5a9a3ad780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T03:50:33.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:33.690+0000 7f5a9a3ad780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T03:50:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:38.692+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:38.686 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:50:38.686 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:50:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:43.694+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:43.688 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:50:43.688 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:50:48.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:48.696+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:48.690 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:50:48.690 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:50:53.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:53.699+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:53.692 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:50:53.692 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:50:58.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:50:58.701+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:58.694 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:50:58.695 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:03.702+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:03.697 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.98.6, , , , 00000000001.34.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:03.697 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:08.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:08.704+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:08.699 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.98.6, , , , 00000000001.34.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:08.699 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:13.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:13.707+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:13.700 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.98.6, , , , 00000000001.34.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:13.701 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:18.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:18.710+0000 7f5a9a3ad780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:18.703 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.98.6, , , , 00000000001.34.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:18.703 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:23.712+0000 7f5a9a3ad780 1 bucket sync caught up with source: 2024-04-12T03:51:23.705 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:23.706 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.98.6, , 00000000001.62.6, , 00000000001.34.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.69.6] 2024-04-12T03:51:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:23.712+0000 7f5a9a3ad780 0 bucket checkpoint complete 2024-04-12T03:51:23.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:51:23.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:23.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:23.724 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:51:23.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:51:23.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:51:23.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:23.833+0000 7f52d6531780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:23.827 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000001.1925.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:23.827 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000003.436.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:28.834+0000 7f52d6531780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:28.828 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000001.1925.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:28.828 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000003.436.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:29.910 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:51:29.917+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:51:33.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:33.836+0000 7f52d6531780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:33.829 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000002.425.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:33.829 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000003.436.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:38.837+0000 7f52d6531780 1 bucket sync caught up with source: 2024-04-12T03:51:38.830 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000003.436.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:38.830 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000003.436.6, 00000000006.2249.6, 00000000004.1491.6] 2024-04-12T03:51:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:38.837+0000 7f52d6531780 0 bucket checkpoint complete 2024-04-12T03:51:38.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:51:38.847 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.847 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.848 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.848 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:38.848 DEBUG:boto:Data: 2024-04-12T03:51:38.848 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.848 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.848 DEBUG:boto:Params: {} 2024-04-12T03:51:38.849 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:51:38.849 DEBUG:boto:Token: None 2024-04-12T03:51:38.849 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/ 2024-04-12T03:51:38.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+sYjZ5vZtAGprRoChgEz1SUKuMg= 2024-04-12T03:51:38.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+sYjZ5vZtAGprRoChgEz1SUKuMg=', 'Content-Length': '0'} 2024-04-12T03:51:38.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083b07aa4dfc7f3ab-006618afca-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.864 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.864 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.864 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.864 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:38.865 DEBUG:boto:Data: 2024-04-12T03:51:38.865 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.865 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.865 DEBUG:boto:Params: {} 2024-04-12T03:51:38.865 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:51:38.865 DEBUG:boto:Token: None 2024-04-12T03:51:38.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/ 2024-04-12T03:51:38.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+sYjZ5vZtAGprRoChgEz1SUKuMg= 2024-04-12T03:51:38.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+sYjZ5vZtAGprRoChgEz1SUKuMg=', 'Content-Length': '0'} 2024-04-12T03:51:38.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043ca217a8790a90a-006618afca-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.878 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:51:38.879 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.879 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.879 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.879 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.879 DEBUG:boto:Method: GET 2024-04-12T03:51:38.879 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.879 DEBUG:boto:Data: 2024-04-12T03:51:38.879 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.880 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.880 DEBUG:boto:Params: {} 2024-04-12T03:51:38.880 DEBUG:boto:Token: None 2024-04-12T03:51:38.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/?versions 2024-04-12T03:51:38.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg= 2024-04-12T03:51:38.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg=', 'Content-Length': '0'} 2024-04-12T03:51:38.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000406424515b893b0e-006618afca-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.883 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:51:38.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:51:38.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:51:38.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:51:38.888 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.888 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.888 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.888 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.888 DEBUG:boto:Method: GET 2024-04-12T03:51:38.888 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.888 DEBUG:boto:Data: 2024-04-12T03:51:38.888 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.888 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.888 DEBUG:boto:Params: {} 2024-04-12T03:51:38.889 DEBUG:boto:Token: None 2024-04-12T03:51:38.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/?versions 2024-04-12T03:51:38.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg= 2024-04-12T03:51:38.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg=', 'Content-Length': '0'} 2024-04-12T03:51:38.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4a1bd38daa9c89-006618afca-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.893 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:51:38.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:51:38.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:51:38.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:51:38.897 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.897 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.897 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.897 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.898 DEBUG:boto:Method: GET 2024-04-12T03:51:38.898 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.898 DEBUG:boto:Data: 2024-04-12T03:51:38.898 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.898 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.898 DEBUG:boto:Params: {} 2024-04-12T03:51:38.898 DEBUG:boto:Token: None 2024-04-12T03:51:38.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/?versions 2024-04-12T03:51:38.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg= 2024-04-12T03:51:38.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg=', 'Content-Length': '0'} 2024-04-12T03:51:38.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6833802c79a3035-006618afca-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.902 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.903 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.903 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.903 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.903 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.904 DEBUG:boto:Method: GET 2024-04-12T03:51:38.904 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:51:38.904 DEBUG:boto:Data: 2024-04-12T03:51:38.904 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.904 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.904 DEBUG:boto:Params: {} 2024-04-12T03:51:38.904 DEBUG:boto:Token: None 2024-04-12T03:51:38.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/?versions 2024-04-12T03:51:38.905 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg= 2024-04-12T03:51:38.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bxU9NFYEsrf+tF7971G540NILKg=', 'Content-Length': '0'} 2024-04-12T03:51:38.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e710c8a66315dd-006618afca-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.909 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:51:38.911 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.911 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.911 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.911 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.911 DEBUG:boto:Method: GET 2024-04-12T03:51:38.911 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.911 DEBUG:boto:Data: 2024-04-12T03:51:38.912 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.912 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.912 DEBUG:boto:Params: {} 2024-04-12T03:51:38.912 DEBUG:boto:Token: None 2024-04-12T03:51:38.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk= 2024-04-12T03:51:38.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk=', 'Content-Length': '0'} 2024-04-12T03:51:38.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef859fc61d66490e-006618afca-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.915 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.915 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.915 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.915 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.915 DEBUG:boto:Method: GET 2024-04-12T03:51:38.915 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.915 DEBUG:boto:Data: 2024-04-12T03:51:38.915 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.915 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.915 DEBUG:boto:Params: {} 2024-04-12T03:51:38.916 DEBUG:boto:Token: None 2024-04-12T03:51:38.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk= 2024-04-12T03:51:38.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk=', 'Content-Length': '0'} 2024-04-12T03:51:38.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477b02359bb153e1-006618afca-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.958 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.958 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.958 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.959 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.959 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.959 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.959 DEBUG:boto:Data: 2024-04-12T03:51:38.959 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.959 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.959 DEBUG:boto:Params: {} 2024-04-12T03:51:38.959 DEBUG:boto:Token: None 2024-04-12T03:51:38.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.960 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he2chpj4nO2aLZQhhZwcqwRvwOA= 2024-04-12T03:51:38.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he2chpj4nO2aLZQhhZwcqwRvwOA=', 'Content-Length': '0'} 2024-04-12T03:51:38.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066969feabb60eac1-006618afca-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.962 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.962 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.962 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.962 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.962 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.962 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.962 DEBUG:boto:Data: 2024-04-12T03:51:38.963 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.963 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.963 DEBUG:boto:Params: {} 2024-04-12T03:51:38.963 DEBUG:boto:Token: None 2024-04-12T03:51:38.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he2chpj4nO2aLZQhhZwcqwRvwOA= 2024-04-12T03:51:38.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he2chpj4nO2aLZQhhZwcqwRvwOA=', 'Content-Length': '0'} 2024-04-12T03:51:38.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dde372cb878083c-006618afca-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:51:38.965 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.966 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.966 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.966 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.966 DEBUG:boto:Method: GET 2024-04-12T03:51:38.966 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.966 DEBUG:boto:Data: 2024-04-12T03:51:38.966 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.966 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.966 DEBUG:boto:Params: {} 2024-04-12T03:51:38.966 DEBUG:boto:Token: None 2024-04-12T03:51:38.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.967 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk= 2024-04-12T03:51:38.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk=', 'Content-Length': '0'} 2024-04-12T03:51:38.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002148a491cb36abfe-006618afca-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.969 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.969 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.969 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.969 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.969 DEBUG:boto:Method: GET 2024-04-12T03:51:38.969 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.969 DEBUG:boto:Data: 2024-04-12T03:51:38.970 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.970 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.970 DEBUG:boto:Params: {} 2024-04-12T03:51:38.970 DEBUG:boto:Token: None 2024-04-12T03:51:38.971 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:51:38.971 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk= 2024-04-12T03:51:38.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5JQApQ/qVobCZrW8DvdBaS9rsk=', 'Content-Length': '0'} 2024-04-12T03:51:38.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ccb351faeedcef8-006618afca-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.974 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.974 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.974 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.974 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.974 DEBUG:boto:Method: GET 2024-04-12T03:51:38.974 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.974 DEBUG:boto:Data: 2024-04-12T03:51:38.975 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.975 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.975 DEBUG:boto:Params: {} 2024-04-12T03:51:38.975 DEBUG:boto:Token: None 2024-04-12T03:51:38.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/ 2024-04-12T03:51:38.975 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tv2y+3ciUJxytWxsrPTjn/YI8T8= 2024-04-12T03:51:38.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tv2y+3ciUJxytWxsrPTjn/YI8T8=', 'Content-Length': '0'} 2024-04-12T03:51:38.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ded51aa1763dcc3d-006618afca-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.978 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.978 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:38.979 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:38.979 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.979 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.979 DEBUG:boto:Method: GET 2024-04-12T03:51:38.979 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:51:38.979 DEBUG:boto:Data: 2024-04-12T03:51:38.979 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.979 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.979 DEBUG:boto:Params: {} 2024-04-12T03:51:38.980 DEBUG:boto:Token: None 2024-04-12T03:51:38.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/ 2024-04-12T03:51:38.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tv2y+3ciUJxytWxsrPTjn/YI8T8= 2024-04-12T03:51:38.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tv2y+3ciUJxytWxsrPTjn/YI8T8=', 'Content-Length': '0'} 2024-04-12T03:51:38.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fad094733c4a74f-006618afca-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.984 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:38.984 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.984 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.984 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.984 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:51:38.984 DEBUG:boto:Data: 2024-04-12T03:51:38.984 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.985 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.985 DEBUG:boto:Params: {} 2024-04-12T03:51:38.985 DEBUG:boto:Token: None 2024-04-12T03:51:38.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11 2024-04-12T03:51:38.985 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:22u3rb1FlQ8/u7o6+EOVovwIxh8= 2024-04-12T03:51:38.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:22u3rb1FlQ8/u7o6+EOVovwIxh8=', 'Content-Length': '0'} 2024-04-12T03:51:38.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044eead400fa721cd-006618afca-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.987 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.987 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.987 DEBUG:boto:Method: HEAD 2024-04-12T03:51:38.987 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:51:38.987 DEBUG:boto:Data: 2024-04-12T03:51:38.988 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:38.988 DEBUG:boto:Port: 8000 2024-04-12T03:51:38.988 DEBUG:boto:Params: {} 2024-04-12T03:51:38.988 DEBUG:boto:Token: None 2024-04-12T03:51:38.988 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11 2024-04-12T03:51:38.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:22u3rb1FlQ8/u7o6+EOVovwIxh8= 2024-04-12T03:51:38.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:22u3rb1FlQ8/u7o6+EOVovwIxh8=', 'Content-Length': '0'} 2024-04-12T03:51:38.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002311730c10ec96a-006618afca-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:38.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:51:38.990 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:38.990 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:38.990 DEBUG:boto:Method: GET 2024-04-12T03:51:38.991 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:51:38.991 DEBUG:boto:Data: 2024-04-12T03:51:38.991 DEBUG:boto:Headers: {} 2024-04-12T03:51:38.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:38.991 DEBUG:boto:Port: 8001 2024-04-12T03:51:38.991 DEBUG:boto:Params: {} 2024-04-12T03:51:38.991 DEBUG:boto:Token: None 2024-04-12T03:51:38.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:38 GMT /dgbpxg-61/obj11 2024-04-12T03:51:38.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NfjwZABYC8ap2z0ENAiZe4Sdy6I= 2024-04-12T03:51:38.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NfjwZABYC8ap2z0ENAiZe4Sdy6I=', 'Content-Length': '0'} 2024-04-12T03:51:38.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa8b5405b87cba3-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.034 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:51:39.034 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:51:39.035 DEBUG:boto:Method: GET 2024-04-12T03:51:39.035 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:51:39.035 DEBUG:boto:Data: 2024-04-12T03:51:39.035 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.035 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.035 DEBUG:boto:Params: {} 2024-04-12T03:51:39.035 DEBUG:boto:Token: None 2024-04-12T03:51:39.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj11 2024-04-12T03:51:39.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ycO7igjLsufwcUySz+MW/+V6m7s= 2024-04-12T03:51:39.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ycO7igjLsufwcUySz+MW/+V6m7s=', 'Content-Length': '0'} 2024-04-12T03:51:39.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055044131c09ad97-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.038 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.038 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.038 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.038 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.038 DEBUG:boto:Method: GET 2024-04-12T03:51:39.038 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.038 DEBUG:boto:Data: 2024-04-12T03:51:39.038 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.039 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.039 DEBUG:boto:Params: {} 2024-04-12T03:51:39.039 DEBUG:boto:Token: None 2024-04-12T03:51:39.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1778e14ba4a4c3-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.042 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.042 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.042 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.042 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.042 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.042 DEBUG:boto:Method: GET 2024-04-12T03:51:39.043 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:51:39.043 DEBUG:boto:Data: 2024-04-12T03:51:39.043 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.043 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.043 DEBUG:boto:Params: {} 2024-04-12T03:51:39.043 DEBUG:boto:Token: None 2024-04-12T03:51:39.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a1dcaa22b82da1-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.047 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:51:39.047 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.048 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.048 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.048 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.048 DEBUG:boto:Method: GET 2024-04-12T03:51:39.048 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.048 DEBUG:boto:Data: 2024-04-12T03:51:39.048 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.048 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.048 DEBUG:boto:Params: {} 2024-04-12T03:51:39.049 DEBUG:boto:Token: None 2024-04-12T03:51:39.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.049 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0= 2024-04-12T03:51:39.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0=', 'Content-Length': '0'} 2024-04-12T03:51:39.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ed663d33f17c2a-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.051 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.051 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.051 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.051 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.051 DEBUG:boto:Method: GET 2024-04-12T03:51:39.051 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.051 DEBUG:boto:Data: 2024-04-12T03:51:39.052 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.052 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.052 DEBUG:boto:Params: {} 2024-04-12T03:51:39.052 DEBUG:boto:Token: None 2024-04-12T03:51:39.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0= 2024-04-12T03:51:39.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0=', 'Content-Length': '0'} 2024-04-12T03:51:39.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4dcde552da079dd-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.098 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.098 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.099 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.099 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.099 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.099 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.099 DEBUG:boto:Data: 2024-04-12T03:51:39.099 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.099 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.099 DEBUG:boto:Params: {} 2024-04-12T03:51:39.099 DEBUG:boto:Token: None 2024-04-12T03:51:39.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YZkWjAn9RUyLOCg8LN9hvHDU3h8= 2024-04-12T03:51:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YZkWjAn9RUyLOCg8LN9hvHDU3h8=', 'Content-Length': '0'} 2024-04-12T03:51:39.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc151a71cab6499-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.101 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.102 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.102 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.102 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.102 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.102 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.102 DEBUG:boto:Data: 2024-04-12T03:51:39.102 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.102 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.102 DEBUG:boto:Params: {} 2024-04-12T03:51:39.102 DEBUG:boto:Token: None 2024-04-12T03:51:39.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YZkWjAn9RUyLOCg8LN9hvHDU3h8= 2024-04-12T03:51:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YZkWjAn9RUyLOCg8LN9hvHDU3h8=', 'Content-Length': '0'} 2024-04-12T03:51:39.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0cb6b6432c3f970-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:51:39.105 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.105 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.105 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.105 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.105 DEBUG:boto:Method: GET 2024-04-12T03:51:39.105 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.105 DEBUG:boto:Data: 2024-04-12T03:51:39.105 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.106 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.106 DEBUG:boto:Params: {} 2024-04-12T03:51:39.106 DEBUG:boto:Token: None 2024-04-12T03:51:39.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0= 2024-04-12T03:51:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0=', 'Content-Length': '0'} 2024-04-12T03:51:39.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0967ce4ad9cd52-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.108 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.108 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.108 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.108 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.108 DEBUG:boto:Method: GET 2024-04-12T03:51:39.108 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.109 DEBUG:boto:Data: 2024-04-12T03:51:39.109 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.109 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.109 DEBUG:boto:Params: {} 2024-04-12T03:51:39.109 DEBUG:boto:Token: None 2024-04-12T03:51:39.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:51:39.109 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0= 2024-04-12T03:51:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e5rPsBI+mq4wpmWZsvH8zB/hkY0=', 'Content-Length': '0'} 2024-04-12T03:51:39.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a7f8357c9141765-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.112 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.112 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.112 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.112 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.112 DEBUG:boto:Method: GET 2024-04-12T03:51:39.112 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.112 DEBUG:boto:Data: 2024-04-12T03:51:39.112 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.113 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.113 DEBUG:boto:Params: {} 2024-04-12T03:51:39.113 DEBUG:boto:Token: None 2024-04-12T03:51:39.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000323c6a336fcfc436-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.115 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.116 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.116 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.116 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.116 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.116 DEBUG:boto:Method: GET 2024-04-12T03:51:39.116 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.116 DEBUG:boto:Data: 2024-04-12T03:51:39.116 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.116 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.117 DEBUG:boto:Params: {} 2024-04-12T03:51:39.117 DEBUG:boto:Token: None 2024-04-12T03:51:39.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf24fb97cd9b6b5-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.120 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.121 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.121 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.121 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.121 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:51:39.121 DEBUG:boto:Data: 2024-04-12T03:51:39.121 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.121 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.121 DEBUG:boto:Params: {} 2024-04-12T03:51:39.121 DEBUG:boto:Token: None 2024-04-12T03:51:39.122 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115 2024-04-12T03:51:39.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lcgqp+OahR5MvPREMXPPH+mF2+Y= 2024-04-12T03:51:39.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lcgqp+OahR5MvPREMXPPH+mF2+Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092395b769e647548-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.124 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.124 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.124 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.124 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:51:39.124 DEBUG:boto:Data: 2024-04-12T03:51:39.124 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.124 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.124 DEBUG:boto:Params: {} 2024-04-12T03:51:39.124 DEBUG:boto:Token: None 2024-04-12T03:51:39.125 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115 2024-04-12T03:51:39.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lcgqp+OahR5MvPREMXPPH+mF2+Y= 2024-04-12T03:51:39.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lcgqp+OahR5MvPREMXPPH+mF2+Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096b5a2e002dc1ae1-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:51:39.127 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.127 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.127 DEBUG:boto:Method: GET 2024-04-12T03:51:39.127 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:51:39.127 DEBUG:boto:Data: 2024-04-12T03:51:39.127 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.127 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.127 DEBUG:boto:Params: {} 2024-04-12T03:51:39.127 DEBUG:boto:Token: None 2024-04-12T03:51:39.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115 2024-04-12T03:51:39.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eMkZEoe5xnhYLROCvO+955pRpMk= 2024-04-12T03:51:39.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eMkZEoe5xnhYLROCvO+955pRpMk=', 'Content-Length': '0'} 2024-04-12T03:51:39.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c9082c194d305c-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.129 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:51:39.130 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:51:39.130 DEBUG:boto:Method: GET 2024-04-12T03:51:39.130 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:51:39.130 DEBUG:boto:Data: 2024-04-12T03:51:39.130 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.130 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.130 DEBUG:boto:Params: {} 2024-04-12T03:51:39.130 DEBUG:boto:Token: None 2024-04-12T03:51:39.130 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj115 2024-04-12T03:51:39.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eMkZEoe5xnhYLROCvO+955pRpMk= 2024-04-12T03:51:39.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eMkZEoe5xnhYLROCvO+955pRpMk=', 'Content-Length': '0'} 2024-04-12T03:51:39.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae8bc487fa035c4-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.174 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.174 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.175 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.175 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.175 DEBUG:boto:Method: GET 2024-04-12T03:51:39.175 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.175 DEBUG:boto:Data: 2024-04-12T03:51:39.175 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.175 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.175 DEBUG:boto:Params: {} 2024-04-12T03:51:39.175 DEBUG:boto:Token: None 2024-04-12T03:51:39.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b26fad02bb17f7-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.178 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.179 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.179 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.179 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.179 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.179 DEBUG:boto:Method: GET 2024-04-12T03:51:39.179 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:51:39.179 DEBUG:boto:Data: 2024-04-12T03:51:39.179 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.180 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.180 DEBUG:boto:Params: {} 2024-04-12T03:51:39.180 DEBUG:boto:Token: None 2024-04-12T03:51:39.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001af9fcd0b8a264bc-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.184 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:51:39.184 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.184 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.184 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.184 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.185 DEBUG:boto:Method: GET 2024-04-12T03:51:39.185 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.185 DEBUG:boto:Data: 2024-04-12T03:51:39.185 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.185 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.185 DEBUG:boto:Params: {} 2024-04-12T03:51:39.185 DEBUG:boto:Token: None 2024-04-12T03:51:39.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8= 2024-04-12T03:51:39.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8=', 'Content-Length': '0'} 2024-04-12T03:51:39.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cc80b9d0fc50124-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.188 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.188 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.188 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.188 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.188 DEBUG:boto:Method: GET 2024-04-12T03:51:39.188 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.188 DEBUG:boto:Data: 2024-04-12T03:51:39.188 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.189 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.189 DEBUG:boto:Params: {} 2024-04-12T03:51:39.189 DEBUG:boto:Token: None 2024-04-12T03:51:39.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.189 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8= 2024-04-12T03:51:39.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8=', 'Content-Length': '0'} 2024-04-12T03:51:39.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead184a81776138f-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.191 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.191 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.191 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.191 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.191 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.191 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.191 DEBUG:boto:Data: 2024-04-12T03:51:39.192 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.192 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.192 DEBUG:boto:Params: {} 2024-04-12T03:51:39.192 DEBUG:boto:Token: None 2024-04-12T03:51:39.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MMUGI0E8DGP0edQ3gq4V53F6JFw= 2024-04-12T03:51:39.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MMUGI0E8DGP0edQ3gq4V53F6JFw=', 'Content-Length': '0'} 2024-04-12T03:51:39.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8115eec989b3fdb-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.194 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.194 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.194 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.195 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.195 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.195 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.195 DEBUG:boto:Data: 2024-04-12T03:51:39.195 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.195 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.195 DEBUG:boto:Params: {} 2024-04-12T03:51:39.195 DEBUG:boto:Token: None 2024-04-12T03:51:39.195 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MMUGI0E8DGP0edQ3gq4V53F6JFw= 2024-04-12T03:51:39.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MMUGI0E8DGP0edQ3gq4V53F6JFw=', 'Content-Length': '0'} 2024-04-12T03:51:39.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82bdf572ffc267d-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:51:39.198 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.198 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.198 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.198 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.198 DEBUG:boto:Method: GET 2024-04-12T03:51:39.198 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.198 DEBUG:boto:Data: 2024-04-12T03:51:39.198 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.198 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.199 DEBUG:boto:Params: {} 2024-04-12T03:51:39.199 DEBUG:boto:Token: None 2024-04-12T03:51:39.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8= 2024-04-12T03:51:39.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8=', 'Content-Length': '0'} 2024-04-12T03:51:39.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026ba19df6d098420-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.201 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.201 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.201 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.201 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.201 DEBUG:boto:Method: GET 2024-04-12T03:51:39.202 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.202 DEBUG:boto:Data: 2024-04-12T03:51:39.202 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.202 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.202 DEBUG:boto:Params: {} 2024-04-12T03:51:39.202 DEBUG:boto:Token: None 2024-04-12T03:51:39.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:51:39.202 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8= 2024-04-12T03:51:39.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SkMrR/iVMt9yuFFZcZm3gDMunm8=', 'Content-Length': '0'} 2024-04-12T03:51:39.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba040099d179c682-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.246 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.247 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.247 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.247 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.247 DEBUG:boto:Method: GET 2024-04-12T03:51:39.247 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.247 DEBUG:boto:Data: 2024-04-12T03:51:39.247 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.247 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.247 DEBUG:boto:Params: {} 2024-04-12T03:51:39.248 DEBUG:boto:Token: None 2024-04-12T03:51:39.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a99f5ac3940ba3-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.250 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.251 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.251 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.251 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.251 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.251 DEBUG:boto:Method: GET 2024-04-12T03:51:39.251 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.251 DEBUG:boto:Data: 2024-04-12T03:51:39.251 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.252 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.252 DEBUG:boto:Params: {} 2024-04-12T03:51:39.252 DEBUG:boto:Token: None 2024-04-12T03:51:39.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e62d81b7d2a43eb-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.256 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.256 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.256 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.256 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.256 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:51:39.256 DEBUG:boto:Data: 2024-04-12T03:51:39.257 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.257 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.257 DEBUG:boto:Params: {} 2024-04-12T03:51:39.257 DEBUG:boto:Token: None 2024-04-12T03:51:39.257 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116 2024-04-12T03:51:39.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gy47JosNCs3w4M2Ysr0njuJ45Mo= 2024-04-12T03:51:39.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gy47JosNCs3w4M2Ysr0njuJ45Mo=', 'Content-Length': '0'} 2024-04-12T03:51:39.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857c3af5a816ffed-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.259 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.259 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.259 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.259 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:51:39.259 DEBUG:boto:Data: 2024-04-12T03:51:39.259 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.260 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.260 DEBUG:boto:Params: {} 2024-04-12T03:51:39.260 DEBUG:boto:Token: None 2024-04-12T03:51:39.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116 2024-04-12T03:51:39.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gy47JosNCs3w4M2Ysr0njuJ45Mo= 2024-04-12T03:51:39.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gy47JosNCs3w4M2Ysr0njuJ45Mo=', 'Content-Length': '0'} 2024-04-12T03:51:39.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6faa835688b6cf7-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:51:39.262 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.263 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.263 DEBUG:boto:Method: GET 2024-04-12T03:51:39.263 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:51:39.263 DEBUG:boto:Data: 2024-04-12T03:51:39.263 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.263 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.263 DEBUG:boto:Params: {} 2024-04-12T03:51:39.263 DEBUG:boto:Token: None 2024-04-12T03:51:39.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116 2024-04-12T03:51:39.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5gin/YHs02xRo68HwJMWWJNRLTw= 2024-04-12T03:51:39.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5gin/YHs02xRo68HwJMWWJNRLTw=', 'Content-Length': '0'} 2024-04-12T03:51:39.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f55a5d5e2b629a-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.310 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:51:39.311 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:51:39.311 DEBUG:boto:Method: GET 2024-04-12T03:51:39.311 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:51:39.311 DEBUG:boto:Data: 2024-04-12T03:51:39.311 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.311 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.311 DEBUG:boto:Params: {} 2024-04-12T03:51:39.311 DEBUG:boto:Token: None 2024-04-12T03:51:39.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj116 2024-04-12T03:51:39.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5gin/YHs02xRo68HwJMWWJNRLTw= 2024-04-12T03:51:39.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5gin/YHs02xRo68HwJMWWJNRLTw=', 'Content-Length': '0'} 2024-04-12T03:51:39.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabebed496cdfbe8-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.314 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.314 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.314 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.314 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.315 DEBUG:boto:Method: GET 2024-04-12T03:51:39.315 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.315 DEBUG:boto:Data: 2024-04-12T03:51:39.315 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.315 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.315 DEBUG:boto:Params: {} 2024-04-12T03:51:39.315 DEBUG:boto:Token: None 2024-04-12T03:51:39.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000af409600b5722a2-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.318 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.318 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.318 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.319 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.319 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.319 DEBUG:boto:Method: GET 2024-04-12T03:51:39.319 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:51:39.319 DEBUG:boto:Data: 2024-04-12T03:51:39.319 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.319 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.319 DEBUG:boto:Params: {} 2024-04-12T03:51:39.319 DEBUG:boto:Token: None 2024-04-12T03:51:39.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0528d5ed6378e93-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.323 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:51:39.324 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.324 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.324 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.324 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.324 DEBUG:boto:Method: GET 2024-04-12T03:51:39.324 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.324 DEBUG:boto:Data: 2024-04-12T03:51:39.324 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.324 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.324 DEBUG:boto:Params: {} 2024-04-12T03:51:39.325 DEBUG:boto:Token: None 2024-04-12T03:51:39.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI= 2024-04-12T03:51:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI=', 'Content-Length': '0'} 2024-04-12T03:51:39.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2a1b9dda503203-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.327 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.327 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.327 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.327 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.327 DEBUG:boto:Method: GET 2024-04-12T03:51:39.328 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.328 DEBUG:boto:Data: 2024-04-12T03:51:39.328 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.328 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.328 DEBUG:boto:Params: {} 2024-04-12T03:51:39.328 DEBUG:boto:Token: None 2024-04-12T03:51:39.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI= 2024-04-12T03:51:39.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI=', 'Content-Length': '0'} 2024-04-12T03:51:39.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6c0c8bc1aa12ee-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.375 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.375 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.375 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.375 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.375 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.375 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.375 DEBUG:boto:Data: 2024-04-12T03:51:39.375 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.375 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.376 DEBUG:boto:Params: {} 2024-04-12T03:51:39.376 DEBUG:boto:Token: None 2024-04-12T03:51:39.376 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u0AVPPU2s1pcCCs3OMwxiz228kA= 2024-04-12T03:51:39.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u0AVPPU2s1pcCCs3OMwxiz228kA=', 'Content-Length': '0'} 2024-04-12T03:51:39.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007388f0dff8d9ff80-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.378 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.378 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.379 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.379 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.379 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.379 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.379 DEBUG:boto:Data: 2024-04-12T03:51:39.379 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.379 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.379 DEBUG:boto:Params: {} 2024-04-12T03:51:39.379 DEBUG:boto:Token: None 2024-04-12T03:51:39.380 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u0AVPPU2s1pcCCs3OMwxiz228kA= 2024-04-12T03:51:39.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u0AVPPU2s1pcCCs3OMwxiz228kA=', 'Content-Length': '0'} 2024-04-12T03:51:39.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05c4eb4b6f8e65b-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:51:39.382 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.382 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.382 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.382 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.382 DEBUG:boto:Method: GET 2024-04-12T03:51:39.382 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.382 DEBUG:boto:Data: 2024-04-12T03:51:39.383 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.383 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.383 DEBUG:boto:Params: {} 2024-04-12T03:51:39.383 DEBUG:boto:Token: None 2024-04-12T03:51:39.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI= 2024-04-12T03:51:39.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI=', 'Content-Length': '0'} 2024-04-12T03:51:39.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aecbb9dac7018aca-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.385 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.385 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.386 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.386 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.386 DEBUG:boto:Method: GET 2024-04-12T03:51:39.386 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.386 DEBUG:boto:Data: 2024-04-12T03:51:39.386 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.386 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.386 DEBUG:boto:Params: {} 2024-04-12T03:51:39.386 DEBUG:boto:Token: None 2024-04-12T03:51:39.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:51:39.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI= 2024-04-12T03:51:39.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga70iI7zCrdTPzK1ljCbSLXnMtI=', 'Content-Length': '0'} 2024-04-12T03:51:39.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a815d9b3c4ba81fc-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.389 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.389 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.389 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.389 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.389 DEBUG:boto:Method: GET 2024-04-12T03:51:39.389 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.389 DEBUG:boto:Data: 2024-04-12T03:51:39.389 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.390 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.390 DEBUG:boto:Params: {} 2024-04-12T03:51:39.390 DEBUG:boto:Token: None 2024-04-12T03:51:39.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000184cb937c8f58cc3-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.392 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.393 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.393 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.393 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.393 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.393 DEBUG:boto:Method: GET 2024-04-12T03:51:39.393 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.393 DEBUG:boto:Data: 2024-04-12T03:51:39.394 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.394 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.394 DEBUG:boto:Params: {} 2024-04-12T03:51:39.394 DEBUG:boto:Token: None 2024-04-12T03:51:39.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.394 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ed25ac06d5977c-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.398 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.398 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.398 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.398 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.398 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:51:39.399 DEBUG:boto:Data: 2024-04-12T03:51:39.399 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.399 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.399 DEBUG:boto:Params: {} 2024-04-12T03:51:39.399 DEBUG:boto:Token: None 2024-04-12T03:51:39.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124 2024-04-12T03:51:39.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mQG6271cnJO5Qhc7WZcgcEr2KJ0= 2024-04-12T03:51:39.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mQG6271cnJO5Qhc7WZcgcEr2KJ0=', 'Content-Length': '0'} 2024-04-12T03:51:39.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae32f4546de00e2-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.401 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.401 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.401 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.401 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:51:39.402 DEBUG:boto:Data: 2024-04-12T03:51:39.402 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.402 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.402 DEBUG:boto:Params: {} 2024-04-12T03:51:39.402 DEBUG:boto:Token: None 2024-04-12T03:51:39.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124 2024-04-12T03:51:39.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mQG6271cnJO5Qhc7WZcgcEr2KJ0= 2024-04-12T03:51:39.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mQG6271cnJO5Qhc7WZcgcEr2KJ0=', 'Content-Length': '0'} 2024-04-12T03:51:39.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614d3e9ca75fdac9-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:51:39.405 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.405 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.405 DEBUG:boto:Method: GET 2024-04-12T03:51:39.405 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:51:39.405 DEBUG:boto:Data: 2024-04-12T03:51:39.405 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.405 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.405 DEBUG:boto:Params: {} 2024-04-12T03:51:39.405 DEBUG:boto:Token: None 2024-04-12T03:51:39.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124 2024-04-12T03:51:39.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67PeIpATtzN7gFB+gznNAymOhBk= 2024-04-12T03:51:39.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67PeIpATtzN7gFB+gznNAymOhBk=', 'Content-Length': '0'} 2024-04-12T03:51:39.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd8d4935577bd0f-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.408 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:51:39.408 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:51:39.408 DEBUG:boto:Method: GET 2024-04-12T03:51:39.408 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:51:39.408 DEBUG:boto:Data: 2024-04-12T03:51:39.408 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.408 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.408 DEBUG:boto:Params: {} 2024-04-12T03:51:39.409 DEBUG:boto:Token: None 2024-04-12T03:51:39.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj124 2024-04-12T03:51:39.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67PeIpATtzN7gFB+gznNAymOhBk= 2024-04-12T03:51:39.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67PeIpATtzN7gFB+gznNAymOhBk=', 'Content-Length': '0'} 2024-04-12T03:51:39.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d98099a3adef14b-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.411 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.411 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.411 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.411 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.411 DEBUG:boto:Method: GET 2024-04-12T03:51:39.411 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.411 DEBUG:boto:Data: 2024-04-12T03:51:39.411 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.411 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.412 DEBUG:boto:Params: {} 2024-04-12T03:51:39.412 DEBUG:boto:Token: None 2024-04-12T03:51:39.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880eca5865d41e2f-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.414 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.415 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.415 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.415 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.415 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.415 DEBUG:boto:Method: GET 2024-04-12T03:51:39.415 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:51:39.415 DEBUG:boto:Data: 2024-04-12T03:51:39.416 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.416 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.416 DEBUG:boto:Params: {} 2024-04-12T03:51:39.416 DEBUG:boto:Token: None 2024-04-12T03:51:39.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30fa016c173d71a-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.419 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:51:39.420 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.420 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.420 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.420 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.420 DEBUG:boto:Method: GET 2024-04-12T03:51:39.420 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.420 DEBUG:boto:Data: 2024-04-12T03:51:39.421 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.421 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.421 DEBUG:boto:Params: {} 2024-04-12T03:51:39.421 DEBUG:boto:Token: None 2024-04-12T03:51:39.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU= 2024-04-12T03:51:39.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU=', 'Content-Length': '0'} 2024-04-12T03:51:39.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e619ce01bb35e6-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.423 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.424 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.424 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.424 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.424 DEBUG:boto:Method: GET 2024-04-12T03:51:39.424 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.424 DEBUG:boto:Data: 2024-04-12T03:51:39.424 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.424 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.424 DEBUG:boto:Params: {} 2024-04-12T03:51:39.424 DEBUG:boto:Token: None 2024-04-12T03:51:39.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.425 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU= 2024-04-12T03:51:39.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU=', 'Content-Length': '0'} 2024-04-12T03:51:39.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e658f89f1cf4f95f-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.470 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.471 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.471 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.471 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.471 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.471 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.471 DEBUG:boto:Data: 2024-04-12T03:51:39.471 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.471 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.471 DEBUG:boto:Params: {} 2024-04-12T03:51:39.472 DEBUG:boto:Token: None 2024-04-12T03:51:39.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZiZL/UzxpNRkJU8BrtXZziCvHM= 2024-04-12T03:51:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZiZL/UzxpNRkJU8BrtXZziCvHM=', 'Content-Length': '0'} 2024-04-12T03:51:39.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb75864b53b54864-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.474 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.474 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.474 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.474 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.474 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.474 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.474 DEBUG:boto:Data: 2024-04-12T03:51:39.475 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.475 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.475 DEBUG:boto:Params: {} 2024-04-12T03:51:39.475 DEBUG:boto:Token: None 2024-04-12T03:51:39.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wZiZL/UzxpNRkJU8BrtXZziCvHM= 2024-04-12T03:51:39.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wZiZL/UzxpNRkJU8BrtXZziCvHM=', 'Content-Length': '0'} 2024-04-12T03:51:39.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f60f9e5921b830-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:51:39.477 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.477 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.478 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.478 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.478 DEBUG:boto:Method: GET 2024-04-12T03:51:39.478 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.478 DEBUG:boto:Data: 2024-04-12T03:51:39.478 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.478 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.478 DEBUG:boto:Params: {} 2024-04-12T03:51:39.479 DEBUG:boto:Token: None 2024-04-12T03:51:39.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU= 2024-04-12T03:51:39.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU=', 'Content-Length': '0'} 2024-04-12T03:51:39.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072719151323480e-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.481 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.481 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.481 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.481 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.481 DEBUG:boto:Method: GET 2024-04-12T03:51:39.481 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.482 DEBUG:boto:Data: 2024-04-12T03:51:39.482 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.482 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.482 DEBUG:boto:Params: {} 2024-04-12T03:51:39.482 DEBUG:boto:Token: None 2024-04-12T03:51:39.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:51:39.482 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU= 2024-04-12T03:51:39.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1c3RHjaCsLnG3/kKtSbvRq+BzIU=', 'Content-Length': '0'} 2024-04-12T03:51:39.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d3f2f6989c2f18-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.485 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.485 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.485 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.485 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.485 DEBUG:boto:Method: GET 2024-04-12T03:51:39.485 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.485 DEBUG:boto:Data: 2024-04-12T03:51:39.485 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.485 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.485 DEBUG:boto:Params: {} 2024-04-12T03:51:39.486 DEBUG:boto:Token: None 2024-04-12T03:51:39.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.486 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36b3618b9af3374-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.488 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.489 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.489 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.489 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.489 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.489 DEBUG:boto:Method: GET 2024-04-12T03:51:39.489 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.489 DEBUG:boto:Data: 2024-04-12T03:51:39.489 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.489 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.490 DEBUG:boto:Params: {} 2024-04-12T03:51:39.490 DEBUG:boto:Token: None 2024-04-12T03:51:39.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000455db7c7822383ea-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.494 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.494 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.494 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.494 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.494 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:51:39.494 DEBUG:boto:Data: 2024-04-12T03:51:39.495 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.495 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.495 DEBUG:boto:Params: {} 2024-04-12T03:51:39.495 DEBUG:boto:Token: None 2024-04-12T03:51:39.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145 2024-04-12T03:51:39.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqweB1sgf0TusDyU2eCR+oyUJWY= 2024-04-12T03:51:39.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqweB1sgf0TusDyU2eCR+oyUJWY=', 'Content-Length': '0'} 2024-04-12T03:51:39.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038acc734cbe271c0-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.497 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.497 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.497 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.497 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:51:39.497 DEBUG:boto:Data: 2024-04-12T03:51:39.498 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.498 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.498 DEBUG:boto:Params: {} 2024-04-12T03:51:39.498 DEBUG:boto:Token: None 2024-04-12T03:51:39.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145 2024-04-12T03:51:39.498 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HqweB1sgf0TusDyU2eCR+oyUJWY= 2024-04-12T03:51:39.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HqweB1sgf0TusDyU2eCR+oyUJWY=', 'Content-Length': '0'} 2024-04-12T03:51:39.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094000f3ecfd40890-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:51:39.500 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.500 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.501 DEBUG:boto:Method: GET 2024-04-12T03:51:39.501 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:51:39.501 DEBUG:boto:Data: 2024-04-12T03:51:39.501 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.501 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.501 DEBUG:boto:Params: {} 2024-04-12T03:51:39.501 DEBUG:boto:Token: None 2024-04-12T03:51:39.501 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145 2024-04-12T03:51:39.501 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1QjUHkmf11bOFg8zSrJUv1H8HLc= 2024-04-12T03:51:39.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1QjUHkmf11bOFg8zSrJUv1H8HLc=', 'Content-Length': '0'} 2024-04-12T03:51:39.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5680fbc10257dca-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.503 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:51:39.504 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:51:39.504 DEBUG:boto:Method: GET 2024-04-12T03:51:39.504 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:51:39.504 DEBUG:boto:Data: 2024-04-12T03:51:39.504 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.504 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.504 DEBUG:boto:Params: {} 2024-04-12T03:51:39.504 DEBUG:boto:Token: None 2024-04-12T03:51:39.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj145 2024-04-12T03:51:39.505 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1QjUHkmf11bOFg8zSrJUv1H8HLc= 2024-04-12T03:51:39.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1QjUHkmf11bOFg8zSrJUv1H8HLc=', 'Content-Length': '0'} 2024-04-12T03:51:39.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1f44813f1be9c2-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.507 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.507 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.507 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.507 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.507 DEBUG:boto:Method: GET 2024-04-12T03:51:39.507 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.508 DEBUG:boto:Data: 2024-04-12T03:51:39.508 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.508 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.508 DEBUG:boto:Params: {} 2024-04-12T03:51:39.508 DEBUG:boto:Token: None 2024-04-12T03:51:39.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9dac40beb96dd1-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.554 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.555 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.555 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.555 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.555 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.555 DEBUG:boto:Method: GET 2024-04-12T03:51:39.555 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:51:39.555 DEBUG:boto:Data: 2024-04-12T03:51:39.555 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.556 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.556 DEBUG:boto:Params: {} 2024-04-12T03:51:39.556 DEBUG:boto:Token: None 2024-04-12T03:51:39.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e005890d96c2d56-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.560 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:51:39.560 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.560 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.560 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.560 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.561 DEBUG:boto:Method: GET 2024-04-12T03:51:39.561 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.561 DEBUG:boto:Data: 2024-04-12T03:51:39.561 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.561 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.561 DEBUG:boto:Params: {} 2024-04-12T03:51:39.561 DEBUG:boto:Token: None 2024-04-12T03:51:39.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34= 2024-04-12T03:51:39.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34=', 'Content-Length': '0'} 2024-04-12T03:51:39.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d8b2f0b4387881-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.564 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.564 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.564 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.564 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.564 DEBUG:boto:Method: GET 2024-04-12T03:51:39.564 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.564 DEBUG:boto:Data: 2024-04-12T03:51:39.564 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.565 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.565 DEBUG:boto:Params: {} 2024-04-12T03:51:39.565 DEBUG:boto:Token: None 2024-04-12T03:51:39.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34= 2024-04-12T03:51:39.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34=', 'Content-Length': '0'} 2024-04-12T03:51:39.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212a3bb432c30358-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.567 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.567 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.567 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.567 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.567 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.567 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.567 DEBUG:boto:Data: 2024-04-12T03:51:39.568 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.568 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.568 DEBUG:boto:Params: {} 2024-04-12T03:51:39.568 DEBUG:boto:Token: None 2024-04-12T03:51:39.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2://mPTV6LYBCrFdzTxK0GZDgX1LE= 2024-04-12T03:51:39.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2://mPTV6LYBCrFdzTxK0GZDgX1LE=', 'Content-Length': '0'} 2024-04-12T03:51:39.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28b06b340a08eb7-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.570 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.570 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.570 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.570 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.570 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.571 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.571 DEBUG:boto:Data: 2024-04-12T03:51:39.571 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.571 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.571 DEBUG:boto:Params: {} 2024-04-12T03:51:39.571 DEBUG:boto:Token: None 2024-04-12T03:51:39.571 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2://mPTV6LYBCrFdzTxK0GZDgX1LE= 2024-04-12T03:51:39.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2://mPTV6LYBCrFdzTxK0GZDgX1LE=', 'Content-Length': '0'} 2024-04-12T03:51:39.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032cd7e99572fdc47-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:51:39.573 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.574 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.574 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.574 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.574 DEBUG:boto:Method: GET 2024-04-12T03:51:39.574 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.574 DEBUG:boto:Data: 2024-04-12T03:51:39.574 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.574 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.574 DEBUG:boto:Params: {} 2024-04-12T03:51:39.575 DEBUG:boto:Token: None 2024-04-12T03:51:39.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34= 2024-04-12T03:51:39.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34=', 'Content-Length': '0'} 2024-04-12T03:51:39.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18adad200200db7-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.577 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.577 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.577 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.577 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.577 DEBUG:boto:Method: GET 2024-04-12T03:51:39.577 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.577 DEBUG:boto:Data: 2024-04-12T03:51:39.577 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.577 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.578 DEBUG:boto:Params: {} 2024-04-12T03:51:39.578 DEBUG:boto:Token: None 2024-04-12T03:51:39.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:51:39.578 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34= 2024-04-12T03:51:39.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p/9PMhsdCI1kkF1tFPC+ZjJOm34=', 'Content-Length': '0'} 2024-04-12T03:51:39.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce6e0f69fdbdc1b-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.622 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.623 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.623 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.623 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.623 DEBUG:boto:Method: GET 2024-04-12T03:51:39.623 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.623 DEBUG:boto:Data: 2024-04-12T03:51:39.623 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.623 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.623 DEBUG:boto:Params: {} 2024-04-12T03:51:39.624 DEBUG:boto:Token: None 2024-04-12T03:51:39.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d92564b6e78017e-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.627 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.627 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.627 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.627 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.627 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.627 DEBUG:boto:Method: GET 2024-04-12T03:51:39.627 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.627 DEBUG:boto:Data: 2024-04-12T03:51:39.628 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.628 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.628 DEBUG:boto:Params: {} 2024-04-12T03:51:39.628 DEBUG:boto:Token: None 2024-04-12T03:51:39.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007fd49c9ebf9fc3c-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.631 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.632 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.632 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.632 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.632 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:51:39.632 DEBUG:boto:Data: 2024-04-12T03:51:39.632 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.632 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.632 DEBUG:boto:Params: {} 2024-04-12T03:51:39.633 DEBUG:boto:Token: None 2024-04-12T03:51:39.633 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155 2024-04-12T03:51:39.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+uegr4jj/LLXtQv1hDikVMC1g00= 2024-04-12T03:51:39.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+uegr4jj/LLXtQv1hDikVMC1g00=', 'Content-Length': '0'} 2024-04-12T03:51:39.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e4c305c5ef366c-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.634 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.635 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.635 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.635 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:51:39.635 DEBUG:boto:Data: 2024-04-12T03:51:39.635 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.635 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.635 DEBUG:boto:Params: {} 2024-04-12T03:51:39.635 DEBUG:boto:Token: None 2024-04-12T03:51:39.635 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155 2024-04-12T03:51:39.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+uegr4jj/LLXtQv1hDikVMC1g00= 2024-04-12T03:51:39.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+uegr4jj/LLXtQv1hDikVMC1g00=', 'Content-Length': '0'} 2024-04-12T03:51:39.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d55c3ebfe4605f-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:51:39.637 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.638 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.638 DEBUG:boto:Method: GET 2024-04-12T03:51:39.638 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:51:39.638 DEBUG:boto:Data: 2024-04-12T03:51:39.638 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.638 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.638 DEBUG:boto:Params: {} 2024-04-12T03:51:39.639 DEBUG:boto:Token: None 2024-04-12T03:51:39.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155 2024-04-12T03:51:39.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:no45hnDfFZUWM+PNwaZpwIH/5Y8= 2024-04-12T03:51:39.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:no45hnDfFZUWM+PNwaZpwIH/5Y8=', 'Content-Length': '0'} 2024-04-12T03:51:39.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000581e7a30115742db-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.682 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:51:39.682 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:51:39.683 DEBUG:boto:Method: GET 2024-04-12T03:51:39.683 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:51:39.683 DEBUG:boto:Data: 2024-04-12T03:51:39.683 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.683 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.683 DEBUG:boto:Params: {} 2024-04-12T03:51:39.683 DEBUG:boto:Token: None 2024-04-12T03:51:39.683 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj155 2024-04-12T03:51:39.683 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:no45hnDfFZUWM+PNwaZpwIH/5Y8= 2024-04-12T03:51:39.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:no45hnDfFZUWM+PNwaZpwIH/5Y8=', 'Content-Length': '0'} 2024-04-12T03:51:39.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb8f54dd2fe213f-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.685 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.686 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.686 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.686 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.686 DEBUG:boto:Method: GET 2024-04-12T03:51:39.686 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.686 DEBUG:boto:Data: 2024-04-12T03:51:39.686 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.686 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.686 DEBUG:boto:Params: {} 2024-04-12T03:51:39.687 DEBUG:boto:Token: None 2024-04-12T03:51:39.687 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.687 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ac3bf14475cc478-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.689 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.689 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.689 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.690 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.690 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.690 DEBUG:boto:Method: GET 2024-04-12T03:51:39.690 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:51:39.690 DEBUG:boto:Data: 2024-04-12T03:51:39.690 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.690 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.690 DEBUG:boto:Params: {} 2024-04-12T03:51:39.690 DEBUG:boto:Token: None 2024-04-12T03:51:39.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5f28e41972448c-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.696 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:51:39.697 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.697 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.697 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.697 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.697 DEBUG:boto:Method: GET 2024-04-12T03:51:39.697 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.697 DEBUG:boto:Data: 2024-04-12T03:51:39.697 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.697 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.698 DEBUG:boto:Params: {} 2024-04-12T03:51:39.698 DEBUG:boto:Token: None 2024-04-12T03:51:39.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ= 2024-04-12T03:51:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0a2d1b4e08dd2f-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.700 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.700 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.700 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.701 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.701 DEBUG:boto:Method: GET 2024-04-12T03:51:39.701 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.701 DEBUG:boto:Data: 2024-04-12T03:51:39.701 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.701 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.701 DEBUG:boto:Params: {} 2024-04-12T03:51:39.701 DEBUG:boto:Token: None 2024-04-12T03:51:39.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ= 2024-04-12T03:51:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004103e208c7caf93d-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.750 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.751 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.751 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.751 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.751 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.751 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.751 DEBUG:boto:Data: 2024-04-12T03:51:39.751 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.752 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.752 DEBUG:boto:Params: {} 2024-04-12T03:51:39.752 DEBUG:boto:Token: None 2024-04-12T03:51:39.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t9P85KHLNYGN70QCZU95yNmB6Kg= 2024-04-12T03:51:39.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t9P85KHLNYGN70QCZU95yNmB6Kg=', 'Content-Length': '0'} 2024-04-12T03:51:39.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a5225c7853a840-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.755 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.755 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.755 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.755 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.755 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.755 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.755 DEBUG:boto:Data: 2024-04-12T03:51:39.755 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.755 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.756 DEBUG:boto:Params: {} 2024-04-12T03:51:39.756 DEBUG:boto:Token: None 2024-04-12T03:51:39.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t9P85KHLNYGN70QCZU95yNmB6Kg= 2024-04-12T03:51:39.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t9P85KHLNYGN70QCZU95yNmB6Kg=', 'Content-Length': '0'} 2024-04-12T03:51:39.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000984b76812376edb1-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:51:39.758 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.758 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.758 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.758 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.758 DEBUG:boto:Method: GET 2024-04-12T03:51:39.759 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.759 DEBUG:boto:Data: 2024-04-12T03:51:39.759 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.759 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.759 DEBUG:boto:Params: {} 2024-04-12T03:51:39.759 DEBUG:boto:Token: None 2024-04-12T03:51:39.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ= 2024-04-12T03:51:39.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8f556f382a9901-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.762 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.762 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.762 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.762 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.762 DEBUG:boto:Method: GET 2024-04-12T03:51:39.762 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.762 DEBUG:boto:Data: 2024-04-12T03:51:39.762 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.763 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.763 DEBUG:boto:Params: {} 2024-04-12T03:51:39.763 DEBUG:boto:Token: None 2024-04-12T03:51:39.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:51:39.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ= 2024-04-12T03:51:39.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Y57iuX8RiguGJdxEW938iy74IQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd4e4c7e09ca09e9-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.765 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.765 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.765 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.765 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.766 DEBUG:boto:Method: GET 2024-04-12T03:51:39.766 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.766 DEBUG:boto:Data: 2024-04-12T03:51:39.766 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.766 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.766 DEBUG:boto:Params: {} 2024-04-12T03:51:39.766 DEBUG:boto:Token: None 2024-04-12T03:51:39.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592e0ce657eb1033-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.769 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.769 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.769 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.769 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.770 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.770 DEBUG:boto:Method: GET 2024-04-12T03:51:39.770 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.770 DEBUG:boto:Data: 2024-04-12T03:51:39.770 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.770 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.770 DEBUG:boto:Params: {} 2024-04-12T03:51:39.770 DEBUG:boto:Token: None 2024-04-12T03:51:39.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748cdcbb45ad807e-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.774 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.775 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.775 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.775 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.775 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:51:39.775 DEBUG:boto:Data: 2024-04-12T03:51:39.775 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.775 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.775 DEBUG:boto:Params: {} 2024-04-12T03:51:39.775 DEBUG:boto:Token: None 2024-04-12T03:51:39.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17 2024-04-12T03:51:39.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rOs1n4WFzJcKgFIK9JOxtO1JzoY= 2024-04-12T03:51:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rOs1n4WFzJcKgFIK9JOxtO1JzoY=', 'Content-Length': '0'} 2024-04-12T03:51:39.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6f2b90b96f95df-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.778 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.778 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.778 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.778 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:51:39.778 DEBUG:boto:Data: 2024-04-12T03:51:39.778 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.779 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.779 DEBUG:boto:Params: {} 2024-04-12T03:51:39.779 DEBUG:boto:Token: None 2024-04-12T03:51:39.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17 2024-04-12T03:51:39.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rOs1n4WFzJcKgFIK9JOxtO1JzoY= 2024-04-12T03:51:39.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rOs1n4WFzJcKgFIK9JOxtO1JzoY=', 'Content-Length': '0'} 2024-04-12T03:51:39.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006001e7da3a333dbc-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:51:39.781 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.781 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.781 DEBUG:boto:Method: GET 2024-04-12T03:51:39.781 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:51:39.781 DEBUG:boto:Data: 2024-04-12T03:51:39.781 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.782 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.782 DEBUG:boto:Params: {} 2024-04-12T03:51:39.782 DEBUG:boto:Token: None 2024-04-12T03:51:39.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17 2024-04-12T03:51:39.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H5iXsFnaElNFTBDHy3ODDrbUOMw= 2024-04-12T03:51:39.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H5iXsFnaElNFTBDHy3ODDrbUOMw=', 'Content-Length': '0'} 2024-04-12T03:51:39.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99571755e455c23-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.784 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:51:39.784 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:51:39.784 DEBUG:boto:Method: GET 2024-04-12T03:51:39.784 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:51:39.785 DEBUG:boto:Data: 2024-04-12T03:51:39.785 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.785 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.785 DEBUG:boto:Params: {} 2024-04-12T03:51:39.785 DEBUG:boto:Token: None 2024-04-12T03:51:39.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj17 2024-04-12T03:51:39.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H5iXsFnaElNFTBDHy3ODDrbUOMw= 2024-04-12T03:51:39.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H5iXsFnaElNFTBDHy3ODDrbUOMw=', 'Content-Length': '0'} 2024-04-12T03:51:39.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd8abe020960112-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.787 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.787 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.787 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.788 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.788 DEBUG:boto:Method: GET 2024-04-12T03:51:39.788 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.788 DEBUG:boto:Data: 2024-04-12T03:51:39.788 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.788 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.788 DEBUG:boto:Params: {} 2024-04-12T03:51:39.788 DEBUG:boto:Token: None 2024-04-12T03:51:39.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16c4d6954ee723d-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.791 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.791 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.792 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.792 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.792 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.792 DEBUG:boto:Method: GET 2024-04-12T03:51:39.792 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:51:39.792 DEBUG:boto:Data: 2024-04-12T03:51:39.792 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.792 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.792 DEBUG:boto:Params: {} 2024-04-12T03:51:39.792 DEBUG:boto:Token: None 2024-04-12T03:51:39.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14b2b9ab70e73cc-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.842 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:51:39.843 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.843 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.843 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.843 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.843 DEBUG:boto:Method: GET 2024-04-12T03:51:39.843 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.844 DEBUG:boto:Data: 2024-04-12T03:51:39.844 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.844 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.844 DEBUG:boto:Params: {} 2024-04-12T03:51:39.844 DEBUG:boto:Token: None 2024-04-12T03:51:39.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I= 2024-04-12T03:51:39.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I=', 'Content-Length': '0'} 2024-04-12T03:51:39.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9541dfafdabdd06-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.846 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.847 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.847 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.847 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.847 DEBUG:boto:Method: GET 2024-04-12T03:51:39.847 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.847 DEBUG:boto:Data: 2024-04-12T03:51:39.847 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.847 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.847 DEBUG:boto:Params: {} 2024-04-12T03:51:39.848 DEBUG:boto:Token: None 2024-04-12T03:51:39.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I= 2024-04-12T03:51:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I=', 'Content-Length': '0'} 2024-04-12T03:51:39.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b38b703640d15ad-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.850 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.850 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.850 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.850 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.850 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.851 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.851 DEBUG:boto:Data: 2024-04-12T03:51:39.851 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.851 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.851 DEBUG:boto:Params: {} 2024-04-12T03:51:39.851 DEBUG:boto:Token: None 2024-04-12T03:51:39.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KyO3OYodkH+DiPQpw3SHlCPyDh8= 2024-04-12T03:51:39.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KyO3OYodkH+DiPQpw3SHlCPyDh8=', 'Content-Length': '0'} 2024-04-12T03:51:39.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af4b5eda3d6d51a-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.854 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.854 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.854 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.854 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.854 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.854 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.854 DEBUG:boto:Data: 2024-04-12T03:51:39.854 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.854 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.855 DEBUG:boto:Params: {} 2024-04-12T03:51:39.855 DEBUG:boto:Token: None 2024-04-12T03:51:39.855 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KyO3OYodkH+DiPQpw3SHlCPyDh8= 2024-04-12T03:51:39.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KyO3OYodkH+DiPQpw3SHlCPyDh8=', 'Content-Length': '0'} 2024-04-12T03:51:39.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043aeca0981ea8f44-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:51:39.857 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.857 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.857 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.857 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.857 DEBUG:boto:Method: GET 2024-04-12T03:51:39.857 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.857 DEBUG:boto:Data: 2024-04-12T03:51:39.858 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.858 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.858 DEBUG:boto:Params: {} 2024-04-12T03:51:39.858 DEBUG:boto:Token: None 2024-04-12T03:51:39.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.858 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I= 2024-04-12T03:51:39.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I=', 'Content-Length': '0'} 2024-04-12T03:51:39.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0bfc9f5dbf6245-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.860 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.860 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.861 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.861 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.861 DEBUG:boto:Method: GET 2024-04-12T03:51:39.861 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.861 DEBUG:boto:Data: 2024-04-12T03:51:39.861 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.861 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.861 DEBUG:boto:Params: {} 2024-04-12T03:51:39.862 DEBUG:boto:Token: None 2024-04-12T03:51:39.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:51:39.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I= 2024-04-12T03:51:39.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5TUmLVgPPv7zWXaSlourqsdb47I=', 'Content-Length': '0'} 2024-04-12T03:51:39.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc48c68caaa760ac-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.864 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.864 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.864 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.864 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.865 DEBUG:boto:Method: GET 2024-04-12T03:51:39.865 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.865 DEBUG:boto:Data: 2024-04-12T03:51:39.865 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.865 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.865 DEBUG:boto:Params: {} 2024-04-12T03:51:39.865 DEBUG:boto:Token: None 2024-04-12T03:51:39.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798be8503a541db0-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.868 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.869 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.869 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.869 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.869 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.869 DEBUG:boto:Method: GET 2024-04-12T03:51:39.869 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.869 DEBUG:boto:Data: 2024-04-12T03:51:39.869 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.870 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.870 DEBUG:boto:Params: {} 2024-04-12T03:51:39.870 DEBUG:boto:Token: None 2024-04-12T03:51:39.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806aa96fb38b57fe-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.873 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.874 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.874 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.874 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.874 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:51:39.874 DEBUG:boto:Data: 2024-04-12T03:51:39.874 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.874 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.874 DEBUG:boto:Params: {} 2024-04-12T03:51:39.875 DEBUG:boto:Token: None 2024-04-12T03:51:39.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19 2024-04-12T03:51:39.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hqiNAs3b4OTM1h7gvJcpusaYudg= 2024-04-12T03:51:39.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hqiNAs3b4OTM1h7gvJcpusaYudg=', 'Content-Length': '0'} 2024-04-12T03:51:39.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6566b942e0abdd-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.877 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.877 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.877 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.877 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:51:39.877 DEBUG:boto:Data: 2024-04-12T03:51:39.878 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.878 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.878 DEBUG:boto:Params: {} 2024-04-12T03:51:39.878 DEBUG:boto:Token: None 2024-04-12T03:51:39.878 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19 2024-04-12T03:51:39.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hqiNAs3b4OTM1h7gvJcpusaYudg= 2024-04-12T03:51:39.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hqiNAs3b4OTM1h7gvJcpusaYudg=', 'Content-Length': '0'} 2024-04-12T03:51:39.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049911496017a076a-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:51:39.881 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.881 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.881 DEBUG:boto:Method: GET 2024-04-12T03:51:39.881 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:51:39.881 DEBUG:boto:Data: 2024-04-12T03:51:39.881 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.881 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.882 DEBUG:boto:Params: {} 2024-04-12T03:51:39.882 DEBUG:boto:Token: None 2024-04-12T03:51:39.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19 2024-04-12T03:51:39.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j0iuDK9zRFMOkwbSfkw9+LiglWc= 2024-04-12T03:51:39.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j0iuDK9zRFMOkwbSfkw9+LiglWc=', 'Content-Length': '0'} 2024-04-12T03:51:39.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69f96be54488b9d-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.884 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:51:39.884 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:51:39.884 DEBUG:boto:Method: GET 2024-04-12T03:51:39.884 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:51:39.884 DEBUG:boto:Data: 2024-04-12T03:51:39.884 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.885 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.885 DEBUG:boto:Params: {} 2024-04-12T03:51:39.885 DEBUG:boto:Token: None 2024-04-12T03:51:39.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj19 2024-04-12T03:51:39.885 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j0iuDK9zRFMOkwbSfkw9+LiglWc= 2024-04-12T03:51:39.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j0iuDK9zRFMOkwbSfkw9+LiglWc=', 'Content-Length': '0'} 2024-04-12T03:51:39.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a366e172e12984d-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.887 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.887 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.888 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.888 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.888 DEBUG:boto:Method: GET 2024-04-12T03:51:39.888 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.888 DEBUG:boto:Data: 2024-04-12T03:51:39.888 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.888 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.888 DEBUG:boto:Params: {} 2024-04-12T03:51:39.889 DEBUG:boto:Token: None 2024-04-12T03:51:39.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f4a7c0ea903eb15-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.891 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.892 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.892 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.892 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.892 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.892 DEBUG:boto:Method: GET 2024-04-12T03:51:39.892 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:51:39.892 DEBUG:boto:Data: 2024-04-12T03:51:39.892 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.893 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.893 DEBUG:boto:Params: {} 2024-04-12T03:51:39.893 DEBUG:boto:Token: None 2024-04-12T03:51:39.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68b870e8a75344e-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.896 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:51:39.897 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.897 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.897 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.897 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.897 DEBUG:boto:Method: GET 2024-04-12T03:51:39.897 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.898 DEBUG:boto:Data: 2024-04-12T03:51:39.898 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.898 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.898 DEBUG:boto:Params: {} 2024-04-12T03:51:39.898 DEBUG:boto:Token: None 2024-04-12T03:51:39.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y= 2024-04-12T03:51:39.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1bc8e82dd8812b-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.900 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.901 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.901 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.901 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.901 DEBUG:boto:Method: GET 2024-04-12T03:51:39.901 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.901 DEBUG:boto:Data: 2024-04-12T03:51:39.901 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.901 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.901 DEBUG:boto:Params: {} 2024-04-12T03:51:39.902 DEBUG:boto:Token: None 2024-04-12T03:51:39.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y= 2024-04-12T03:51:39.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce08b0e2012b817-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.946 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.947 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.947 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.947 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.947 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.947 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.947 DEBUG:boto:Data: 2024-04-12T03:51:39.948 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.948 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.948 DEBUG:boto:Params: {} 2024-04-12T03:51:39.948 DEBUG:boto:Token: None 2024-04-12T03:51:39.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:whp2wSFhsE/J4vDWwdLvQnKE2SQ= 2024-04-12T03:51:39.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:whp2wSFhsE/J4vDWwdLvQnKE2SQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ea72294e3094a0-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.950 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.950 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.950 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.951 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.951 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.951 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.951 DEBUG:boto:Data: 2024-04-12T03:51:39.951 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.951 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.951 DEBUG:boto:Params: {} 2024-04-12T03:51:39.951 DEBUG:boto:Token: None 2024-04-12T03:51:39.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:whp2wSFhsE/J4vDWwdLvQnKE2SQ= 2024-04-12T03:51:39.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:whp2wSFhsE/J4vDWwdLvQnKE2SQ=', 'Content-Length': '0'} 2024-04-12T03:51:39.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005146cb22386a77ca-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:51:39.954 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.954 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.954 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.954 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.954 DEBUG:boto:Method: GET 2024-04-12T03:51:39.954 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.954 DEBUG:boto:Data: 2024-04-12T03:51:39.954 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.955 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.955 DEBUG:boto:Params: {} 2024-04-12T03:51:39.955 DEBUG:boto:Token: None 2024-04-12T03:51:39.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y= 2024-04-12T03:51:39.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b1dda653d90e37-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.957 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.957 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.958 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.958 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.958 DEBUG:boto:Method: GET 2024-04-12T03:51:39.958 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.958 DEBUG:boto:Data: 2024-04-12T03:51:39.958 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.958 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.958 DEBUG:boto:Params: {} 2024-04-12T03:51:39.959 DEBUG:boto:Token: None 2024-04-12T03:51:39.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:51:39.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y= 2024-04-12T03:51:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O4unuZsDosnWP7cvhe0UqVntB2Y=', 'Content-Length': '0'} 2024-04-12T03:51:39.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81ba573ddac0029-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.961 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.962 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.962 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.962 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.962 DEBUG:boto:Method: GET 2024-04-12T03:51:39.962 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.962 DEBUG:boto:Data: 2024-04-12T03:51:39.962 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.963 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.963 DEBUG:boto:Params: {} 2024-04-12T03:51:39.963 DEBUG:boto:Token: None 2024-04-12T03:51:39.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb69fea7e9f8b0d-006618afcb-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.967 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.968 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:39.968 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:39.968 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.968 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.968 DEBUG:boto:Method: GET 2024-04-12T03:51:39.969 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:51:39.969 DEBUG:boto:Data: 2024-04-12T03:51:39.969 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.969 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.969 DEBUG:boto:Params: {} 2024-04-12T03:51:39.969 DEBUG:boto:Token: None 2024-04-12T03:51:39.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/ 2024-04-12T03:51:39.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U= 2024-04-12T03:51:39.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vK2RAqQc9zMp6ONUFLXX8V45F2U=', 'Content-Length': '0'} 2024-04-12T03:51:39.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cf4d21f8680c31-006618afcb-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.973 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:39.973 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.973 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.973 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.973 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:51:39.974 DEBUG:boto:Data: 2024-04-12T03:51:39.974 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.974 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.974 DEBUG:boto:Params: {} 2024-04-12T03:51:39.974 DEBUG:boto:Token: None 2024-04-12T03:51:39.974 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22 2024-04-12T03:51:39.974 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eqfQsEv+Z/oV1zhxZMe+pilWePE= 2024-04-12T03:51:39.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eqfQsEv+Z/oV1zhxZMe+pilWePE=', 'Content-Length': '0'} 2024-04-12T03:51:39.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0b793a9a5cb812-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.976 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.976 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.976 DEBUG:boto:Method: HEAD 2024-04-12T03:51:39.977 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:51:39.977 DEBUG:boto:Data: 2024-04-12T03:51:39.977 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:39.977 DEBUG:boto:Port: 8000 2024-04-12T03:51:39.977 DEBUG:boto:Params: {} 2024-04-12T03:51:39.977 DEBUG:boto:Token: None 2024-04-12T03:51:39.977 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22 2024-04-12T03:51:39.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eqfQsEv+Z/oV1zhxZMe+pilWePE= 2024-04-12T03:51:39.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eqfQsEv+Z/oV1zhxZMe+pilWePE=', 'Content-Length': '0'} 2024-04-12T03:51:39.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf9125fbe68edbe-006618afcb-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:39.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:51:39.979 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:39.979 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:39.980 DEBUG:boto:Method: GET 2024-04-12T03:51:39.980 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:51:39.980 DEBUG:boto:Data: 2024-04-12T03:51:39.980 DEBUG:boto:Headers: {} 2024-04-12T03:51:39.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:39.980 DEBUG:boto:Port: 8001 2024-04-12T03:51:39.980 DEBUG:boto:Params: {} 2024-04-12T03:51:39.980 DEBUG:boto:Token: None 2024-04-12T03:51:39.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:39 GMT /dgbpxg-61/obj22 2024-04-12T03:51:39.981 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XaF0UKaHe0E+MjIFTW2EghOLit0= 2024-04-12T03:51:39.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XaF0UKaHe0E+MjIFTW2EghOLit0=', 'Content-Length': '0'} 2024-04-12T03:51:39.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49aa4b4b6ffb64c-006618afcb-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.026 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:51:40.027 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:51:40.027 DEBUG:boto:Method: GET 2024-04-12T03:51:40.027 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:51:40.027 DEBUG:boto:Data: 2024-04-12T03:51:40.027 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.027 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.027 DEBUG:boto:Params: {} 2024-04-12T03:51:40.027 DEBUG:boto:Token: None 2024-04-12T03:51:40.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj22 2024-04-12T03:51:40.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fi2qFNQmpLOxioFf+b0EXTrYcb0= 2024-04-12T03:51:40.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fi2qFNQmpLOxioFf+b0EXTrYcb0=', 'Content-Length': '0'} 2024-04-12T03:51:40.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faa8871ab1f9226-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.030 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.030 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.030 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.030 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.030 DEBUG:boto:Method: GET 2024-04-12T03:51:40.031 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.031 DEBUG:boto:Data: 2024-04-12T03:51:40.031 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.031 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.031 DEBUG:boto:Params: {} 2024-04-12T03:51:40.031 DEBUG:boto:Token: None 2024-04-12T03:51:40.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b2d88ca282d1ea-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.033 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.034 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.034 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.034 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.034 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.034 DEBUG:boto:Method: GET 2024-04-12T03:51:40.034 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:51:40.034 DEBUG:boto:Data: 2024-04-12T03:51:40.035 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.035 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.035 DEBUG:boto:Params: {} 2024-04-12T03:51:40.035 DEBUG:boto:Token: None 2024-04-12T03:51:40.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d34b7eceb5d5b0-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.038 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:51:40.039 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.039 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.039 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.039 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.039 DEBUG:boto:Method: GET 2024-04-12T03:51:40.039 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.039 DEBUG:boto:Data: 2024-04-12T03:51:40.040 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.040 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.040 DEBUG:boto:Params: {} 2024-04-12T03:51:40.040 DEBUG:boto:Token: None 2024-04-12T03:51:40.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA= 2024-04-12T03:51:40.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA=', 'Content-Length': '0'} 2024-04-12T03:51:40.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fc2743b0fda6bd-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.042 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.042 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.042 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.043 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.043 DEBUG:boto:Method: GET 2024-04-12T03:51:40.043 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.043 DEBUG:boto:Data: 2024-04-12T03:51:40.043 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.043 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.044 DEBUG:boto:Params: {} 2024-04-12T03:51:40.044 DEBUG:boto:Token: None 2024-04-12T03:51:40.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA= 2024-04-12T03:51:40.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA=', 'Content-Length': '0'} 2024-04-12T03:51:40.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded33bb700a15cf2-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.086 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.086 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.087 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.087 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.087 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.087 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.087 DEBUG:boto:Data: 2024-04-12T03:51:40.087 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.087 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.087 DEBUG:boto:Params: {} 2024-04-12T03:51:40.087 DEBUG:boto:Token: None 2024-04-12T03:51:40.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0T14AOvqVjo4NQFSAtWV2PXwBwQ= 2024-04-12T03:51:40.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0T14AOvqVjo4NQFSAtWV2PXwBwQ=', 'Content-Length': '0'} 2024-04-12T03:51:40.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d937b3484aee141-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.090 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.090 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.090 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.090 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.090 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.090 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.090 DEBUG:boto:Data: 2024-04-12T03:51:40.090 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.091 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.091 DEBUG:boto:Params: {} 2024-04-12T03:51:40.091 DEBUG:boto:Token: None 2024-04-12T03:51:40.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.091 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0T14AOvqVjo4NQFSAtWV2PXwBwQ= 2024-04-12T03:51:40.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0T14AOvqVjo4NQFSAtWV2PXwBwQ=', 'Content-Length': '0'} 2024-04-12T03:51:40.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f0b6c8a5212d73-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:51:40.093 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.093 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.093 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.093 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.093 DEBUG:boto:Method: GET 2024-04-12T03:51:40.093 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.093 DEBUG:boto:Data: 2024-04-12T03:51:40.094 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.094 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.094 DEBUG:boto:Params: {} 2024-04-12T03:51:40.094 DEBUG:boto:Token: None 2024-04-12T03:51:40.094 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA= 2024-04-12T03:51:40.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA=', 'Content-Length': '0'} 2024-04-12T03:51:40.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa06826e7add313-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.096 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.096 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.097 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.097 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.097 DEBUG:boto:Method: GET 2024-04-12T03:51:40.097 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.097 DEBUG:boto:Data: 2024-04-12T03:51:40.097 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.097 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.097 DEBUG:boto:Params: {} 2024-04-12T03:51:40.097 DEBUG:boto:Token: None 2024-04-12T03:51:40.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:51:40.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA= 2024-04-12T03:51:40.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QTNQw7wGkjmKbF5d0Fwwy1m7uoA=', 'Content-Length': '0'} 2024-04-12T03:51:40.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6fc99e0e7c60b0c-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.100 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.100 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.100 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.100 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.100 DEBUG:boto:Method: GET 2024-04-12T03:51:40.100 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.100 DEBUG:boto:Data: 2024-04-12T03:51:40.100 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.100 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.101 DEBUG:boto:Params: {} 2024-04-12T03:51:40.101 DEBUG:boto:Token: None 2024-04-12T03:51:40.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033026575789d2c15-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.103 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.104 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.104 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.104 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.104 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.104 DEBUG:boto:Method: GET 2024-04-12T03:51:40.104 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.104 DEBUG:boto:Data: 2024-04-12T03:51:40.104 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.105 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.105 DEBUG:boto:Params: {} 2024-04-12T03:51:40.105 DEBUG:boto:Token: None 2024-04-12T03:51:40.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4fa904488438820-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.109 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.109 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.109 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.109 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.109 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:51:40.109 DEBUG:boto:Data: 2024-04-12T03:51:40.110 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.110 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.110 DEBUG:boto:Params: {} 2024-04-12T03:51:40.110 DEBUG:boto:Token: None 2024-04-12T03:51:40.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23 2024-04-12T03:51:40.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+x66J5qXQmF9P6QFHLDYEK+opk= 2024-04-12T03:51:40.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+x66J5qXQmF9P6QFHLDYEK+opk=', 'Content-Length': '0'} 2024-04-12T03:51:40.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007104e1b72aeba011-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.112 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.113 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.113 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.113 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:51:40.113 DEBUG:boto:Data: 2024-04-12T03:51:40.113 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.113 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.113 DEBUG:boto:Params: {} 2024-04-12T03:51:40.113 DEBUG:boto:Token: None 2024-04-12T03:51:40.113 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23 2024-04-12T03:51:40.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+x66J5qXQmF9P6QFHLDYEK+opk= 2024-04-12T03:51:40.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+x66J5qXQmF9P6QFHLDYEK+opk=', 'Content-Length': '0'} 2024-04-12T03:51:40.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb8e9641ae1f093f-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:51:40.115 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.116 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.116 DEBUG:boto:Method: GET 2024-04-12T03:51:40.116 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:51:40.116 DEBUG:boto:Data: 2024-04-12T03:51:40.116 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.116 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.116 DEBUG:boto:Params: {} 2024-04-12T03:51:40.116 DEBUG:boto:Token: None 2024-04-12T03:51:40.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23 2024-04-12T03:51:40.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GZtDu9r39pl5r/W596YR+7QeVJY= 2024-04-12T03:51:40.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GZtDu9r39pl5r/W596YR+7QeVJY=', 'Content-Length': '0'} 2024-04-12T03:51:40.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af15ca03c2aef04a-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.118 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:51:40.118 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:51:40.119 DEBUG:boto:Method: GET 2024-04-12T03:51:40.119 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:51:40.119 DEBUG:boto:Data: 2024-04-12T03:51:40.119 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.119 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.119 DEBUG:boto:Params: {} 2024-04-12T03:51:40.119 DEBUG:boto:Token: None 2024-04-12T03:51:40.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj23 2024-04-12T03:51:40.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GZtDu9r39pl5r/W596YR+7QeVJY= 2024-04-12T03:51:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GZtDu9r39pl5r/W596YR+7QeVJY=', 'Content-Length': '0'} 2024-04-12T03:51:40.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004232a4cc0cf95177-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.162 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.162 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.163 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.163 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.163 DEBUG:boto:Method: GET 2024-04-12T03:51:40.163 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.163 DEBUG:boto:Data: 2024-04-12T03:51:40.163 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.163 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.163 DEBUG:boto:Params: {} 2024-04-12T03:51:40.163 DEBUG:boto:Token: None 2024-04-12T03:51:40.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9cef7803e7d6a7d-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.166 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.166 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.167 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.167 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.167 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.167 DEBUG:boto:Method: GET 2024-04-12T03:51:40.167 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:51:40.167 DEBUG:boto:Data: 2024-04-12T03:51:40.167 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.167 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.167 DEBUG:boto:Params: {} 2024-04-12T03:51:40.168 DEBUG:boto:Token: None 2024-04-12T03:51:40.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.168 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624edb2244ef6ae8-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.171 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:51:40.173 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.173 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.173 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.173 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.173 DEBUG:boto:Method: GET 2024-04-12T03:51:40.173 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.174 DEBUG:boto:Data: 2024-04-12T03:51:40.174 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.174 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.174 DEBUG:boto:Params: {} 2024-04-12T03:51:40.174 DEBUG:boto:Token: None 2024-04-12T03:51:40.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.174 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w= 2024-04-12T03:51:40.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w=', 'Content-Length': '0'} 2024-04-12T03:51:40.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176646e999da55a5-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.176 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.176 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.176 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.176 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.177 DEBUG:boto:Method: GET 2024-04-12T03:51:40.177 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.177 DEBUG:boto:Data: 2024-04-12T03:51:40.177 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.177 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.177 DEBUG:boto:Params: {} 2024-04-12T03:51:40.177 DEBUG:boto:Token: None 2024-04-12T03:51:40.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.177 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w= 2024-04-12T03:51:40.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w=', 'Content-Length': '0'} 2024-04-12T03:51:40.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007276c1f55b966f1e-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.180 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.180 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.180 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.180 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.180 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.180 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.180 DEBUG:boto:Data: 2024-04-12T03:51:40.180 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.181 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.181 DEBUG:boto:Params: {} 2024-04-12T03:51:40.181 DEBUG:boto:Token: None 2024-04-12T03:51:40.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lZ+hmFHmEtmKsjqUSzPVRSg+ORk= 2024-04-12T03:51:40.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lZ+hmFHmEtmKsjqUSzPVRSg+ORk=', 'Content-Length': '0'} 2024-04-12T03:51:40.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d311ac9b0e729ecb-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.183 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.183 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.183 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.183 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.184 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.184 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.184 DEBUG:boto:Data: 2024-04-12T03:51:40.184 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.184 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.184 DEBUG:boto:Params: {} 2024-04-12T03:51:40.184 DEBUG:boto:Token: None 2024-04-12T03:51:40.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lZ+hmFHmEtmKsjqUSzPVRSg+ORk= 2024-04-12T03:51:40.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lZ+hmFHmEtmKsjqUSzPVRSg+ORk=', 'Content-Length': '0'} 2024-04-12T03:51:40.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f04386260147e80-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:51:40.186 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.186 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.187 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.187 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.187 DEBUG:boto:Method: GET 2024-04-12T03:51:40.187 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.187 DEBUG:boto:Data: 2024-04-12T03:51:40.187 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.187 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.187 DEBUG:boto:Params: {} 2024-04-12T03:51:40.187 DEBUG:boto:Token: None 2024-04-12T03:51:40.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w= 2024-04-12T03:51:40.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w=', 'Content-Length': '0'} 2024-04-12T03:51:40.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096efd2b05a259ee2-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.234 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.234 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.235 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.235 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.235 DEBUG:boto:Method: GET 2024-04-12T03:51:40.235 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.235 DEBUG:boto:Data: 2024-04-12T03:51:40.235 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.235 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.235 DEBUG:boto:Params: {} 2024-04-12T03:51:40.235 DEBUG:boto:Token: None 2024-04-12T03:51:40.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:51:40.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w= 2024-04-12T03:51:40.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S3/CBjq5rBvNUl20IYHxkQ2Zt0w=', 'Content-Length': '0'} 2024-04-12T03:51:40.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82781fe52f44daf-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.238 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.238 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.238 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.238 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.238 DEBUG:boto:Method: GET 2024-04-12T03:51:40.238 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.238 DEBUG:boto:Data: 2024-04-12T03:51:40.238 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.238 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.239 DEBUG:boto:Params: {} 2024-04-12T03:51:40.239 DEBUG:boto:Token: None 2024-04-12T03:51:40.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045371c8035850a97-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.241 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.242 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.242 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.242 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.242 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.242 DEBUG:boto:Method: GET 2024-04-12T03:51:40.242 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.242 DEBUG:boto:Data: 2024-04-12T03:51:40.242 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.243 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.243 DEBUG:boto:Params: {} 2024-04-12T03:51:40.243 DEBUG:boto:Token: None 2024-04-12T03:51:40.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000774631b465bb7e43-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.246 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.247 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.247 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.247 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.247 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:51:40.247 DEBUG:boto:Data: 2024-04-12T03:51:40.247 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.247 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.247 DEBUG:boto:Params: {} 2024-04-12T03:51:40.248 DEBUG:boto:Token: None 2024-04-12T03:51:40.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29 2024-04-12T03:51:40.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KXzZ8IUCVUEdu3NWpOvhqEieoNM= 2024-04-12T03:51:40.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KXzZ8IUCVUEdu3NWpOvhqEieoNM=', 'Content-Length': '0'} 2024-04-12T03:51:40.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034b8d561529f4c0f-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.250 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.250 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.250 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.250 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:51:40.250 DEBUG:boto:Data: 2024-04-12T03:51:40.250 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.250 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.250 DEBUG:boto:Params: {} 2024-04-12T03:51:40.251 DEBUG:boto:Token: None 2024-04-12T03:51:40.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29 2024-04-12T03:51:40.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KXzZ8IUCVUEdu3NWpOvhqEieoNM= 2024-04-12T03:51:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KXzZ8IUCVUEdu3NWpOvhqEieoNM=', 'Content-Length': '0'} 2024-04-12T03:51:40.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005286fe3a6f1ec7a8-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:51:40.253 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.253 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.253 DEBUG:boto:Method: GET 2024-04-12T03:51:40.253 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:51:40.253 DEBUG:boto:Data: 2024-04-12T03:51:40.253 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.253 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.253 DEBUG:boto:Params: {} 2024-04-12T03:51:40.254 DEBUG:boto:Token: None 2024-04-12T03:51:40.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29 2024-04-12T03:51:40.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N30v/0uQzL2SM4b6gyqF69En7XY= 2024-04-12T03:51:40.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N30v/0uQzL2SM4b6gyqF69En7XY=', 'Content-Length': '0'} 2024-04-12T03:51:40.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b89dc0141aa9bb-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.256 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:51:40.256 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:51:40.256 DEBUG:boto:Method: GET 2024-04-12T03:51:40.256 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:51:40.256 DEBUG:boto:Data: 2024-04-12T03:51:40.256 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.256 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.256 DEBUG:boto:Params: {} 2024-04-12T03:51:40.257 DEBUG:boto:Token: None 2024-04-12T03:51:40.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj29 2024-04-12T03:51:40.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N30v/0uQzL2SM4b6gyqF69En7XY= 2024-04-12T03:51:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N30v/0uQzL2SM4b6gyqF69En7XY=', 'Content-Length': '0'} 2024-04-12T03:51:40.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000007ed9e90e30cff-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.259 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.259 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.259 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.259 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.259 DEBUG:boto:Method: GET 2024-04-12T03:51:40.259 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.259 DEBUG:boto:Data: 2024-04-12T03:51:40.259 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.260 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.260 DEBUG:boto:Params: {} 2024-04-12T03:51:40.260 DEBUG:boto:Token: None 2024-04-12T03:51:40.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5d2cde98e515d66-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.262 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.263 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.263 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.263 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.263 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.263 DEBUG:boto:Method: GET 2024-04-12T03:51:40.263 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:51:40.263 DEBUG:boto:Data: 2024-04-12T03:51:40.263 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.263 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.263 DEBUG:boto:Params: {} 2024-04-12T03:51:40.264 DEBUG:boto:Token: None 2024-04-12T03:51:40.264 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000528c501da98fddf-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.267 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:51:40.267 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.268 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.268 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.268 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.268 DEBUG:boto:Method: GET 2024-04-12T03:51:40.268 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.268 DEBUG:boto:Data: 2024-04-12T03:51:40.268 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.268 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.268 DEBUG:boto:Params: {} 2024-04-12T03:51:40.269 DEBUG:boto:Token: None 2024-04-12T03:51:40.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck= 2024-04-12T03:51:40.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck=', 'Content-Length': '0'} 2024-04-12T03:51:40.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015c12e00a461e59b-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.271 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.271 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.271 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.271 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.272 DEBUG:boto:Method: GET 2024-04-12T03:51:40.272 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.272 DEBUG:boto:Data: 2024-04-12T03:51:40.272 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.272 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.272 DEBUG:boto:Params: {} 2024-04-12T03:51:40.272 DEBUG:boto:Token: None 2024-04-12T03:51:40.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck= 2024-04-12T03:51:40.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck=', 'Content-Length': '0'} 2024-04-12T03:51:40.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94cca08fb986092-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.314 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.314 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.315 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.315 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.315 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.315 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.315 DEBUG:boto:Data: 2024-04-12T03:51:40.315 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.315 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.315 DEBUG:boto:Params: {} 2024-04-12T03:51:40.315 DEBUG:boto:Token: None 2024-04-12T03:51:40.315 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:76JQv2WSKhVY5yEJuKXS4pvQmjA= 2024-04-12T03:51:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:76JQv2WSKhVY5yEJuKXS4pvQmjA=', 'Content-Length': '0'} 2024-04-12T03:51:40.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f78377afd2cc27-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.318 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.318 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.318 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.318 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.318 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.318 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.318 DEBUG:boto:Data: 2024-04-12T03:51:40.318 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.319 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.319 DEBUG:boto:Params: {} 2024-04-12T03:51:40.319 DEBUG:boto:Token: None 2024-04-12T03:51:40.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:76JQv2WSKhVY5yEJuKXS4pvQmjA= 2024-04-12T03:51:40.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:76JQv2WSKhVY5yEJuKXS4pvQmjA=', 'Content-Length': '0'} 2024-04-12T03:51:40.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f17ecf53b2e8e32-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:51:40.321 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.321 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.321 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.321 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.321 DEBUG:boto:Method: GET 2024-04-12T03:51:40.321 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.321 DEBUG:boto:Data: 2024-04-12T03:51:40.322 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.322 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.322 DEBUG:boto:Params: {} 2024-04-12T03:51:40.322 DEBUG:boto:Token: None 2024-04-12T03:51:40.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck= 2024-04-12T03:51:40.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck=', 'Content-Length': '0'} 2024-04-12T03:51:40.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a36a41349c176b7c-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.324 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.324 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.324 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.324 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.325 DEBUG:boto:Method: GET 2024-04-12T03:51:40.325 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.325 DEBUG:boto:Data: 2024-04-12T03:51:40.325 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.325 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.325 DEBUG:boto:Params: {} 2024-04-12T03:51:40.325 DEBUG:boto:Token: None 2024-04-12T03:51:40.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:51:40.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck= 2024-04-12T03:51:40.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XRZGhfdkiqpSiGAYgOGFhZtfpck=', 'Content-Length': '0'} 2024-04-12T03:51:40.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ba8755672d9e2a-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.327 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.327 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.327 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.328 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.328 DEBUG:boto:Method: GET 2024-04-12T03:51:40.328 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.328 DEBUG:boto:Data: 2024-04-12T03:51:40.328 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.328 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.328 DEBUG:boto:Params: {} 2024-04-12T03:51:40.328 DEBUG:boto:Token: None 2024-04-12T03:51:40.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047656b9d0963c236-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.331 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.331 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.331 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.331 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.331 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.331 DEBUG:boto:Method: GET 2024-04-12T03:51:40.331 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.332 DEBUG:boto:Data: 2024-04-12T03:51:40.332 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.332 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.332 DEBUG:boto:Params: {} 2024-04-12T03:51:40.332 DEBUG:boto:Token: None 2024-04-12T03:51:40.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3cc27fad191c28-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.336 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.336 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.336 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.336 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.336 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:51:40.336 DEBUG:boto:Data: 2024-04-12T03:51:40.336 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.337 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.337 DEBUG:boto:Params: {} 2024-04-12T03:51:40.337 DEBUG:boto:Token: None 2024-04-12T03:51:40.337 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31 2024-04-12T03:51:40.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl/kKJ5lEoHip2GrAneT1MHHvLY= 2024-04-12T03:51:40.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl/kKJ5lEoHip2GrAneT1MHHvLY=', 'Content-Length': '0'} 2024-04-12T03:51:40.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b80ff0855cb5f68-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.339 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.339 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.339 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.339 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:51:40.339 DEBUG:boto:Data: 2024-04-12T03:51:40.339 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.339 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.340 DEBUG:boto:Params: {} 2024-04-12T03:51:40.340 DEBUG:boto:Token: None 2024-04-12T03:51:40.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31 2024-04-12T03:51:40.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl/kKJ5lEoHip2GrAneT1MHHvLY= 2024-04-12T03:51:40.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl/kKJ5lEoHip2GrAneT1MHHvLY=', 'Content-Length': '0'} 2024-04-12T03:51:40.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbe33c4c37a3c15-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:51:40.342 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.342 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.342 DEBUG:boto:Method: GET 2024-04-12T03:51:40.342 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:51:40.342 DEBUG:boto:Data: 2024-04-12T03:51:40.342 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.343 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.343 DEBUG:boto:Params: {} 2024-04-12T03:51:40.343 DEBUG:boto:Token: None 2024-04-12T03:51:40.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31 2024-04-12T03:51:40.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kfojRnCUg9L3G6ksfFQJuPGevJ4= 2024-04-12T03:51:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kfojRnCUg9L3G6ksfFQJuPGevJ4=', 'Content-Length': '0'} 2024-04-12T03:51:40.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a43b2713c0c978-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.386 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:51:40.386 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:51:40.387 DEBUG:boto:Method: GET 2024-04-12T03:51:40.387 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:51:40.387 DEBUG:boto:Data: 2024-04-12T03:51:40.387 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.387 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.387 DEBUG:boto:Params: {} 2024-04-12T03:51:40.387 DEBUG:boto:Token: None 2024-04-12T03:51:40.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj31 2024-04-12T03:51:40.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kfojRnCUg9L3G6ksfFQJuPGevJ4= 2024-04-12T03:51:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kfojRnCUg9L3G6ksfFQJuPGevJ4=', 'Content-Length': '0'} 2024-04-12T03:51:40.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b814a44a460c670b-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.390 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.390 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.390 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.390 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.390 DEBUG:boto:Method: GET 2024-04-12T03:51:40.390 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.390 DEBUG:boto:Data: 2024-04-12T03:51:40.391 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.391 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.391 DEBUG:boto:Params: {} 2024-04-12T03:51:40.391 DEBUG:boto:Token: None 2024-04-12T03:51:40.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f17bc97025f205f-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.394 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.394 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.394 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.394 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.394 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.395 DEBUG:boto:Method: GET 2024-04-12T03:51:40.395 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:51:40.395 DEBUG:boto:Data: 2024-04-12T03:51:40.395 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.395 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.395 DEBUG:boto:Params: {} 2024-04-12T03:51:40.395 DEBUG:boto:Token: None 2024-04-12T03:51:40.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f667af5c491d5b-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.399 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:51:40.399 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.399 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.399 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.399 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.399 DEBUG:boto:Method: GET 2024-04-12T03:51:40.399 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.400 DEBUG:boto:Data: 2024-04-12T03:51:40.400 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.400 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.400 DEBUG:boto:Params: {} 2024-04-12T03:51:40.400 DEBUG:boto:Token: None 2024-04-12T03:51:40.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw= 2024-04-12T03:51:40.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw=', 'Content-Length': '0'} 2024-04-12T03:51:40.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05f3dc738287937-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.402 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.402 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.403 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.403 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.403 DEBUG:boto:Method: GET 2024-04-12T03:51:40.403 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.403 DEBUG:boto:Data: 2024-04-12T03:51:40.403 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.403 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.403 DEBUG:boto:Params: {} 2024-04-12T03:51:40.404 DEBUG:boto:Token: None 2024-04-12T03:51:40.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw= 2024-04-12T03:51:40.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw=', 'Content-Length': '0'} 2024-04-12T03:51:40.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970e3d59f3853080-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.406 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.406 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.406 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.406 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.407 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.407 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.407 DEBUG:boto:Data: 2024-04-12T03:51:40.407 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.407 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.407 DEBUG:boto:Params: {} 2024-04-12T03:51:40.407 DEBUG:boto:Token: None 2024-04-12T03:51:40.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGJ8eqaPGSTKB8v8FmXe80m60mI= 2024-04-12T03:51:40.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGJ8eqaPGSTKB8v8FmXe80m60mI=', 'Content-Length': '0'} 2024-04-12T03:51:40.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e6bcb44f45b58d-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.409 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.409 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.410 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.410 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.410 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.410 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.410 DEBUG:boto:Data: 2024-04-12T03:51:40.410 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.410 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.410 DEBUG:boto:Params: {} 2024-04-12T03:51:40.410 DEBUG:boto:Token: None 2024-04-12T03:51:40.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGJ8eqaPGSTKB8v8FmXe80m60mI= 2024-04-12T03:51:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGJ8eqaPGSTKB8v8FmXe80m60mI=', 'Content-Length': '0'} 2024-04-12T03:51:40.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605c71e829045025-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:51:40.413 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.413 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.413 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.413 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.413 DEBUG:boto:Method: GET 2024-04-12T03:51:40.413 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.413 DEBUG:boto:Data: 2024-04-12T03:51:40.413 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.414 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.414 DEBUG:boto:Params: {} 2024-04-12T03:51:40.414 DEBUG:boto:Token: None 2024-04-12T03:51:40.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw= 2024-04-12T03:51:40.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw=', 'Content-Length': '0'} 2024-04-12T03:51:40.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001cc099c6de5de95-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.416 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.416 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.416 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.416 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.416 DEBUG:boto:Method: GET 2024-04-12T03:51:40.416 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.417 DEBUG:boto:Data: 2024-04-12T03:51:40.417 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.417 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.417 DEBUG:boto:Params: {} 2024-04-12T03:51:40.417 DEBUG:boto:Token: None 2024-04-12T03:51:40.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:51:40.417 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw= 2024-04-12T03:51:40.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQgSRYsXByhP+yxLypLL8PbpAnw=', 'Content-Length': '0'} 2024-04-12T03:51:40.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f414315a631b6028-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.419 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.419 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.419 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.419 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.419 DEBUG:boto:Method: GET 2024-04-12T03:51:40.420 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.420 DEBUG:boto:Data: 2024-04-12T03:51:40.420 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.420 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.420 DEBUG:boto:Params: {} 2024-04-12T03:51:40.420 DEBUG:boto:Token: None 2024-04-12T03:51:40.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.420 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb8b87269ca4375-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.423 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.423 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.423 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.423 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.423 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.424 DEBUG:boto:Method: GET 2024-04-12T03:51:40.424 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.424 DEBUG:boto:Data: 2024-04-12T03:51:40.424 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.424 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.424 DEBUG:boto:Params: {} 2024-04-12T03:51:40.424 DEBUG:boto:Token: None 2024-04-12T03:51:40.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d39d16fe8e304a9d-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.427 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.428 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.428 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.428 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.428 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:51:40.428 DEBUG:boto:Data: 2024-04-12T03:51:40.428 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.428 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.428 DEBUG:boto:Params: {} 2024-04-12T03:51:40.428 DEBUG:boto:Token: None 2024-04-12T03:51:40.429 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33 2024-04-12T03:51:40.429 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88dgHvuWgCAiNWt7yXhhzdqxu0E= 2024-04-12T03:51:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88dgHvuWgCAiNWt7yXhhzdqxu0E=', 'Content-Length': '0'} 2024-04-12T03:51:40.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5470fb423965cc0-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.430 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.430 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.431 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.431 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:51:40.431 DEBUG:boto:Data: 2024-04-12T03:51:40.431 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.431 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.431 DEBUG:boto:Params: {} 2024-04-12T03:51:40.431 DEBUG:boto:Token: None 2024-04-12T03:51:40.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33 2024-04-12T03:51:40.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88dgHvuWgCAiNWt7yXhhzdqxu0E= 2024-04-12T03:51:40.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88dgHvuWgCAiNWt7yXhhzdqxu0E=', 'Content-Length': '0'} 2024-04-12T03:51:40.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d5df73832abd004-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:51:40.433 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.434 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.434 DEBUG:boto:Method: GET 2024-04-12T03:51:40.434 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:51:40.434 DEBUG:boto:Data: 2024-04-12T03:51:40.434 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.434 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.434 DEBUG:boto:Params: {} 2024-04-12T03:51:40.434 DEBUG:boto:Token: None 2024-04-12T03:51:40.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33 2024-04-12T03:51:40.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8kuSVzyPOA2h8vAqvhoOgsQy/SA= 2024-04-12T03:51:40.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8kuSVzyPOA2h8vAqvhoOgsQy/SA=', 'Content-Length': '0'} 2024-04-12T03:51:40.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb8d6fd9973841a-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.436 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:51:40.436 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:51:40.436 DEBUG:boto:Method: GET 2024-04-12T03:51:40.437 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:51:40.437 DEBUG:boto:Data: 2024-04-12T03:51:40.437 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.437 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.437 DEBUG:boto:Params: {} 2024-04-12T03:51:40.437 DEBUG:boto:Token: None 2024-04-12T03:51:40.437 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj33 2024-04-12T03:51:40.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8kuSVzyPOA2h8vAqvhoOgsQy/SA= 2024-04-12T03:51:40.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8kuSVzyPOA2h8vAqvhoOgsQy/SA=', 'Content-Length': '0'} 2024-04-12T03:51:40.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022614712b77da968-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.482 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.482 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.483 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.483 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.483 DEBUG:boto:Method: GET 2024-04-12T03:51:40.483 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.483 DEBUG:boto:Data: 2024-04-12T03:51:40.483 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.483 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.483 DEBUG:boto:Params: {} 2024-04-12T03:51:40.483 DEBUG:boto:Token: None 2024-04-12T03:51:40.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d1d44e7a4ee38a-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.486 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.486 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.487 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.487 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.487 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.487 DEBUG:boto:Method: GET 2024-04-12T03:51:40.487 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:51:40.487 DEBUG:boto:Data: 2024-04-12T03:51:40.487 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.487 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.487 DEBUG:boto:Params: {} 2024-04-12T03:51:40.487 DEBUG:boto:Token: None 2024-04-12T03:51:40.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e7a03187f06888-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.491 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:51:40.491 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.491 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.491 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.491 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.492 DEBUG:boto:Method: GET 2024-04-12T03:51:40.492 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.492 DEBUG:boto:Data: 2024-04-12T03:51:40.492 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.492 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.492 DEBUG:boto:Params: {} 2024-04-12T03:51:40.492 DEBUG:boto:Token: None 2024-04-12T03:51:40.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM= 2024-04-12T03:51:40.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM=', 'Content-Length': '0'} 2024-04-12T03:51:40.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004483b3e6b3224100-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.495 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.495 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.495 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.495 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.495 DEBUG:boto:Method: GET 2024-04-12T03:51:40.495 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.495 DEBUG:boto:Data: 2024-04-12T03:51:40.495 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.495 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.496 DEBUG:boto:Params: {} 2024-04-12T03:51:40.496 DEBUG:boto:Token: None 2024-04-12T03:51:40.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM= 2024-04-12T03:51:40.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM=', 'Content-Length': '0'} 2024-04-12T03:51:40.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a0284fc2ece9d4-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.498 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.498 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.498 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.498 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.499 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.499 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.499 DEBUG:boto:Data: 2024-04-12T03:51:40.499 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.499 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.499 DEBUG:boto:Params: {} 2024-04-12T03:51:40.499 DEBUG:boto:Token: None 2024-04-12T03:51:40.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LF0UztkkNP/WQf3k/bIjgmWEZU8= 2024-04-12T03:51:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LF0UztkkNP/WQf3k/bIjgmWEZU8=', 'Content-Length': '0'} 2024-04-12T03:51:40.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8873a4f89f15a8a-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.501 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.501 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.502 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.502 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.502 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.502 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.502 DEBUG:boto:Data: 2024-04-12T03:51:40.502 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.502 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.502 DEBUG:boto:Params: {} 2024-04-12T03:51:40.502 DEBUG:boto:Token: None 2024-04-12T03:51:40.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LF0UztkkNP/WQf3k/bIjgmWEZU8= 2024-04-12T03:51:40.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LF0UztkkNP/WQf3k/bIjgmWEZU8=', 'Content-Length': '0'} 2024-04-12T03:51:40.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd661283fb23a99-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:51:40.505 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.505 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.505 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.505 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.505 DEBUG:boto:Method: GET 2024-04-12T03:51:40.505 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.505 DEBUG:boto:Data: 2024-04-12T03:51:40.505 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.506 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.506 DEBUG:boto:Params: {} 2024-04-12T03:51:40.506 DEBUG:boto:Token: None 2024-04-12T03:51:40.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM= 2024-04-12T03:51:40.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM=', 'Content-Length': '0'} 2024-04-12T03:51:40.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070fc21892e0c88bc-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.550 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.550 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.551 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.551 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.551 DEBUG:boto:Method: GET 2024-04-12T03:51:40.551 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.551 DEBUG:boto:Data: 2024-04-12T03:51:40.551 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.551 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.551 DEBUG:boto:Params: {} 2024-04-12T03:51:40.551 DEBUG:boto:Token: None 2024-04-12T03:51:40.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:51:40.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM= 2024-04-12T03:51:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNe3Jao2En1gdWjJVl7vXYaF4zM=', 'Content-Length': '0'} 2024-04-12T03:51:40.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b036d2b8fdb0566a-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.554 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.554 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.554 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.554 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.554 DEBUG:boto:Method: GET 2024-04-12T03:51:40.554 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.554 DEBUG:boto:Data: 2024-04-12T03:51:40.554 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.554 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.555 DEBUG:boto:Params: {} 2024-04-12T03:51:40.555 DEBUG:boto:Token: None 2024-04-12T03:51:40.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f24d42fd93b4e10-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.557 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.558 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.558 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.558 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.558 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.558 DEBUG:boto:Method: GET 2024-04-12T03:51:40.558 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.558 DEBUG:boto:Data: 2024-04-12T03:51:40.558 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.559 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.559 DEBUG:boto:Params: {} 2024-04-12T03:51:40.559 DEBUG:boto:Token: None 2024-04-12T03:51:40.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.559 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5da943e8b0e3100-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.562 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.563 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.563 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.563 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.563 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:51:40.563 DEBUG:boto:Data: 2024-04-12T03:51:40.563 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.564 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.564 DEBUG:boto:Params: {} 2024-04-12T03:51:40.564 DEBUG:boto:Token: None 2024-04-12T03:51:40.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34 2024-04-12T03:51:40.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mE0IZDKwfVexGA4zosTkdnbPbmA= 2024-04-12T03:51:40.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mE0IZDKwfVexGA4zosTkdnbPbmA=', 'Content-Length': '0'} 2024-04-12T03:51:40.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f16e2b312c9c1d-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.566 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.566 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.566 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.566 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:51:40.566 DEBUG:boto:Data: 2024-04-12T03:51:40.567 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.567 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.567 DEBUG:boto:Params: {} 2024-04-12T03:51:40.567 DEBUG:boto:Token: None 2024-04-12T03:51:40.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34 2024-04-12T03:51:40.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mE0IZDKwfVexGA4zosTkdnbPbmA= 2024-04-12T03:51:40.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mE0IZDKwfVexGA4zosTkdnbPbmA=', 'Content-Length': '0'} 2024-04-12T03:51:40.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5788eb7260e3ed-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:51:40.569 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.569 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.569 DEBUG:boto:Method: GET 2024-04-12T03:51:40.570 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:51:40.570 DEBUG:boto:Data: 2024-04-12T03:51:40.570 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.570 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.570 DEBUG:boto:Params: {} 2024-04-12T03:51:40.570 DEBUG:boto:Token: None 2024-04-12T03:51:40.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34 2024-04-12T03:51:40.570 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kq6yIlMpMbnqtnFAg/UFslWQioc= 2024-04-12T03:51:40.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kq6yIlMpMbnqtnFAg/UFslWQioc=', 'Content-Length': '0'} 2024-04-12T03:51:40.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc29247e11946853-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.572 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:51:40.573 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:51:40.573 DEBUG:boto:Method: GET 2024-04-12T03:51:40.573 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:51:40.573 DEBUG:boto:Data: 2024-04-12T03:51:40.573 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.573 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.573 DEBUG:boto:Params: {} 2024-04-12T03:51:40.573 DEBUG:boto:Token: None 2024-04-12T03:51:40.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj34 2024-04-12T03:51:40.574 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kq6yIlMpMbnqtnFAg/UFslWQioc= 2024-04-12T03:51:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kq6yIlMpMbnqtnFAg/UFslWQioc=', 'Content-Length': '0'} 2024-04-12T03:51:40.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4eba57341ca6298-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.618 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.619 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.619 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.619 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.619 DEBUG:boto:Method: GET 2024-04-12T03:51:40.619 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.619 DEBUG:boto:Data: 2024-04-12T03:51:40.619 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.620 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.620 DEBUG:boto:Params: {} 2024-04-12T03:51:40.620 DEBUG:boto:Token: None 2024-04-12T03:51:40.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0d95ae9b895227-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.622 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.623 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.623 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.623 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.623 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.623 DEBUG:boto:Method: GET 2024-04-12T03:51:40.623 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:51:40.623 DEBUG:boto:Data: 2024-04-12T03:51:40.623 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.624 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.624 DEBUG:boto:Params: {} 2024-04-12T03:51:40.624 DEBUG:boto:Token: None 2024-04-12T03:51:40.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d5d1aae4192371-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.628 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:51:40.628 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.628 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.628 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.628 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.628 DEBUG:boto:Method: GET 2024-04-12T03:51:40.629 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.629 DEBUG:boto:Data: 2024-04-12T03:51:40.629 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.629 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.629 DEBUG:boto:Params: {} 2024-04-12T03:51:40.629 DEBUG:boto:Token: None 2024-04-12T03:51:40.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o= 2024-04-12T03:51:40.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o=', 'Content-Length': '0'} 2024-04-12T03:51:40.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e76e78606e55ff-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.631 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.632 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.632 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.632 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.632 DEBUG:boto:Method: GET 2024-04-12T03:51:40.632 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.632 DEBUG:boto:Data: 2024-04-12T03:51:40.632 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.632 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.632 DEBUG:boto:Params: {} 2024-04-12T03:51:40.632 DEBUG:boto:Token: None 2024-04-12T03:51:40.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o= 2024-04-12T03:51:40.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o=', 'Content-Length': '0'} 2024-04-12T03:51:40.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3149a99ef149575-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.635 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.635 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.635 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.635 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.635 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.635 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.635 DEBUG:boto:Data: 2024-04-12T03:51:40.635 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.635 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.635 DEBUG:boto:Params: {} 2024-04-12T03:51:40.636 DEBUG:boto:Token: None 2024-04-12T03:51:40.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:itYjawHfLBZQtB4H4zuofQbcu9c= 2024-04-12T03:51:40.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:itYjawHfLBZQtB4H4zuofQbcu9c=', 'Content-Length': '0'} 2024-04-12T03:51:40.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000837d2bc60f974ba6-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.638 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.638 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.638 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.638 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.638 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.638 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.638 DEBUG:boto:Data: 2024-04-12T03:51:40.639 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.639 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.639 DEBUG:boto:Params: {} 2024-04-12T03:51:40.639 DEBUG:boto:Token: None 2024-04-12T03:51:40.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:itYjawHfLBZQtB4H4zuofQbcu9c= 2024-04-12T03:51:40.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:itYjawHfLBZQtB4H4zuofQbcu9c=', 'Content-Length': '0'} 2024-04-12T03:51:40.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9ec763d1efe6fd-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:51:40.641 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.641 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.641 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.641 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.641 DEBUG:boto:Method: GET 2024-04-12T03:51:40.641 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.641 DEBUG:boto:Data: 2024-04-12T03:51:40.641 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.642 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.642 DEBUG:boto:Params: {} 2024-04-12T03:51:40.642 DEBUG:boto:Token: None 2024-04-12T03:51:40.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o= 2024-04-12T03:51:40.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o=', 'Content-Length': '0'} 2024-04-12T03:51:40.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0cee27459b28a42-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.686 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.686 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.687 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.687 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.687 DEBUG:boto:Method: GET 2024-04-12T03:51:40.687 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.687 DEBUG:boto:Data: 2024-04-12T03:51:40.687 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.687 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.687 DEBUG:boto:Params: {} 2024-04-12T03:51:40.687 DEBUG:boto:Token: None 2024-04-12T03:51:40.687 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:51:40.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o= 2024-04-12T03:51:40.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88hGTabcce1WTeS9ndWFCguTD7o=', 'Content-Length': '0'} 2024-04-12T03:51:40.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e6c2c9a064d59d-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.690 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.690 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.690 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.690 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.691 DEBUG:boto:Method: GET 2024-04-12T03:51:40.691 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.691 DEBUG:boto:Data: 2024-04-12T03:51:40.691 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.691 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.691 DEBUG:boto:Params: {} 2024-04-12T03:51:40.691 DEBUG:boto:Token: None 2024-04-12T03:51:40.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.691 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dbd6c2c071e360f-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.694 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.694 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.694 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.694 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.694 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.695 DEBUG:boto:Method: GET 2024-04-12T03:51:40.695 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.695 DEBUG:boto:Data: 2024-04-12T03:51:40.695 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.695 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.695 DEBUG:boto:Params: {} 2024-04-12T03:51:40.695 DEBUG:boto:Token: None 2024-04-12T03:51:40.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fac72bde692d43f-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.699 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.699 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.699 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.699 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.699 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:51:40.699 DEBUG:boto:Data: 2024-04-12T03:51:40.699 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.700 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.700 DEBUG:boto:Params: {} 2024-04-12T03:51:40.700 DEBUG:boto:Token: None 2024-04-12T03:51:40.700 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38 2024-04-12T03:51:40.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5FvFMHggxtSl8qytlcR/04qwP4= 2024-04-12T03:51:40.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5FvFMHggxtSl8qytlcR/04qwP4=', 'Content-Length': '0'} 2024-04-12T03:51:40.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000161ae5d2a7617eab-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.702 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.702 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.702 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.703 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:51:40.703 DEBUG:boto:Data: 2024-04-12T03:51:40.703 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.703 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.703 DEBUG:boto:Params: {} 2024-04-12T03:51:40.703 DEBUG:boto:Token: None 2024-04-12T03:51:40.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38 2024-04-12T03:51:40.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B5FvFMHggxtSl8qytlcR/04qwP4= 2024-04-12T03:51:40.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B5FvFMHggxtSl8qytlcR/04qwP4=', 'Content-Length': '0'} 2024-04-12T03:51:40.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc7178d3c9c00a9-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:51:40.705 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.705 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.705 DEBUG:boto:Method: GET 2024-04-12T03:51:40.705 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:51:40.705 DEBUG:boto:Data: 2024-04-12T03:51:40.706 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.706 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.706 DEBUG:boto:Params: {} 2024-04-12T03:51:40.706 DEBUG:boto:Token: None 2024-04-12T03:51:40.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38 2024-04-12T03:51:40.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5Y2SoiZf1fqQCtLLU8hbTw9wog= 2024-04-12T03:51:40.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5Y2SoiZf1fqQCtLLU8hbTw9wog=', 'Content-Length': '0'} 2024-04-12T03:51:40.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3fbd389e414b7d-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.708 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:51:40.708 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:51:40.708 DEBUG:boto:Method: GET 2024-04-12T03:51:40.708 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:51:40.708 DEBUG:boto:Data: 2024-04-12T03:51:40.709 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.709 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.709 DEBUG:boto:Params: {} 2024-04-12T03:51:40.709 DEBUG:boto:Token: None 2024-04-12T03:51:40.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj38 2024-04-12T03:51:40.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R5Y2SoiZf1fqQCtLLU8hbTw9wog= 2024-04-12T03:51:40.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R5Y2SoiZf1fqQCtLLU8hbTw9wog=', 'Content-Length': '0'} 2024-04-12T03:51:40.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2df92269ca1079-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.711 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.711 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.711 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.712 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.712 DEBUG:boto:Method: GET 2024-04-12T03:51:40.712 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.712 DEBUG:boto:Data: 2024-04-12T03:51:40.712 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.712 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.712 DEBUG:boto:Params: {} 2024-04-12T03:51:40.712 DEBUG:boto:Token: None 2024-04-12T03:51:40.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.713 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff966c335add79b0-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.715 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.716 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.716 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.716 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.716 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.716 DEBUG:boto:Method: GET 2024-04-12T03:51:40.716 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:51:40.716 DEBUG:boto:Data: 2024-04-12T03:51:40.716 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.716 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.717 DEBUG:boto:Params: {} 2024-04-12T03:51:40.717 DEBUG:boto:Token: None 2024-04-12T03:51:40.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170c9ea79ed5ec35-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.720 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:51:40.721 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.721 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.721 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.721 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.721 DEBUG:boto:Method: GET 2024-04-12T03:51:40.721 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.721 DEBUG:boto:Data: 2024-04-12T03:51:40.721 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.721 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.722 DEBUG:boto:Params: {} 2024-04-12T03:51:40.722 DEBUG:boto:Token: None 2024-04-12T03:51:40.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.722 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0= 2024-04-12T03:51:40.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0=', 'Content-Length': '0'} 2024-04-12T03:51:40.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009968087c146c50bd-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.724 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.724 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.724 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.724 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.724 DEBUG:boto:Method: GET 2024-04-12T03:51:40.724 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.724 DEBUG:boto:Data: 2024-04-12T03:51:40.725 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.725 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.725 DEBUG:boto:Params: {} 2024-04-12T03:51:40.725 DEBUG:boto:Token: None 2024-04-12T03:51:40.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0= 2024-04-12T03:51:40.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0=', 'Content-Length': '0'} 2024-04-12T03:51:40.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859580c29e1d1449-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.774 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.775 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.775 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.775 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.775 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.775 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.775 DEBUG:boto:Data: 2024-04-12T03:51:40.775 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.775 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.775 DEBUG:boto:Params: {} 2024-04-12T03:51:40.775 DEBUG:boto:Token: None 2024-04-12T03:51:40.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tnJSjwWr51w22CXvgvEAADKyQeg= 2024-04-12T03:51:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tnJSjwWr51w22CXvgvEAADKyQeg=', 'Content-Length': '0'} 2024-04-12T03:51:40.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a41af07938ec16e-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.778 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.778 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.778 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.778 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.778 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.778 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.778 DEBUG:boto:Data: 2024-04-12T03:51:40.778 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.778 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.778 DEBUG:boto:Params: {} 2024-04-12T03:51:40.779 DEBUG:boto:Token: None 2024-04-12T03:51:40.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tnJSjwWr51w22CXvgvEAADKyQeg= 2024-04-12T03:51:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tnJSjwWr51w22CXvgvEAADKyQeg=', 'Content-Length': '0'} 2024-04-12T03:51:40.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e024d7ac250713c4-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:51:40.784 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.784 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.784 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.784 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.784 DEBUG:boto:Method: GET 2024-04-12T03:51:40.785 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.785 DEBUG:boto:Data: 2024-04-12T03:51:40.785 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.785 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.785 DEBUG:boto:Params: {} 2024-04-12T03:51:40.785 DEBUG:boto:Token: None 2024-04-12T03:51:40.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0= 2024-04-12T03:51:40.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0=', 'Content-Length': '0'} 2024-04-12T03:51:40.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002442d385ad8f1d94-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.787 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.787 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.787 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.787 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.788 DEBUG:boto:Method: GET 2024-04-12T03:51:40.788 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.788 DEBUG:boto:Data: 2024-04-12T03:51:40.788 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.788 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.788 DEBUG:boto:Params: {} 2024-04-12T03:51:40.788 DEBUG:boto:Token: None 2024-04-12T03:51:40.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:51:40.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0= 2024-04-12T03:51:40.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hJCRvGzhajnY2HLZKUNrmPueNw0=', 'Content-Length': '0'} 2024-04-12T03:51:40.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98787e8bbbcf3f8-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.790 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.790 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.790 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.790 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.791 DEBUG:boto:Method: GET 2024-04-12T03:51:40.791 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.791 DEBUG:boto:Data: 2024-04-12T03:51:40.791 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.791 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.791 DEBUG:boto:Params: {} 2024-04-12T03:51:40.791 DEBUG:boto:Token: None 2024-04-12T03:51:40.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aeed392df4eb4f3-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.838 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.839 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.839 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.839 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.839 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.839 DEBUG:boto:Method: GET 2024-04-12T03:51:40.839 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.839 DEBUG:boto:Data: 2024-04-12T03:51:40.839 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.840 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.840 DEBUG:boto:Params: {} 2024-04-12T03:51:40.840 DEBUG:boto:Token: None 2024-04-12T03:51:40.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf5c471ee947fa0-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.844 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.844 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.844 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.844 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.845 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:51:40.845 DEBUG:boto:Data: 2024-04-12T03:51:40.845 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.845 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.845 DEBUG:boto:Params: {} 2024-04-12T03:51:40.845 DEBUG:boto:Token: None 2024-04-12T03:51:40.845 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44 2024-04-12T03:51:40.845 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hFFUM3EMr8PUhvaP7pUhYVrqsRw= 2024-04-12T03:51:40.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hFFUM3EMr8PUhvaP7pUhYVrqsRw=', 'Content-Length': '0'} 2024-04-12T03:51:40.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ff9cb6b315ad9c-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.847 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.847 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.847 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.847 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:51:40.847 DEBUG:boto:Data: 2024-04-12T03:51:40.847 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.848 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.848 DEBUG:boto:Params: {} 2024-04-12T03:51:40.848 DEBUG:boto:Token: None 2024-04-12T03:51:40.848 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44 2024-04-12T03:51:40.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hFFUM3EMr8PUhvaP7pUhYVrqsRw= 2024-04-12T03:51:40.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hFFUM3EMr8PUhvaP7pUhYVrqsRw=', 'Content-Length': '0'} 2024-04-12T03:51:40.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e1a8d4e29fd194-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:51:40.850 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.850 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.850 DEBUG:boto:Method: GET 2024-04-12T03:51:40.850 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:51:40.850 DEBUG:boto:Data: 2024-04-12T03:51:40.850 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.851 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.851 DEBUG:boto:Params: {} 2024-04-12T03:51:40.851 DEBUG:boto:Token: None 2024-04-12T03:51:40.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44 2024-04-12T03:51:40.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Olykv8/cOu6CUtxRV2qDfwZV54= 2024-04-12T03:51:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Olykv8/cOu6CUtxRV2qDfwZV54=', 'Content-Length': '0'} 2024-04-12T03:51:40.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1556527357dce3-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.853 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:51:40.853 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:51:40.853 DEBUG:boto:Method: GET 2024-04-12T03:51:40.853 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:51:40.853 DEBUG:boto:Data: 2024-04-12T03:51:40.853 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.853 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.853 DEBUG:boto:Params: {} 2024-04-12T03:51:40.854 DEBUG:boto:Token: None 2024-04-12T03:51:40.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj44 2024-04-12T03:51:40.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Olykv8/cOu6CUtxRV2qDfwZV54= 2024-04-12T03:51:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Olykv8/cOu6CUtxRV2qDfwZV54=', 'Content-Length': '0'} 2024-04-12T03:51:40.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819f574b81bd2ed3-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.856 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.856 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.856 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.856 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.856 DEBUG:boto:Method: GET 2024-04-12T03:51:40.856 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.856 DEBUG:boto:Data: 2024-04-12T03:51:40.856 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.856 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.856 DEBUG:boto:Params: {} 2024-04-12T03:51:40.857 DEBUG:boto:Token: None 2024-04-12T03:51:40.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7177c90118b0e54-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.859 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.859 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.859 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.860 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.860 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.860 DEBUG:boto:Method: GET 2024-04-12T03:51:40.860 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:51:40.860 DEBUG:boto:Data: 2024-04-12T03:51:40.860 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.860 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.860 DEBUG:boto:Params: {} 2024-04-12T03:51:40.860 DEBUG:boto:Token: None 2024-04-12T03:51:40.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da90c96bc4efede6-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.864 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:51:40.864 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.864 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.864 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.864 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.864 DEBUG:boto:Method: GET 2024-04-12T03:51:40.864 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.865 DEBUG:boto:Data: 2024-04-12T03:51:40.865 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.865 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.865 DEBUG:boto:Params: {} 2024-04-12T03:51:40.865 DEBUG:boto:Token: None 2024-04-12T03:51:40.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI= 2024-04-12T03:51:40.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI=', 'Content-Length': '0'} 2024-04-12T03:51:40.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f2c3d60a64ca92-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.914 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.915 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.915 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.915 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.915 DEBUG:boto:Method: GET 2024-04-12T03:51:40.915 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.915 DEBUG:boto:Data: 2024-04-12T03:51:40.915 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.915 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.915 DEBUG:boto:Params: {} 2024-04-12T03:51:40.916 DEBUG:boto:Token: None 2024-04-12T03:51:40.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI= 2024-04-12T03:51:40.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI=', 'Content-Length': '0'} 2024-04-12T03:51:40.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038613161885de7cf-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.918 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.918 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.918 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.918 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.919 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.919 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.919 DEBUG:boto:Data: 2024-04-12T03:51:40.919 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.919 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.919 DEBUG:boto:Params: {} 2024-04-12T03:51:40.919 DEBUG:boto:Token: None 2024-04-12T03:51:40.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fF3KH5mfKhMdvFOytsX9hlONAx4= 2024-04-12T03:51:40.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fF3KH5mfKhMdvFOytsX9hlONAx4=', 'Content-Length': '0'} 2024-04-12T03:51:40.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e031a94a381030a-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.922 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.922 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.922 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.922 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.923 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.923 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.923 DEBUG:boto:Data: 2024-04-12T03:51:40.923 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.923 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.923 DEBUG:boto:Params: {} 2024-04-12T03:51:40.923 DEBUG:boto:Token: None 2024-04-12T03:51:40.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fF3KH5mfKhMdvFOytsX9hlONAx4= 2024-04-12T03:51:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fF3KH5mfKhMdvFOytsX9hlONAx4=', 'Content-Length': '0'} 2024-04-12T03:51:40.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7d4047541c9dd2-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:51:40.925 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.925 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.926 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.926 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.926 DEBUG:boto:Method: GET 2024-04-12T03:51:40.926 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.926 DEBUG:boto:Data: 2024-04-12T03:51:40.926 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.926 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.926 DEBUG:boto:Params: {} 2024-04-12T03:51:40.926 DEBUG:boto:Token: None 2024-04-12T03:51:40.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI= 2024-04-12T03:51:40.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI=', 'Content-Length': '0'} 2024-04-12T03:51:40.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c1ff8e19db4b1a-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.929 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.929 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.929 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.929 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.929 DEBUG:boto:Method: GET 2024-04-12T03:51:40.929 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.929 DEBUG:boto:Data: 2024-04-12T03:51:40.929 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.930 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.930 DEBUG:boto:Params: {} 2024-04-12T03:51:40.930 DEBUG:boto:Token: None 2024-04-12T03:51:40.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:51:40.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI= 2024-04-12T03:51:40.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m6igQDt5448au8gWkMOSL+oLPmI=', 'Content-Length': '0'} 2024-04-12T03:51:40.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ad8cf69b4f3b84-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.932 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.932 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.932 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.932 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.932 DEBUG:boto:Method: GET 2024-04-12T03:51:40.933 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.933 DEBUG:boto:Data: 2024-04-12T03:51:40.933 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.933 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.933 DEBUG:boto:Params: {} 2024-04-12T03:51:40.933 DEBUG:boto:Token: None 2024-04-12T03:51:40.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003505ce574c45257a-006618afcc-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.935 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.936 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.936 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.936 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.936 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.936 DEBUG:boto:Method: GET 2024-04-12T03:51:40.936 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.936 DEBUG:boto:Data: 2024-04-12T03:51:40.936 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.937 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.937 DEBUG:boto:Params: {} 2024-04-12T03:51:40.937 DEBUG:boto:Token: None 2024-04-12T03:51:40.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9fd44143d35e18-006618afcc-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.940 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.941 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.941 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.941 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.941 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:51:40.941 DEBUG:boto:Data: 2024-04-12T03:51:40.941 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.941 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.942 DEBUG:boto:Params: {} 2024-04-12T03:51:40.942 DEBUG:boto:Token: None 2024-04-12T03:51:40.942 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46 2024-04-12T03:51:40.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:myCWLYhiApFQWE/lfHDsFVFbt9Q= 2024-04-12T03:51:40.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:myCWLYhiApFQWE/lfHDsFVFbt9Q=', 'Content-Length': '0'} 2024-04-12T03:51:40.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb135b879b74fe9-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.944 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.944 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.944 DEBUG:boto:Method: HEAD 2024-04-12T03:51:40.944 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:51:40.944 DEBUG:boto:Data: 2024-04-12T03:51:40.944 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.945 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.945 DEBUG:boto:Params: {} 2024-04-12T03:51:40.945 DEBUG:boto:Token: None 2024-04-12T03:51:40.945 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46 2024-04-12T03:51:40.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:myCWLYhiApFQWE/lfHDsFVFbt9Q= 2024-04-12T03:51:40.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:myCWLYhiApFQWE/lfHDsFVFbt9Q=', 'Content-Length': '0'} 2024-04-12T03:51:40.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3aac1dbb0921104-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:51:40.947 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.947 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.947 DEBUG:boto:Method: GET 2024-04-12T03:51:40.947 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:51:40.947 DEBUG:boto:Data: 2024-04-12T03:51:40.948 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.948 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.948 DEBUG:boto:Params: {} 2024-04-12T03:51:40.948 DEBUG:boto:Token: None 2024-04-12T03:51:40.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46 2024-04-12T03:51:40.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GipAluEaJVDZ4IXozpRbdNv/8Ck= 2024-04-12T03:51:40.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GipAluEaJVDZ4IXozpRbdNv/8Ck=', 'Content-Length': '0'} 2024-04-12T03:51:40.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ac2e2d9d69fffa-006618afcc-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.950 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:51:40.950 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:51:40.950 DEBUG:boto:Method: GET 2024-04-12T03:51:40.951 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:51:40.951 DEBUG:boto:Data: 2024-04-12T03:51:40.951 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.951 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.951 DEBUG:boto:Params: {} 2024-04-12T03:51:40.951 DEBUG:boto:Token: None 2024-04-12T03:51:40.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/obj46 2024-04-12T03:51:40.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GipAluEaJVDZ4IXozpRbdNv/8Ck= 2024-04-12T03:51:40.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GipAluEaJVDZ4IXozpRbdNv/8Ck=', 'Content-Length': '0'} 2024-04-12T03:51:40.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115f3c34c156d9ca-006618afcc-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.994 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.995 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.995 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.995 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.995 DEBUG:boto:Method: GET 2024-04-12T03:51:40.995 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.995 DEBUG:boto:Data: 2024-04-12T03:51:40.995 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:40.995 DEBUG:boto:Port: 8001 2024-04-12T03:51:40.995 DEBUG:boto:Params: {} 2024-04-12T03:51:40.996 DEBUG:boto:Token: None 2024-04-12T03:51:40.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:40 GMT /dgbpxg-61/ 2024-04-12T03:51:40.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0= 2024-04-12T03:51:40.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:68J2YUUoFXwKNOmyzT1CKJwo2E0=', 'Content-Length': '0'} 2024-04-12T03:51:40.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec540f99546808b-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:40.998 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:40.999 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:40.999 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:40.999 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.999 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.999 DEBUG:boto:Method: GET 2024-04-12T03:51:40.999 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:51:40.999 DEBUG:boto:Data: 2024-04-12T03:51:40.999 DEBUG:boto:Headers: {} 2024-04-12T03:51:40.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:40.999 DEBUG:boto:Port: 8000 2024-04-12T03:51:40.999 DEBUG:boto:Params: {} 2024-04-12T03:51:41.000 DEBUG:boto:Token: None 2024-04-12T03:51:41.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cf6424da3237aa-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.003 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:51:41.003 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.004 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.004 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.004 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.004 DEBUG:boto:Method: GET 2024-04-12T03:51:41.004 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.004 DEBUG:boto:Data: 2024-04-12T03:51:41.004 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.004 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.004 DEBUG:boto:Params: {} 2024-04-12T03:51:41.005 DEBUG:boto:Token: None 2024-04-12T03:51:41.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.005 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4= 2024-04-12T03:51:41.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4=', 'Content-Length': '0'} 2024-04-12T03:51:41.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dadd2fae75e835-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.007 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.007 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.007 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.007 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.007 DEBUG:boto:Method: GET 2024-04-12T03:51:41.007 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.007 DEBUG:boto:Data: 2024-04-12T03:51:41.007 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.008 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.008 DEBUG:boto:Params: {} 2024-04-12T03:51:41.008 DEBUG:boto:Token: None 2024-04-12T03:51:41.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4= 2024-04-12T03:51:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4=', 'Content-Length': '0'} 2024-04-12T03:51:41.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed5fb354968556d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.010 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.010 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.010 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.010 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.011 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.011 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.011 DEBUG:boto:Data: 2024-04-12T03:51:41.011 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.011 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.011 DEBUG:boto:Params: {} 2024-04-12T03:51:41.011 DEBUG:boto:Token: None 2024-04-12T03:51:41.011 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9x5P7NO7j2kBuU1uUlWKgA0bd3g= 2024-04-12T03:51:41.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9x5P7NO7j2kBuU1uUlWKgA0bd3g=', 'Content-Length': '0'} 2024-04-12T03:51:41.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e786025c826466-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.013 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.014 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.014 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.014 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.014 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.014 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.014 DEBUG:boto:Data: 2024-04-12T03:51:41.014 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.014 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.014 DEBUG:boto:Params: {} 2024-04-12T03:51:41.015 DEBUG:boto:Token: None 2024-04-12T03:51:41.015 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.015 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9x5P7NO7j2kBuU1uUlWKgA0bd3g= 2024-04-12T03:51:41.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9x5P7NO7j2kBuU1uUlWKgA0bd3g=', 'Content-Length': '0'} 2024-04-12T03:51:41.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b1ba3ae5ac693d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:51:41.017 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.017 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.017 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.017 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.017 DEBUG:boto:Method: GET 2024-04-12T03:51:41.017 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.017 DEBUG:boto:Data: 2024-04-12T03:51:41.017 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.017 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.017 DEBUG:boto:Params: {} 2024-04-12T03:51:41.018 DEBUG:boto:Token: None 2024-04-12T03:51:41.018 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.018 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4= 2024-04-12T03:51:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4=', 'Content-Length': '0'} 2024-04-12T03:51:41.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d04461ef3207e26-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.066 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.066 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.067 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.067 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.067 DEBUG:boto:Method: GET 2024-04-12T03:51:41.067 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.067 DEBUG:boto:Data: 2024-04-12T03:51:41.067 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.067 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.067 DEBUG:boto:Params: {} 2024-04-12T03:51:41.067 DEBUG:boto:Token: None 2024-04-12T03:51:41.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:51:41.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4= 2024-04-12T03:51:41.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hG2dBLvssv89uPgQlRW4OUcm0L4=', 'Content-Length': '0'} 2024-04-12T03:51:41.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8cfddec8aa99abe-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.070 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.070 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.070 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.070 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.070 DEBUG:boto:Method: GET 2024-04-12T03:51:41.070 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.070 DEBUG:boto:Data: 2024-04-12T03:51:41.070 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.070 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.070 DEBUG:boto:Params: {} 2024-04-12T03:51:41.071 DEBUG:boto:Token: None 2024-04-12T03:51:41.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a82aa25e6d7f402-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.073 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.074 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.074 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.074 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.074 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.074 DEBUG:boto:Method: GET 2024-04-12T03:51:41.074 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.074 DEBUG:boto:Data: 2024-04-12T03:51:41.074 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.075 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.075 DEBUG:boto:Params: {} 2024-04-12T03:51:41.075 DEBUG:boto:Token: None 2024-04-12T03:51:41.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007ba824fac98a30e-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.078 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.078 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.079 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.079 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.079 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:51:41.079 DEBUG:boto:Data: 2024-04-12T03:51:41.079 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.079 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.079 DEBUG:boto:Params: {} 2024-04-12T03:51:41.079 DEBUG:boto:Token: None 2024-04-12T03:51:41.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51 2024-04-12T03:51:41.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yj64cVXCHhP021/O0QAT6dU/FE= 2024-04-12T03:51:41.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yj64cVXCHhP021/O0QAT6dU/FE=', 'Content-Length': '0'} 2024-04-12T03:51:41.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000077974164f8319ac-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.081 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.081 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.082 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.082 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:51:41.082 DEBUG:boto:Data: 2024-04-12T03:51:41.082 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.082 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.082 DEBUG:boto:Params: {} 2024-04-12T03:51:41.082 DEBUG:boto:Token: None 2024-04-12T03:51:41.082 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51 2024-04-12T03:51:41.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yj64cVXCHhP021/O0QAT6dU/FE= 2024-04-12T03:51:41.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yj64cVXCHhP021/O0QAT6dU/FE=', 'Content-Length': '0'} 2024-04-12T03:51:41.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdee6890e6704384-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:51:41.084 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.084 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.085 DEBUG:boto:Method: GET 2024-04-12T03:51:41.085 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:51:41.085 DEBUG:boto:Data: 2024-04-12T03:51:41.085 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.085 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.085 DEBUG:boto:Params: {} 2024-04-12T03:51:41.085 DEBUG:boto:Token: None 2024-04-12T03:51:41.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51 2024-04-12T03:51:41.086 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k46rY45eNBdELcZarkVIiIL8jnE= 2024-04-12T03:51:41.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k46rY45eNBdELcZarkVIiIL8jnE=', 'Content-Length': '0'} 2024-04-12T03:51:41.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c00afe5d236df0a-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.087 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:51:41.087 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:51:41.088 DEBUG:boto:Method: GET 2024-04-12T03:51:41.088 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:51:41.088 DEBUG:boto:Data: 2024-04-12T03:51:41.088 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.088 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.088 DEBUG:boto:Params: {} 2024-04-12T03:51:41.088 DEBUG:boto:Token: None 2024-04-12T03:51:41.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj51 2024-04-12T03:51:41.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k46rY45eNBdELcZarkVIiIL8jnE= 2024-04-12T03:51:41.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k46rY45eNBdELcZarkVIiIL8jnE=', 'Content-Length': '0'} 2024-04-12T03:51:41.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72dd5dd67b3fa34-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.091 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.091 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.091 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.091 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.091 DEBUG:boto:Method: GET 2024-04-12T03:51:41.091 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.091 DEBUG:boto:Data: 2024-04-12T03:51:41.091 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.091 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.091 DEBUG:boto:Params: {} 2024-04-12T03:51:41.092 DEBUG:boto:Token: None 2024-04-12T03:51:41.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade291e2b427d285-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.094 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.094 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.095 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.095 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.095 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.095 DEBUG:boto:Method: GET 2024-04-12T03:51:41.095 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:51:41.095 DEBUG:boto:Data: 2024-04-12T03:51:41.095 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.095 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.095 DEBUG:boto:Params: {} 2024-04-12T03:51:41.095 DEBUG:boto:Token: None 2024-04-12T03:51:41.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000debe0711a08fb62b-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.142 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:51:41.143 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.143 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.143 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.143 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.143 DEBUG:boto:Method: GET 2024-04-12T03:51:41.143 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.143 DEBUG:boto:Data: 2024-04-12T03:51:41.143 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.143 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.144 DEBUG:boto:Params: {} 2024-04-12T03:51:41.144 DEBUG:boto:Token: None 2024-04-12T03:51:41.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y= 2024-04-12T03:51:41.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y=', 'Content-Length': '0'} 2024-04-12T03:51:41.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d27bda103a09a6-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.146 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.146 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.147 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.147 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.147 DEBUG:boto:Method: GET 2024-04-12T03:51:41.147 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.147 DEBUG:boto:Data: 2024-04-12T03:51:41.147 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.147 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.147 DEBUG:boto:Params: {} 2024-04-12T03:51:41.147 DEBUG:boto:Token: None 2024-04-12T03:51:41.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y= 2024-04-12T03:51:41.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y=', 'Content-Length': '0'} 2024-04-12T03:51:41.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a560170dfa43eb-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.149 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.149 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.150 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.150 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.150 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.150 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.150 DEBUG:boto:Data: 2024-04-12T03:51:41.150 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.150 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.150 DEBUG:boto:Params: {} 2024-04-12T03:51:41.150 DEBUG:boto:Token: None 2024-04-12T03:51:41.150 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gEyZdGBu80yXInZAVB5KEPqOgQU= 2024-04-12T03:51:41.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gEyZdGBu80yXInZAVB5KEPqOgQU=', 'Content-Length': '0'} 2024-04-12T03:51:41.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61e01d65f9fc170-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.152 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.152 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.153 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.153 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.153 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.153 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.153 DEBUG:boto:Data: 2024-04-12T03:51:41.153 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.153 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.153 DEBUG:boto:Params: {} 2024-04-12T03:51:41.153 DEBUG:boto:Token: None 2024-04-12T03:51:41.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.154 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gEyZdGBu80yXInZAVB5KEPqOgQU= 2024-04-12T03:51:41.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gEyZdGBu80yXInZAVB5KEPqOgQU=', 'Content-Length': '0'} 2024-04-12T03:51:41.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01eae807c2ca26d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:51:41.155 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.156 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.156 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.156 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.156 DEBUG:boto:Method: GET 2024-04-12T03:51:41.156 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.156 DEBUG:boto:Data: 2024-04-12T03:51:41.156 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.156 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.156 DEBUG:boto:Params: {} 2024-04-12T03:51:41.156 DEBUG:boto:Token: None 2024-04-12T03:51:41.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y= 2024-04-12T03:51:41.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y=', 'Content-Length': '0'} 2024-04-12T03:51:41.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039dcb3c3a9ca014e-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.158 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.159 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.159 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.159 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.159 DEBUG:boto:Method: GET 2024-04-12T03:51:41.159 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.159 DEBUG:boto:Data: 2024-04-12T03:51:41.159 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.159 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.159 DEBUG:boto:Params: {} 2024-04-12T03:51:41.159 DEBUG:boto:Token: None 2024-04-12T03:51:41.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:51:41.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y= 2024-04-12T03:51:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0q093kHF3HihGp4Qg+ns5jJm/2Y=', 'Content-Length': '0'} 2024-04-12T03:51:41.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfd82f5bb4c2ce8-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.162 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.162 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.162 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.162 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.162 DEBUG:boto:Method: GET 2024-04-12T03:51:41.162 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.162 DEBUG:boto:Data: 2024-04-12T03:51:41.162 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.162 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.162 DEBUG:boto:Params: {} 2024-04-12T03:51:41.163 DEBUG:boto:Token: None 2024-04-12T03:51:41.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015768aae5d3bb99b-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.165 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.165 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.165 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.166 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.166 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.166 DEBUG:boto:Method: GET 2024-04-12T03:51:41.166 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.166 DEBUG:boto:Data: 2024-04-12T03:51:41.166 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.166 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.166 DEBUG:boto:Params: {} 2024-04-12T03:51:41.166 DEBUG:boto:Token: None 2024-04-12T03:51:41.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008991243f89835181-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.170 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.170 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.171 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.171 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.171 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:51:41.171 DEBUG:boto:Data: 2024-04-12T03:51:41.171 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.171 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.171 DEBUG:boto:Params: {} 2024-04-12T03:51:41.171 DEBUG:boto:Token: None 2024-04-12T03:51:41.171 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55 2024-04-12T03:51:41.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9siUxM+vLSokZcKRemr5yDW6GJk= 2024-04-12T03:51:41.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9siUxM+vLSokZcKRemr5yDW6GJk=', 'Content-Length': '0'} 2024-04-12T03:51:41.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025de78edca31274a-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.173 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.174 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.174 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.174 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:51:41.174 DEBUG:boto:Data: 2024-04-12T03:51:41.174 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.174 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.174 DEBUG:boto:Params: {} 2024-04-12T03:51:41.174 DEBUG:boto:Token: None 2024-04-12T03:51:41.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55 2024-04-12T03:51:41.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9siUxM+vLSokZcKRemr5yDW6GJk= 2024-04-12T03:51:41.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9siUxM+vLSokZcKRemr5yDW6GJk=', 'Content-Length': '0'} 2024-04-12T03:51:41.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110be5b383fb9385-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:51:41.177 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.177 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.177 DEBUG:boto:Method: GET 2024-04-12T03:51:41.177 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:51:41.177 DEBUG:boto:Data: 2024-04-12T03:51:41.177 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.177 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.177 DEBUG:boto:Params: {} 2024-04-12T03:51:41.177 DEBUG:boto:Token: None 2024-04-12T03:51:41.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55 2024-04-12T03:51:41.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ry0KymXUNatRpTUU9Ov1niEU3MA= 2024-04-12T03:51:41.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ry0KymXUNatRpTUU9Ov1niEU3MA=', 'Content-Length': '0'} 2024-04-12T03:51:41.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007887b550c904a04a-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.180 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:51:41.180 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:51:41.180 DEBUG:boto:Method: GET 2024-04-12T03:51:41.180 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:51:41.180 DEBUG:boto:Data: 2024-04-12T03:51:41.180 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.180 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.180 DEBUG:boto:Params: {} 2024-04-12T03:51:41.180 DEBUG:boto:Token: None 2024-04-12T03:51:41.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj55 2024-04-12T03:51:41.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ry0KymXUNatRpTUU9Ov1niEU3MA= 2024-04-12T03:51:41.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ry0KymXUNatRpTUU9Ov1niEU3MA=', 'Content-Length': '0'} 2024-04-12T03:51:41.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb2f443fcbc397a-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.182 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.183 DEBUG:boto:Method: GET 2024-04-12T03:51:41.183 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.183 DEBUG:boto:Data: 2024-04-12T03:51:41.183 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.183 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.183 DEBUG:boto:Params: {} 2024-04-12T03:51:41.184 DEBUG:boto:Token: None 2024-04-12T03:51:41.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e38166d77b0bbc4a-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.186 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.186 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.186 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.186 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.187 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.187 DEBUG:boto:Method: GET 2024-04-12T03:51:41.187 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:51:41.187 DEBUG:boto:Data: 2024-04-12T03:51:41.187 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.187 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.187 DEBUG:boto:Params: {} 2024-04-12T03:51:41.187 DEBUG:boto:Token: None 2024-04-12T03:51:41.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a68d117c54bc39de-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.234 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:51:41.235 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.235 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.235 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.235 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.235 DEBUG:boto:Method: GET 2024-04-12T03:51:41.235 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.235 DEBUG:boto:Data: 2024-04-12T03:51:41.235 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.236 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.236 DEBUG:boto:Params: {} 2024-04-12T03:51:41.236 DEBUG:boto:Token: None 2024-04-12T03:51:41.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas= 2024-04-12T03:51:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas=', 'Content-Length': '0'} 2024-04-12T03:51:41.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ea16edf5681a8d-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.238 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.238 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.238 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.239 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.239 DEBUG:boto:Method: GET 2024-04-12T03:51:41.239 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.239 DEBUG:boto:Data: 2024-04-12T03:51:41.239 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.239 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.239 DEBUG:boto:Params: {} 2024-04-12T03:51:41.239 DEBUG:boto:Token: None 2024-04-12T03:51:41.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas= 2024-04-12T03:51:41.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas=', 'Content-Length': '0'} 2024-04-12T03:51:41.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea32a1311b226d86-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.241 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.241 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.241 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.241 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.242 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.242 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.242 DEBUG:boto:Data: 2024-04-12T03:51:41.242 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.242 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.242 DEBUG:boto:Params: {} 2024-04-12T03:51:41.242 DEBUG:boto:Token: None 2024-04-12T03:51:41.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NwHXw9RWhs4vDDk8Nsj7oDlf5UA= 2024-04-12T03:51:41.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NwHXw9RWhs4vDDk8Nsj7oDlf5UA=', 'Content-Length': '0'} 2024-04-12T03:51:41.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f81fd9260560849-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.244 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.244 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.244 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.244 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.245 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.245 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.245 DEBUG:boto:Data: 2024-04-12T03:51:41.245 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.245 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.245 DEBUG:boto:Params: {} 2024-04-12T03:51:41.245 DEBUG:boto:Token: None 2024-04-12T03:51:41.245 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NwHXw9RWhs4vDDk8Nsj7oDlf5UA= 2024-04-12T03:51:41.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NwHXw9RWhs4vDDk8Nsj7oDlf5UA=', 'Content-Length': '0'} 2024-04-12T03:51:41.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f4c300108b0ab8-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:51:41.248 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.248 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.248 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.248 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.248 DEBUG:boto:Method: GET 2024-04-12T03:51:41.248 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.248 DEBUG:boto:Data: 2024-04-12T03:51:41.249 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.249 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.249 DEBUG:boto:Params: {} 2024-04-12T03:51:41.249 DEBUG:boto:Token: None 2024-04-12T03:51:41.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas= 2024-04-12T03:51:41.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas=', 'Content-Length': '0'} 2024-04-12T03:51:41.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021643c6ef1e6a8fc-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.251 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.251 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.251 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.251 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.252 DEBUG:boto:Method: GET 2024-04-12T03:51:41.252 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.252 DEBUG:boto:Data: 2024-04-12T03:51:41.252 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.252 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.252 DEBUG:boto:Params: {} 2024-04-12T03:51:41.252 DEBUG:boto:Token: None 2024-04-12T03:51:41.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:51:41.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas= 2024-04-12T03:51:41.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GR3Qp4PIjIoNWf4s5puDOctYcas=', 'Content-Length': '0'} 2024-04-12T03:51:41.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ead56f2319ab1b8-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.254 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.255 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.255 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.255 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.255 DEBUG:boto:Method: GET 2024-04-12T03:51:41.255 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.255 DEBUG:boto:Data: 2024-04-12T03:51:41.255 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.256 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.256 DEBUG:boto:Params: {} 2024-04-12T03:51:41.256 DEBUG:boto:Token: None 2024-04-12T03:51:41.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000545e9abbcc884b7b-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.306 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.307 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.307 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.307 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.307 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.307 DEBUG:boto:Method: GET 2024-04-12T03:51:41.307 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.307 DEBUG:boto:Data: 2024-04-12T03:51:41.308 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.308 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.308 DEBUG:boto:Params: {} 2024-04-12T03:51:41.308 DEBUG:boto:Token: None 2024-04-12T03:51:41.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041b7e2c018311581-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.312 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.312 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.312 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.312 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.312 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:51:41.312 DEBUG:boto:Data: 2024-04-12T03:51:41.313 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.313 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.313 DEBUG:boto:Params: {} 2024-04-12T03:51:41.313 DEBUG:boto:Token: None 2024-04-12T03:51:41.313 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57 2024-04-12T03:51:41.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1KycZegM4k/ePLQ4CBirqVUFcWE= 2024-04-12T03:51:41.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1KycZegM4k/ePLQ4CBirqVUFcWE=', 'Content-Length': '0'} 2024-04-12T03:51:41.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083112a6eabaf216c-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.315 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.315 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.315 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.315 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:51:41.315 DEBUG:boto:Data: 2024-04-12T03:51:41.315 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.315 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.316 DEBUG:boto:Params: {} 2024-04-12T03:51:41.316 DEBUG:boto:Token: None 2024-04-12T03:51:41.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57 2024-04-12T03:51:41.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1KycZegM4k/ePLQ4CBirqVUFcWE= 2024-04-12T03:51:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1KycZegM4k/ePLQ4CBirqVUFcWE=', 'Content-Length': '0'} 2024-04-12T03:51:41.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b082236b9ddcf57-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:51:41.318 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.318 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.318 DEBUG:boto:Method: GET 2024-04-12T03:51:41.318 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:51:41.318 DEBUG:boto:Data: 2024-04-12T03:51:41.318 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.319 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.319 DEBUG:boto:Params: {} 2024-04-12T03:51:41.319 DEBUG:boto:Token: None 2024-04-12T03:51:41.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57 2024-04-12T03:51:41.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bE/UBRxwVeqbrg+VA1oP+bYCsDA= 2024-04-12T03:51:41.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bE/UBRxwVeqbrg+VA1oP+bYCsDA=', 'Content-Length': '0'} 2024-04-12T03:51:41.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb7d8d6411d9255-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.321 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:51:41.321 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:51:41.321 DEBUG:boto:Method: GET 2024-04-12T03:51:41.321 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:51:41.321 DEBUG:boto:Data: 2024-04-12T03:51:41.321 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.321 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.321 DEBUG:boto:Params: {} 2024-04-12T03:51:41.322 DEBUG:boto:Token: None 2024-04-12T03:51:41.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj57 2024-04-12T03:51:41.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bE/UBRxwVeqbrg+VA1oP+bYCsDA= 2024-04-12T03:51:41.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bE/UBRxwVeqbrg+VA1oP+bYCsDA=', 'Content-Length': '0'} 2024-04-12T03:51:41.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e39f0130299d134-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.324 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.324 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.324 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.324 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.324 DEBUG:boto:Method: GET 2024-04-12T03:51:41.324 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.324 DEBUG:boto:Data: 2024-04-12T03:51:41.324 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.324 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.324 DEBUG:boto:Params: {} 2024-04-12T03:51:41.325 DEBUG:boto:Token: None 2024-04-12T03:51:41.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.325 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518fb3759c605ab8-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.327 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.327 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.328 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.328 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.328 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.328 DEBUG:boto:Method: GET 2024-04-12T03:51:41.328 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:51:41.328 DEBUG:boto:Data: 2024-04-12T03:51:41.328 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.328 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.328 DEBUG:boto:Params: {} 2024-04-12T03:51:41.328 DEBUG:boto:Token: None 2024-04-12T03:51:41.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000560ccb316bb0be8-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.332 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:51:41.332 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.333 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.333 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.333 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.333 DEBUG:boto:Method: GET 2024-04-12T03:51:41.333 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.333 DEBUG:boto:Data: 2024-04-12T03:51:41.333 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.333 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.333 DEBUG:boto:Params: {} 2024-04-12T03:51:41.333 DEBUG:boto:Token: None 2024-04-12T03:51:41.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U= 2024-04-12T03:51:41.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U=', 'Content-Length': '0'} 2024-04-12T03:51:41.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf9f4907e6a1565-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.378 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.379 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.379 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.379 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.379 DEBUG:boto:Method: GET 2024-04-12T03:51:41.379 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.379 DEBUG:boto:Data: 2024-04-12T03:51:41.379 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.379 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.379 DEBUG:boto:Params: {} 2024-04-12T03:51:41.380 DEBUG:boto:Token: None 2024-04-12T03:51:41.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U= 2024-04-12T03:51:41.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U=', 'Content-Length': '0'} 2024-04-12T03:51:41.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbdda084d737d70c-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.382 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.382 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.382 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.383 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.383 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.383 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.383 DEBUG:boto:Data: 2024-04-12T03:51:41.383 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.383 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.383 DEBUG:boto:Params: {} 2024-04-12T03:51:41.383 DEBUG:boto:Token: None 2024-04-12T03:51:41.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.383 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PLo9+mPEJh26qbH0PNkrLnRC+uU= 2024-04-12T03:51:41.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PLo9+mPEJh26qbH0PNkrLnRC+uU=', 'Content-Length': '0'} 2024-04-12T03:51:41.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f3dd00cc97799c-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.385 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.386 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.386 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.386 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.386 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.386 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.386 DEBUG:boto:Data: 2024-04-12T03:51:41.386 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.386 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.386 DEBUG:boto:Params: {} 2024-04-12T03:51:41.386 DEBUG:boto:Token: None 2024-04-12T03:51:41.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PLo9+mPEJh26qbH0PNkrLnRC+uU= 2024-04-12T03:51:41.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PLo9+mPEJh26qbH0PNkrLnRC+uU=', 'Content-Length': '0'} 2024-04-12T03:51:41.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e82541b02b681bf-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:51:41.389 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.389 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.389 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.389 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.389 DEBUG:boto:Method: GET 2024-04-12T03:51:41.389 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.390 DEBUG:boto:Data: 2024-04-12T03:51:41.390 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.390 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.390 DEBUG:boto:Params: {} 2024-04-12T03:51:41.390 DEBUG:boto:Token: None 2024-04-12T03:51:41.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.390 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U= 2024-04-12T03:51:41.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U=', 'Content-Length': '0'} 2024-04-12T03:51:41.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba95465cd22aba75-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.392 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.392 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.392 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.392 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.393 DEBUG:boto:Method: GET 2024-04-12T03:51:41.393 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.393 DEBUG:boto:Data: 2024-04-12T03:51:41.393 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.393 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.393 DEBUG:boto:Params: {} 2024-04-12T03:51:41.393 DEBUG:boto:Token: None 2024-04-12T03:51:41.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:51:41.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U= 2024-04-12T03:51:41.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JkrTG2iI+F2mKDY2xxLWBv+Ql8U=', 'Content-Length': '0'} 2024-04-12T03:51:41.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d695683c7f7b16-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.397 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.398 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.398 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.398 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.398 DEBUG:boto:Method: GET 2024-04-12T03:51:41.398 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.398 DEBUG:boto:Data: 2024-04-12T03:51:41.398 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.398 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.398 DEBUG:boto:Params: {} 2024-04-12T03:51:41.399 DEBUG:boto:Token: None 2024-04-12T03:51:41.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066e2810ba583e199-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.401 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.402 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.402 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.402 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.402 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.402 DEBUG:boto:Method: GET 2024-04-12T03:51:41.402 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.402 DEBUG:boto:Data: 2024-04-12T03:51:41.402 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.402 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.402 DEBUG:boto:Params: {} 2024-04-12T03:51:41.403 DEBUG:boto:Token: None 2024-04-12T03:51:41.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081628513d095c264-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.446 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.447 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.447 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.447 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.447 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:51:41.447 DEBUG:boto:Data: 2024-04-12T03:51:41.447 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.448 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.448 DEBUG:boto:Params: {} 2024-04-12T03:51:41.448 DEBUG:boto:Token: None 2024-04-12T03:51:41.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58 2024-04-12T03:51:41.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WULXIjwzUEW1MkiBjyXKoibG/U= 2024-04-12T03:51:41.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WULXIjwzUEW1MkiBjyXKoibG/U=', 'Content-Length': '0'} 2024-04-12T03:51:41.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088dc22f03585414c-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.450 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.450 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.450 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.450 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:51:41.451 DEBUG:boto:Data: 2024-04-12T03:51:41.451 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.451 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.451 DEBUG:boto:Params: {} 2024-04-12T03:51:41.451 DEBUG:boto:Token: None 2024-04-12T03:51:41.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58 2024-04-12T03:51:41.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5WULXIjwzUEW1MkiBjyXKoibG/U= 2024-04-12T03:51:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5WULXIjwzUEW1MkiBjyXKoibG/U=', 'Content-Length': '0'} 2024-04-12T03:51:41.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8de1c8538cfabf8-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:51:41.453 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.453 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.454 DEBUG:boto:Method: GET 2024-04-12T03:51:41.454 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:51:41.454 DEBUG:boto:Data: 2024-04-12T03:51:41.454 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.454 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.454 DEBUG:boto:Params: {} 2024-04-12T03:51:41.454 DEBUG:boto:Token: None 2024-04-12T03:51:41.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58 2024-04-12T03:51:41.455 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xl3dADLNAdhZr0kXEAxYbKziH6M= 2024-04-12T03:51:41.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xl3dADLNAdhZr0kXEAxYbKziH6M=', 'Content-Length': '0'} 2024-04-12T03:51:41.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1681681eb9e7f46-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.457 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:51:41.457 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:51:41.457 DEBUG:boto:Method: GET 2024-04-12T03:51:41.457 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:51:41.457 DEBUG:boto:Data: 2024-04-12T03:51:41.457 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.457 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.457 DEBUG:boto:Params: {} 2024-04-12T03:51:41.458 DEBUG:boto:Token: None 2024-04-12T03:51:41.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj58 2024-04-12T03:51:41.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xl3dADLNAdhZr0kXEAxYbKziH6M= 2024-04-12T03:51:41.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xl3dADLNAdhZr0kXEAxYbKziH6M=', 'Content-Length': '0'} 2024-04-12T03:51:41.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e569144d1844c6-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.460 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.460 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.460 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.460 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.460 DEBUG:boto:Method: GET 2024-04-12T03:51:41.460 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.461 DEBUG:boto:Data: 2024-04-12T03:51:41.461 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.461 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.461 DEBUG:boto:Params: {} 2024-04-12T03:51:41.461 DEBUG:boto:Token: None 2024-04-12T03:51:41.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f24f4c42a174e9-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.464 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.464 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.464 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.464 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.464 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.465 DEBUG:boto:Method: GET 2024-04-12T03:51:41.465 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:51:41.465 DEBUG:boto:Data: 2024-04-12T03:51:41.465 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.465 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.465 DEBUG:boto:Params: {} 2024-04-12T03:51:41.465 DEBUG:boto:Token: None 2024-04-12T03:51:41.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec6baaa76f992c7-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.469 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:51:41.469 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.469 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.469 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.469 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.469 DEBUG:boto:Method: GET 2024-04-12T03:51:41.470 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.470 DEBUG:boto:Data: 2024-04-12T03:51:41.470 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.470 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.470 DEBUG:boto:Params: {} 2024-04-12T03:51:41.470 DEBUG:boto:Token: None 2024-04-12T03:51:41.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA= 2024-04-12T03:51:41.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA=', 'Content-Length': '0'} 2024-04-12T03:51:41.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafce098b40aa857-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.473 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.473 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.473 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.473 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.473 DEBUG:boto:Method: GET 2024-04-12T03:51:41.473 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.473 DEBUG:boto:Data: 2024-04-12T03:51:41.473 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.473 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.474 DEBUG:boto:Params: {} 2024-04-12T03:51:41.474 DEBUG:boto:Token: None 2024-04-12T03:51:41.474 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.474 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA= 2024-04-12T03:51:41.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA=', 'Content-Length': '0'} 2024-04-12T03:51:41.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038399072b58f5736-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.476 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.476 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.476 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.476 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.476 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.476 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.476 DEBUG:boto:Data: 2024-04-12T03:51:41.476 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.477 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.477 DEBUG:boto:Params: {} 2024-04-12T03:51:41.477 DEBUG:boto:Token: None 2024-04-12T03:51:41.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.477 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v45ES/2mDJ/W7gQYQ29y74XuCGU= 2024-04-12T03:51:41.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v45ES/2mDJ/W7gQYQ29y74XuCGU=', 'Content-Length': '0'} 2024-04-12T03:51:41.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f113fdf0eabd8415-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.479 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.479 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.479 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.479 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.479 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.479 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.479 DEBUG:boto:Data: 2024-04-12T03:51:41.479 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.479 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.480 DEBUG:boto:Params: {} 2024-04-12T03:51:41.480 DEBUG:boto:Token: None 2024-04-12T03:51:41.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v45ES/2mDJ/W7gQYQ29y74XuCGU= 2024-04-12T03:51:41.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v45ES/2mDJ/W7gQYQ29y74XuCGU=', 'Content-Length': '0'} 2024-04-12T03:51:41.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ddab6eecd1275d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:51:41.482 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.482 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.482 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.482 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.482 DEBUG:boto:Method: GET 2024-04-12T03:51:41.482 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.483 DEBUG:boto:Data: 2024-04-12T03:51:41.483 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.483 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.483 DEBUG:boto:Params: {} 2024-04-12T03:51:41.483 DEBUG:boto:Token: None 2024-04-12T03:51:41.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA= 2024-04-12T03:51:41.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA=', 'Content-Length': '0'} 2024-04-12T03:51:41.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf4ae84f409d6fa-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.526 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.526 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.526 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.527 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.527 DEBUG:boto:Method: GET 2024-04-12T03:51:41.527 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.527 DEBUG:boto:Data: 2024-04-12T03:51:41.527 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.527 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.527 DEBUG:boto:Params: {} 2024-04-12T03:51:41.527 DEBUG:boto:Token: None 2024-04-12T03:51:41.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:51:41.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA= 2024-04-12T03:51:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZGtkberu/CicL60eSKptWppI8oA=', 'Content-Length': '0'} 2024-04-12T03:51:41.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b5d495a8589caa-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.529 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.530 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.530 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.530 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.530 DEBUG:boto:Method: GET 2024-04-12T03:51:41.530 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.530 DEBUG:boto:Data: 2024-04-12T03:51:41.530 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.530 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.530 DEBUG:boto:Params: {} 2024-04-12T03:51:41.531 DEBUG:boto:Token: None 2024-04-12T03:51:41.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b10704c3c3a0636-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.533 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.533 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.534 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.534 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.534 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.534 DEBUG:boto:Method: GET 2024-04-12T03:51:41.534 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.534 DEBUG:boto:Data: 2024-04-12T03:51:41.534 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.534 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.534 DEBUG:boto:Params: {} 2024-04-12T03:51:41.535 DEBUG:boto:Token: None 2024-04-12T03:51:41.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000726bbf7e3624506-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.538 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.539 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.539 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.539 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.539 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:51:41.539 DEBUG:boto:Data: 2024-04-12T03:51:41.539 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.539 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.540 DEBUG:boto:Params: {} 2024-04-12T03:51:41.540 DEBUG:boto:Token: None 2024-04-12T03:51:41.540 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62 2024-04-12T03:51:41.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:09wLPSGXqhSsJpTGLdiaBF+tmag= 2024-04-12T03:51:41.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:09wLPSGXqhSsJpTGLdiaBF+tmag=', 'Content-Length': '0'} 2024-04-12T03:51:41.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6088f85dfee8b2d-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.542 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.542 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.542 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.542 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:51:41.542 DEBUG:boto:Data: 2024-04-12T03:51:41.542 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.543 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.543 DEBUG:boto:Params: {} 2024-04-12T03:51:41.543 DEBUG:boto:Token: None 2024-04-12T03:51:41.543 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62 2024-04-12T03:51:41.543 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:09wLPSGXqhSsJpTGLdiaBF+tmag= 2024-04-12T03:51:41.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:09wLPSGXqhSsJpTGLdiaBF+tmag=', 'Content-Length': '0'} 2024-04-12T03:51:41.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027043ed6736298d2-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:51:41.545 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.545 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.545 DEBUG:boto:Method: GET 2024-04-12T03:51:41.545 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:51:41.545 DEBUG:boto:Data: 2024-04-12T03:51:41.545 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.545 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.545 DEBUG:boto:Params: {} 2024-04-12T03:51:41.545 DEBUG:boto:Token: None 2024-04-12T03:51:41.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62 2024-04-12T03:51:41.546 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cmrTpm9jx7L5AAO82PTaTnyb3YE= 2024-04-12T03:51:41.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cmrTpm9jx7L5AAO82PTaTnyb3YE=', 'Content-Length': '0'} 2024-04-12T03:51:41.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305925d1981fd7f2-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.547 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:51:41.548 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:51:41.548 DEBUG:boto:Method: GET 2024-04-12T03:51:41.548 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:51:41.548 DEBUG:boto:Data: 2024-04-12T03:51:41.548 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.548 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.548 DEBUG:boto:Params: {} 2024-04-12T03:51:41.548 DEBUG:boto:Token: None 2024-04-12T03:51:41.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj62 2024-04-12T03:51:41.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cmrTpm9jx7L5AAO82PTaTnyb3YE= 2024-04-12T03:51:41.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cmrTpm9jx7L5AAO82PTaTnyb3YE=', 'Content-Length': '0'} 2024-04-12T03:51:41.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d61050f23a4d1f4-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.594 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.595 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.595 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.595 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.595 DEBUG:boto:Method: GET 2024-04-12T03:51:41.595 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.595 DEBUG:boto:Data: 2024-04-12T03:51:41.595 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.595 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.595 DEBUG:boto:Params: {} 2024-04-12T03:51:41.596 DEBUG:boto:Token: None 2024-04-12T03:51:41.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b728279e7fd7a1-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.598 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.599 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.599 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.599 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.599 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.599 DEBUG:boto:Method: GET 2024-04-12T03:51:41.599 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:51:41.599 DEBUG:boto:Data: 2024-04-12T03:51:41.599 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.601 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.601 DEBUG:boto:Params: {} 2024-04-12T03:51:41.601 DEBUG:boto:Token: None 2024-04-12T03:51:41.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b043818f818b62-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.605 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:51:41.605 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.605 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.605 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.605 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.606 DEBUG:boto:Method: GET 2024-04-12T03:51:41.606 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.606 DEBUG:boto:Data: 2024-04-12T03:51:41.606 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.606 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.606 DEBUG:boto:Params: {} 2024-04-12T03:51:41.606 DEBUG:boto:Token: None 2024-04-12T03:51:41.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY= 2024-04-12T03:51:41.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY=', 'Content-Length': '0'} 2024-04-12T03:51:41.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d6a5bfb4de415e-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.609 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.609 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.609 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.610 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.610 DEBUG:boto:Method: GET 2024-04-12T03:51:41.610 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.610 DEBUG:boto:Data: 2024-04-12T03:51:41.610 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.610 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.610 DEBUG:boto:Params: {} 2024-04-12T03:51:41.610 DEBUG:boto:Token: None 2024-04-12T03:51:41.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY= 2024-04-12T03:51:41.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY=', 'Content-Length': '0'} 2024-04-12T03:51:41.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034db07f8dc65727d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.613 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.613 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.614 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.614 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.614 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.614 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.614 DEBUG:boto:Data: 2024-04-12T03:51:41.614 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.614 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.614 DEBUG:boto:Params: {} 2024-04-12T03:51:41.614 DEBUG:boto:Token: None 2024-04-12T03:51:41.615 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qqQkYUvBGl4Uh0vqlSPIYUsBl6M= 2024-04-12T03:51:41.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qqQkYUvBGl4Uh0vqlSPIYUsBl6M=', 'Content-Length': '0'} 2024-04-12T03:51:41.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3f147067b340ae-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.616 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.617 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.617 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.617 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.617 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.617 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.617 DEBUG:boto:Data: 2024-04-12T03:51:41.617 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.617 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.617 DEBUG:boto:Params: {} 2024-04-12T03:51:41.617 DEBUG:boto:Token: None 2024-04-12T03:51:41.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qqQkYUvBGl4Uh0vqlSPIYUsBl6M= 2024-04-12T03:51:41.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qqQkYUvBGl4Uh0vqlSPIYUsBl6M=', 'Content-Length': '0'} 2024-04-12T03:51:41.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241e0e2dd1f209fe-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:51:41.620 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.620 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.620 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.620 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.620 DEBUG:boto:Method: GET 2024-04-12T03:51:41.620 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.620 DEBUG:boto:Data: 2024-04-12T03:51:41.620 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.620 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.621 DEBUG:boto:Params: {} 2024-04-12T03:51:41.621 DEBUG:boto:Token: None 2024-04-12T03:51:41.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.621 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY= 2024-04-12T03:51:41.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY=', 'Content-Length': '0'} 2024-04-12T03:51:41.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cc4f1f598d5c35-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.623 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.623 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.623 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.623 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.623 DEBUG:boto:Method: GET 2024-04-12T03:51:41.623 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.623 DEBUG:boto:Data: 2024-04-12T03:51:41.624 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.624 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.624 DEBUG:boto:Params: {} 2024-04-12T03:51:41.624 DEBUG:boto:Token: None 2024-04-12T03:51:41.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:51:41.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY= 2024-04-12T03:51:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VwIg2JwQ7/VCdQshzOC/n4a3LDY=', 'Content-Length': '0'} 2024-04-12T03:51:41.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cfdbd91b95ae197-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.626 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.626 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.626 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.627 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.627 DEBUG:boto:Method: GET 2024-04-12T03:51:41.627 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.627 DEBUG:boto:Data: 2024-04-12T03:51:41.627 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.627 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.627 DEBUG:boto:Params: {} 2024-04-12T03:51:41.627 DEBUG:boto:Token: None 2024-04-12T03:51:41.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002338d3dfc2d94487-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.630 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.630 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.631 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.631 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.631 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.631 DEBUG:boto:Method: GET 2024-04-12T03:51:41.631 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.631 DEBUG:boto:Data: 2024-04-12T03:51:41.631 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.631 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.631 DEBUG:boto:Params: {} 2024-04-12T03:51:41.632 DEBUG:boto:Token: None 2024-04-12T03:51:41.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d25a3786f88ac59-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.635 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.636 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.636 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.636 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.636 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:51:41.636 DEBUG:boto:Data: 2024-04-12T03:51:41.636 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.636 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.636 DEBUG:boto:Params: {} 2024-04-12T03:51:41.636 DEBUG:boto:Token: None 2024-04-12T03:51:41.637 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68 2024-04-12T03:51:41.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UM2ghH87v0Th7iv0uqSPsgvpkbQ= 2024-04-12T03:51:41.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UM2ghH87v0Th7iv0uqSPsgvpkbQ=', 'Content-Length': '0'} 2024-04-12T03:51:41.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075128cdcd5738f36-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.638 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.639 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.639 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.639 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:51:41.639 DEBUG:boto:Data: 2024-04-12T03:51:41.639 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.639 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.639 DEBUG:boto:Params: {} 2024-04-12T03:51:41.639 DEBUG:boto:Token: None 2024-04-12T03:51:41.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68 2024-04-12T03:51:41.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UM2ghH87v0Th7iv0uqSPsgvpkbQ= 2024-04-12T03:51:41.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UM2ghH87v0Th7iv0uqSPsgvpkbQ=', 'Content-Length': '0'} 2024-04-12T03:51:41.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cadb5000dd6a73f-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:51:41.642 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.642 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.642 DEBUG:boto:Method: GET 2024-04-12T03:51:41.642 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:51:41.642 DEBUG:boto:Data: 2024-04-12T03:51:41.642 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.642 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.642 DEBUG:boto:Params: {} 2024-04-12T03:51:41.642 DEBUG:boto:Token: None 2024-04-12T03:51:41.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68 2024-04-12T03:51:41.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sPHGVjZ4aB3gPrbu7M9V4XaH/OM= 2024-04-12T03:51:41.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sPHGVjZ4aB3gPrbu7M9V4XaH/OM=', 'Content-Length': '0'} 2024-04-12T03:51:41.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdd5e8c9561430e-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.645 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:51:41.645 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:51:41.645 DEBUG:boto:Method: GET 2024-04-12T03:51:41.645 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:51:41.645 DEBUG:boto:Data: 2024-04-12T03:51:41.645 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.645 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.645 DEBUG:boto:Params: {} 2024-04-12T03:51:41.646 DEBUG:boto:Token: None 2024-04-12T03:51:41.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj68 2024-04-12T03:51:41.646 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sPHGVjZ4aB3gPrbu7M9V4XaH/OM= 2024-04-12T03:51:41.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sPHGVjZ4aB3gPrbu7M9V4XaH/OM=', 'Content-Length': '0'} 2024-04-12T03:51:41.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e0f8deb06423c2-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.690 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.691 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.691 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.691 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.691 DEBUG:boto:Method: GET 2024-04-12T03:51:41.691 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.691 DEBUG:boto:Data: 2024-04-12T03:51:41.691 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.691 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.691 DEBUG:boto:Params: {} 2024-04-12T03:51:41.691 DEBUG:boto:Token: None 2024-04-12T03:51:41.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1d7ae4b46b345c-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.694 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.695 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.695 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.695 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.695 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.695 DEBUG:boto:Method: GET 2024-04-12T03:51:41.695 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:51:41.695 DEBUG:boto:Data: 2024-04-12T03:51:41.695 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.696 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.696 DEBUG:boto:Params: {} 2024-04-12T03:51:41.696 DEBUG:boto:Token: None 2024-04-12T03:51:41.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3929728ebbd78d4-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.699 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:51:41.700 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.700 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.700 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.700 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.700 DEBUG:boto:Method: GET 2024-04-12T03:51:41.700 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.700 DEBUG:boto:Data: 2024-04-12T03:51:41.701 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.701 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.701 DEBUG:boto:Params: {} 2024-04-12T03:51:41.701 DEBUG:boto:Token: None 2024-04-12T03:51:41.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.701 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA= 2024-04-12T03:51:41.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA=', 'Content-Length': '0'} 2024-04-12T03:51:41.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76a9ca192fbbb1a-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.746 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.747 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.747 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.747 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.747 DEBUG:boto:Method: GET 2024-04-12T03:51:41.747 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.747 DEBUG:boto:Data: 2024-04-12T03:51:41.747 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.747 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.747 DEBUG:boto:Params: {} 2024-04-12T03:51:41.748 DEBUG:boto:Token: None 2024-04-12T03:51:41.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.748 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA= 2024-04-12T03:51:41.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA=', 'Content-Length': '0'} 2024-04-12T03:51:41.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c226e85408257beb-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.750 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.750 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.751 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.751 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.751 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.751 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.751 DEBUG:boto:Data: 2024-04-12T03:51:41.751 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.751 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.751 DEBUG:boto:Params: {} 2024-04-12T03:51:41.751 DEBUG:boto:Token: None 2024-04-12T03:51:41.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eqLnFkMe4vlH9Mv9gIu2B3Ww/Hs= 2024-04-12T03:51:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eqLnFkMe4vlH9Mv9gIu2B3Ww/Hs=', 'Content-Length': '0'} 2024-04-12T03:51:41.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e56a918fd94fc95-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.754 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.754 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.754 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.754 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.754 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.754 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.754 DEBUG:boto:Data: 2024-04-12T03:51:41.754 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.754 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.754 DEBUG:boto:Params: {} 2024-04-12T03:51:41.755 DEBUG:boto:Token: None 2024-04-12T03:51:41.755 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eqLnFkMe4vlH9Mv9gIu2B3Ww/Hs= 2024-04-12T03:51:41.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eqLnFkMe4vlH9Mv9gIu2B3Ww/Hs=', 'Content-Length': '0'} 2024-04-12T03:51:41.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8487f805e3666dc-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:51:41.757 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.757 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.757 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.757 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.757 DEBUG:boto:Method: GET 2024-04-12T03:51:41.757 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.758 DEBUG:boto:Data: 2024-04-12T03:51:41.758 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.758 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.758 DEBUG:boto:Params: {} 2024-04-12T03:51:41.758 DEBUG:boto:Token: None 2024-04-12T03:51:41.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.758 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA= 2024-04-12T03:51:41.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA=', 'Content-Length': '0'} 2024-04-12T03:51:41.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e77e0779e84556da-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.760 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.761 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.761 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.761 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.761 DEBUG:boto:Method: GET 2024-04-12T03:51:41.761 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.761 DEBUG:boto:Data: 2024-04-12T03:51:41.761 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.761 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.761 DEBUG:boto:Params: {} 2024-04-12T03:51:41.762 DEBUG:boto:Token: None 2024-04-12T03:51:41.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:51:41.762 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA= 2024-04-12T03:51:41.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ALfPnSzLObFVuDTBZqL7iOCrphA=', 'Content-Length': '0'} 2024-04-12T03:51:41.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fca3c190d367c12-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.764 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.764 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.765 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.765 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.765 DEBUG:boto:Method: GET 2024-04-12T03:51:41.765 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.765 DEBUG:boto:Data: 2024-04-12T03:51:41.765 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.765 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.765 DEBUG:boto:Params: {} 2024-04-12T03:51:41.765 DEBUG:boto:Token: None 2024-04-12T03:51:41.765 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c587d7031c4e0af8-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.768 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.768 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.768 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.768 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.769 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.769 DEBUG:boto:Method: GET 2024-04-12T03:51:41.769 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.769 DEBUG:boto:Data: 2024-04-12T03:51:41.769 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.769 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.769 DEBUG:boto:Params: {} 2024-04-12T03:51:41.769 DEBUG:boto:Token: None 2024-04-12T03:51:41.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d95262a7fc9a24-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.773 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.773 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.773 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.773 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.773 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:51:41.773 DEBUG:boto:Data: 2024-04-12T03:51:41.774 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.774 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.774 DEBUG:boto:Params: {} 2024-04-12T03:51:41.774 DEBUG:boto:Token: None 2024-04-12T03:51:41.774 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69 2024-04-12T03:51:41.774 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QH8U8KMZQexn1FmB7VcGyzmlhCw= 2024-04-12T03:51:41.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QH8U8KMZQexn1FmB7VcGyzmlhCw=', 'Content-Length': '0'} 2024-04-12T03:51:41.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb140386d19aedb3-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.776 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.776 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.776 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.776 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:51:41.776 DEBUG:boto:Data: 2024-04-12T03:51:41.776 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.776 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.776 DEBUG:boto:Params: {} 2024-04-12T03:51:41.777 DEBUG:boto:Token: None 2024-04-12T03:51:41.777 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69 2024-04-12T03:51:41.777 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QH8U8KMZQexn1FmB7VcGyzmlhCw= 2024-04-12T03:51:41.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QH8U8KMZQexn1FmB7VcGyzmlhCw=', 'Content-Length': '0'} 2024-04-12T03:51:41.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f41ae90c5515caf-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:51:41.779 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.779 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.779 DEBUG:boto:Method: GET 2024-04-12T03:51:41.779 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:51:41.779 DEBUG:boto:Data: 2024-04-12T03:51:41.779 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.780 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.780 DEBUG:boto:Params: {} 2024-04-12T03:51:41.780 DEBUG:boto:Token: None 2024-04-12T03:51:41.780 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69 2024-04-12T03:51:41.780 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xNnfJ4KTwULqamdvF/gU42OZJyk= 2024-04-12T03:51:41.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xNnfJ4KTwULqamdvF/gU42OZJyk=', 'Content-Length': '0'} 2024-04-12T03:51:41.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a129fae511932ce-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.782 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:51:41.782 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:51:41.782 DEBUG:boto:Method: GET 2024-04-12T03:51:41.782 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:51:41.782 DEBUG:boto:Data: 2024-04-12T03:51:41.782 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.783 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.783 DEBUG:boto:Params: {} 2024-04-12T03:51:41.783 DEBUG:boto:Token: None 2024-04-12T03:51:41.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj69 2024-04-12T03:51:41.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xNnfJ4KTwULqamdvF/gU42OZJyk= 2024-04-12T03:51:41.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xNnfJ4KTwULqamdvF/gU42OZJyk=', 'Content-Length': '0'} 2024-04-12T03:51:41.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd62e4ecd9bf498d-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.826 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.826 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.827 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.827 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.827 DEBUG:boto:Method: GET 2024-04-12T03:51:41.827 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.827 DEBUG:boto:Data: 2024-04-12T03:51:41.827 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.827 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.827 DEBUG:boto:Params: {} 2024-04-12T03:51:41.827 DEBUG:boto:Token: None 2024-04-12T03:51:41.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a585eca160cf8082-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.830 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.830 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.830 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.830 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.831 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.831 DEBUG:boto:Method: GET 2024-04-12T03:51:41.831 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:51:41.831 DEBUG:boto:Data: 2024-04-12T03:51:41.831 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.831 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.831 DEBUG:boto:Params: {} 2024-04-12T03:51:41.831 DEBUG:boto:Token: None 2024-04-12T03:51:41.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676d972dfa0befa1-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.835 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:51:41.835 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.835 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.835 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.835 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.835 DEBUG:boto:Method: GET 2024-04-12T03:51:41.835 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.836 DEBUG:boto:Data: 2024-04-12T03:51:41.836 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.836 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.836 DEBUG:boto:Params: {} 2024-04-12T03:51:41.836 DEBUG:boto:Token: None 2024-04-12T03:51:41.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw= 2024-04-12T03:51:41.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw=', 'Content-Length': '0'} 2024-04-12T03:51:41.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afce984989ca7e8-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.838 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.838 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.839 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.839 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.839 DEBUG:boto:Method: GET 2024-04-12T03:51:41.839 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.839 DEBUG:boto:Data: 2024-04-12T03:51:41.839 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.839 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.839 DEBUG:boto:Params: {} 2024-04-12T03:51:41.839 DEBUG:boto:Token: None 2024-04-12T03:51:41.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw= 2024-04-12T03:51:41.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw=', 'Content-Length': '0'} 2024-04-12T03:51:41.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001699f1db4f4394c8-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.842 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.842 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.842 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.842 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.842 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.842 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.842 DEBUG:boto:Data: 2024-04-12T03:51:41.843 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.843 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.843 DEBUG:boto:Params: {} 2024-04-12T03:51:41.843 DEBUG:boto:Token: None 2024-04-12T03:51:41.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7p9GznhOykU33i10rJovS2Ig0pE= 2024-04-12T03:51:41.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7p9GznhOykU33i10rJovS2Ig0pE=', 'Content-Length': '0'} 2024-04-12T03:51:41.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002470f47255bda7a0-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.845 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.845 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.845 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.845 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.845 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.845 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.845 DEBUG:boto:Data: 2024-04-12T03:51:41.845 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.846 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.846 DEBUG:boto:Params: {} 2024-04-12T03:51:41.846 DEBUG:boto:Token: None 2024-04-12T03:51:41.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7p9GznhOykU33i10rJovS2Ig0pE= 2024-04-12T03:51:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7p9GznhOykU33i10rJovS2Ig0pE=', 'Content-Length': '0'} 2024-04-12T03:51:41.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504b6126c882633b-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:51:41.848 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.848 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.848 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.848 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.848 DEBUG:boto:Method: GET 2024-04-12T03:51:41.848 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.848 DEBUG:boto:Data: 2024-04-12T03:51:41.848 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.849 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.849 DEBUG:boto:Params: {} 2024-04-12T03:51:41.849 DEBUG:boto:Token: None 2024-04-12T03:51:41.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw= 2024-04-12T03:51:41.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw=', 'Content-Length': '0'} 2024-04-12T03:51:41.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcd266febf9311d-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.851 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.851 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.851 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.851 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.851 DEBUG:boto:Method: GET 2024-04-12T03:51:41.851 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.851 DEBUG:boto:Data: 2024-04-12T03:51:41.852 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.852 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.852 DEBUG:boto:Params: {} 2024-04-12T03:51:41.852 DEBUG:boto:Token: None 2024-04-12T03:51:41.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:51:41.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw= 2024-04-12T03:51:41.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:etRG3z/7Fg92MbdtmZQr6W4Dgkw=', 'Content-Length': '0'} 2024-04-12T03:51:41.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f83cfebcf81a942-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.854 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.854 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.854 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.854 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.855 DEBUG:boto:Method: GET 2024-04-12T03:51:41.855 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.855 DEBUG:boto:Data: 2024-04-12T03:51:41.855 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.855 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.855 DEBUG:boto:Params: {} 2024-04-12T03:51:41.855 DEBUG:boto:Token: None 2024-04-12T03:51:41.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe2a5b0924f95a6-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.858 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.858 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.858 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.858 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.858 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.858 DEBUG:boto:Method: GET 2024-04-12T03:51:41.859 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.859 DEBUG:boto:Data: 2024-04-12T03:51:41.859 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.859 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.859 DEBUG:boto:Params: {} 2024-04-12T03:51:41.859 DEBUG:boto:Token: None 2024-04-12T03:51:41.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b0e9acc0b70e96f-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.862 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.863 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.863 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.863 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.863 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:51:41.863 DEBUG:boto:Data: 2024-04-12T03:51:41.863 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.863 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.863 DEBUG:boto:Params: {} 2024-04-12T03:51:41.863 DEBUG:boto:Token: None 2024-04-12T03:51:41.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76 2024-04-12T03:51:41.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/GmAZFe53oRRc2U3CHMt/EobucI= 2024-04-12T03:51:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/GmAZFe53oRRc2U3CHMt/EobucI=', 'Content-Length': '0'} 2024-04-12T03:51:41.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd034b9d52c3302-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.865 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.865 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.866 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.866 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:51:41.866 DEBUG:boto:Data: 2024-04-12T03:51:41.866 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.866 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.866 DEBUG:boto:Params: {} 2024-04-12T03:51:41.866 DEBUG:boto:Token: None 2024-04-12T03:51:41.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76 2024-04-12T03:51:41.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/GmAZFe53oRRc2U3CHMt/EobucI= 2024-04-12T03:51:41.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/GmAZFe53oRRc2U3CHMt/EobucI=', 'Content-Length': '0'} 2024-04-12T03:51:41.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b5b512ba328880-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:51:41.869 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.869 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.869 DEBUG:boto:Method: GET 2024-04-12T03:51:41.869 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:51:41.869 DEBUG:boto:Data: 2024-04-12T03:51:41.869 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.869 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.869 DEBUG:boto:Params: {} 2024-04-12T03:51:41.870 DEBUG:boto:Token: None 2024-04-12T03:51:41.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76 2024-04-12T03:51:41.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OMw76WP+ZPaH/jicrQFyepIwt44= 2024-04-12T03:51:41.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OMw76WP+ZPaH/jicrQFyepIwt44=', 'Content-Length': '0'} 2024-04-12T03:51:41.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062684c79d149f4cd-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.872 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:51:41.872 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:51:41.872 DEBUG:boto:Method: GET 2024-04-12T03:51:41.872 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:51:41.872 DEBUG:boto:Data: 2024-04-12T03:51:41.872 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.872 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.872 DEBUG:boto:Params: {} 2024-04-12T03:51:41.873 DEBUG:boto:Token: None 2024-04-12T03:51:41.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj76 2024-04-12T03:51:41.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OMw76WP+ZPaH/jicrQFyepIwt44= 2024-04-12T03:51:41.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OMw76WP+ZPaH/jicrQFyepIwt44=', 'Content-Length': '0'} 2024-04-12T03:51:41.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095862d42e1f4c348-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.922 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.923 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.923 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.923 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.923 DEBUG:boto:Method: GET 2024-04-12T03:51:41.923 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.923 DEBUG:boto:Data: 2024-04-12T03:51:41.923 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.923 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.923 DEBUG:boto:Params: {} 2024-04-12T03:51:41.924 DEBUG:boto:Token: None 2024-04-12T03:51:41.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8427a9267f3c3c0-006618afcd-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.926 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.927 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.927 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.927 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.927 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.927 DEBUG:boto:Method: GET 2024-04-12T03:51:41.927 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:51:41.927 DEBUG:boto:Data: 2024-04-12T03:51:41.927 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.927 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.928 DEBUG:boto:Params: {} 2024-04-12T03:51:41.928 DEBUG:boto:Token: None 2024-04-12T03:51:41.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:41.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd6e3bf2aa8e8355-006618afcd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.931 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:41.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:51:41.932 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.932 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.932 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.932 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.932 DEBUG:boto:Method: GET 2024-04-12T03:51:41.932 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.932 DEBUG:boto:Data: 2024-04-12T03:51:41.932 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.933 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.933 DEBUG:boto:Params: {} 2024-04-12T03:51:41.933 DEBUG:boto:Token: None 2024-04-12T03:51:41.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s= 2024-04-12T03:51:41.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051019476396ea5cb-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.935 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.935 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.935 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.935 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.936 DEBUG:boto:Method: GET 2024-04-12T03:51:41.936 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.936 DEBUG:boto:Data: 2024-04-12T03:51:41.936 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.936 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.936 DEBUG:boto:Params: {} 2024-04-12T03:51:41.936 DEBUG:boto:Token: None 2024-04-12T03:51:41.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s= 2024-04-12T03:51:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7ee8c597cf18bd-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.938 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.939 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.939 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.939 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.939 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.939 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.939 DEBUG:boto:Data: 2024-04-12T03:51:41.939 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.939 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.939 DEBUG:boto:Params: {} 2024-04-12T03:51:41.940 DEBUG:boto:Token: None 2024-04-12T03:51:41.940 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bUhWLyXmBcUAWeRhsZa//7mRG/s= 2024-04-12T03:51:41.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bUhWLyXmBcUAWeRhsZa//7mRG/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4dae2b8e954bd0-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.942 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.942 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.942 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.942 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.942 DEBUG:boto:Method: HEAD 2024-04-12T03:51:41.942 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.943 DEBUG:boto:Data: 2024-04-12T03:51:41.943 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.943 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.943 DEBUG:boto:Params: {} 2024-04-12T03:51:41.943 DEBUG:boto:Token: None 2024-04-12T03:51:41.943 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bUhWLyXmBcUAWeRhsZa//7mRG/s= 2024-04-12T03:51:41.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bUhWLyXmBcUAWeRhsZa//7mRG/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285bbd721c76c5a5-006618afcd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:51:41.945 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.945 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.946 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.946 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.946 DEBUG:boto:Method: GET 2024-04-12T03:51:41.946 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.946 DEBUG:boto:Data: 2024-04-12T03:51:41.946 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.946 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.946 DEBUG:boto:Params: {} 2024-04-12T03:51:41.946 DEBUG:boto:Token: None 2024-04-12T03:51:41.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s= 2024-04-12T03:51:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1599cee9f6c7bd-006618afcd-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.994 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:41.995 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:41.995 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.995 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.995 DEBUG:boto:Method: GET 2024-04-12T03:51:41.995 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.995 DEBUG:boto:Data: 2024-04-12T03:51:41.995 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:41.995 DEBUG:boto:Port: 8000 2024-04-12T03:51:41.995 DEBUG:boto:Params: {} 2024-04-12T03:51:41.996 DEBUG:boto:Token: None 2024-04-12T03:51:41.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:51:41.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s= 2024-04-12T03:51:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E+QvIIphv6xmbFJ24QlGhgdnm/s=', 'Content-Length': '0'} 2024-04-12T03:51:41.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e6c52dff9df0a6-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:41.998 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:41.998 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:41.998 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:41.998 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:41.998 DEBUG:boto:Method: GET 2024-04-12T03:51:41.998 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:51:41.999 DEBUG:boto:Data: 2024-04-12T03:51:41.999 DEBUG:boto:Headers: {} 2024-04-12T03:51:41.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:41.999 DEBUG:boto:Port: 8001 2024-04-12T03:51:41.999 DEBUG:boto:Params: {} 2024-04-12T03:51:41.999 DEBUG:boto:Token: None 2024-04-12T03:51:41.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:41 GMT /dgbpxg-61/ 2024-04-12T03:51:41.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs= 2024-04-12T03:51:41.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bwrpdYRkApGkkf1PcHOkXd8MPNs=', 'Content-Length': '0'} 2024-04-12T03:51:42.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c4c7534ae521bd-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.002 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.002 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.002 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.002 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.002 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.002 DEBUG:boto:Method: GET 2024-04-12T03:51:42.003 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.003 DEBUG:boto:Data: 2024-04-12T03:51:42.003 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.003 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.003 DEBUG:boto:Params: {} 2024-04-12T03:51:42.003 DEBUG:boto:Token: None 2024-04-12T03:51:42.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012be747960a82c7f-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.009 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.009 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:42.009 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:42.010 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.010 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:51:42.010 DEBUG:boto:Data: 2024-04-12T03:51:42.010 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.010 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.010 DEBUG:boto:Params: {} 2024-04-12T03:51:42.010 DEBUG:boto:Token: None 2024-04-12T03:51:42.010 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj85 2024-04-12T03:51:42.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QBJ8Iqp+1um9PUkdsjHOploA1m8= 2024-04-12T03:51:42.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QBJ8Iqp+1um9PUkdsjHOploA1m8=', 'Content-Length': '0'} 2024-04-12T03:51:42.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3dc7b7f1113f9b-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.012 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:42.012 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:42.013 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.013 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:51:42.013 DEBUG:boto:Data: 2024-04-12T03:51:42.013 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.013 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.013 DEBUG:boto:Params: {} 2024-04-12T03:51:42.013 DEBUG:boto:Token: None 2024-04-12T03:51:42.013 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj85 2024-04-12T03:51:42.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QBJ8Iqp+1um9PUkdsjHOploA1m8= 2024-04-12T03:51:42.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QBJ8Iqp+1um9PUkdsjHOploA1m8=', 'Content-Length': '0'} 2024-04-12T03:51:42.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09b64d75188d342-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:51:42.016 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:42.016 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:42.016 DEBUG:boto:Method: GET 2024-04-12T03:51:42.016 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:51:42.016 DEBUG:boto:Data: 2024-04-12T03:51:42.016 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.016 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.016 DEBUG:boto:Params: {} 2024-04-12T03:51:42.016 DEBUG:boto:Token: None 2024-04-12T03:51:42.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj85 2024-04-12T03:51:42.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A3qKzpLBc/h6bmN/Te/Xygug5fU= 2024-04-12T03:51:42.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A3qKzpLBc/h6bmN/Te/Xygug5fU=', 'Content-Length': '0'} 2024-04-12T03:51:42.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f08aaf2e82ad0b-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.019 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:51:42.019 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:51:42.019 DEBUG:boto:Method: GET 2024-04-12T03:51:42.019 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:51:42.019 DEBUG:boto:Data: 2024-04-12T03:51:42.019 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.019 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.020 DEBUG:boto:Params: {} 2024-04-12T03:51:42.020 DEBUG:boto:Token: None 2024-04-12T03:51:42.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj85 2024-04-12T03:51:42.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:A3qKzpLBc/h6bmN/Te/Xygug5fU= 2024-04-12T03:51:42.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A3qKzpLBc/h6bmN/Te/Xygug5fU=', 'Content-Length': '0'} 2024-04-12T03:51:42.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57b3997441a97c2-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.022 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.022 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.022 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.022 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.023 DEBUG:boto:Method: GET 2024-04-12T03:51:42.023 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.023 DEBUG:boto:Data: 2024-04-12T03:51:42.023 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.023 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.023 DEBUG:boto:Params: {} 2024-04-12T03:51:42.023 DEBUG:boto:Token: None 2024-04-12T03:51:42.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841c3e7d24265a2f-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.026 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.026 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.026 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.026 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.026 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.026 DEBUG:boto:Method: GET 2024-04-12T03:51:42.026 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:51:42.027 DEBUG:boto:Data: 2024-04-12T03:51:42.027 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.027 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.027 DEBUG:boto:Params: {} 2024-04-12T03:51:42.027 DEBUG:boto:Token: None 2024-04-12T03:51:42.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bdd90c30f59c2e-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.030 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:51:42.031 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.031 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.031 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.031 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.031 DEBUG:boto:Method: GET 2024-04-12T03:51:42.031 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.031 DEBUG:boto:Data: 2024-04-12T03:51:42.031 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.032 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.032 DEBUG:boto:Params: {} 2024-04-12T03:51:42.032 DEBUG:boto:Token: None 2024-04-12T03:51:42.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg= 2024-04-12T03:51:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg=', 'Content-Length': '0'} 2024-04-12T03:51:42.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708696bc660bc3a7-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.034 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.034 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.034 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.034 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.034 DEBUG:boto:Method: GET 2024-04-12T03:51:42.034 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.035 DEBUG:boto:Data: 2024-04-12T03:51:42.035 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.035 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.035 DEBUG:boto:Params: {} 2024-04-12T03:51:42.035 DEBUG:boto:Token: None 2024-04-12T03:51:42.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg= 2024-04-12T03:51:42.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg=', 'Content-Length': '0'} 2024-04-12T03:51:42.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97ed203a423deae-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.078 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.078 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.079 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.079 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.079 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.079 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.079 DEBUG:boto:Data: 2024-04-12T03:51:42.079 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.079 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.079 DEBUG:boto:Params: {} 2024-04-12T03:51:42.079 DEBUG:boto:Token: None 2024-04-12T03:51:42.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zgBCPWuLS+XnP1CS4pLcmudESOo= 2024-04-12T03:51:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zgBCPWuLS+XnP1CS4pLcmudESOo=', 'Content-Length': '0'} 2024-04-12T03:51:42.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007501ad672e39d6ae-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.082 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.082 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.082 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.082 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.082 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.082 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.082 DEBUG:boto:Data: 2024-04-12T03:51:42.082 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.083 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.083 DEBUG:boto:Params: {} 2024-04-12T03:51:42.083 DEBUG:boto:Token: None 2024-04-12T03:51:42.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zgBCPWuLS+XnP1CS4pLcmudESOo= 2024-04-12T03:51:42.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zgBCPWuLS+XnP1CS4pLcmudESOo=', 'Content-Length': '0'} 2024-04-12T03:51:42.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ff2810ad461661-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:51:42.085 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.086 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.086 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.086 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.086 DEBUG:boto:Method: GET 2024-04-12T03:51:42.086 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.086 DEBUG:boto:Data: 2024-04-12T03:51:42.086 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.086 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.087 DEBUG:boto:Params: {} 2024-04-12T03:51:42.087 DEBUG:boto:Token: None 2024-04-12T03:51:42.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg= 2024-04-12T03:51:42.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg=', 'Content-Length': '0'} 2024-04-12T03:51:42.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0f16f7c71da564c-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.089 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.089 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.089 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.089 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.089 DEBUG:boto:Method: GET 2024-04-12T03:51:42.089 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.089 DEBUG:boto:Data: 2024-04-12T03:51:42.089 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.090 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.090 DEBUG:boto:Params: {} 2024-04-12T03:51:42.090 DEBUG:boto:Token: None 2024-04-12T03:51:42.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:51:42.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg= 2024-04-12T03:51:42.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iKRp67F8QB9L+bXJKu7xjftMQOg=', 'Content-Length': '0'} 2024-04-12T03:51:42.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d59dcba97fbc83-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.092 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.092 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.092 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.092 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.092 DEBUG:boto:Method: GET 2024-04-12T03:51:42.092 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.092 DEBUG:boto:Data: 2024-04-12T03:51:42.093 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.093 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.093 DEBUG:boto:Params: {} 2024-04-12T03:51:42.093 DEBUG:boto:Token: None 2024-04-12T03:51:42.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4790195b6cd4fbb-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.095 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.096 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.096 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.096 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.096 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.096 DEBUG:boto:Method: GET 2024-04-12T03:51:42.096 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.096 DEBUG:boto:Data: 2024-04-12T03:51:42.096 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.097 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.097 DEBUG:boto:Params: {} 2024-04-12T03:51:42.097 DEBUG:boto:Token: None 2024-04-12T03:51:42.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492702d71bbc28c7-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.100 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.101 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.101 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.101 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.101 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:51:42.101 DEBUG:boto:Data: 2024-04-12T03:51:42.101 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.101 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.101 DEBUG:boto:Params: {} 2024-04-12T03:51:42.102 DEBUG:boto:Token: None 2024-04-12T03:51:42.102 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87 2024-04-12T03:51:42.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CQ/M/qx+LlBQQQmFqcoc7qvD3PE= 2024-04-12T03:51:42.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CQ/M/qx+LlBQQQmFqcoc7qvD3PE=', 'Content-Length': '0'} 2024-04-12T03:51:42.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcb74b7e58a05cd4-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.104 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.104 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.104 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.104 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:51:42.104 DEBUG:boto:Data: 2024-04-12T03:51:42.104 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.105 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.105 DEBUG:boto:Params: {} 2024-04-12T03:51:42.105 DEBUG:boto:Token: None 2024-04-12T03:51:42.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87 2024-04-12T03:51:42.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CQ/M/qx+LlBQQQmFqcoc7qvD3PE= 2024-04-12T03:51:42.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CQ/M/qx+LlBQQQmFqcoc7qvD3PE=', 'Content-Length': '0'} 2024-04-12T03:51:42.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4c9d91e2c9ae11-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:51:42.107 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.107 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.107 DEBUG:boto:Method: GET 2024-04-12T03:51:42.107 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:51:42.107 DEBUG:boto:Data: 2024-04-12T03:51:42.107 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.107 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.107 DEBUG:boto:Params: {} 2024-04-12T03:51:42.107 DEBUG:boto:Token: None 2024-04-12T03:51:42.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87 2024-04-12T03:51:42.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J4RfxrmD0NCBNf4oUWVH6G96Xgw= 2024-04-12T03:51:42.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J4RfxrmD0NCBNf4oUWVH6G96Xgw=', 'Content-Length': '0'} 2024-04-12T03:51:42.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc4d5df205843c6-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.150 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:51:42.150 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:51:42.151 DEBUG:boto:Method: GET 2024-04-12T03:51:42.151 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:51:42.151 DEBUG:boto:Data: 2024-04-12T03:51:42.151 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.151 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.151 DEBUG:boto:Params: {} 2024-04-12T03:51:42.151 DEBUG:boto:Token: None 2024-04-12T03:51:42.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj87 2024-04-12T03:51:42.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J4RfxrmD0NCBNf4oUWVH6G96Xgw= 2024-04-12T03:51:42.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J4RfxrmD0NCBNf4oUWVH6G96Xgw=', 'Content-Length': '0'} 2024-04-12T03:51:42.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcde6894e80d8c5-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.153 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.154 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.154 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.154 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.154 DEBUG:boto:Method: GET 2024-04-12T03:51:42.154 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.154 DEBUG:boto:Data: 2024-04-12T03:51:42.154 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.154 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.154 DEBUG:boto:Params: {} 2024-04-12T03:51:42.155 DEBUG:boto:Token: None 2024-04-12T03:51:42.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342e19d82c8db941-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.157 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.157 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.157 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.157 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.157 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.158 DEBUG:boto:Method: GET 2024-04-12T03:51:42.158 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:51:42.158 DEBUG:boto:Data: 2024-04-12T03:51:42.158 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.158 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.158 DEBUG:boto:Params: {} 2024-04-12T03:51:42.158 DEBUG:boto:Token: None 2024-04-12T03:51:42.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb058f41e7ee02b2-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.162 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:51:42.162 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.162 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.163 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.163 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.163 DEBUG:boto:Method: GET 2024-04-12T03:51:42.163 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.163 DEBUG:boto:Data: 2024-04-12T03:51:42.163 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.163 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.163 DEBUG:boto:Params: {} 2024-04-12T03:51:42.163 DEBUG:boto:Token: None 2024-04-12T03:51:42.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E= 2024-04-12T03:51:42.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E=', 'Content-Length': '0'} 2024-04-12T03:51:42.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49e8a73b6b0dc86-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.166 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.166 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.166 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.166 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.166 DEBUG:boto:Method: GET 2024-04-12T03:51:42.166 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.166 DEBUG:boto:Data: 2024-04-12T03:51:42.166 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.167 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.167 DEBUG:boto:Params: {} 2024-04-12T03:51:42.167 DEBUG:boto:Token: None 2024-04-12T03:51:42.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E= 2024-04-12T03:51:42.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E=', 'Content-Length': '0'} 2024-04-12T03:51:42.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c1e269d4e973b9-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.169 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.169 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.169 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.169 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.170 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.170 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.170 DEBUG:boto:Data: 2024-04-12T03:51:42.170 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.170 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.170 DEBUG:boto:Params: {} 2024-04-12T03:51:42.170 DEBUG:boto:Token: None 2024-04-12T03:51:42.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.171 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DwKF68xFiWxejMEX5V5mmTrJtOo= 2024-04-12T03:51:42.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DwKF68xFiWxejMEX5V5mmTrJtOo=', 'Content-Length': '0'} 2024-04-12T03:51:42.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e145e98ce4bc2151-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.172 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.172 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.173 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.173 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.173 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.173 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.173 DEBUG:boto:Data: 2024-04-12T03:51:42.173 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.173 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.173 DEBUG:boto:Params: {} 2024-04-12T03:51:42.173 DEBUG:boto:Token: None 2024-04-12T03:51:42.174 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.174 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DwKF68xFiWxejMEX5V5mmTrJtOo= 2024-04-12T03:51:42.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DwKF68xFiWxejMEX5V5mmTrJtOo=', 'Content-Length': '0'} 2024-04-12T03:51:42.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7ad50cb488012b-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:51:42.176 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.176 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.176 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.176 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.176 DEBUG:boto:Method: GET 2024-04-12T03:51:42.176 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.176 DEBUG:boto:Data: 2024-04-12T03:51:42.176 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.177 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.177 DEBUG:boto:Params: {} 2024-04-12T03:51:42.177 DEBUG:boto:Token: None 2024-04-12T03:51:42.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.177 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E= 2024-04-12T03:51:42.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E=', 'Content-Length': '0'} 2024-04-12T03:51:42.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069e16b0e74f3128a-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.179 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.179 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.179 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.179 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.179 DEBUG:boto:Method: GET 2024-04-12T03:51:42.179 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.179 DEBUG:boto:Data: 2024-04-12T03:51:42.179 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.180 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.180 DEBUG:boto:Params: {} 2024-04-12T03:51:42.180 DEBUG:boto:Token: None 2024-04-12T03:51:42.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:51:42.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E= 2024-04-12T03:51:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CZ3u1PulP6V915jDbg5bh5n+y1E=', 'Content-Length': '0'} 2024-04-12T03:51:42.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014c00da9c6824813-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.182 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.183 DEBUG:boto:Method: GET 2024-04-12T03:51:42.183 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.183 DEBUG:boto:Data: 2024-04-12T03:51:42.183 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.183 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.183 DEBUG:boto:Params: {} 2024-04-12T03:51:42.183 DEBUG:boto:Token: None 2024-04-12T03:51:42.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9ed6956db8b57c3-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.186 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.186 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.187 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.187 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.187 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.187 DEBUG:boto:Method: GET 2024-04-12T03:51:42.187 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.187 DEBUG:boto:Data: 2024-04-12T03:51:42.187 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.187 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.187 DEBUG:boto:Params: {} 2024-04-12T03:51:42.187 DEBUG:boto:Token: None 2024-04-12T03:51:42.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1db20982b088b45-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.191 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.191 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.191 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.191 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.191 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:51:42.191 DEBUG:boto:Data: 2024-04-12T03:51:42.192 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.192 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.192 DEBUG:boto:Params: {} 2024-04-12T03:51:42.192 DEBUG:boto:Token: None 2024-04-12T03:51:42.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92 2024-04-12T03:51:42.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9PQtf5Emp6K1YP2RaJOH0MOR6oE= 2024-04-12T03:51:42.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9PQtf5Emp6K1YP2RaJOH0MOR6oE=', 'Content-Length': '0'} 2024-04-12T03:51:42.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db749efa9d13d726-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.194 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.194 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.194 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.195 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:51:42.195 DEBUG:boto:Data: 2024-04-12T03:51:42.195 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.195 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.195 DEBUG:boto:Params: {} 2024-04-12T03:51:42.195 DEBUG:boto:Token: None 2024-04-12T03:51:42.195 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92 2024-04-12T03:51:42.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9PQtf5Emp6K1YP2RaJOH0MOR6oE= 2024-04-12T03:51:42.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9PQtf5Emp6K1YP2RaJOH0MOR6oE=', 'Content-Length': '0'} 2024-04-12T03:51:42.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a1e557b12ad52f-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:51:42.197 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.198 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.198 DEBUG:boto:Method: GET 2024-04-12T03:51:42.198 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:51:42.198 DEBUG:boto:Data: 2024-04-12T03:51:42.198 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.198 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.198 DEBUG:boto:Params: {} 2024-04-12T03:51:42.198 DEBUG:boto:Token: None 2024-04-12T03:51:42.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92 2024-04-12T03:51:42.198 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpEgiCCbc5UFSoU6wJVmicj6/dk= 2024-04-12T03:51:42.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpEgiCCbc5UFSoU6wJVmicj6/dk=', 'Content-Length': '0'} 2024-04-12T03:51:42.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b171813e5ff51ac4-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.200 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:51:42.200 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:51:42.200 DEBUG:boto:Method: GET 2024-04-12T03:51:42.201 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:51:42.201 DEBUG:boto:Data: 2024-04-12T03:51:42.201 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.201 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.201 DEBUG:boto:Params: {} 2024-04-12T03:51:42.201 DEBUG:boto:Token: None 2024-04-12T03:51:42.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj92 2024-04-12T03:51:42.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpEgiCCbc5UFSoU6wJVmicj6/dk= 2024-04-12T03:51:42.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpEgiCCbc5UFSoU6wJVmicj6/dk=', 'Content-Length': '0'} 2024-04-12T03:51:42.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfa5fdc956fe65c-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.246 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.246 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.247 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.247 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.247 DEBUG:boto:Method: GET 2024-04-12T03:51:42.247 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.247 DEBUG:boto:Data: 2024-04-12T03:51:42.247 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.247 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.247 DEBUG:boto:Params: {} 2024-04-12T03:51:42.247 DEBUG:boto:Token: None 2024-04-12T03:51:42.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eea21a286e42101e-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.250 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.250 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.251 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.251 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.251 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.251 DEBUG:boto:Method: GET 2024-04-12T03:51:42.251 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:51:42.251 DEBUG:boto:Data: 2024-04-12T03:51:42.251 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.251 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.251 DEBUG:boto:Params: {} 2024-04-12T03:51:42.251 DEBUG:boto:Token: None 2024-04-12T03:51:42.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.252 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000986fecfb56a19b1a-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.255 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:51:42.255 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.255 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.256 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.256 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.256 DEBUG:boto:Method: GET 2024-04-12T03:51:42.256 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.256 DEBUG:boto:Data: 2024-04-12T03:51:42.256 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.256 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.256 DEBUG:boto:Params: {} 2024-04-12T03:51:42.256 DEBUG:boto:Token: None 2024-04-12T03:51:42.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w= 2024-04-12T03:51:42.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w=', 'Content-Length': '0'} 2024-04-12T03:51:42.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fad9c5b7f5aa9e-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.259 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.259 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.259 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.259 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.259 DEBUG:boto:Method: GET 2024-04-12T03:51:42.259 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.259 DEBUG:boto:Data: 2024-04-12T03:51:42.259 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.260 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.260 DEBUG:boto:Params: {} 2024-04-12T03:51:42.260 DEBUG:boto:Token: None 2024-04-12T03:51:42.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w= 2024-04-12T03:51:42.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w=', 'Content-Length': '0'} 2024-04-12T03:51:42.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c759549a565573-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.262 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.262 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.262 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.262 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.263 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.263 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.263 DEBUG:boto:Data: 2024-04-12T03:51:42.263 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.263 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.263 DEBUG:boto:Params: {} 2024-04-12T03:51:42.263 DEBUG:boto:Token: None 2024-04-12T03:51:42.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UWmwpFaO+esXZKoEGq0Dr2IFECk= 2024-04-12T03:51:42.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UWmwpFaO+esXZKoEGq0Dr2IFECk=', 'Content-Length': '0'} 2024-04-12T03:51:42.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422cf5ae22b858c5-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.265 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.265 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.265 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.265 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.266 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.266 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.266 DEBUG:boto:Data: 2024-04-12T03:51:42.266 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.266 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.266 DEBUG:boto:Params: {} 2024-04-12T03:51:42.266 DEBUG:boto:Token: None 2024-04-12T03:51:42.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UWmwpFaO+esXZKoEGq0Dr2IFECk= 2024-04-12T03:51:42.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UWmwpFaO+esXZKoEGq0Dr2IFECk=', 'Content-Length': '0'} 2024-04-12T03:51:42.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f93c5925efb115-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:51:42.268 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.268 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.268 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.269 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.269 DEBUG:boto:Method: GET 2024-04-12T03:51:42.269 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.269 DEBUG:boto:Data: 2024-04-12T03:51:42.269 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.269 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.269 DEBUG:boto:Params: {} 2024-04-12T03:51:42.269 DEBUG:boto:Token: None 2024-04-12T03:51:42.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w= 2024-04-12T03:51:42.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w=', 'Content-Length': '0'} 2024-04-12T03:51:42.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba25c4f4a2b719f3-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.314 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.314 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.315 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.315 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.315 DEBUG:boto:Method: GET 2024-04-12T03:51:42.315 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.315 DEBUG:boto:Data: 2024-04-12T03:51:42.315 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.315 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.315 DEBUG:boto:Params: {} 2024-04-12T03:51:42.315 DEBUG:boto:Token: None 2024-04-12T03:51:42.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:51:42.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w= 2024-04-12T03:51:42.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RQx9ckib4fBEUvmFcJKU7IfKs1w=', 'Content-Length': '0'} 2024-04-12T03:51:42.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655252926a0e445f-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.317 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.318 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.318 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.318 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.318 DEBUG:boto:Method: GET 2024-04-12T03:51:42.318 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.318 DEBUG:boto:Data: 2024-04-12T03:51:42.318 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.318 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.318 DEBUG:boto:Params: {} 2024-04-12T03:51:42.319 DEBUG:boto:Token: None 2024-04-12T03:51:42.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.319 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a039fa33fdb1c8-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.321 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.321 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.322 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.322 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.322 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.322 DEBUG:boto:Method: GET 2024-04-12T03:51:42.322 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.322 DEBUG:boto:Data: 2024-04-12T03:51:42.322 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.322 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.322 DEBUG:boto:Params: {} 2024-04-12T03:51:42.323 DEBUG:boto:Token: None 2024-04-12T03:51:42.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee5f4b2dab21f26-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.326 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.327 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.327 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.327 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.327 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:51:42.327 DEBUG:boto:Data: 2024-04-12T03:51:42.327 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.327 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.327 DEBUG:boto:Params: {} 2024-04-12T03:51:42.328 DEBUG:boto:Token: None 2024-04-12T03:51:42.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93 2024-04-12T03:51:42.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dSpe8SBN4OkhAUL9H1xhG1YvHSw= 2024-04-12T03:51:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dSpe8SBN4OkhAUL9H1xhG1YvHSw=', 'Content-Length': '0'} 2024-04-12T03:51:42.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5472a07312f934-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.330 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.330 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.330 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.330 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:51:42.330 DEBUG:boto:Data: 2024-04-12T03:51:42.330 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.331 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.331 DEBUG:boto:Params: {} 2024-04-12T03:51:42.331 DEBUG:boto:Token: None 2024-04-12T03:51:42.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93 2024-04-12T03:51:42.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dSpe8SBN4OkhAUL9H1xhG1YvHSw= 2024-04-12T03:51:42.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dSpe8SBN4OkhAUL9H1xhG1YvHSw=', 'Content-Length': '0'} 2024-04-12T03:51:42.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7579fc556439a7-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:51:42.333 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.333 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.333 DEBUG:boto:Method: GET 2024-04-12T03:51:42.333 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:51:42.333 DEBUG:boto:Data: 2024-04-12T03:51:42.333 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.334 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.334 DEBUG:boto:Params: {} 2024-04-12T03:51:42.334 DEBUG:boto:Token: None 2024-04-12T03:51:42.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93 2024-04-12T03:51:42.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cClL3m/XReNYjIcxnCY4udjGo/8= 2024-04-12T03:51:42.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cClL3m/XReNYjIcxnCY4udjGo/8=', 'Content-Length': '0'} 2024-04-12T03:51:42.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063066d4969c99fea-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.336 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:51:42.336 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:51:42.336 DEBUG:boto:Method: GET 2024-04-12T03:51:42.336 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:51:42.336 DEBUG:boto:Data: 2024-04-12T03:51:42.336 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.337 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.337 DEBUG:boto:Params: {} 2024-04-12T03:51:42.337 DEBUG:boto:Token: None 2024-04-12T03:51:42.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj93 2024-04-12T03:51:42.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cClL3m/XReNYjIcxnCY4udjGo/8= 2024-04-12T03:51:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cClL3m/XReNYjIcxnCY4udjGo/8=', 'Content-Length': '0'} 2024-04-12T03:51:42.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004efc0627cc01972d-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.382 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.382 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.383 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.383 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.383 DEBUG:boto:Method: GET 2024-04-12T03:51:42.383 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.383 DEBUG:boto:Data: 2024-04-12T03:51:42.383 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.383 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.383 DEBUG:boto:Params: {} 2024-04-12T03:51:42.383 DEBUG:boto:Token: None 2024-04-12T03:51:42.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094621c7118ee3af-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.386 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.386 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.386 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.386 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.386 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.387 DEBUG:boto:Method: GET 2024-04-12T03:51:42.387 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:51:42.387 DEBUG:boto:Data: 2024-04-12T03:51:42.387 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.387 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.387 DEBUG:boto:Params: {} 2024-04-12T03:51:42.387 DEBUG:boto:Token: None 2024-04-12T03:51:42.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d6ae534946f004-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.391 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:51:42.391 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.391 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.391 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.391 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.391 DEBUG:boto:Method: GET 2024-04-12T03:51:42.392 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.392 DEBUG:boto:Data: 2024-04-12T03:51:42.392 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.392 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.392 DEBUG:boto:Params: {} 2024-04-12T03:51:42.392 DEBUG:boto:Token: None 2024-04-12T03:51:42.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0= 2024-04-12T03:51:42.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0=', 'Content-Length': '0'} 2024-04-12T03:51:42.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab6974b68341dd8-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.394 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.395 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.395 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.395 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.395 DEBUG:boto:Method: GET 2024-04-12T03:51:42.395 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.395 DEBUG:boto:Data: 2024-04-12T03:51:42.395 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.395 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.395 DEBUG:boto:Params: {} 2024-04-12T03:51:42.395 DEBUG:boto:Token: None 2024-04-12T03:51:42.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0= 2024-04-12T03:51:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0=', 'Content-Length': '0'} 2024-04-12T03:51:42.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f6e92468ad6934-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.397 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.397 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.398 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.398 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.398 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.398 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.398 DEBUG:boto:Data: 2024-04-12T03:51:42.398 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.398 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.398 DEBUG:boto:Params: {} 2024-04-12T03:51:42.398 DEBUG:boto:Token: None 2024-04-12T03:51:42.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.399 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iZubycSSynzMAd/rtXtbWKLyZmM= 2024-04-12T03:51:42.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iZubycSSynzMAd/rtXtbWKLyZmM=', 'Content-Length': '0'} 2024-04-12T03:51:42.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005138de9b945b455b-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.401 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.401 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.401 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.401 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.401 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.401 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.401 DEBUG:boto:Data: 2024-04-12T03:51:42.401 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.402 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.402 DEBUG:boto:Params: {} 2024-04-12T03:51:42.402 DEBUG:boto:Token: None 2024-04-12T03:51:42.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iZubycSSynzMAd/rtXtbWKLyZmM= 2024-04-12T03:51:42.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iZubycSSynzMAd/rtXtbWKLyZmM=', 'Content-Length': '0'} 2024-04-12T03:51:42.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc372e1fe492b18-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:51:42.404 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.404 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.404 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.404 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.404 DEBUG:boto:Method: GET 2024-04-12T03:51:42.404 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.404 DEBUG:boto:Data: 2024-04-12T03:51:42.404 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.405 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.405 DEBUG:boto:Params: {} 2024-04-12T03:51:42.405 DEBUG:boto:Token: None 2024-04-12T03:51:42.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0= 2024-04-12T03:51:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0=', 'Content-Length': '0'} 2024-04-12T03:51:42.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bef54ebeba39e1c-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.407 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.407 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.407 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.407 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.407 DEBUG:boto:Method: GET 2024-04-12T03:51:42.407 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.407 DEBUG:boto:Data: 2024-04-12T03:51:42.407 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.408 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.408 DEBUG:boto:Params: {} 2024-04-12T03:51:42.408 DEBUG:boto:Token: None 2024-04-12T03:51:42.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:51:42.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0= 2024-04-12T03:51:42.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByCoCfjx0Ly/ftYK5YK6nS3CqL0=', 'Content-Length': '0'} 2024-04-12T03:51:42.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1816a7be8e8eeb-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.410 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.410 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.410 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.410 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.410 DEBUG:boto:Method: GET 2024-04-12T03:51:42.410 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.411 DEBUG:boto:Data: 2024-04-12T03:51:42.411 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.411 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.411 DEBUG:boto:Params: {} 2024-04-12T03:51:42.411 DEBUG:boto:Token: None 2024-04-12T03:51:42.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f2fb90fcf9a703-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.454 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.455 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.455 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.455 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.455 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.455 DEBUG:boto:Method: GET 2024-04-12T03:51:42.455 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.455 DEBUG:boto:Data: 2024-04-12T03:51:42.455 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.456 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.456 DEBUG:boto:Params: {} 2024-04-12T03:51:42.456 DEBUG:boto:Token: None 2024-04-12T03:51:42.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000288ddd72515405d8-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.459 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.460 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.460 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.460 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.460 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:51:42.460 DEBUG:boto:Data: 2024-04-12T03:51:42.460 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.460 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.460 DEBUG:boto:Params: {} 2024-04-12T03:51:42.461 DEBUG:boto:Token: None 2024-04-12T03:51:42.461 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95 2024-04-12T03:51:42.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LF/zJDZyx5s88Qe1YvugtonBs50= 2024-04-12T03:51:42.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LF/zJDZyx5s88Qe1YvugtonBs50=', 'Content-Length': '0'} 2024-04-12T03:51:42.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6bb89d0a9be3ffb-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.463 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.463 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.463 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.463 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:51:42.463 DEBUG:boto:Data: 2024-04-12T03:51:42.463 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.463 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.463 DEBUG:boto:Params: {} 2024-04-12T03:51:42.463 DEBUG:boto:Token: None 2024-04-12T03:51:42.464 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95 2024-04-12T03:51:42.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LF/zJDZyx5s88Qe1YvugtonBs50= 2024-04-12T03:51:42.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LF/zJDZyx5s88Qe1YvugtonBs50=', 'Content-Length': '0'} 2024-04-12T03:51:42.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9210f62ceadde9-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:51:42.465 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.466 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.466 DEBUG:boto:Method: GET 2024-04-12T03:51:42.466 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:51:42.466 DEBUG:boto:Data: 2024-04-12T03:51:42.466 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.466 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.466 DEBUG:boto:Params: {} 2024-04-12T03:51:42.466 DEBUG:boto:Token: None 2024-04-12T03:51:42.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95 2024-04-12T03:51:42.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:91XF7SuiGmiDu+exYqQEGpaJ5QI= 2024-04-12T03:51:42.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:91XF7SuiGmiDu+exYqQEGpaJ5QI=', 'Content-Length': '0'} 2024-04-12T03:51:42.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050289825f198c5cd-006618afce-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.469 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:51:42.469 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:51:42.469 DEBUG:boto:Method: GET 2024-04-12T03:51:42.469 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:51:42.469 DEBUG:boto:Data: 2024-04-12T03:51:42.469 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.469 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.469 DEBUG:boto:Params: {} 2024-04-12T03:51:42.469 DEBUG:boto:Token: None 2024-04-12T03:51:42.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj95 2024-04-12T03:51:42.470 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:91XF7SuiGmiDu+exYqQEGpaJ5QI= 2024-04-12T03:51:42.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:91XF7SuiGmiDu+exYqQEGpaJ5QI=', 'Content-Length': '0'} 2024-04-12T03:51:42.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729b96eebf37abe3-006618afce-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.514 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.514 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.515 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.515 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.515 DEBUG:boto:Method: GET 2024-04-12T03:51:42.515 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.515 DEBUG:boto:Data: 2024-04-12T03:51:42.515 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:51:42.515 DEBUG:boto:Port: 8001 2024-04-12T03:51:42.515 DEBUG:boto:Params: {} 2024-04-12T03:51:42.515 DEBUG:boto:Token: None 2024-04-12T03:51:42.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee85101009b29b6b-006618afce-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.518 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.518 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.519 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.519 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.519 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.519 DEBUG:boto:Method: GET 2024-04-12T03:51:42.519 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:51:42.519 DEBUG:boto:Data: 2024-04-12T03:51:42.519 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.519 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.519 DEBUG:boto:Params: {} 2024-04-12T03:51:42.520 DEBUG:boto:Token: None 2024-04-12T03:51:42.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0= 2024-04-12T03:51:42.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqyl7vdgP2ILeq42FzKgbSktES0=', 'Content-Length': '0'} 2024-04-12T03:51:42.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007257397e2fa3d5f2-006618afce-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.523 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:51:42.524 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:51:42.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:51:42.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:51:42.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:51:42.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:51:42.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:51:42.624 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:51:42.625 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-61 2024-04-12T03:51:42.636 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2024-04-12T03:51:42.695 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.695 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.696 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.696 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:42.696 DEBUG:boto:Data: 2024-04-12T03:51:42.696 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.696 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.696 DEBUG:boto:Params: {} 2024-04-12T03:51:42.696 DEBUG:boto:Token: None 2024-04-12T03:51:42.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.697 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8= 2024-04-12T03:51:42.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8=', 'Content-Length': '0'} 2024-04-12T03:51:42.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da4d78e33f44ccac-006618afce-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.701 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:51:42.701 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:51:42.701 DEBUG:boto:Method: PUT 2024-04-12T03:51:42.701 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:51:42.701 DEBUG:boto:Data: 2024-04-12T03:51:42.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:51:42.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.701 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.702 DEBUG:boto:Params: {} 2024-04-12T03:51:42.702 DEBUG:boto:Token: None 2024-04-12T03:51:42.702 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj37 2024-04-12T03:51:42.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xEiSunY8IpKHRd3rsz7n1CoQkRg= 2024-04-12T03:51:42.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xEiSunY8IpKHRd3rsz7n1CoQkRg='} 2024-04-12T03:51:42.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000338c61f7bcb6f1dc-006618afce-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.707 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.707 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.707 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.707 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:42.707 DEBUG:boto:Data: 2024-04-12T03:51:42.708 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.708 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.708 DEBUG:boto:Params: {} 2024-04-12T03:51:42.708 DEBUG:boto:Token: None 2024-04-12T03:51:42.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8= 2024-04-12T03:51:42.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8=', 'Content-Length': '0'} 2024-04-12T03:51:42.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ca83f13c0ea8b5c-006618afce-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.711 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:51:42.712 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:51:42.712 DEBUG:boto:Method: PUT 2024-04-12T03:51:42.712 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:51:42.712 DEBUG:boto:Data: 2024-04-12T03:51:42.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:51:42.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.712 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.712 DEBUG:boto:Params: {} 2024-04-12T03:51:42.712 DEBUG:boto:Token: None 2024-04-12T03:51:42.713 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj74 2024-04-12T03:51:42.713 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NmgGcGGstyqwEon+pXLpkJQrIPc= 2024-04-12T03:51:42.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NmgGcGGstyqwEon+pXLpkJQrIPc='} 2024-04-12T03:51:42.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d198e320526eb060-006618afce-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.755 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.755 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.755 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.755 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:42.755 DEBUG:boto:Data: 2024-04-12T03:51:42.755 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.755 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.755 DEBUG:boto:Params: {} 2024-04-12T03:51:42.756 DEBUG:boto:Token: None 2024-04-12T03:51:42.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8= 2024-04-12T03:51:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8=', 'Content-Length': '0'} 2024-04-12T03:51:42.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a8fee7774a3ca97-006618afce-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.759 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:51:42.759 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:51:42.760 DEBUG:boto:Method: PUT 2024-04-12T03:51:42.760 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:51:42.760 DEBUG:boto:Data: 2024-04-12T03:51:42.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:51:42.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.760 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.760 DEBUG:boto:Params: {} 2024-04-12T03:51:42.760 DEBUG:boto:Token: None 2024-04-12T03:51:42.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj111 2024-04-12T03:51:42.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JqDwGiLvDQH6bSS7wi5divwKIVk= 2024-04-12T03:51:42.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JqDwGiLvDQH6bSS7wi5divwKIVk='} 2024-04-12T03:51:42.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089106324817aae2f-006618afce-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.766 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.766 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.766 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.766 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:42.766 DEBUG:boto:Data: 2024-04-12T03:51:42.766 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.766 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.766 DEBUG:boto:Params: {} 2024-04-12T03:51:42.767 DEBUG:boto:Token: None 2024-04-12T03:51:42.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8= 2024-04-12T03:51:42.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8=', 'Content-Length': '0'} 2024-04-12T03:51:42.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa28dd60193377b-006618afce-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.770 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:51:42.770 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:51:42.770 DEBUG:boto:Method: PUT 2024-04-12T03:51:42.771 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:51:42.771 DEBUG:boto:Data: 2024-04-12T03:51:42.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:51:42.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.771 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.771 DEBUG:boto:Params: {} 2024-04-12T03:51:42.771 DEBUG:boto:Token: None 2024-04-12T03:51:42.771 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj148 2024-04-12T03:51:42.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h4YwufQ/zZQOewVJ+I/iHMgdW3s= 2024-04-12T03:51:42.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h4YwufQ/zZQOewVJ+I/iHMgdW3s='} 2024-04-12T03:51:42.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c4ca1f6a508f78c-006618afce-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.815 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:51:42.815 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:51:42.815 DEBUG:boto:Method: HEAD 2024-04-12T03:51:42.815 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:51:42.815 DEBUG:boto:Data: 2024-04-12T03:51:42.815 DEBUG:boto:Headers: {} 2024-04-12T03:51:42.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.815 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.815 DEBUG:boto:Params: {} 2024-04-12T03:51:42.815 DEBUG:boto:Token: None 2024-04-12T03:51:42.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/ 2024-04-12T03:51:42.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8= 2024-04-12T03:51:42.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hYWRLkcI/quJhu2x6QD2dxK9kt8=', 'Content-Length': '0'} 2024-04-12T03:51:42.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087e2677922672ae6-006618afce-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.819 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:51:42.819 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:51:42.819 DEBUG:boto:Method: PUT 2024-04-12T03:51:42.820 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:51:42.820 DEBUG:boto:Data: 2024-04-12T03:51:42.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:51:42.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:51:42.820 DEBUG:boto:Port: 8000 2024-04-12T03:51:42.820 DEBUG:boto:Params: {} 2024-04-12T03:51:42.820 DEBUG:boto:Token: None 2024-04-12T03:51:42.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:51:42 GMT /dgbpxg-61/obj185 2024-04-12T03:51:42.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PNKbHNHxrzCh4F49toIgYxUz23g= 2024-04-12T03:51:42.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:51:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PNKbHNHxrzCh4F49toIgYxUz23g='} 2024-04-12T03:51:42.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc2bb15cc066aa16-006618afce-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:51:42.826 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:51:42.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:42.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:42.826 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:51:42.881 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:51:42.882 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:51:42.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:42.924+0000 7f354ca71780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T03:51:47.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:47.925+0000 7f354ca71780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T03:51:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:52.925+0000 7f354ca71780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T03:51:57.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:51:57.927+0000 7f354ca71780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T03:52:02.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:02.930+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:02.923 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:02.923 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:07.932+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:07.925 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:07.925 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:12.935+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:12.927 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:12.927 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:17.937+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:17.930 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:17.930 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:22.940+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:22.932 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:22.932 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:27.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:27.942+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:27.935 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:27.935 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:32.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:32.944+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:32.937 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:32.937 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:37.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:37.948+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:37.940 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:37.941 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:42.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:42.950+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:42.942 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:42.942 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:47.953+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:47.945 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:47.945 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:52.955+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:52.947 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:52.947 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:52:57.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:52:57.958+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:57.950 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:52:57.950 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:02.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:02.960+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:02.952 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:53:02.952 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:07.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:07.963+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:07.955 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:53:07.955 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:12.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:12.965+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:12.957 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:53:12.957 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:17.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:17.967+0000 7f354ca71780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:17.960 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T03:53:17.960 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:22.972+0000 7f354ca71780 1 bucket sync caught up with source: 2024-04-12T03:53:22.964 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:22.964 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:22.972+0000 7f354ca71780 0 bucket checkpoint complete 2024-04-12T03:53:22.981 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:53:22.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:22.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:22.982 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:53:23.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:23.044 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:23.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:23.090+0000 7f031df44780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:23.081 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000006.2249.6, 00000000005.1502.6] 2024-04-12T03:53:23.081 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:28.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:28.091+0000 7f031df44780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:28.083 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000006.2249.6, 00000000005.1502.6] 2024-04-12T03:53:28.083 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:33.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:33.093+0000 7f031df44780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:33.084 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000004.162.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000006.2249.6, 00000000005.1502.6] 2024-04-12T03:53:33.084 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:38.094+0000 7f031df44780 1 bucket sync caught up with source: 2024-04-12T03:53:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1083.6, 00000000002.2097.6, 00000000002.1936.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000001.857.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:38.094+0000 7f031df44780 0 bucket checkpoint complete 2024-04-12T03:53:38.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:53:38.100 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.100 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.100 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.101 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:53:38.101 DEBUG:boto:Data: 2024-04-12T03:53:38.101 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.101 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.101 DEBUG:boto:Params: {} 2024-04-12T03:53:38.102 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:53:38.102 DEBUG:boto:Token: None 2024-04-12T03:53:38.102 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9FbqIFG5kVenBEXpGtKCNZG5t0g= 2024-04-12T03:53:38.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9FbqIFG5kVenBEXpGtKCNZG5t0g=', 'Content-Length': '0'} 2024-04-12T03:53:38.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c775ea1e7ee6a566-006618b042-1e69-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.116 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.116 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.116 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.116 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:53:38.116 DEBUG:boto:Data: 2024-04-12T03:53:38.116 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.116 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.116 DEBUG:boto:Params: {} 2024-04-12T03:53:38.116 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:53:38.117 DEBUG:boto:Token: None 2024-04-12T03:53:38.117 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9FbqIFG5kVenBEXpGtKCNZG5t0g= 2024-04-12T03:53:38.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9FbqIFG5kVenBEXpGtKCNZG5t0g=', 'Content-Length': '0'} 2024-04-12T03:53:38.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898df9ade9d08eef-006618b042-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:53:38.132 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.132 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.132 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.133 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.133 DEBUG:boto:Method: GET 2024-04-12T03:53:38.133 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.133 DEBUG:boto:Data: 2024-04-12T03:53:38.133 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.133 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.133 DEBUG:boto:Params: {} 2024-04-12T03:53:38.133 DEBUG:boto:Token: None 2024-04-12T03:53:38.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/?versions 2024-04-12T03:53:38.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk= 2024-04-12T03:53:38.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk=', 'Content-Length': '0'} 2024-04-12T03:53:38.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c87e19d07c1775-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.136 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:53:38.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:53:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:53:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:53:38.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:53:38.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:53:38.142 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.142 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.142 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.142 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.142 DEBUG:boto:Method: GET 2024-04-12T03:53:38.142 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.142 DEBUG:boto:Data: 2024-04-12T03:53:38.143 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.143 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.143 DEBUG:boto:Params: {} 2024-04-12T03:53:38.143 DEBUG:boto:Token: None 2024-04-12T03:53:38.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/?versions 2024-04-12T03:53:38.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk= 2024-04-12T03:53:38.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk=', 'Content-Length': '0'} 2024-04-12T03:53:38.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000932d6ff64b6fe343-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.147 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:53:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:53:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:53:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:53:38.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:53:38.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:53:38.152 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.152 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.152 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.152 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.152 DEBUG:boto:Method: GET 2024-04-12T03:53:38.152 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.152 DEBUG:boto:Data: 2024-04-12T03:53:38.152 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.152 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.152 DEBUG:boto:Params: {} 2024-04-12T03:53:38.153 DEBUG:boto:Token: None 2024-04-12T03:53:38.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/?versions 2024-04-12T03:53:38.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk= 2024-04-12T03:53:38.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk=', 'Content-Length': '0'} 2024-04-12T03:53:38.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e9bab19a1c2854-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.155 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.157 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.157 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.157 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.157 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.157 DEBUG:boto:Method: GET 2024-04-12T03:53:38.157 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:53:38.158 DEBUG:boto:Data: 2024-04-12T03:53:38.158 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.158 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.158 DEBUG:boto:Params: {} 2024-04-12T03:53:38.158 DEBUG:boto:Token: None 2024-04-12T03:53:38.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/?versions 2024-04-12T03:53:38.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk= 2024-04-12T03:53:38.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ykJIB5jvR2qqOAT5hQdbJIDJuSk=', 'Content-Length': '0'} 2024-04-12T03:53:38.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e90618467a32d909-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.162 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:53:38.164 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.164 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.164 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.165 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.165 DEBUG:boto:Method: GET 2024-04-12T03:53:38.165 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.165 DEBUG:boto:Data: 2024-04-12T03:53:38.165 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.165 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.165 DEBUG:boto:Params: {} 2024-04-12T03:53:38.165 DEBUG:boto:Token: None 2024-04-12T03:53:38.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.166 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM= 2024-04-12T03:53:38.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM=', 'Content-Length': '0'} 2024-04-12T03:53:38.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd272288b3ffec68-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.210 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.210 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.211 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.211 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.211 DEBUG:boto:Method: GET 2024-04-12T03:53:38.211 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.211 DEBUG:boto:Data: 2024-04-12T03:53:38.211 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.211 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.211 DEBUG:boto:Params: {} 2024-04-12T03:53:38.211 DEBUG:boto:Token: None 2024-04-12T03:53:38.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM= 2024-04-12T03:53:38.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM=', 'Content-Length': '0'} 2024-04-12T03:53:38.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb6412444080765-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.214 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.214 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.214 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.214 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.214 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.214 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.214 DEBUG:boto:Data: 2024-04-12T03:53:38.215 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.215 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.215 DEBUG:boto:Params: {} 2024-04-12T03:53:38.215 DEBUG:boto:Token: None 2024-04-12T03:53:38.215 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vsnhBswJbNSC0S9irV8eqL38sVM= 2024-04-12T03:53:38.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vsnhBswJbNSC0S9irV8eqL38sVM=', 'Content-Length': '0'} 2024-04-12T03:53:38.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbfffaf7556e0012-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.217 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.217 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.217 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.217 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.217 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.217 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.218 DEBUG:boto:Data: 2024-04-12T03:53:38.218 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.218 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.218 DEBUG:boto:Params: {} 2024-04-12T03:53:38.218 DEBUG:boto:Token: None 2024-04-12T03:53:38.218 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vsnhBswJbNSC0S9irV8eqL38sVM= 2024-04-12T03:53:38.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vsnhBswJbNSC0S9irV8eqL38sVM=', 'Content-Length': '0'} 2024-04-12T03:53:38.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326ea2cbb149e859-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:53:38.220 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.220 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.220 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.220 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.221 DEBUG:boto:Method: GET 2024-04-12T03:53:38.221 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.221 DEBUG:boto:Data: 2024-04-12T03:53:38.221 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.221 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.221 DEBUG:boto:Params: {} 2024-04-12T03:53:38.221 DEBUG:boto:Token: None 2024-04-12T03:53:38.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM= 2024-04-12T03:53:38.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM=', 'Content-Length': '0'} 2024-04-12T03:53:38.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000709e5ded97d96ae9-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.223 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.223 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.224 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.224 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.224 DEBUG:boto:Method: GET 2024-04-12T03:53:38.224 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.224 DEBUG:boto:Data: 2024-04-12T03:53:38.224 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.224 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.224 DEBUG:boto:Params: {} 2024-04-12T03:53:38.224 DEBUG:boto:Token: None 2024-04-12T03:53:38.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:53:38.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM= 2024-04-12T03:53:38.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XhsQllDjjTrBLeMAYq+FVgcGKHM=', 'Content-Length': '0'} 2024-04-12T03:53:38.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aabe88c7db149e24-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.227 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.227 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.227 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.227 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.227 DEBUG:boto:Method: GET 2024-04-12T03:53:38.227 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.227 DEBUG:boto:Data: 2024-04-12T03:53:38.227 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.227 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.227 DEBUG:boto:Params: {} 2024-04-12T03:53:38.228 DEBUG:boto:Token: None 2024-04-12T03:53:38.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004955b5fa8ee71bba-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.230 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.231 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.231 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.231 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.231 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.231 DEBUG:boto:Method: GET 2024-04-12T03:53:38.231 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.231 DEBUG:boto:Data: 2024-04-12T03:53:38.231 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.231 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.231 DEBUG:boto:Params: {} 2024-04-12T03:53:38.232 DEBUG:boto:Token: None 2024-04-12T03:53:38.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0bd5e407dfc0937-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.236 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.236 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.236 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.237 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.237 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:53:38.237 DEBUG:boto:Data: 2024-04-12T03:53:38.237 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.237 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.237 DEBUG:boto:Params: {} 2024-04-12T03:53:38.237 DEBUG:boto:Token: None 2024-04-12T03:53:38.237 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11 2024-04-12T03:53:38.237 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K6g9LpxkrI29sOiBpKKBDPFXfwI= 2024-04-12T03:53:38.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K6g9LpxkrI29sOiBpKKBDPFXfwI=', 'Content-Length': '0'} 2024-04-12T03:53:38.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4f35c3636b614e-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.239 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.239 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.240 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.240 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:53:38.240 DEBUG:boto:Data: 2024-04-12T03:53:38.240 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.240 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.240 DEBUG:boto:Params: {} 2024-04-12T03:53:38.240 DEBUG:boto:Token: None 2024-04-12T03:53:38.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11 2024-04-12T03:53:38.240 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K6g9LpxkrI29sOiBpKKBDPFXfwI= 2024-04-12T03:53:38.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K6g9LpxkrI29sOiBpKKBDPFXfwI=', 'Content-Length': '0'} 2024-04-12T03:53:38.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8092fc3e61ceb36-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:53:38.242 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.242 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.242 DEBUG:boto:Method: GET 2024-04-12T03:53:38.243 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:53:38.243 DEBUG:boto:Data: 2024-04-12T03:53:38.243 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.243 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.243 DEBUG:boto:Params: {} 2024-04-12T03:53:38.243 DEBUG:boto:Token: None 2024-04-12T03:53:38.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11 2024-04-12T03:53:38.243 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+oyBI9WYRFIVS4Xa24X1UKGhh4= 2024-04-12T03:53:38.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+oyBI9WYRFIVS4Xa24X1UKGhh4=', 'Content-Length': '0'} 2024-04-12T03:53:38.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6a3e4e549c033d-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.245 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:53:38.245 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:53:38.245 DEBUG:boto:Method: GET 2024-04-12T03:53:38.245 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:53:38.246 DEBUG:boto:Data: 2024-04-12T03:53:38.246 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.246 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.246 DEBUG:boto:Params: {} 2024-04-12T03:53:38.246 DEBUG:boto:Token: None 2024-04-12T03:53:38.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj11 2024-04-12T03:53:38.246 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:v+oyBI9WYRFIVS4Xa24X1UKGhh4= 2024-04-12T03:53:38.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+oyBI9WYRFIVS4Xa24X1UKGhh4=', 'Content-Length': '0'} 2024-04-12T03:53:38.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087551d942cbf9f23-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.290 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.291 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.291 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.291 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.291 DEBUG:boto:Method: GET 2024-04-12T03:53:38.291 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.291 DEBUG:boto:Data: 2024-04-12T03:53:38.291 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.291 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.291 DEBUG:boto:Params: {} 2024-04-12T03:53:38.292 DEBUG:boto:Token: None 2024-04-12T03:53:38.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f779f7b4f63f7d-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.294 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.295 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.295 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.295 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.295 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.295 DEBUG:boto:Method: GET 2024-04-12T03:53:38.295 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:53:38.295 DEBUG:boto:Data: 2024-04-12T03:53:38.295 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.295 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.296 DEBUG:boto:Params: {} 2024-04-12T03:53:38.296 DEBUG:boto:Token: None 2024-04-12T03:53:38.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8339e31091b7b6-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.300 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:53:38.301 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.301 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.301 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.301 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.301 DEBUG:boto:Method: GET 2024-04-12T03:53:38.301 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.301 DEBUG:boto:Data: 2024-04-12T03:53:38.301 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.301 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.301 DEBUG:boto:Params: {} 2024-04-12T03:53:38.302 DEBUG:boto:Token: None 2024-04-12T03:53:38.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.302 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0= 2024-04-12T03:53:38.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0=', 'Content-Length': '0'} 2024-04-12T03:53:38.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003749a3f5aa31031e-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.304 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.304 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.304 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.305 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.305 DEBUG:boto:Method: GET 2024-04-12T03:53:38.305 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.305 DEBUG:boto:Data: 2024-04-12T03:53:38.305 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.305 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.305 DEBUG:boto:Params: {} 2024-04-12T03:53:38.305 DEBUG:boto:Token: None 2024-04-12T03:53:38.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0= 2024-04-12T03:53:38.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0=', 'Content-Length': '0'} 2024-04-12T03:53:38.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5fc77394c2728d-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.307 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.308 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.308 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.308 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.308 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.308 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.308 DEBUG:boto:Data: 2024-04-12T03:53:38.308 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.308 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.308 DEBUG:boto:Params: {} 2024-04-12T03:53:38.309 DEBUG:boto:Token: None 2024-04-12T03:53:38.309 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.309 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JRJBqqOSI1iYImex50EnhOjrjco= 2024-04-12T03:53:38.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRJBqqOSI1iYImex50EnhOjrjco=', 'Content-Length': '0'} 2024-04-12T03:53:38.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d27c594a79a6991-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.311 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.311 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.311 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.311 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.311 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.311 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.311 DEBUG:boto:Data: 2024-04-12T03:53:38.312 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.312 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.312 DEBUG:boto:Params: {} 2024-04-12T03:53:38.312 DEBUG:boto:Token: None 2024-04-12T03:53:38.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JRJBqqOSI1iYImex50EnhOjrjco= 2024-04-12T03:53:38.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRJBqqOSI1iYImex50EnhOjrjco=', 'Content-Length': '0'} 2024-04-12T03:53:38.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bab81f9eb16a4dad-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:53:38.314 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.314 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.314 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.315 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.315 DEBUG:boto:Method: GET 2024-04-12T03:53:38.315 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.315 DEBUG:boto:Data: 2024-04-12T03:53:38.315 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.315 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.315 DEBUG:boto:Params: {} 2024-04-12T03:53:38.315 DEBUG:boto:Token: None 2024-04-12T03:53:38.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0= 2024-04-12T03:53:38.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0=', 'Content-Length': '0'} 2024-04-12T03:53:38.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081b512e6a705bab7-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.358 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.358 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.358 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.359 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.359 DEBUG:boto:Method: GET 2024-04-12T03:53:38.359 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.359 DEBUG:boto:Data: 2024-04-12T03:53:38.359 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.359 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.359 DEBUG:boto:Params: {} 2024-04-12T03:53:38.359 DEBUG:boto:Token: None 2024-04-12T03:53:38.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:53:38.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0= 2024-04-12T03:53:38.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8nsZqG47osDxZUGU3JQTh6eOAp0=', 'Content-Length': '0'} 2024-04-12T03:53:38.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f8a89c162a4502-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.362 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.362 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.362 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.362 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.362 DEBUG:boto:Method: GET 2024-04-12T03:53:38.362 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.362 DEBUG:boto:Data: 2024-04-12T03:53:38.362 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.362 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.362 DEBUG:boto:Params: {} 2024-04-12T03:53:38.363 DEBUG:boto:Token: None 2024-04-12T03:53:38.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e71153e4a8fd280-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.365 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.366 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.366 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.366 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.366 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.366 DEBUG:boto:Method: GET 2024-04-12T03:53:38.366 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.366 DEBUG:boto:Data: 2024-04-12T03:53:38.366 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.366 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.367 DEBUG:boto:Params: {} 2024-04-12T03:53:38.367 DEBUG:boto:Token: None 2024-04-12T03:53:38.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.367 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000130799fde3ffd297-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.372 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.372 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.372 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.372 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.372 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:53:38.372 DEBUG:boto:Data: 2024-04-12T03:53:38.372 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.373 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.373 DEBUG:boto:Params: {} 2024-04-12T03:53:38.373 DEBUG:boto:Token: None 2024-04-12T03:53:38.373 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111 2024-04-12T03:53:38.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPo6N/AFMlJW59FWvMmm7jNR70E= 2024-04-12T03:53:38.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPo6N/AFMlJW59FWvMmm7jNR70E=', 'Content-Length': '0'} 2024-04-12T03:53:38.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fba2420ec4b89995-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.375 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.375 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.375 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.375 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:53:38.376 DEBUG:boto:Data: 2024-04-12T03:53:38.376 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.376 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.376 DEBUG:boto:Params: {} 2024-04-12T03:53:38.376 DEBUG:boto:Token: None 2024-04-12T03:53:38.376 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111 2024-04-12T03:53:38.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPo6N/AFMlJW59FWvMmm7jNR70E= 2024-04-12T03:53:38.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPo6N/AFMlJW59FWvMmm7jNR70E=', 'Content-Length': '0'} 2024-04-12T03:53:38.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001563a2582ae08eac-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:53:38.378 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.378 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.379 DEBUG:boto:Method: GET 2024-04-12T03:53:38.379 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:53:38.379 DEBUG:boto:Data: 2024-04-12T03:53:38.379 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.379 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.379 DEBUG:boto:Params: {} 2024-04-12T03:53:38.379 DEBUG:boto:Token: None 2024-04-12T03:53:38.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111 2024-04-12T03:53:38.379 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L85edOIQkmze4F6f8c6S+ifJsNA= 2024-04-12T03:53:38.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L85edOIQkmze4F6f8c6S+ifJsNA=', 'Content-Length': '0'} 2024-04-12T03:53:38.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a60b26edb7ee05-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.381 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:53:38.381 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:53:38.381 DEBUG:boto:Method: GET 2024-04-12T03:53:38.382 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:53:38.382 DEBUG:boto:Data: 2024-04-12T03:53:38.382 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.382 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.382 DEBUG:boto:Params: {} 2024-04-12T03:53:38.382 DEBUG:boto:Token: None 2024-04-12T03:53:38.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj111 2024-04-12T03:53:38.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L85edOIQkmze4F6f8c6S+ifJsNA= 2024-04-12T03:53:38.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L85edOIQkmze4F6f8c6S+ifJsNA=', 'Content-Length': '0'} 2024-04-12T03:53:38.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095faa5a31d38af7f-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.426 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.426 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.426 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.427 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.427 DEBUG:boto:Method: GET 2024-04-12T03:53:38.427 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.427 DEBUG:boto:Data: 2024-04-12T03:53:38.427 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.427 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.427 DEBUG:boto:Params: {} 2024-04-12T03:53:38.427 DEBUG:boto:Token: None 2024-04-12T03:53:38.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694c33eb6e9a38d3-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.430 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.431 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.431 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.431 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.431 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.431 DEBUG:boto:Method: GET 2024-04-12T03:53:38.431 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:53:38.431 DEBUG:boto:Data: 2024-04-12T03:53:38.431 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.432 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.432 DEBUG:boto:Params: {} 2024-04-12T03:53:38.432 DEBUG:boto:Token: None 2024-04-12T03:53:38.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001002aa8b7e0b00e5-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.436 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:53:38.437 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.437 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.437 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.437 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.437 DEBUG:boto:Method: GET 2024-04-12T03:53:38.437 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.438 DEBUG:boto:Data: 2024-04-12T03:53:38.438 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.438 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.438 DEBUG:boto:Params: {} 2024-04-12T03:53:38.438 DEBUG:boto:Token: None 2024-04-12T03:53:38.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0= 2024-04-12T03:53:38.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0=', 'Content-Length': '0'} 2024-04-12T03:53:38.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4c16f2a9bf7577-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.486 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.486 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.487 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.487 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.487 DEBUG:boto:Method: GET 2024-04-12T03:53:38.487 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.487 DEBUG:boto:Data: 2024-04-12T03:53:38.487 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.487 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.487 DEBUG:boto:Params: {} 2024-04-12T03:53:38.487 DEBUG:boto:Token: None 2024-04-12T03:53:38.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0= 2024-04-12T03:53:38.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0=', 'Content-Length': '0'} 2024-04-12T03:53:38.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001134715a4f2a66f5-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.490 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.490 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.490 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.490 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.490 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.490 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.490 DEBUG:boto:Data: 2024-04-12T03:53:38.490 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.490 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.491 DEBUG:boto:Params: {} 2024-04-12T03:53:38.491 DEBUG:boto:Token: None 2024-04-12T03:53:38.491 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:khlnuixfZC6qwfu/yme/lSunJqY= 2024-04-12T03:53:38.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:khlnuixfZC6qwfu/yme/lSunJqY=', 'Content-Length': '0'} 2024-04-12T03:53:38.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd50532d73dff8f-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.493 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.493 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.493 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.493 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.493 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.493 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.493 DEBUG:boto:Data: 2024-04-12T03:53:38.493 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.494 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.494 DEBUG:boto:Params: {} 2024-04-12T03:53:38.494 DEBUG:boto:Token: None 2024-04-12T03:53:38.494 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.494 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:khlnuixfZC6qwfu/yme/lSunJqY= 2024-04-12T03:53:38.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:khlnuixfZC6qwfu/yme/lSunJqY=', 'Content-Length': '0'} 2024-04-12T03:53:38.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcda94d8315d431c-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:53:38.496 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.496 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.496 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.496 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.496 DEBUG:boto:Method: GET 2024-04-12T03:53:38.497 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.497 DEBUG:boto:Data: 2024-04-12T03:53:38.497 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.497 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.497 DEBUG:boto:Params: {} 2024-04-12T03:53:38.497 DEBUG:boto:Token: None 2024-04-12T03:53:38.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.497 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0= 2024-04-12T03:53:38.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0=', 'Content-Length': '0'} 2024-04-12T03:53:38.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8223d7d5f85084-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.499 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.499 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.499 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.499 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.500 DEBUG:boto:Method: GET 2024-04-12T03:53:38.500 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.500 DEBUG:boto:Data: 2024-04-12T03:53:38.500 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.500 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.500 DEBUG:boto:Params: {} 2024-04-12T03:53:38.500 DEBUG:boto:Token: None 2024-04-12T03:53:38.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:53:38.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0= 2024-04-12T03:53:38.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:W1lSkikvuxrlVNIUtu0n9DISZw0=', 'Content-Length': '0'} 2024-04-12T03:53:38.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5655811ab6026fe-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.502 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.502 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.502 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.502 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.503 DEBUG:boto:Method: GET 2024-04-12T03:53:38.503 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.503 DEBUG:boto:Data: 2024-04-12T03:53:38.503 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.503 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.503 DEBUG:boto:Params: {} 2024-04-12T03:53:38.503 DEBUG:boto:Token: None 2024-04-12T03:53:38.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee5ef5f37f78f0c-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.505 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.506 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.506 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.506 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.506 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.506 DEBUG:boto:Method: GET 2024-04-12T03:53:38.506 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.506 DEBUG:boto:Data: 2024-04-12T03:53:38.506 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.507 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.507 DEBUG:boto:Params: {} 2024-04-12T03:53:38.507 DEBUG:boto:Token: None 2024-04-12T03:53:38.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bf46b7aa7e04ed-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.511 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.511 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.511 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.511 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.511 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:53:38.511 DEBUG:boto:Data: 2024-04-12T03:53:38.511 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.512 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.512 DEBUG:boto:Params: {} 2024-04-12T03:53:38.512 DEBUG:boto:Token: None 2024-04-12T03:53:38.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115 2024-04-12T03:53:38.512 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xswwwdX3Aun5ui0rM04Y8zBr9iE= 2024-04-12T03:53:38.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xswwwdX3Aun5ui0rM04Y8zBr9iE=', 'Content-Length': '0'} 2024-04-12T03:53:38.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb2de5cb37ff819-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.514 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.514 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.514 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.514 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:53:38.514 DEBUG:boto:Data: 2024-04-12T03:53:38.515 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.515 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.515 DEBUG:boto:Params: {} 2024-04-12T03:53:38.515 DEBUG:boto:Token: None 2024-04-12T03:53:38.515 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115 2024-04-12T03:53:38.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xswwwdX3Aun5ui0rM04Y8zBr9iE= 2024-04-12T03:53:38.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xswwwdX3Aun5ui0rM04Y8zBr9iE=', 'Content-Length': '0'} 2024-04-12T03:53:38.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000684bcbfcef4d630-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:53:38.517 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.517 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.517 DEBUG:boto:Method: GET 2024-04-12T03:53:38.517 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:53:38.517 DEBUG:boto:Data: 2024-04-12T03:53:38.517 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.518 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.518 DEBUG:boto:Params: {} 2024-04-12T03:53:38.518 DEBUG:boto:Token: None 2024-04-12T03:53:38.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115 2024-04-12T03:53:38.518 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JebLB1nhuJFqnNyIzWeWNFz+/j0= 2024-04-12T03:53:38.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JebLB1nhuJFqnNyIzWeWNFz+/j0=', 'Content-Length': '0'} 2024-04-12T03:53:38.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e421ffe7594215-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.520 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:53:38.520 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:53:38.520 DEBUG:boto:Method: GET 2024-04-12T03:53:38.520 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:53:38.521 DEBUG:boto:Data: 2024-04-12T03:53:38.521 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.521 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.521 DEBUG:boto:Params: {} 2024-04-12T03:53:38.521 DEBUG:boto:Token: None 2024-04-12T03:53:38.521 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj115 2024-04-12T03:53:38.521 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JebLB1nhuJFqnNyIzWeWNFz+/j0= 2024-04-12T03:53:38.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JebLB1nhuJFqnNyIzWeWNFz+/j0=', 'Content-Length': '0'} 2024-04-12T03:53:38.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5500f0a60d97fb4-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.523 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.523 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.523 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.523 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.523 DEBUG:boto:Method: GET 2024-04-12T03:53:38.523 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.523 DEBUG:boto:Data: 2024-04-12T03:53:38.523 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.524 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.524 DEBUG:boto:Params: {} 2024-04-12T03:53:38.524 DEBUG:boto:Token: None 2024-04-12T03:53:38.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d080d280311b22d-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.526 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.526 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.526 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.527 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.527 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.527 DEBUG:boto:Method: GET 2024-04-12T03:53:38.527 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:53:38.527 DEBUG:boto:Data: 2024-04-12T03:53:38.527 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.527 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.527 DEBUG:boto:Params: {} 2024-04-12T03:53:38.527 DEBUG:boto:Token: None 2024-04-12T03:53:38.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d137a01bf24589-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.574 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:53:38.575 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.575 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.575 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.575 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.575 DEBUG:boto:Method: GET 2024-04-12T03:53:38.575 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.575 DEBUG:boto:Data: 2024-04-12T03:53:38.575 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.575 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.575 DEBUG:boto:Params: {} 2024-04-12T03:53:38.576 DEBUG:boto:Token: None 2024-04-12T03:53:38.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600= 2024-04-12T03:53:38.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600=', 'Content-Length': '0'} 2024-04-12T03:53:38.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3070b4b1ad40da0-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.578 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.578 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.578 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.578 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.578 DEBUG:boto:Method: GET 2024-04-12T03:53:38.578 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.578 DEBUG:boto:Data: 2024-04-12T03:53:38.578 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.579 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.579 DEBUG:boto:Params: {} 2024-04-12T03:53:38.579 DEBUG:boto:Token: None 2024-04-12T03:53:38.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600= 2024-04-12T03:53:38.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600=', 'Content-Length': '0'} 2024-04-12T03:53:38.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2451746a2e5360c-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.581 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.581 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.581 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.581 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.581 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.582 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.582 DEBUG:boto:Data: 2024-04-12T03:53:38.582 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.582 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.582 DEBUG:boto:Params: {} 2024-04-12T03:53:38.582 DEBUG:boto:Token: None 2024-04-12T03:53:38.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PevIZ5bHvWHifBr/+5+UW9BLQ3g= 2024-04-12T03:53:38.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PevIZ5bHvWHifBr/+5+UW9BLQ3g=', 'Content-Length': '0'} 2024-04-12T03:53:38.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfcfab2356ae304-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.584 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.584 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.584 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.584 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.584 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.584 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.585 DEBUG:boto:Data: 2024-04-12T03:53:38.585 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.585 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.585 DEBUG:boto:Params: {} 2024-04-12T03:53:38.585 DEBUG:boto:Token: None 2024-04-12T03:53:38.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PevIZ5bHvWHifBr/+5+UW9BLQ3g= 2024-04-12T03:53:38.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PevIZ5bHvWHifBr/+5+UW9BLQ3g=', 'Content-Length': '0'} 2024-04-12T03:53:38.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeef9adfad85e3bb-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:53:38.587 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.587 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.587 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.587 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.588 DEBUG:boto:Method: GET 2024-04-12T03:53:38.588 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.588 DEBUG:boto:Data: 2024-04-12T03:53:38.588 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.588 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.588 DEBUG:boto:Params: {} 2024-04-12T03:53:38.588 DEBUG:boto:Token: None 2024-04-12T03:53:38.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600= 2024-04-12T03:53:38.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600=', 'Content-Length': '0'} 2024-04-12T03:53:38.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db33004675e59935-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.590 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.590 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.590 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.590 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.590 DEBUG:boto:Method: GET 2024-04-12T03:53:38.591 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.591 DEBUG:boto:Data: 2024-04-12T03:53:38.591 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.591 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.591 DEBUG:boto:Params: {} 2024-04-12T03:53:38.591 DEBUG:boto:Token: None 2024-04-12T03:53:38.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:53:38.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600= 2024-04-12T03:53:38.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGu6dBtz5P6sYfOQ5JNdjQuJ600=', 'Content-Length': '0'} 2024-04-12T03:53:38.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7bef405d4146566-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.593 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.593 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.594 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.594 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.594 DEBUG:boto:Method: GET 2024-04-12T03:53:38.594 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.594 DEBUG:boto:Data: 2024-04-12T03:53:38.594 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.594 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.594 DEBUG:boto:Params: {} 2024-04-12T03:53:38.594 DEBUG:boto:Token: None 2024-04-12T03:53:38.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001c880669da9d49b-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.597 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.597 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.597 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.597 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.597 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.598 DEBUG:boto:Method: GET 2024-04-12T03:53:38.598 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.598 DEBUG:boto:Data: 2024-04-12T03:53:38.598 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.598 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.598 DEBUG:boto:Params: {} 2024-04-12T03:53:38.598 DEBUG:boto:Token: None 2024-04-12T03:53:38.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31ff4cfc1864b1d-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.602 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.602 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.602 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.602 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.602 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:53:38.602 DEBUG:boto:Data: 2024-04-12T03:53:38.603 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.603 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.603 DEBUG:boto:Params: {} 2024-04-12T03:53:38.603 DEBUG:boto:Token: None 2024-04-12T03:53:38.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116 2024-04-12T03:53:38.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GFO7+DTUAlNHeEU5Yi2dqG3kh2Q= 2024-04-12T03:53:38.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GFO7+DTUAlNHeEU5Yi2dqG3kh2Q=', 'Content-Length': '0'} 2024-04-12T03:53:38.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7ae2baf18bfe70-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.605 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.605 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.605 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.605 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:53:38.605 DEBUG:boto:Data: 2024-04-12T03:53:38.605 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.605 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.606 DEBUG:boto:Params: {} 2024-04-12T03:53:38.606 DEBUG:boto:Token: None 2024-04-12T03:53:38.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116 2024-04-12T03:53:38.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GFO7+DTUAlNHeEU5Yi2dqG3kh2Q= 2024-04-12T03:53:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GFO7+DTUAlNHeEU5Yi2dqG3kh2Q=', 'Content-Length': '0'} 2024-04-12T03:53:38.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001816fddc3a8d9199-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:53:38.608 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.608 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.608 DEBUG:boto:Method: GET 2024-04-12T03:53:38.608 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:53:38.608 DEBUG:boto:Data: 2024-04-12T03:53:38.608 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.608 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.608 DEBUG:boto:Params: {} 2024-04-12T03:53:38.609 DEBUG:boto:Token: None 2024-04-12T03:53:38.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116 2024-04-12T03:53:38.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oz1b/d+VKi1HkwRKbrjExca+5fk= 2024-04-12T03:53:38.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oz1b/d+VKi1HkwRKbrjExca+5fk=', 'Content-Length': '0'} 2024-04-12T03:53:38.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb5243bfbe8e82b-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.611 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:53:38.611 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:53:38.611 DEBUG:boto:Method: GET 2024-04-12T03:53:38.611 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:53:38.611 DEBUG:boto:Data: 2024-04-12T03:53:38.611 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.611 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.611 DEBUG:boto:Params: {} 2024-04-12T03:53:38.611 DEBUG:boto:Token: None 2024-04-12T03:53:38.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj116 2024-04-12T03:53:38.612 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oz1b/d+VKi1HkwRKbrjExca+5fk= 2024-04-12T03:53:38.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oz1b/d+VKi1HkwRKbrjExca+5fk=', 'Content-Length': '0'} 2024-04-12T03:53:38.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f214937d816fa539-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.613 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.613 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.613 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.613 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.614 DEBUG:boto:Method: GET 2024-04-12T03:53:38.614 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.614 DEBUG:boto:Data: 2024-04-12T03:53:38.614 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.614 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.614 DEBUG:boto:Params: {} 2024-04-12T03:53:38.614 DEBUG:boto:Token: None 2024-04-12T03:53:38.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb362b0775c2dca-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.658 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.658 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.659 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.659 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.659 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.659 DEBUG:boto:Method: GET 2024-04-12T03:53:38.659 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:53:38.659 DEBUG:boto:Data: 2024-04-12T03:53:38.659 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.659 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.659 DEBUG:boto:Params: {} 2024-04-12T03:53:38.659 DEBUG:boto:Token: None 2024-04-12T03:53:38.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4ea4a4a8fc04f62-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.663 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:53:38.664 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.664 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.664 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.664 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.664 DEBUG:boto:Method: GET 2024-04-12T03:53:38.664 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.664 DEBUG:boto:Data: 2024-04-12T03:53:38.664 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.665 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.665 DEBUG:boto:Params: {} 2024-04-12T03:53:38.665 DEBUG:boto:Token: None 2024-04-12T03:53:38.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.665 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk= 2024-04-12T03:53:38.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk=', 'Content-Length': '0'} 2024-04-12T03:53:38.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000700e1f879e3c42a-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.667 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.667 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.667 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.667 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.667 DEBUG:boto:Method: GET 2024-04-12T03:53:38.667 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.668 DEBUG:boto:Data: 2024-04-12T03:53:38.668 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.668 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.668 DEBUG:boto:Params: {} 2024-04-12T03:53:38.668 DEBUG:boto:Token: None 2024-04-12T03:53:38.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk= 2024-04-12T03:53:38.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk=', 'Content-Length': '0'} 2024-04-12T03:53:38.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f19e1c8bc06873f-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.710 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.711 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.711 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.711 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.711 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.711 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.711 DEBUG:boto:Data: 2024-04-12T03:53:38.711 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.711 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.711 DEBUG:boto:Params: {} 2024-04-12T03:53:38.712 DEBUG:boto:Token: None 2024-04-12T03:53:38.712 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BEQFe08JJugyTJztcTYTK5MoG/g= 2024-04-12T03:53:38.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BEQFe08JJugyTJztcTYTK5MoG/g=', 'Content-Length': '0'} 2024-04-12T03:53:38.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad59ef20a15f7fe5-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.714 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.714 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.714 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.714 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.714 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.714 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.714 DEBUG:boto:Data: 2024-04-12T03:53:38.714 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.714 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.715 DEBUG:boto:Params: {} 2024-04-12T03:53:38.715 DEBUG:boto:Token: None 2024-04-12T03:53:38.715 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BEQFe08JJugyTJztcTYTK5MoG/g= 2024-04-12T03:53:38.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BEQFe08JJugyTJztcTYTK5MoG/g=', 'Content-Length': '0'} 2024-04-12T03:53:38.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e75bf30aee0cf10-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:53:38.717 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.717 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.717 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.717 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.717 DEBUG:boto:Method: GET 2024-04-12T03:53:38.717 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.717 DEBUG:boto:Data: 2024-04-12T03:53:38.717 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.718 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.718 DEBUG:boto:Params: {} 2024-04-12T03:53:38.718 DEBUG:boto:Token: None 2024-04-12T03:53:38.718 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk= 2024-04-12T03:53:38.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk=', 'Content-Length': '0'} 2024-04-12T03:53:38.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005806676038c1a448-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.720 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.720 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.720 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.720 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.720 DEBUG:boto:Method: GET 2024-04-12T03:53:38.720 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.720 DEBUG:boto:Data: 2024-04-12T03:53:38.720 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.720 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.720 DEBUG:boto:Params: {} 2024-04-12T03:53:38.720 DEBUG:boto:Token: None 2024-04-12T03:53:38.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:53:38.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk= 2024-04-12T03:53:38.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoF5+kkKIK3pN0Z0wLnGtFzOGdk=', 'Content-Length': '0'} 2024-04-12T03:53:38.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a825089f0e20949f-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.723 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.723 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.723 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.723 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.723 DEBUG:boto:Method: GET 2024-04-12T03:53:38.723 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.723 DEBUG:boto:Data: 2024-04-12T03:53:38.723 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.723 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.723 DEBUG:boto:Params: {} 2024-04-12T03:53:38.724 DEBUG:boto:Token: None 2024-04-12T03:53:38.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c49013ba0bc13be0-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.726 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.726 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.727 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.727 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.727 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.727 DEBUG:boto:Method: GET 2024-04-12T03:53:38.727 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.727 DEBUG:boto:Data: 2024-04-12T03:53:38.727 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.727 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.727 DEBUG:boto:Params: {} 2024-04-12T03:53:38.727 DEBUG:boto:Token: None 2024-04-12T03:53:38.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001918a2595c3315c3-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.731 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.731 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.731 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.731 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.732 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:53:38.732 DEBUG:boto:Data: 2024-04-12T03:53:38.732 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.732 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.732 DEBUG:boto:Params: {} 2024-04-12T03:53:38.732 DEBUG:boto:Token: None 2024-04-12T03:53:38.732 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124 2024-04-12T03:53:38.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OJ8UvHqhU2FZO46bUC4+1aS83ZY= 2024-04-12T03:53:38.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OJ8UvHqhU2FZO46bUC4+1aS83ZY=', 'Content-Length': '0'} 2024-04-12T03:53:38.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836566575eb06c77-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.734 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.734 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.734 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.734 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:53:38.734 DEBUG:boto:Data: 2024-04-12T03:53:38.734 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.734 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.735 DEBUG:boto:Params: {} 2024-04-12T03:53:38.735 DEBUG:boto:Token: None 2024-04-12T03:53:38.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124 2024-04-12T03:53:38.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OJ8UvHqhU2FZO46bUC4+1aS83ZY= 2024-04-12T03:53:38.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OJ8UvHqhU2FZO46bUC4+1aS83ZY=', 'Content-Length': '0'} 2024-04-12T03:53:38.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed94283f7d81da1-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:53:38.737 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.737 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.737 DEBUG:boto:Method: GET 2024-04-12T03:53:38.737 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:53:38.737 DEBUG:boto:Data: 2024-04-12T03:53:38.737 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.738 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.738 DEBUG:boto:Params: {} 2024-04-12T03:53:38.738 DEBUG:boto:Token: None 2024-04-12T03:53:38.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124 2024-04-12T03:53:38.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p0ETq/jwCwP7CBfFelSttG2PYb4= 2024-04-12T03:53:38.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p0ETq/jwCwP7CBfFelSttG2PYb4=', 'Content-Length': '0'} 2024-04-12T03:53:38.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357b99de35780ff9-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.740 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:53:38.740 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:53:38.740 DEBUG:boto:Method: GET 2024-04-12T03:53:38.740 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:53:38.740 DEBUG:boto:Data: 2024-04-12T03:53:38.740 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.741 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.741 DEBUG:boto:Params: {} 2024-04-12T03:53:38.741 DEBUG:boto:Token: None 2024-04-12T03:53:38.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj124 2024-04-12T03:53:38.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p0ETq/jwCwP7CBfFelSttG2PYb4= 2024-04-12T03:53:38.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p0ETq/jwCwP7CBfFelSttG2PYb4=', 'Content-Length': '0'} 2024-04-12T03:53:38.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1524ae19e4008d-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.786 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.786 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.787 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.787 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.787 DEBUG:boto:Method: GET 2024-04-12T03:53:38.787 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.787 DEBUG:boto:Data: 2024-04-12T03:53:38.787 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.787 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.787 DEBUG:boto:Params: {} 2024-04-12T03:53:38.787 DEBUG:boto:Token: None 2024-04-12T03:53:38.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dde1afd120387b24-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.790 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.790 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.790 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.790 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.790 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.790 DEBUG:boto:Method: GET 2024-04-12T03:53:38.790 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:53:38.791 DEBUG:boto:Data: 2024-04-12T03:53:38.791 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.791 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.791 DEBUG:boto:Params: {} 2024-04-12T03:53:38.791 DEBUG:boto:Token: None 2024-04-12T03:53:38.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060d15cbc9efaef8a-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.795 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:53:38.795 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.796 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.796 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.796 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.796 DEBUG:boto:Method: GET 2024-04-12T03:53:38.796 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.796 DEBUG:boto:Data: 2024-04-12T03:53:38.796 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.796 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.796 DEBUG:boto:Params: {} 2024-04-12T03:53:38.796 DEBUG:boto:Token: None 2024-04-12T03:53:38.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE= 2024-04-12T03:53:38.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE=', 'Content-Length': '0'} 2024-04-12T03:53:38.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684fc4ddf8ce3569-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.798 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.799 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.799 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.799 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.799 DEBUG:boto:Method: GET 2024-04-12T03:53:38.799 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.799 DEBUG:boto:Data: 2024-04-12T03:53:38.799 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.799 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.799 DEBUG:boto:Params: {} 2024-04-12T03:53:38.799 DEBUG:boto:Token: None 2024-04-12T03:53:38.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE= 2024-04-12T03:53:38.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE=', 'Content-Length': '0'} 2024-04-12T03:53:38.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c38ab22e7f628c7-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.802 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.802 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.802 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.802 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.802 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.803 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.803 DEBUG:boto:Data: 2024-04-12T03:53:38.803 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.803 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.803 DEBUG:boto:Params: {} 2024-04-12T03:53:38.803 DEBUG:boto:Token: None 2024-04-12T03:53:38.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EsLbl2gpFlpo4XHZ7bILEunj0X4= 2024-04-12T03:53:38.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EsLbl2gpFlpo4XHZ7bILEunj0X4=', 'Content-Length': '0'} 2024-04-12T03:53:38.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75b3d75d0d64170-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.805 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.805 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.805 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.805 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.805 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.805 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.806 DEBUG:boto:Data: 2024-04-12T03:53:38.806 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.806 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.806 DEBUG:boto:Params: {} 2024-04-12T03:53:38.806 DEBUG:boto:Token: None 2024-04-12T03:53:38.806 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EsLbl2gpFlpo4XHZ7bILEunj0X4= 2024-04-12T03:53:38.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EsLbl2gpFlpo4XHZ7bILEunj0X4=', 'Content-Length': '0'} 2024-04-12T03:53:38.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b47893e9deefb8-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:53:38.808 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.809 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.809 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.809 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.809 DEBUG:boto:Method: GET 2024-04-12T03:53:38.809 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.809 DEBUG:boto:Data: 2024-04-12T03:53:38.809 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.809 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.809 DEBUG:boto:Params: {} 2024-04-12T03:53:38.809 DEBUG:boto:Token: None 2024-04-12T03:53:38.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE= 2024-04-12T03:53:38.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE=', 'Content-Length': '0'} 2024-04-12T03:53:38.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2f5c66dffc2ba8-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.811 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.811 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.812 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.812 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.812 DEBUG:boto:Method: GET 2024-04-12T03:53:38.812 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.812 DEBUG:boto:Data: 2024-04-12T03:53:38.812 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.812 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.812 DEBUG:boto:Params: {} 2024-04-12T03:53:38.812 DEBUG:boto:Token: None 2024-04-12T03:53:38.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:53:38.813 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE= 2024-04-12T03:53:38.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oRi47EO40X40rtsJ0BthzdBtZmE=', 'Content-Length': '0'} 2024-04-12T03:53:38.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542f490b080d1b20-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.815 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.815 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.815 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.815 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.815 DEBUG:boto:Method: GET 2024-04-12T03:53:38.815 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.815 DEBUG:boto:Data: 2024-04-12T03:53:38.815 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.815 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.816 DEBUG:boto:Params: {} 2024-04-12T03:53:38.816 DEBUG:boto:Token: None 2024-04-12T03:53:38.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740e1d71c527983d-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.818 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.819 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.819 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.819 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.819 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.819 DEBUG:boto:Method: GET 2024-04-12T03:53:38.819 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.819 DEBUG:boto:Data: 2024-04-12T03:53:38.819 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.819 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.819 DEBUG:boto:Params: {} 2024-04-12T03:53:38.820 DEBUG:boto:Token: None 2024-04-12T03:53:38.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.820 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795173adbfbf3efb-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.866 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.867 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.867 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.867 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.867 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:53:38.867 DEBUG:boto:Data: 2024-04-12T03:53:38.867 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.867 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.867 DEBUG:boto:Params: {} 2024-04-12T03:53:38.868 DEBUG:boto:Token: None 2024-04-12T03:53:38.868 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145 2024-04-12T03:53:38.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V1MHHaLjWyGonkdW2JjEk2ZwgBs= 2024-04-12T03:53:38.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V1MHHaLjWyGonkdW2JjEk2ZwgBs=', 'Content-Length': '0'} 2024-04-12T03:53:38.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d1d086b65c178d-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.870 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.870 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.870 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.870 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:53:38.870 DEBUG:boto:Data: 2024-04-12T03:53:38.870 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.870 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.870 DEBUG:boto:Params: {} 2024-04-12T03:53:38.871 DEBUG:boto:Token: None 2024-04-12T03:53:38.871 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145 2024-04-12T03:53:38.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V1MHHaLjWyGonkdW2JjEk2ZwgBs= 2024-04-12T03:53:38.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V1MHHaLjWyGonkdW2JjEk2ZwgBs=', 'Content-Length': '0'} 2024-04-12T03:53:38.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddfa1b4226425ce7-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:53:38.873 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.873 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.873 DEBUG:boto:Method: GET 2024-04-12T03:53:38.873 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:53:38.873 DEBUG:boto:Data: 2024-04-12T03:53:38.873 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.873 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.873 DEBUG:boto:Params: {} 2024-04-12T03:53:38.873 DEBUG:boto:Token: None 2024-04-12T03:53:38.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145 2024-04-12T03:53:38.874 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0NgV5KS99RykekZ4U9gjI5enWI= 2024-04-12T03:53:38.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0NgV5KS99RykekZ4U9gjI5enWI=', 'Content-Length': '0'} 2024-04-12T03:53:38.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56100255120a6b3-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.875 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:53:38.876 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:53:38.876 DEBUG:boto:Method: GET 2024-04-12T03:53:38.876 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:53:38.876 DEBUG:boto:Data: 2024-04-12T03:53:38.876 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.876 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.876 DEBUG:boto:Params: {} 2024-04-12T03:53:38.876 DEBUG:boto:Token: None 2024-04-12T03:53:38.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj145 2024-04-12T03:53:38.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U0NgV5KS99RykekZ4U9gjI5enWI= 2024-04-12T03:53:38.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U0NgV5KS99RykekZ4U9gjI5enWI=', 'Content-Length': '0'} 2024-04-12T03:53:38.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b1d44a8f09d5f6-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.878 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.878 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.879 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.879 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.879 DEBUG:boto:Method: GET 2024-04-12T03:53:38.879 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.879 DEBUG:boto:Data: 2024-04-12T03:53:38.879 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.879 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.879 DEBUG:boto:Params: {} 2024-04-12T03:53:38.879 DEBUG:boto:Token: None 2024-04-12T03:53:38.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000212bd291036bb63c-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.882 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.882 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.882 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.882 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.882 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.882 DEBUG:boto:Method: GET 2024-04-12T03:53:38.882 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:53:38.883 DEBUG:boto:Data: 2024-04-12T03:53:38.883 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.883 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.883 DEBUG:boto:Params: {} 2024-04-12T03:53:38.883 DEBUG:boto:Token: None 2024-04-12T03:53:38.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009509ff9bb2b31b83-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.887 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:53:38.887 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.887 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.887 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.888 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.888 DEBUG:boto:Method: GET 2024-04-12T03:53:38.888 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.888 DEBUG:boto:Data: 2024-04-12T03:53:38.888 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.888 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.888 DEBUG:boto:Params: {} 2024-04-12T03:53:38.888 DEBUG:boto:Token: None 2024-04-12T03:53:38.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc= 2024-04-12T03:53:38.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc=', 'Content-Length': '0'} 2024-04-12T03:53:38.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307e9d11cefa523e-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.890 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.890 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.891 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.891 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.891 DEBUG:boto:Method: GET 2024-04-12T03:53:38.891 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.891 DEBUG:boto:Data: 2024-04-12T03:53:38.891 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.891 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.891 DEBUG:boto:Params: {} 2024-04-12T03:53:38.891 DEBUG:boto:Token: None 2024-04-12T03:53:38.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc= 2024-04-12T03:53:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc=', 'Content-Length': '0'} 2024-04-12T03:53:38.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9926a3d30b34be1-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.934 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.935 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.935 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.935 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.935 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.935 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.935 DEBUG:boto:Data: 2024-04-12T03:53:38.935 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.935 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.935 DEBUG:boto:Params: {} 2024-04-12T03:53:38.936 DEBUG:boto:Token: None 2024-04-12T03:53:38.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hAqa+knFlRq55bYHVnkkaNgFrMw= 2024-04-12T03:53:38.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hAqa+knFlRq55bYHVnkkaNgFrMw=', 'Content-Length': '0'} 2024-04-12T03:53:38.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fa12161f8d3f14-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.938 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.938 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.938 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.938 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.938 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.938 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.938 DEBUG:boto:Data: 2024-04-12T03:53:38.938 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.939 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.939 DEBUG:boto:Params: {} 2024-04-12T03:53:38.939 DEBUG:boto:Token: None 2024-04-12T03:53:38.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hAqa+knFlRq55bYHVnkkaNgFrMw= 2024-04-12T03:53:38.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hAqa+knFlRq55bYHVnkkaNgFrMw=', 'Content-Length': '0'} 2024-04-12T03:53:38.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888f21b737a2bb83-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:53:38.941 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.941 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.941 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.941 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.941 DEBUG:boto:Method: GET 2024-04-12T03:53:38.941 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.942 DEBUG:boto:Data: 2024-04-12T03:53:38.942 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.942 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.942 DEBUG:boto:Params: {} 2024-04-12T03:53:38.942 DEBUG:boto:Token: None 2024-04-12T03:53:38.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc= 2024-04-12T03:53:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc=', 'Content-Length': '0'} 2024-04-12T03:53:38.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1649dd444010d7-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.944 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.944 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.944 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.944 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.945 DEBUG:boto:Method: GET 2024-04-12T03:53:38.945 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.945 DEBUG:boto:Data: 2024-04-12T03:53:38.945 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.945 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.945 DEBUG:boto:Params: {} 2024-04-12T03:53:38.945 DEBUG:boto:Token: None 2024-04-12T03:53:38.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:53:38.946 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc= 2024-04-12T03:53:38.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yH+XCJHDtE3TjhGmV3/+oPj/Qdc=', 'Content-Length': '0'} 2024-04-12T03:53:38.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a42ee7cb763f841-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.947 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.948 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.948 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.948 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.948 DEBUG:boto:Method: GET 2024-04-12T03:53:38.948 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.948 DEBUG:boto:Data: 2024-04-12T03:53:38.948 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.948 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.948 DEBUG:boto:Params: {} 2024-04-12T03:53:38.948 DEBUG:boto:Token: None 2024-04-12T03:53:38.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.949 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021876ee9feeb03d-006618b042-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.951 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.951 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:38.951 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:38.952 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.952 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.952 DEBUG:boto:Method: GET 2024-04-12T03:53:38.952 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:53:38.952 DEBUG:boto:Data: 2024-04-12T03:53:38.952 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.952 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.952 DEBUG:boto:Params: {} 2024-04-12T03:53:38.952 DEBUG:boto:Token: None 2024-04-12T03:53:38.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/ 2024-04-12T03:53:38.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA= 2024-04-12T03:53:38.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jyJusTmI1ZrOb/XvZy/5MA7OouA=', 'Content-Length': '0'} 2024-04-12T03:53:38.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcc9fc2f2e9b6d4-006618b042-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.956 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:38.957 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.957 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.957 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.957 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:53:38.957 DEBUG:boto:Data: 2024-04-12T03:53:38.957 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.957 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.957 DEBUG:boto:Params: {} 2024-04-12T03:53:38.958 DEBUG:boto:Token: None 2024-04-12T03:53:38.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148 2024-04-12T03:53:38.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G4TUvQBnvnFwOj4VbiBOWsEzHGU= 2024-04-12T03:53:38.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G4TUvQBnvnFwOj4VbiBOWsEzHGU=', 'Content-Length': '0'} 2024-04-12T03:53:38.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023dcf2376dd0d0fa-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.960 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.960 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.960 DEBUG:boto:Method: HEAD 2024-04-12T03:53:38.960 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:53:38.960 DEBUG:boto:Data: 2024-04-12T03:53:38.960 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:38.960 DEBUG:boto:Port: 8000 2024-04-12T03:53:38.960 DEBUG:boto:Params: {} 2024-04-12T03:53:38.960 DEBUG:boto:Token: None 2024-04-12T03:53:38.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148 2024-04-12T03:53:38.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G4TUvQBnvnFwOj4VbiBOWsEzHGU= 2024-04-12T03:53:38.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G4TUvQBnvnFwOj4VbiBOWsEzHGU=', 'Content-Length': '0'} 2024-04-12T03:53:38.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cd6bdc0b947242-006618b042-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:38.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:53:38.963 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:38.963 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:38.963 DEBUG:boto:Method: GET 2024-04-12T03:53:38.963 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:53:38.963 DEBUG:boto:Data: 2024-04-12T03:53:38.963 DEBUG:boto:Headers: {} 2024-04-12T03:53:38.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:38.964 DEBUG:boto:Port: 8001 2024-04-12T03:53:38.964 DEBUG:boto:Params: {} 2024-04-12T03:53:38.964 DEBUG:boto:Token: None 2024-04-12T03:53:38.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:38 GMT /dgbpxg-61/obj148 2024-04-12T03:53:38.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7LV5MoGAi6vwC3W0pn1zUSjejBQ= 2024-04-12T03:53:38.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7LV5MoGAi6vwC3W0pn1zUSjejBQ=', 'Content-Length': '0'} 2024-04-12T03:53:38.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e2c48ec625d0ca-006618b042-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.006 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:53:39.007 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:53:39.007 DEBUG:boto:Method: GET 2024-04-12T03:53:39.007 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:53:39.007 DEBUG:boto:Data: 2024-04-12T03:53:39.007 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.007 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.007 DEBUG:boto:Params: {} 2024-04-12T03:53:39.007 DEBUG:boto:Token: None 2024-04-12T03:53:39.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj148 2024-04-12T03:53:39.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sJk2guLj5wGweIEkJShxDxfxl/k= 2024-04-12T03:53:39.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sJk2guLj5wGweIEkJShxDxfxl/k=', 'Content-Length': '0'} 2024-04-12T03:53:39.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce11f37b40cbb66a-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.010 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.010 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.010 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.010 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.010 DEBUG:boto:Method: GET 2024-04-12T03:53:39.010 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.010 DEBUG:boto:Data: 2024-04-12T03:53:39.010 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.010 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.010 DEBUG:boto:Params: {} 2024-04-12T03:53:39.011 DEBUG:boto:Token: None 2024-04-12T03:53:39.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed5ec80a1de2aa2-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.013 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.013 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.013 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.013 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.013 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.013 DEBUG:boto:Method: GET 2024-04-12T03:53:39.014 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:53:39.014 DEBUG:boto:Data: 2024-04-12T03:53:39.014 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.014 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.014 DEBUG:boto:Params: {} 2024-04-12T03:53:39.014 DEBUG:boto:Token: None 2024-04-12T03:53:39.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb9835eefda49af-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.018 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:53:39.018 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.019 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.019 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.019 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.019 DEBUG:boto:Method: GET 2024-04-12T03:53:39.019 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.019 DEBUG:boto:Data: 2024-04-12T03:53:39.019 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.019 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.019 DEBUG:boto:Params: {} 2024-04-12T03:53:39.019 DEBUG:boto:Token: None 2024-04-12T03:53:39.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc= 2024-04-12T03:53:39.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc=', 'Content-Length': '0'} 2024-04-12T03:53:39.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e143644899f6ca3-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.021 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.021 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.021 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.021 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.022 DEBUG:boto:Method: GET 2024-04-12T03:53:39.022 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.022 DEBUG:boto:Data: 2024-04-12T03:53:39.022 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.022 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.022 DEBUG:boto:Params: {} 2024-04-12T03:53:39.022 DEBUG:boto:Token: None 2024-04-12T03:53:39.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc= 2024-04-12T03:53:39.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc=', 'Content-Length': '0'} 2024-04-12T03:53:39.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93316a0ed5c5cd1-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.024 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.024 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.024 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.025 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.025 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.025 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.025 DEBUG:boto:Data: 2024-04-12T03:53:39.025 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.025 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.025 DEBUG:boto:Params: {} 2024-04-12T03:53:39.025 DEBUG:boto:Token: None 2024-04-12T03:53:39.025 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UC5X1NkWl2sIZJ2J/WRciJ6eYpI= 2024-04-12T03:53:39.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UC5X1NkWl2sIZJ2J/WRciJ6eYpI=', 'Content-Length': '0'} 2024-04-12T03:53:39.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629ab103a94da2e5-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.027 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.028 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.028 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.028 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.028 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.028 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.028 DEBUG:boto:Data: 2024-04-12T03:53:39.028 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.028 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.028 DEBUG:boto:Params: {} 2024-04-12T03:53:39.028 DEBUG:boto:Token: None 2024-04-12T03:53:39.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UC5X1NkWl2sIZJ2J/WRciJ6eYpI= 2024-04-12T03:53:39.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UC5X1NkWl2sIZJ2J/WRciJ6eYpI=', 'Content-Length': '0'} 2024-04-12T03:53:39.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20cd258a75b6e9d-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:53:39.030 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.030 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.031 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.031 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.031 DEBUG:boto:Method: GET 2024-04-12T03:53:39.031 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.031 DEBUG:boto:Data: 2024-04-12T03:53:39.031 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.031 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.031 DEBUG:boto:Params: {} 2024-04-12T03:53:39.031 DEBUG:boto:Token: None 2024-04-12T03:53:39.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc= 2024-04-12T03:53:39.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc=', 'Content-Length': '0'} 2024-04-12T03:53:39.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef526d135ba849c1-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.033 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.033 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.033 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.033 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.033 DEBUG:boto:Method: GET 2024-04-12T03:53:39.034 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.034 DEBUG:boto:Data: 2024-04-12T03:53:39.034 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.034 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.034 DEBUG:boto:Params: {} 2024-04-12T03:53:39.034 DEBUG:boto:Token: None 2024-04-12T03:53:39.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:53:39.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc= 2024-04-12T03:53:39.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2uPNFxfpBxyRPN/I0o/p9nESqbc=', 'Content-Length': '0'} 2024-04-12T03:53:39.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbf9c681a6ecd0d-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.036 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.036 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.036 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.036 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.037 DEBUG:boto:Method: GET 2024-04-12T03:53:39.037 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.037 DEBUG:boto:Data: 2024-04-12T03:53:39.037 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.037 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.037 DEBUG:boto:Params: {} 2024-04-12T03:53:39.037 DEBUG:boto:Token: None 2024-04-12T03:53:39.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f762fbfd2d4cd0-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.040 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.040 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.040 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.040 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.040 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.040 DEBUG:boto:Method: GET 2024-04-12T03:53:39.040 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.040 DEBUG:boto:Data: 2024-04-12T03:53:39.041 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.041 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.041 DEBUG:boto:Params: {} 2024-04-12T03:53:39.041 DEBUG:boto:Token: None 2024-04-12T03:53:39.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031aba7c5b27945d5-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.046 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.046 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.046 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.046 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.046 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:53:39.047 DEBUG:boto:Data: 2024-04-12T03:53:39.047 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.047 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.047 DEBUG:boto:Params: {} 2024-04-12T03:53:39.047 DEBUG:boto:Token: None 2024-04-12T03:53:39.047 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155 2024-04-12T03:53:39.047 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dx9JaZZGZmAEPEBzVHKtcYkl8G8= 2024-04-12T03:53:39.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dx9JaZZGZmAEPEBzVHKtcYkl8G8=', 'Content-Length': '0'} 2024-04-12T03:53:39.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dba7376c0fd0e9b-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.049 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.049 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.049 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.049 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:53:39.049 DEBUG:boto:Data: 2024-04-12T03:53:39.049 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.049 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.050 DEBUG:boto:Params: {} 2024-04-12T03:53:39.050 DEBUG:boto:Token: None 2024-04-12T03:53:39.050 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155 2024-04-12T03:53:39.050 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dx9JaZZGZmAEPEBzVHKtcYkl8G8= 2024-04-12T03:53:39.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dx9JaZZGZmAEPEBzVHKtcYkl8G8=', 'Content-Length': '0'} 2024-04-12T03:53:39.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca921873d0c18c28-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:53:39.052 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.052 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.052 DEBUG:boto:Method: GET 2024-04-12T03:53:39.052 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:53:39.052 DEBUG:boto:Data: 2024-04-12T03:53:39.052 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.052 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.053 DEBUG:boto:Params: {} 2024-04-12T03:53:39.053 DEBUG:boto:Token: None 2024-04-12T03:53:39.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155 2024-04-12T03:53:39.053 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oG/9sjhcXkbuetLwIekjm2xaUVQ= 2024-04-12T03:53:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oG/9sjhcXkbuetLwIekjm2xaUVQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd0c34c464a64f6-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.098 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:53:39.099 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:53:39.099 DEBUG:boto:Method: GET 2024-04-12T03:53:39.099 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:53:39.099 DEBUG:boto:Data: 2024-04-12T03:53:39.099 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.099 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.099 DEBUG:boto:Params: {} 2024-04-12T03:53:39.099 DEBUG:boto:Token: None 2024-04-12T03:53:39.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj155 2024-04-12T03:53:39.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oG/9sjhcXkbuetLwIekjm2xaUVQ= 2024-04-12T03:53:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oG/9sjhcXkbuetLwIekjm2xaUVQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007178e583aa665f25-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.101 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.101 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.102 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.102 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.102 DEBUG:boto:Method: GET 2024-04-12T03:53:39.102 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.102 DEBUG:boto:Data: 2024-04-12T03:53:39.102 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.102 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.102 DEBUG:boto:Params: {} 2024-04-12T03:53:39.102 DEBUG:boto:Token: None 2024-04-12T03:53:39.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d79e7e7c6e8db325-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.105 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.106 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.106 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.106 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.106 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.106 DEBUG:boto:Method: GET 2024-04-12T03:53:39.106 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:53:39.106 DEBUG:boto:Data: 2024-04-12T03:53:39.107 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.107 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.107 DEBUG:boto:Params: {} 2024-04-12T03:53:39.107 DEBUG:boto:Token: None 2024-04-12T03:53:39.107 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.107 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052525778ee918f03-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.111 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:53:39.111 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.111 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.111 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.111 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.112 DEBUG:boto:Method: GET 2024-04-12T03:53:39.112 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.112 DEBUG:boto:Data: 2024-04-12T03:53:39.112 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.112 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.112 DEBUG:boto:Params: {} 2024-04-12T03:53:39.112 DEBUG:boto:Token: None 2024-04-12T03:53:39.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk= 2024-04-12T03:53:39.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk=', 'Content-Length': '0'} 2024-04-12T03:53:39.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc0f76ee46709db-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.114 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.114 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.114 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.114 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.115 DEBUG:boto:Method: GET 2024-04-12T03:53:39.115 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.115 DEBUG:boto:Data: 2024-04-12T03:53:39.115 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.115 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.115 DEBUG:boto:Params: {} 2024-04-12T03:53:39.115 DEBUG:boto:Token: None 2024-04-12T03:53:39.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk= 2024-04-12T03:53:39.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk=', 'Content-Length': '0'} 2024-04-12T03:53:39.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000905536fbd3c7386f-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.117 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.117 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.118 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.118 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.118 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.118 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.118 DEBUG:boto:Data: 2024-04-12T03:53:39.118 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.118 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.118 DEBUG:boto:Params: {} 2024-04-12T03:53:39.118 DEBUG:boto:Token: None 2024-04-12T03:53:39.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9g0HbHyqg66bqu0eptwDRzfjJ7c= 2024-04-12T03:53:39.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9g0HbHyqg66bqu0eptwDRzfjJ7c=', 'Content-Length': '0'} 2024-04-12T03:53:39.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b971235b7fced65a-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.120 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.120 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.120 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.120 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.121 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.121 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.121 DEBUG:boto:Data: 2024-04-12T03:53:39.121 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.121 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.121 DEBUG:boto:Params: {} 2024-04-12T03:53:39.121 DEBUG:boto:Token: None 2024-04-12T03:53:39.121 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9g0HbHyqg66bqu0eptwDRzfjJ7c= 2024-04-12T03:53:39.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9g0HbHyqg66bqu0eptwDRzfjJ7c=', 'Content-Length': '0'} 2024-04-12T03:53:39.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a27234b1660f7bf-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:53:39.123 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.123 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.123 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.123 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.123 DEBUG:boto:Method: GET 2024-04-12T03:53:39.123 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.124 DEBUG:boto:Data: 2024-04-12T03:53:39.124 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.124 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.124 DEBUG:boto:Params: {} 2024-04-12T03:53:39.124 DEBUG:boto:Token: None 2024-04-12T03:53:39.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk= 2024-04-12T03:53:39.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk=', 'Content-Length': '0'} 2024-04-12T03:53:39.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068570ed5b5c33d7b-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.126 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.126 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.126 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.126 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.126 DEBUG:boto:Method: GET 2024-04-12T03:53:39.126 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.126 DEBUG:boto:Data: 2024-04-12T03:53:39.127 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.127 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.127 DEBUG:boto:Params: {} 2024-04-12T03:53:39.127 DEBUG:boto:Token: None 2024-04-12T03:53:39.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:53:39.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk= 2024-04-12T03:53:39.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E6kuKsRSaoETicDRSZGfJIdlRGk=', 'Content-Length': '0'} 2024-04-12T03:53:39.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b89d6d2def9ead-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.170 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.170 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.171 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.171 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.171 DEBUG:boto:Method: GET 2024-04-12T03:53:39.171 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.171 DEBUG:boto:Data: 2024-04-12T03:53:39.171 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.171 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.171 DEBUG:boto:Params: {} 2024-04-12T03:53:39.171 DEBUG:boto:Token: None 2024-04-12T03:53:39.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.172 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec07bfd4319b2b1-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.174 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.174 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.174 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.174 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.174 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.175 DEBUG:boto:Method: GET 2024-04-12T03:53:39.175 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.175 DEBUG:boto:Data: 2024-04-12T03:53:39.175 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.175 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.175 DEBUG:boto:Params: {} 2024-04-12T03:53:39.175 DEBUG:boto:Token: None 2024-04-12T03:53:39.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.175 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103708c95967f907-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.179 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.180 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.180 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.180 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.180 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:53:39.180 DEBUG:boto:Data: 2024-04-12T03:53:39.180 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.180 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.180 DEBUG:boto:Params: {} 2024-04-12T03:53:39.180 DEBUG:boto:Token: None 2024-04-12T03:53:39.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17 2024-04-12T03:53:39.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2f2NyEZF7iaGuavSAElrlMYNaIg= 2024-04-12T03:53:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2f2NyEZF7iaGuavSAElrlMYNaIg=', 'Content-Length': '0'} 2024-04-12T03:53:39.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000214e8324cdcf83c-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.182 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.183 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.183 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.183 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:53:39.183 DEBUG:boto:Data: 2024-04-12T03:53:39.183 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.183 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.183 DEBUG:boto:Params: {} 2024-04-12T03:53:39.183 DEBUG:boto:Token: None 2024-04-12T03:53:39.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17 2024-04-12T03:53:39.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2f2NyEZF7iaGuavSAElrlMYNaIg= 2024-04-12T03:53:39.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2f2NyEZF7iaGuavSAElrlMYNaIg=', 'Content-Length': '0'} 2024-04-12T03:53:39.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867c3f4ef4520f53-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:53:39.185 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.186 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.186 DEBUG:boto:Method: GET 2024-04-12T03:53:39.186 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:53:39.186 DEBUG:boto:Data: 2024-04-12T03:53:39.186 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.186 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.186 DEBUG:boto:Params: {} 2024-04-12T03:53:39.186 DEBUG:boto:Token: None 2024-04-12T03:53:39.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17 2024-04-12T03:53:39.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/clSW9QXfj79HlJBPYDo4sGxYgU= 2024-04-12T03:53:39.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/clSW9QXfj79HlJBPYDo4sGxYgU=', 'Content-Length': '0'} 2024-04-12T03:53:39.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e5023fce5969787-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.189 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:53:39.189 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:53:39.189 DEBUG:boto:Method: GET 2024-04-12T03:53:39.189 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:53:39.189 DEBUG:boto:Data: 2024-04-12T03:53:39.189 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.189 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.189 DEBUG:boto:Params: {} 2024-04-12T03:53:39.189 DEBUG:boto:Token: None 2024-04-12T03:53:39.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj17 2024-04-12T03:53:39.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/clSW9QXfj79HlJBPYDo4sGxYgU= 2024-04-12T03:53:39.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/clSW9QXfj79HlJBPYDo4sGxYgU=', 'Content-Length': '0'} 2024-04-12T03:53:39.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009cf00aac32dbc7e-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.191 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.192 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.192 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.192 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.192 DEBUG:boto:Method: GET 2024-04-12T03:53:39.192 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.192 DEBUG:boto:Data: 2024-04-12T03:53:39.192 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.192 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.192 DEBUG:boto:Params: {} 2024-04-12T03:53:39.193 DEBUG:boto:Token: None 2024-04-12T03:53:39.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.193 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466c62c751e1b15a-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.195 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.195 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.195 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.196 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.196 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.196 DEBUG:boto:Method: GET 2024-04-12T03:53:39.196 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:53:39.196 DEBUG:boto:Data: 2024-04-12T03:53:39.196 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.196 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.196 DEBUG:boto:Params: {} 2024-04-12T03:53:39.196 DEBUG:boto:Token: None 2024-04-12T03:53:39.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2785c368dfd3f18-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.200 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:53:39.201 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.201 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.201 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.201 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.201 DEBUG:boto:Method: GET 2024-04-12T03:53:39.201 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.201 DEBUG:boto:Data: 2024-04-12T03:53:39.201 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.202 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.202 DEBUG:boto:Params: {} 2024-04-12T03:53:39.202 DEBUG:boto:Token: None 2024-04-12T03:53:39.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.202 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs= 2024-04-12T03:53:39.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs=', 'Content-Length': '0'} 2024-04-12T03:53:39.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003402e802859110de-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.246 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.246 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.247 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.247 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.247 DEBUG:boto:Method: GET 2024-04-12T03:53:39.247 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.247 DEBUG:boto:Data: 2024-04-12T03:53:39.247 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.247 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.247 DEBUG:boto:Params: {} 2024-04-12T03:53:39.247 DEBUG:boto:Token: None 2024-04-12T03:53:39.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs= 2024-04-12T03:53:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs=', 'Content-Length': '0'} 2024-04-12T03:53:39.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f94c992c436b88-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.250 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.250 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.250 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.250 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.250 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.250 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.250 DEBUG:boto:Data: 2024-04-12T03:53:39.250 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.250 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.251 DEBUG:boto:Params: {} 2024-04-12T03:53:39.251 DEBUG:boto:Token: None 2024-04-12T03:53:39.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j4jk1uPHLzxg2lYTukC7QV6Rtrk= 2024-04-12T03:53:39.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j4jk1uPHLzxg2lYTukC7QV6Rtrk=', 'Content-Length': '0'} 2024-04-12T03:53:39.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1fc80c9c8e7ae36-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.253 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.253 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.253 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.253 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.253 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.253 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.253 DEBUG:boto:Data: 2024-04-12T03:53:39.253 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.253 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.253 DEBUG:boto:Params: {} 2024-04-12T03:53:39.254 DEBUG:boto:Token: None 2024-04-12T03:53:39.254 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:j4jk1uPHLzxg2lYTukC7QV6Rtrk= 2024-04-12T03:53:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j4jk1uPHLzxg2lYTukC7QV6Rtrk=', 'Content-Length': '0'} 2024-04-12T03:53:39.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69ff14982b713d9-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:53:39.256 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.256 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.256 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.256 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.256 DEBUG:boto:Method: GET 2024-04-12T03:53:39.256 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.257 DEBUG:boto:Data: 2024-04-12T03:53:39.257 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.257 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.257 DEBUG:boto:Params: {} 2024-04-12T03:53:39.257 DEBUG:boto:Token: None 2024-04-12T03:53:39.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs= 2024-04-12T03:53:39.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs=', 'Content-Length': '0'} 2024-04-12T03:53:39.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026cb7e43d86fb519-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.259 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.259 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.259 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.259 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.259 DEBUG:boto:Method: GET 2024-04-12T03:53:39.259 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.259 DEBUG:boto:Data: 2024-04-12T03:53:39.259 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.260 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.260 DEBUG:boto:Params: {} 2024-04-12T03:53:39.260 DEBUG:boto:Token: None 2024-04-12T03:53:39.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:53:39.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs= 2024-04-12T03:53:39.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wu+na+FSzXESxiRh723B6fHCTOs=', 'Content-Length': '0'} 2024-04-12T03:53:39.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c951a0d2f6fe66-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.262 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.262 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.262 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.262 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.262 DEBUG:boto:Method: GET 2024-04-12T03:53:39.262 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.262 DEBUG:boto:Data: 2024-04-12T03:53:39.262 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.263 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.263 DEBUG:boto:Params: {} 2024-04-12T03:53:39.263 DEBUG:boto:Token: None 2024-04-12T03:53:39.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa1e2cf69df1ed1-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.265 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.266 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.266 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.266 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.266 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.266 DEBUG:boto:Method: GET 2024-04-12T03:53:39.266 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.266 DEBUG:boto:Data: 2024-04-12T03:53:39.266 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.267 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.267 DEBUG:boto:Params: {} 2024-04-12T03:53:39.267 DEBUG:boto:Token: None 2024-04-12T03:53:39.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77d4851263b6576-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.270 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.271 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.271 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.271 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.271 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:53:39.271 DEBUG:boto:Data: 2024-04-12T03:53:39.271 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.271 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.271 DEBUG:boto:Params: {} 2024-04-12T03:53:39.271 DEBUG:boto:Token: None 2024-04-12T03:53:39.272 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185 2024-04-12T03:53:39.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TQkb4gCzVXnd9PArFQxQ03N6fIU= 2024-04-12T03:53:39.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TQkb4gCzVXnd9PArFQxQ03N6fIU=', 'Content-Length': '0'} 2024-04-12T03:53:39.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523670541f38cf34-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.273 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.273 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.273 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.274 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:53:39.274 DEBUG:boto:Data: 2024-04-12T03:53:39.274 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.274 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.274 DEBUG:boto:Params: {} 2024-04-12T03:53:39.274 DEBUG:boto:Token: None 2024-04-12T03:53:39.274 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185 2024-04-12T03:53:39.274 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TQkb4gCzVXnd9PArFQxQ03N6fIU= 2024-04-12T03:53:39.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TQkb4gCzVXnd9PArFQxQ03N6fIU=', 'Content-Length': '0'} 2024-04-12T03:53:39.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe43290f5d98e6b-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:53:39.276 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.276 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.276 DEBUG:boto:Method: GET 2024-04-12T03:53:39.276 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:53:39.276 DEBUG:boto:Data: 2024-04-12T03:53:39.276 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.277 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.277 DEBUG:boto:Params: {} 2024-04-12T03:53:39.277 DEBUG:boto:Token: None 2024-04-12T03:53:39.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185 2024-04-12T03:53:39.277 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WO88C7wmNa69gZ9QtEv+40lwfmo= 2024-04-12T03:53:39.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WO88C7wmNa69gZ9QtEv+40lwfmo=', 'Content-Length': '0'} 2024-04-12T03:53:39.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbeebc82bdae357-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.279 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:53:39.279 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:53:39.279 DEBUG:boto:Method: GET 2024-04-12T03:53:39.279 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:53:39.279 DEBUG:boto:Data: 2024-04-12T03:53:39.279 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.279 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.279 DEBUG:boto:Params: {} 2024-04-12T03:53:39.279 DEBUG:boto:Token: None 2024-04-12T03:53:39.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj185 2024-04-12T03:53:39.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WO88C7wmNa69gZ9QtEv+40lwfmo= 2024-04-12T03:53:39.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WO88C7wmNa69gZ9QtEv+40lwfmo=', 'Content-Length': '0'} 2024-04-12T03:53:39.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007194d131fc12cb33-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.322 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.322 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.322 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.323 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.323 DEBUG:boto:Method: GET 2024-04-12T03:53:39.323 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.323 DEBUG:boto:Data: 2024-04-12T03:53:39.323 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.323 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.323 DEBUG:boto:Params: {} 2024-04-12T03:53:39.323 DEBUG:boto:Token: None 2024-04-12T03:53:39.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.323 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc1393dce291e65-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.326 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.326 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.326 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.326 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.326 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.326 DEBUG:boto:Method: GET 2024-04-12T03:53:39.326 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:53:39.326 DEBUG:boto:Data: 2024-04-12T03:53:39.327 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.327 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.327 DEBUG:boto:Params: {} 2024-04-12T03:53:39.327 DEBUG:boto:Token: None 2024-04-12T03:53:39.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.327 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28f1afcb82fb66b-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.330 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:53:39.331 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.331 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.331 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.331 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.331 DEBUG:boto:Method: GET 2024-04-12T03:53:39.331 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.331 DEBUG:boto:Data: 2024-04-12T03:53:39.331 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.332 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.332 DEBUG:boto:Params: {} 2024-04-12T03:53:39.332 DEBUG:boto:Token: None 2024-04-12T03:53:39.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ= 2024-04-12T03:53:39.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a433faa7104c9ee4-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.334 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.334 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.334 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.334 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.334 DEBUG:boto:Method: GET 2024-04-12T03:53:39.334 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.334 DEBUG:boto:Data: 2024-04-12T03:53:39.334 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.334 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.334 DEBUG:boto:Params: {} 2024-04-12T03:53:39.335 DEBUG:boto:Token: None 2024-04-12T03:53:39.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ= 2024-04-12T03:53:39.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca910c50c73e2109-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.337 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.337 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.337 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.337 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.337 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.337 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.337 DEBUG:boto:Data: 2024-04-12T03:53:39.337 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.337 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.337 DEBUG:boto:Params: {} 2024-04-12T03:53:39.338 DEBUG:boto:Token: None 2024-04-12T03:53:39.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hvmcsqpi8M79QS560ys5v0VuY64= 2024-04-12T03:53:39.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hvmcsqpi8M79QS560ys5v0VuY64=', 'Content-Length': '0'} 2024-04-12T03:53:39.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3fdc338c24b41f-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.340 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.340 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.340 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.340 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.340 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.340 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.340 DEBUG:boto:Data: 2024-04-12T03:53:39.340 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.341 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.341 DEBUG:boto:Params: {} 2024-04-12T03:53:39.341 DEBUG:boto:Token: None 2024-04-12T03:53:39.341 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hvmcsqpi8M79QS560ys5v0VuY64= 2024-04-12T03:53:39.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hvmcsqpi8M79QS560ys5v0VuY64=', 'Content-Length': '0'} 2024-04-12T03:53:39.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ad96c8a458e489-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:53:39.343 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.343 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.343 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.343 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.343 DEBUG:boto:Method: GET 2024-04-12T03:53:39.343 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.344 DEBUG:boto:Data: 2024-04-12T03:53:39.344 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.344 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.344 DEBUG:boto:Params: {} 2024-04-12T03:53:39.344 DEBUG:boto:Token: None 2024-04-12T03:53:39.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ= 2024-04-12T03:53:39.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016254f547e61844a-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.346 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.346 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.346 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.346 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.347 DEBUG:boto:Method: GET 2024-04-12T03:53:39.347 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.347 DEBUG:boto:Data: 2024-04-12T03:53:39.347 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.347 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.347 DEBUG:boto:Params: {} 2024-04-12T03:53:39.347 DEBUG:boto:Token: None 2024-04-12T03:53:39.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:53:39.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ= 2024-04-12T03:53:39.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FUDOafXURH7dO0MbkNxhP/Z3giQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a359f54b4088fd6c-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.390 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.390 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.390 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.391 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.391 DEBUG:boto:Method: GET 2024-04-12T03:53:39.391 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.391 DEBUG:boto:Data: 2024-04-12T03:53:39.391 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.391 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.391 DEBUG:boto:Params: {} 2024-04-12T03:53:39.391 DEBUG:boto:Token: None 2024-04-12T03:53:39.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2562c7fc5b1732e-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.393 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.394 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.394 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.394 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.394 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.394 DEBUG:boto:Method: GET 2024-04-12T03:53:39.394 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.394 DEBUG:boto:Data: 2024-04-12T03:53:39.394 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.395 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.395 DEBUG:boto:Params: {} 2024-04-12T03:53:39.395 DEBUG:boto:Token: None 2024-04-12T03:53:39.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526dcc5b0350fe73-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.399 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.399 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.399 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.400 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.400 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:53:39.400 DEBUG:boto:Data: 2024-04-12T03:53:39.400 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.400 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.400 DEBUG:boto:Params: {} 2024-04-12T03:53:39.400 DEBUG:boto:Token: None 2024-04-12T03:53:39.400 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19 2024-04-12T03:53:39.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1JjDz3K6/HLx426aTNXXPesBx2c= 2024-04-12T03:53:39.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1JjDz3K6/HLx426aTNXXPesBx2c=', 'Content-Length': '0'} 2024-04-12T03:53:39.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d06e080c9140ae-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.402 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.402 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.402 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.402 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:53:39.402 DEBUG:boto:Data: 2024-04-12T03:53:39.402 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.403 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.403 DEBUG:boto:Params: {} 2024-04-12T03:53:39.403 DEBUG:boto:Token: None 2024-04-12T03:53:39.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19 2024-04-12T03:53:39.403 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1JjDz3K6/HLx426aTNXXPesBx2c= 2024-04-12T03:53:39.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1JjDz3K6/HLx426aTNXXPesBx2c=', 'Content-Length': '0'} 2024-04-12T03:53:39.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095a39b05f9a49c3-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:53:39.405 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.405 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.405 DEBUG:boto:Method: GET 2024-04-12T03:53:39.406 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:53:39.406 DEBUG:boto:Data: 2024-04-12T03:53:39.406 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.406 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.406 DEBUG:boto:Params: {} 2024-04-12T03:53:39.406 DEBUG:boto:Token: None 2024-04-12T03:53:39.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19 2024-04-12T03:53:39.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FQUIrb9wgXkUbN6xrivJ+4SFOT4= 2024-04-12T03:53:39.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FQUIrb9wgXkUbN6xrivJ+4SFOT4=', 'Content-Length': '0'} 2024-04-12T03:53:39.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e150a0f363ae987d-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.408 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:53:39.408 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:53:39.408 DEBUG:boto:Method: GET 2024-04-12T03:53:39.408 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:53:39.408 DEBUG:boto:Data: 2024-04-12T03:53:39.408 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.409 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.409 DEBUG:boto:Params: {} 2024-04-12T03:53:39.409 DEBUG:boto:Token: None 2024-04-12T03:53:39.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj19 2024-04-12T03:53:39.409 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FQUIrb9wgXkUbN6xrivJ+4SFOT4= 2024-04-12T03:53:39.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FQUIrb9wgXkUbN6xrivJ+4SFOT4=', 'Content-Length': '0'} 2024-04-12T03:53:39.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5b6f04da401572-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.411 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.411 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.411 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.411 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.411 DEBUG:boto:Method: GET 2024-04-12T03:53:39.411 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.411 DEBUG:boto:Data: 2024-04-12T03:53:39.411 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.411 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.412 DEBUG:boto:Params: {} 2024-04-12T03:53:39.412 DEBUG:boto:Token: None 2024-04-12T03:53:39.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021bf673eb329bb2-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.414 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.414 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.414 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.414 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.414 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.414 DEBUG:boto:Method: GET 2024-04-12T03:53:39.415 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:53:39.415 DEBUG:boto:Data: 2024-04-12T03:53:39.415 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.415 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.415 DEBUG:boto:Params: {} 2024-04-12T03:53:39.415 DEBUG:boto:Token: None 2024-04-12T03:53:39.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.415 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01cdcf8ed6eb0ef-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.462 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:53:39.463 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.463 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.463 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.463 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.463 DEBUG:boto:Method: GET 2024-04-12T03:53:39.463 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.463 DEBUG:boto:Data: 2024-04-12T03:53:39.464 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.464 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.464 DEBUG:boto:Params: {} 2024-04-12T03:53:39.464 DEBUG:boto:Token: None 2024-04-12T03:53:39.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c= 2024-04-12T03:53:39.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c=', 'Content-Length': '0'} 2024-04-12T03:53:39.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007208d7b87cf3820b-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.467 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.467 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.467 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.467 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.467 DEBUG:boto:Method: GET 2024-04-12T03:53:39.467 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.468 DEBUG:boto:Data: 2024-04-12T03:53:39.468 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.468 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.468 DEBUG:boto:Params: {} 2024-04-12T03:53:39.468 DEBUG:boto:Token: None 2024-04-12T03:53:39.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c= 2024-04-12T03:53:39.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c=', 'Content-Length': '0'} 2024-04-12T03:53:39.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0e238898472818-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.470 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.471 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.471 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.471 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.471 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.471 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.471 DEBUG:boto:Data: 2024-04-12T03:53:39.471 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.471 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.471 DEBUG:boto:Params: {} 2024-04-12T03:53:39.472 DEBUG:boto:Token: None 2024-04-12T03:53:39.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/jJv+duaVbPLMvXyGxGrFrnmpPY= 2024-04-12T03:53:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/jJv+duaVbPLMvXyGxGrFrnmpPY=', 'Content-Length': '0'} 2024-04-12T03:53:39.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a39eec339ffe24-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.474 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.474 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.474 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.474 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.474 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.474 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.474 DEBUG:boto:Data: 2024-04-12T03:53:39.474 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.474 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.474 DEBUG:boto:Params: {} 2024-04-12T03:53:39.475 DEBUG:boto:Token: None 2024-04-12T03:53:39.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/jJv+duaVbPLMvXyGxGrFrnmpPY= 2024-04-12T03:53:39.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/jJv+duaVbPLMvXyGxGrFrnmpPY=', 'Content-Length': '0'} 2024-04-12T03:53:39.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49fbcad13bc6b2e-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:53:39.477 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.477 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.477 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.478 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.478 DEBUG:boto:Method: GET 2024-04-12T03:53:39.478 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.478 DEBUG:boto:Data: 2024-04-12T03:53:39.478 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.478 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.478 DEBUG:boto:Params: {} 2024-04-12T03:53:39.478 DEBUG:boto:Token: None 2024-04-12T03:53:39.478 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c= 2024-04-12T03:53:39.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c=', 'Content-Length': '0'} 2024-04-12T03:53:39.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f22d60bd3a2582-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.522 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.522 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.523 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.523 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.523 DEBUG:boto:Method: GET 2024-04-12T03:53:39.523 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.523 DEBUG:boto:Data: 2024-04-12T03:53:39.523 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.523 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.523 DEBUG:boto:Params: {} 2024-04-12T03:53:39.523 DEBUG:boto:Token: None 2024-04-12T03:53:39.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:53:39.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c= 2024-04-12T03:53:39.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZOquryc1pZsI7HN7Puoh6KYHa9c=', 'Content-Length': '0'} 2024-04-12T03:53:39.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c415ee70e75dd7a3-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.526 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.526 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.526 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.526 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.526 DEBUG:boto:Method: GET 2024-04-12T03:53:39.526 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.526 DEBUG:boto:Data: 2024-04-12T03:53:39.526 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.526 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.527 DEBUG:boto:Params: {} 2024-04-12T03:53:39.527 DEBUG:boto:Token: None 2024-04-12T03:53:39.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b5e742c1ec27337-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.529 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.529 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.530 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.530 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.530 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.530 DEBUG:boto:Method: GET 2024-04-12T03:53:39.530 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.530 DEBUG:boto:Data: 2024-04-12T03:53:39.530 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.530 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.530 DEBUG:boto:Params: {} 2024-04-12T03:53:39.530 DEBUG:boto:Token: None 2024-04-12T03:53:39.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf6cfd221b18b27-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.535 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.535 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.535 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.535 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.535 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:53:39.535 DEBUG:boto:Data: 2024-04-12T03:53:39.536 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.536 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.536 DEBUG:boto:Params: {} 2024-04-12T03:53:39.536 DEBUG:boto:Token: None 2024-04-12T03:53:39.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22 2024-04-12T03:53:39.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yCYNw9dGYQsue9SsnkDUJQ5PBw= 2024-04-12T03:53:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yCYNw9dGYQsue9SsnkDUJQ5PBw=', 'Content-Length': '0'} 2024-04-12T03:53:39.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8a4e68121da4c6-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.538 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.538 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.538 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.538 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:53:39.538 DEBUG:boto:Data: 2024-04-12T03:53:39.538 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.539 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.539 DEBUG:boto:Params: {} 2024-04-12T03:53:39.539 DEBUG:boto:Token: None 2024-04-12T03:53:39.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22 2024-04-12T03:53:39.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+yCYNw9dGYQsue9SsnkDUJQ5PBw= 2024-04-12T03:53:39.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+yCYNw9dGYQsue9SsnkDUJQ5PBw=', 'Content-Length': '0'} 2024-04-12T03:53:39.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e37dcc4722be36-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:53:39.541 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.541 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.541 DEBUG:boto:Method: GET 2024-04-12T03:53:39.541 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:53:39.541 DEBUG:boto:Data: 2024-04-12T03:53:39.542 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.542 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.542 DEBUG:boto:Params: {} 2024-04-12T03:53:39.542 DEBUG:boto:Token: None 2024-04-12T03:53:39.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22 2024-04-12T03:53:39.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qci2rDAqVFyfxQ11V/ckGWJzjPQ= 2024-04-12T03:53:39.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qci2rDAqVFyfxQ11V/ckGWJzjPQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334386475f3c1f0c-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.544 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:53:39.544 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:53:39.544 DEBUG:boto:Method: GET 2024-04-12T03:53:39.544 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:53:39.545 DEBUG:boto:Data: 2024-04-12T03:53:39.545 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.545 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.545 DEBUG:boto:Params: {} 2024-04-12T03:53:39.545 DEBUG:boto:Token: None 2024-04-12T03:53:39.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj22 2024-04-12T03:53:39.545 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qci2rDAqVFyfxQ11V/ckGWJzjPQ= 2024-04-12T03:53:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qci2rDAqVFyfxQ11V/ckGWJzjPQ=', 'Content-Length': '0'} 2024-04-12T03:53:39.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d4e9f5db4478ae2-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.590 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.590 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.591 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.591 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.591 DEBUG:boto:Method: GET 2024-04-12T03:53:39.591 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.591 DEBUG:boto:Data: 2024-04-12T03:53:39.591 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.591 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.591 DEBUG:boto:Params: {} 2024-04-12T03:53:39.591 DEBUG:boto:Token: None 2024-04-12T03:53:39.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f461527471573ffb-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.594 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.594 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.594 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.594 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.594 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.595 DEBUG:boto:Method: GET 2024-04-12T03:53:39.595 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:53:39.595 DEBUG:boto:Data: 2024-04-12T03:53:39.595 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.595 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.595 DEBUG:boto:Params: {} 2024-04-12T03:53:39.595 DEBUG:boto:Token: None 2024-04-12T03:53:39.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.596 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123816cd77e2143d-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.599 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:53:39.599 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.599 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.599 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.600 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.600 DEBUG:boto:Method: GET 2024-04-12T03:53:39.600 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.600 DEBUG:boto:Data: 2024-04-12T03:53:39.600 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.600 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.600 DEBUG:boto:Params: {} 2024-04-12T03:53:39.600 DEBUG:boto:Token: None 2024-04-12T03:53:39.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0= 2024-04-12T03:53:39.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0=', 'Content-Length': '0'} 2024-04-12T03:53:39.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd9f6ae8a3ef6fc-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.603 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.603 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.603 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.603 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.603 DEBUG:boto:Method: GET 2024-04-12T03:53:39.603 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.603 DEBUG:boto:Data: 2024-04-12T03:53:39.603 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.603 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.603 DEBUG:boto:Params: {} 2024-04-12T03:53:39.604 DEBUG:boto:Token: None 2024-04-12T03:53:39.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0= 2024-04-12T03:53:39.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0=', 'Content-Length': '0'} 2024-04-12T03:53:39.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d0f6c628b210fd-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.606 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.606 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.606 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.606 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.606 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.606 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.606 DEBUG:boto:Data: 2024-04-12T03:53:39.606 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.607 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.607 DEBUG:boto:Params: {} 2024-04-12T03:53:39.607 DEBUG:boto:Token: None 2024-04-12T03:53:39.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zs1xTN5fn0Yy3MzBoi+C1QGgBy0= 2024-04-12T03:53:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zs1xTN5fn0Yy3MzBoi+C1QGgBy0=', 'Content-Length': '0'} 2024-04-12T03:53:39.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000360ca67018d7be3-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.609 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.609 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.609 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.609 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.609 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.609 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.609 DEBUG:boto:Data: 2024-04-12T03:53:39.609 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.610 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.610 DEBUG:boto:Params: {} 2024-04-12T03:53:39.610 DEBUG:boto:Token: None 2024-04-12T03:53:39.610 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.610 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zs1xTN5fn0Yy3MzBoi+C1QGgBy0= 2024-04-12T03:53:39.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zs1xTN5fn0Yy3MzBoi+C1QGgBy0=', 'Content-Length': '0'} 2024-04-12T03:53:39.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0ea146acc25401-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:53:39.612 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.612 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.612 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.612 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.612 DEBUG:boto:Method: GET 2024-04-12T03:53:39.612 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.612 DEBUG:boto:Data: 2024-04-12T03:53:39.613 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.613 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.613 DEBUG:boto:Params: {} 2024-04-12T03:53:39.613 DEBUG:boto:Token: None 2024-04-12T03:53:39.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0= 2024-04-12T03:53:39.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0=', 'Content-Length': '0'} 2024-04-12T03:53:39.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a92a2b237a5c299-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.615 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.615 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.615 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.616 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.616 DEBUG:boto:Method: GET 2024-04-12T03:53:39.616 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.616 DEBUG:boto:Data: 2024-04-12T03:53:39.616 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.616 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.616 DEBUG:boto:Params: {} 2024-04-12T03:53:39.616 DEBUG:boto:Token: None 2024-04-12T03:53:39.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:53:39.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0= 2024-04-12T03:53:39.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:h8RCaIfslcRkSPS8AKCDST5SEW0=', 'Content-Length': '0'} 2024-04-12T03:53:39.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f1f8b8023838368-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.618 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.619 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.619 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.619 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.619 DEBUG:boto:Method: GET 2024-04-12T03:53:39.619 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.619 DEBUG:boto:Data: 2024-04-12T03:53:39.619 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.619 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.620 DEBUG:boto:Params: {} 2024-04-12T03:53:39.620 DEBUG:boto:Token: None 2024-04-12T03:53:39.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f0e0d1e4ac82079-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.662 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.666 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.666 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.666 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.666 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.666 DEBUG:boto:Method: GET 2024-04-12T03:53:39.666 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.666 DEBUG:boto:Data: 2024-04-12T03:53:39.666 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.667 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.667 DEBUG:boto:Params: {} 2024-04-12T03:53:39.667 DEBUG:boto:Token: None 2024-04-12T03:53:39.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.667 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f69e4c164e756c-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.672 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.672 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.672 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.672 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.672 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:53:39.672 DEBUG:boto:Data: 2024-04-12T03:53:39.673 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.673 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.673 DEBUG:boto:Params: {} 2024-04-12T03:53:39.673 DEBUG:boto:Token: None 2024-04-12T03:53:39.673 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23 2024-04-12T03:53:39.673 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YqBZGqMuQYXvhlgpkupeNts9cVo= 2024-04-12T03:53:39.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YqBZGqMuQYXvhlgpkupeNts9cVo=', 'Content-Length': '0'} 2024-04-12T03:53:39.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c31437f1be4346-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.676 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.676 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.676 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.676 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:53:39.676 DEBUG:boto:Data: 2024-04-12T03:53:39.676 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.676 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.677 DEBUG:boto:Params: {} 2024-04-12T03:53:39.677 DEBUG:boto:Token: None 2024-04-12T03:53:39.677 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23 2024-04-12T03:53:39.677 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YqBZGqMuQYXvhlgpkupeNts9cVo= 2024-04-12T03:53:39.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YqBZGqMuQYXvhlgpkupeNts9cVo=', 'Content-Length': '0'} 2024-04-12T03:53:39.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049673561082762ed-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:53:39.680 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.680 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.680 DEBUG:boto:Method: GET 2024-04-12T03:53:39.680 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:53:39.680 DEBUG:boto:Data: 2024-04-12T03:53:39.680 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.680 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.680 DEBUG:boto:Params: {} 2024-04-12T03:53:39.680 DEBUG:boto:Token: None 2024-04-12T03:53:39.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23 2024-04-12T03:53:39.681 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mpUDxBlTwHrPz6tQWTUDBFYP3Jk= 2024-04-12T03:53:39.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mpUDxBlTwHrPz6tQWTUDBFYP3Jk=', 'Content-Length': '0'} 2024-04-12T03:53:39.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032cc7d68d349b2c1-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.683 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:53:39.683 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:53:39.683 DEBUG:boto:Method: GET 2024-04-12T03:53:39.683 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:53:39.683 DEBUG:boto:Data: 2024-04-12T03:53:39.683 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.683 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.683 DEBUG:boto:Params: {} 2024-04-12T03:53:39.684 DEBUG:boto:Token: None 2024-04-12T03:53:39.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj23 2024-04-12T03:53:39.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mpUDxBlTwHrPz6tQWTUDBFYP3Jk= 2024-04-12T03:53:39.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mpUDxBlTwHrPz6tQWTUDBFYP3Jk=', 'Content-Length': '0'} 2024-04-12T03:53:39.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003455db0e8fea76ab-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.727 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.727 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.727 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.727 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.727 DEBUG:boto:Method: GET 2024-04-12T03:53:39.727 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.727 DEBUG:boto:Data: 2024-04-12T03:53:39.727 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.728 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.728 DEBUG:boto:Params: {} 2024-04-12T03:53:39.728 DEBUG:boto:Token: None 2024-04-12T03:53:39.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005118c1b59c4938be-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.731 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.732 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.732 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.732 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.732 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.732 DEBUG:boto:Method: GET 2024-04-12T03:53:39.732 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:53:39.733 DEBUG:boto:Data: 2024-04-12T03:53:39.733 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.733 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.733 DEBUG:boto:Params: {} 2024-04-12T03:53:39.733 DEBUG:boto:Token: None 2024-04-12T03:53:39.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf40cc2c71389bba-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.739 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:53:39.740 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.740 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.740 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.740 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.740 DEBUG:boto:Method: GET 2024-04-12T03:53:39.740 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.741 DEBUG:boto:Data: 2024-04-12T03:53:39.741 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.741 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.741 DEBUG:boto:Params: {} 2024-04-12T03:53:39.741 DEBUG:boto:Token: None 2024-04-12T03:53:39.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew= 2024-04-12T03:53:39.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew=', 'Content-Length': '0'} 2024-04-12T03:53:39.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012c81a4cd34d562c-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.786 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.786 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.787 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.787 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.787 DEBUG:boto:Method: GET 2024-04-12T03:53:39.787 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.787 DEBUG:boto:Data: 2024-04-12T03:53:39.787 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.787 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.787 DEBUG:boto:Params: {} 2024-04-12T03:53:39.787 DEBUG:boto:Token: None 2024-04-12T03:53:39.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew= 2024-04-12T03:53:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew=', 'Content-Length': '0'} 2024-04-12T03:53:39.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ba49c0af9a828f-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.790 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.790 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.790 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.790 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.790 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.790 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.791 DEBUG:boto:Data: 2024-04-12T03:53:39.791 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.791 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.791 DEBUG:boto:Params: {} 2024-04-12T03:53:39.791 DEBUG:boto:Token: None 2024-04-12T03:53:39.791 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0VJarif+T03Z2sRPzJLPLDv+i20= 2024-04-12T03:53:39.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0VJarif+T03Z2sRPzJLPLDv+i20=', 'Content-Length': '0'} 2024-04-12T03:53:39.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000782d01c844ede0f8-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.793 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.793 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.793 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.793 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.793 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.793 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.794 DEBUG:boto:Data: 2024-04-12T03:53:39.794 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.794 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.794 DEBUG:boto:Params: {} 2024-04-12T03:53:39.794 DEBUG:boto:Token: None 2024-04-12T03:53:39.794 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.794 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0VJarif+T03Z2sRPzJLPLDv+i20= 2024-04-12T03:53:39.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0VJarif+T03Z2sRPzJLPLDv+i20=', 'Content-Length': '0'} 2024-04-12T03:53:39.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f82e6de97fb3b6f-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:53:39.796 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.796 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.796 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.797 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.797 DEBUG:boto:Method: GET 2024-04-12T03:53:39.797 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.797 DEBUG:boto:Data: 2024-04-12T03:53:39.797 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.797 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.797 DEBUG:boto:Params: {} 2024-04-12T03:53:39.797 DEBUG:boto:Token: None 2024-04-12T03:53:39.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.798 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew= 2024-04-12T03:53:39.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew=', 'Content-Length': '0'} 2024-04-12T03:53:39.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76ac447a7505e24-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.800 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.800 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.800 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.800 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.800 DEBUG:boto:Method: GET 2024-04-12T03:53:39.800 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.800 DEBUG:boto:Data: 2024-04-12T03:53:39.800 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.800 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.801 DEBUG:boto:Params: {} 2024-04-12T03:53:39.801 DEBUG:boto:Token: None 2024-04-12T03:53:39.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:53:39.801 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew= 2024-04-12T03:53:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88/voz9sQ2REeUkWtMcbq/Yovew=', 'Content-Length': '0'} 2024-04-12T03:53:39.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81f2a92abcf5823-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.803 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.803 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.803 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.803 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.803 DEBUG:boto:Method: GET 2024-04-12T03:53:39.803 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.804 DEBUG:boto:Data: 2024-04-12T03:53:39.804 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.804 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.804 DEBUG:boto:Params: {} 2024-04-12T03:53:39.804 DEBUG:boto:Token: None 2024-04-12T03:53:39.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.804 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b197b187f978e2-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.807 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.807 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.807 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.807 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.807 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.808 DEBUG:boto:Method: GET 2024-04-12T03:53:39.808 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.808 DEBUG:boto:Data: 2024-04-12T03:53:39.808 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.808 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.808 DEBUG:boto:Params: {} 2024-04-12T03:53:39.808 DEBUG:boto:Token: None 2024-04-12T03:53:39.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0740776e15ecf95-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.811 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.812 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.812 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.812 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.812 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:53:39.812 DEBUG:boto:Data: 2024-04-12T03:53:39.812 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.812 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.813 DEBUG:boto:Params: {} 2024-04-12T03:53:39.813 DEBUG:boto:Token: None 2024-04-12T03:53:39.813 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29 2024-04-12T03:53:39.813 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/OfOJf4cnTIlqp+PBddK3zPYqE= 2024-04-12T03:53:39.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/OfOJf4cnTIlqp+PBddK3zPYqE=', 'Content-Length': '0'} 2024-04-12T03:53:39.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019db992a58db61f5-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.815 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.815 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.815 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.815 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:53:39.815 DEBUG:boto:Data: 2024-04-12T03:53:39.815 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.815 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.815 DEBUG:boto:Params: {} 2024-04-12T03:53:39.815 DEBUG:boto:Token: None 2024-04-12T03:53:39.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29 2024-04-12T03:53:39.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/OfOJf4cnTIlqp+PBddK3zPYqE= 2024-04-12T03:53:39.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/OfOJf4cnTIlqp+PBddK3zPYqE=', 'Content-Length': '0'} 2024-04-12T03:53:39.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686100c04f6e2ace-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:53:39.818 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.818 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.818 DEBUG:boto:Method: GET 2024-04-12T03:53:39.818 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:53:39.818 DEBUG:boto:Data: 2024-04-12T03:53:39.818 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.818 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.818 DEBUG:boto:Params: {} 2024-04-12T03:53:39.819 DEBUG:boto:Token: None 2024-04-12T03:53:39.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29 2024-04-12T03:53:39.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1owgY0tCBninsEXnjmpM6NPHhDg= 2024-04-12T03:53:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1owgY0tCBninsEXnjmpM6NPHhDg=', 'Content-Length': '0'} 2024-04-12T03:53:39.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68bcc28cde4c749-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.862 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:53:39.862 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:53:39.863 DEBUG:boto:Method: GET 2024-04-12T03:53:39.863 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:53:39.863 DEBUG:boto:Data: 2024-04-12T03:53:39.863 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.863 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.863 DEBUG:boto:Params: {} 2024-04-12T03:53:39.863 DEBUG:boto:Token: None 2024-04-12T03:53:39.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj29 2024-04-12T03:53:39.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1owgY0tCBninsEXnjmpM6NPHhDg= 2024-04-12T03:53:39.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1owgY0tCBninsEXnjmpM6NPHhDg=', 'Content-Length': '0'} 2024-04-12T03:53:39.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad642e76e2bf1271-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.865 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.865 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.865 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.866 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.866 DEBUG:boto:Method: GET 2024-04-12T03:53:39.866 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.866 DEBUG:boto:Data: 2024-04-12T03:53:39.866 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.866 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.866 DEBUG:boto:Params: {} 2024-04-12T03:53:39.866 DEBUG:boto:Token: None 2024-04-12T03:53:39.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58b180927845501-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.869 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.869 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.870 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.870 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.870 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.870 DEBUG:boto:Method: GET 2024-04-12T03:53:39.870 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:53:39.870 DEBUG:boto:Data: 2024-04-12T03:53:39.870 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.870 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.870 DEBUG:boto:Params: {} 2024-04-12T03:53:39.871 DEBUG:boto:Token: None 2024-04-12T03:53:39.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea981e98a7a4a459-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.875 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:53:39.875 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.875 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.875 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.875 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.876 DEBUG:boto:Method: GET 2024-04-12T03:53:39.876 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.876 DEBUG:boto:Data: 2024-04-12T03:53:39.876 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.876 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.876 DEBUG:boto:Params: {} 2024-04-12T03:53:39.876 DEBUG:boto:Token: None 2024-04-12T03:53:39.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E= 2024-04-12T03:53:39.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E=', 'Content-Length': '0'} 2024-04-12T03:53:39.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8c390341b606ea-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.878 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.879 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.879 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.879 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.879 DEBUG:boto:Method: GET 2024-04-12T03:53:39.879 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.879 DEBUG:boto:Data: 2024-04-12T03:53:39.879 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.879 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.879 DEBUG:boto:Params: {} 2024-04-12T03:53:39.880 DEBUG:boto:Token: None 2024-04-12T03:53:39.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E= 2024-04-12T03:53:39.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E=', 'Content-Length': '0'} 2024-04-12T03:53:39.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110d660b4917b4f7-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.882 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.882 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.882 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.882 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.882 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.883 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.883 DEBUG:boto:Data: 2024-04-12T03:53:39.883 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.883 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.883 DEBUG:boto:Params: {} 2024-04-12T03:53:39.883 DEBUG:boto:Token: None 2024-04-12T03:53:39.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8muEEicvOzRvjc8KAiReESnUijI= 2024-04-12T03:53:39.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8muEEicvOzRvjc8KAiReESnUijI=', 'Content-Length': '0'} 2024-04-12T03:53:39.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b5ec119e63ebf2-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.885 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.885 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.886 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.886 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.886 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.886 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.886 DEBUG:boto:Data: 2024-04-12T03:53:39.886 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.886 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.886 DEBUG:boto:Params: {} 2024-04-12T03:53:39.886 DEBUG:boto:Token: None 2024-04-12T03:53:39.886 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8muEEicvOzRvjc8KAiReESnUijI= 2024-04-12T03:53:39.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8muEEicvOzRvjc8KAiReESnUijI=', 'Content-Length': '0'} 2024-04-12T03:53:39.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e42d01d47cf6011c-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:53:39.889 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.889 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.889 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.889 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.889 DEBUG:boto:Method: GET 2024-04-12T03:53:39.889 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.889 DEBUG:boto:Data: 2024-04-12T03:53:39.889 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.889 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.889 DEBUG:boto:Params: {} 2024-04-12T03:53:39.890 DEBUG:boto:Token: None 2024-04-12T03:53:39.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E= 2024-04-12T03:53:39.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E=', 'Content-Length': '0'} 2024-04-12T03:53:39.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3025f23f57d633-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.892 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.892 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.892 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.892 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.892 DEBUG:boto:Method: GET 2024-04-12T03:53:39.892 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.892 DEBUG:boto:Data: 2024-04-12T03:53:39.892 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.893 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.893 DEBUG:boto:Params: {} 2024-04-12T03:53:39.893 DEBUG:boto:Token: None 2024-04-12T03:53:39.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:53:39.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E= 2024-04-12T03:53:39.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S4Bwq2/J9n5CJmFUifjjxc24p7E=', 'Content-Length': '0'} 2024-04-12T03:53:39.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023fa4c7e7b7f8044-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.895 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.895 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.895 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.895 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.895 DEBUG:boto:Method: GET 2024-04-12T03:53:39.896 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.896 DEBUG:boto:Data: 2024-04-12T03:53:39.896 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.896 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.896 DEBUG:boto:Params: {} 2024-04-12T03:53:39.896 DEBUG:boto:Token: None 2024-04-12T03:53:39.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4751811b52d438f-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.898 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.899 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.899 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.899 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.899 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.899 DEBUG:boto:Method: GET 2024-04-12T03:53:39.899 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.899 DEBUG:boto:Data: 2024-04-12T03:53:39.899 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.900 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.900 DEBUG:boto:Params: {} 2024-04-12T03:53:39.900 DEBUG:boto:Token: None 2024-04-12T03:53:39.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea4e35b7401bee5e-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.904 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.904 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.904 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.904 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.904 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:53:39.904 DEBUG:boto:Data: 2024-04-12T03:53:39.904 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.905 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.905 DEBUG:boto:Params: {} 2024-04-12T03:53:39.905 DEBUG:boto:Token: None 2024-04-12T03:53:39.905 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31 2024-04-12T03:53:39.905 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5V/+P3Gtec81qqBxhss0CQIZmug= 2024-04-12T03:53:39.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5V/+P3Gtec81qqBxhss0CQIZmug=', 'Content-Length': '0'} 2024-04-12T03:53:39.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bad2aef9d089e02-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.907 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.907 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.907 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.907 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:53:39.907 DEBUG:boto:Data: 2024-04-12T03:53:39.908 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.908 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.908 DEBUG:boto:Params: {} 2024-04-12T03:53:39.908 DEBUG:boto:Token: None 2024-04-12T03:53:39.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31 2024-04-12T03:53:39.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5V/+P3Gtec81qqBxhss0CQIZmug= 2024-04-12T03:53:39.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5V/+P3Gtec81qqBxhss0CQIZmug=', 'Content-Length': '0'} 2024-04-12T03:53:39.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb076a92dbc1c15-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:53:39.910 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.910 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.910 DEBUG:boto:Method: GET 2024-04-12T03:53:39.911 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:53:39.911 DEBUG:boto:Data: 2024-04-12T03:53:39.911 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.911 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.911 DEBUG:boto:Params: {} 2024-04-12T03:53:39.911 DEBUG:boto:Token: None 2024-04-12T03:53:39.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31 2024-04-12T03:53:39.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DkOnn02whRiG6lb2ZurQSfqPQGs= 2024-04-12T03:53:39.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DkOnn02whRiG6lb2ZurQSfqPQGs=', 'Content-Length': '0'} 2024-04-12T03:53:39.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912be456ec8ae699-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.954 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:53:39.954 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:53:39.955 DEBUG:boto:Method: GET 2024-04-12T03:53:39.955 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:53:39.955 DEBUG:boto:Data: 2024-04-12T03:53:39.955 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.955 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.955 DEBUG:boto:Params: {} 2024-04-12T03:53:39.955 DEBUG:boto:Token: None 2024-04-12T03:53:39.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj31 2024-04-12T03:53:39.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DkOnn02whRiG6lb2ZurQSfqPQGs= 2024-04-12T03:53:39.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DkOnn02whRiG6lb2ZurQSfqPQGs=', 'Content-Length': '0'} 2024-04-12T03:53:39.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d41afb399e5e752c-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.958 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.958 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.958 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.958 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.958 DEBUG:boto:Method: GET 2024-04-12T03:53:39.958 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.958 DEBUG:boto:Data: 2024-04-12T03:53:39.958 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.959 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.959 DEBUG:boto:Params: {} 2024-04-12T03:53:39.959 DEBUG:boto:Token: None 2024-04-12T03:53:39.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.959 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042928f5a3db9c8e5-006618b043-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.961 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.962 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.962 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.962 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.962 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.962 DEBUG:boto:Method: GET 2024-04-12T03:53:39.962 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:53:39.962 DEBUG:boto:Data: 2024-04-12T03:53:39.962 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.963 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.963 DEBUG:boto:Params: {} 2024-04-12T03:53:39.963 DEBUG:boto:Token: None 2024-04-12T03:53:39.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.963 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2913de230a9f1cd-006618b043-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.967 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:53:39.967 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.968 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.968 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.968 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.968 DEBUG:boto:Method: GET 2024-04-12T03:53:39.968 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.968 DEBUG:boto:Data: 2024-04-12T03:53:39.968 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.968 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.968 DEBUG:boto:Params: {} 2024-04-12T03:53:39.969 DEBUG:boto:Token: None 2024-04-12T03:53:39.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.969 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA= 2024-04-12T03:53:39.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA=', 'Content-Length': '0'} 2024-04-12T03:53:39.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd606d75d5e679ac-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.971 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.971 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.971 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.971 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.971 DEBUG:boto:Method: GET 2024-04-12T03:53:39.971 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.971 DEBUG:boto:Data: 2024-04-12T03:53:39.971 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.972 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.972 DEBUG:boto:Params: {} 2024-04-12T03:53:39.972 DEBUG:boto:Token: None 2024-04-12T03:53:39.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA= 2024-04-12T03:53:39.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA=', 'Content-Length': '0'} 2024-04-12T03:53:39.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc36e3ec7ebef97-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.974 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.974 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.975 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.975 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.975 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.975 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.975 DEBUG:boto:Data: 2024-04-12T03:53:39.975 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.975 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.975 DEBUG:boto:Params: {} 2024-04-12T03:53:39.976 DEBUG:boto:Token: None 2024-04-12T03:53:39.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDcAj2MlD7OvXnxM0pH44UjdDIM= 2024-04-12T03:53:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDcAj2MlD7OvXnxM0pH44UjdDIM=', 'Content-Length': '0'} 2024-04-12T03:53:39.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081aadb5be9a58b7e-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.978 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.978 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.978 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.978 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.978 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.978 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.979 DEBUG:boto:Data: 2024-04-12T03:53:39.979 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.979 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.979 DEBUG:boto:Params: {} 2024-04-12T03:53:39.979 DEBUG:boto:Token: None 2024-04-12T03:53:39.979 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.979 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DDcAj2MlD7OvXnxM0pH44UjdDIM= 2024-04-12T03:53:39.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DDcAj2MlD7OvXnxM0pH44UjdDIM=', 'Content-Length': '0'} 2024-04-12T03:53:39.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e457564e326c98e6-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:53:39.981 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.981 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.982 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.982 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.982 DEBUG:boto:Method: GET 2024-04-12T03:53:39.982 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.982 DEBUG:boto:Data: 2024-04-12T03:53:39.982 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.982 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.982 DEBUG:boto:Params: {} 2024-04-12T03:53:39.982 DEBUG:boto:Token: None 2024-04-12T03:53:39.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA= 2024-04-12T03:53:39.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA=', 'Content-Length': '0'} 2024-04-12T03:53:39.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012730611b15b0a21-006618b043-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.984 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.985 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.985 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.985 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.985 DEBUG:boto:Method: GET 2024-04-12T03:53:39.985 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.985 DEBUG:boto:Data: 2024-04-12T03:53:39.985 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.985 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.986 DEBUG:boto:Params: {} 2024-04-12T03:53:39.986 DEBUG:boto:Token: None 2024-04-12T03:53:39.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:53:39.986 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA= 2024-04-12T03:53:39.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GheGkNdwlvflicfDaZAAiUqaWlA=', 'Content-Length': '0'} 2024-04-12T03:53:39.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5e48f5d0986187-006618b043-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.988 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.988 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.988 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.988 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.989 DEBUG:boto:Method: GET 2024-04-12T03:53:39.989 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.989 DEBUG:boto:Data: 2024-04-12T03:53:39.989 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.989 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.989 DEBUG:boto:Params: {} 2024-04-12T03:53:39.989 DEBUG:boto:Token: None 2024-04-12T03:53:39.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41493cdb5983b84-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.992 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.992 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:39.992 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:39.992 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.993 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.993 DEBUG:boto:Method: GET 2024-04-12T03:53:39.993 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:53:39.993 DEBUG:boto:Data: 2024-04-12T03:53:39.993 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:39.993 DEBUG:boto:Port: 8000 2024-04-12T03:53:39.993 DEBUG:boto:Params: {} 2024-04-12T03:53:39.993 DEBUG:boto:Token: None 2024-04-12T03:53:39.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/ 2024-04-12T03:53:39.994 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg= 2024-04-12T03:53:39.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ccRKKpujTeplgFaPqI5Povp1/Dg=', 'Content-Length': '0'} 2024-04-12T03:53:39.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724ab20042af731f-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:39.998 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:39.998 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:39.998 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:39.998 DEBUG:boto:Method: HEAD 2024-04-12T03:53:39.998 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:53:39.999 DEBUG:boto:Data: 2024-04-12T03:53:39.999 DEBUG:boto:Headers: {} 2024-04-12T03:53:39.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:39.999 DEBUG:boto:Port: 8001 2024-04-12T03:53:39.999 DEBUG:boto:Params: {} 2024-04-12T03:53:39.999 DEBUG:boto:Token: None 2024-04-12T03:53:39.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:39 GMT /dgbpxg-61/obj33 2024-04-12T03:53:39.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3rCRg9QNke2rT8xTWw1F+G2dIhk= 2024-04-12T03:53:40.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3rCRg9QNke2rT8xTWw1F+G2dIhk=', 'Content-Length': '0'} 2024-04-12T03:53:40.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d997017a2b6b8c97-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.001 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:40.001 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:40.001 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.002 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:53:40.002 DEBUG:boto:Data: 2024-04-12T03:53:40.002 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.002 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.002 DEBUG:boto:Params: {} 2024-04-12T03:53:40.002 DEBUG:boto:Token: None 2024-04-12T03:53:40.002 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj33 2024-04-12T03:53:40.002 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TgDmfuZTHpCWMhhTxM8sTswsUY4= 2024-04-12T03:53:40.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TgDmfuZTHpCWMhhTxM8sTswsUY4=', 'Content-Length': '0'} 2024-04-12T03:53:40.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a98aae6ef964ac3-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:53:40.004 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:40.005 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:40.005 DEBUG:boto:Method: GET 2024-04-12T03:53:40.005 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:53:40.005 DEBUG:boto:Data: 2024-04-12T03:53:40.005 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.005 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.005 DEBUG:boto:Params: {} 2024-04-12T03:53:40.005 DEBUG:boto:Token: None 2024-04-12T03:53:40.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj33 2024-04-12T03:53:40.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hj6GYBce4Y2ZeuzASfGpPN1IMDg= 2024-04-12T03:53:40.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hj6GYBce4Y2ZeuzASfGpPN1IMDg=', 'Content-Length': '0'} 2024-04-12T03:53:40.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53a822cb3891d08-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.008 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:53:40.008 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:53:40.008 DEBUG:boto:Method: GET 2024-04-12T03:53:40.008 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:53:40.008 DEBUG:boto:Data: 2024-04-12T03:53:40.008 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.009 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.009 DEBUG:boto:Params: {} 2024-04-12T03:53:40.009 DEBUG:boto:Token: None 2024-04-12T03:53:40.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj33 2024-04-12T03:53:40.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hj6GYBce4Y2ZeuzASfGpPN1IMDg= 2024-04-12T03:53:40.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hj6GYBce4Y2ZeuzASfGpPN1IMDg=', 'Content-Length': '0'} 2024-04-12T03:53:40.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab3bd8651f41d9c-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.011 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.011 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.011 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.011 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.011 DEBUG:boto:Method: GET 2024-04-12T03:53:40.012 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.012 DEBUG:boto:Data: 2024-04-12T03:53:40.012 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.012 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.012 DEBUG:boto:Params: {} 2024-04-12T03:53:40.012 DEBUG:boto:Token: None 2024-04-12T03:53:40.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1f855af65acfe9-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.015 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.015 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.015 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.015 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.015 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.016 DEBUG:boto:Method: GET 2024-04-12T03:53:40.016 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:53:40.016 DEBUG:boto:Data: 2024-04-12T03:53:40.016 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.016 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.016 DEBUG:boto:Params: {} 2024-04-12T03:53:40.016 DEBUG:boto:Token: None 2024-04-12T03:53:40.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a984a9906320b057-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.020 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:53:40.021 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.021 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.021 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.021 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.021 DEBUG:boto:Method: GET 2024-04-12T03:53:40.021 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.021 DEBUG:boto:Data: 2024-04-12T03:53:40.021 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.022 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.022 DEBUG:boto:Params: {} 2024-04-12T03:53:40.022 DEBUG:boto:Token: None 2024-04-12T03:53:40.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc= 2024-04-12T03:53:40.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc=', 'Content-Length': '0'} 2024-04-12T03:53:40.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa06c7f11e48da41-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.066 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.067 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.067 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.067 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.067 DEBUG:boto:Method: GET 2024-04-12T03:53:40.067 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.067 DEBUG:boto:Data: 2024-04-12T03:53:40.067 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.067 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.067 DEBUG:boto:Params: {} 2024-04-12T03:53:40.068 DEBUG:boto:Token: None 2024-04-12T03:53:40.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc= 2024-04-12T03:53:40.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc=', 'Content-Length': '0'} 2024-04-12T03:53:40.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005752c9a49b1ed737-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.070 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.070 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.070 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.071 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.071 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.071 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.071 DEBUG:boto:Data: 2024-04-12T03:53:40.073 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.074 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.074 DEBUG:boto:Params: {} 2024-04-12T03:53:40.074 DEBUG:boto:Token: None 2024-04-12T03:53:40.074 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8jPQgOBHQ7Jt6PVXiNVmBArlutI= 2024-04-12T03:53:40.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8jPQgOBHQ7Jt6PVXiNVmBArlutI=', 'Content-Length': '0'} 2024-04-12T03:53:40.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004750c8ef742df37a-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.076 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.076 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.076 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.077 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.077 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.077 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.077 DEBUG:boto:Data: 2024-04-12T03:53:40.077 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.077 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.077 DEBUG:boto:Params: {} 2024-04-12T03:53:40.077 DEBUG:boto:Token: None 2024-04-12T03:53:40.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8jPQgOBHQ7Jt6PVXiNVmBArlutI= 2024-04-12T03:53:40.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8jPQgOBHQ7Jt6PVXiNVmBArlutI=', 'Content-Length': '0'} 2024-04-12T03:53:40.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983b2982c75a708f-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:53:40.080 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.080 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.080 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.080 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.080 DEBUG:boto:Method: GET 2024-04-12T03:53:40.080 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.080 DEBUG:boto:Data: 2024-04-12T03:53:40.080 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.080 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.081 DEBUG:boto:Params: {} 2024-04-12T03:53:40.081 DEBUG:boto:Token: None 2024-04-12T03:53:40.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc= 2024-04-12T03:53:40.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc=', 'Content-Length': '0'} 2024-04-12T03:53:40.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c114899f5297e6-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.083 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.083 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.083 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.083 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.083 DEBUG:boto:Method: GET 2024-04-12T03:53:40.083 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.084 DEBUG:boto:Data: 2024-04-12T03:53:40.084 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.084 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.084 DEBUG:boto:Params: {} 2024-04-12T03:53:40.084 DEBUG:boto:Token: None 2024-04-12T03:53:40.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:53:40.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc= 2024-04-12T03:53:40.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NgiawyNAzWjhWTHJkwsyzI6cvVc=', 'Content-Length': '0'} 2024-04-12T03:53:40.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986cb17eae41b39a-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.086 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.087 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.087 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.087 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.087 DEBUG:boto:Method: GET 2024-04-12T03:53:40.087 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.087 DEBUG:boto:Data: 2024-04-12T03:53:40.087 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.087 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.088 DEBUG:boto:Params: {} 2024-04-12T03:53:40.088 DEBUG:boto:Token: None 2024-04-12T03:53:40.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4b050e3d48adf8-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.091 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.091 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.091 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.091 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.091 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.092 DEBUG:boto:Method: GET 2024-04-12T03:53:40.092 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.092 DEBUG:boto:Data: 2024-04-12T03:53:40.092 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.092 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.092 DEBUG:boto:Params: {} 2024-04-12T03:53:40.092 DEBUG:boto:Token: None 2024-04-12T03:53:40.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.093 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf6f988811531493-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.138 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.139 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.139 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.139 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.139 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:53:40.139 DEBUG:boto:Data: 2024-04-12T03:53:40.139 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.139 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.139 DEBUG:boto:Params: {} 2024-04-12T03:53:40.140 DEBUG:boto:Token: None 2024-04-12T03:53:40.140 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34 2024-04-12T03:53:40.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8fIHiFis56uvtUv9A2Az802010= 2024-04-12T03:53:40.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fIHiFis56uvtUv9A2Az802010=', 'Content-Length': '0'} 2024-04-12T03:53:40.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bba10b4fa573a79-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.142 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.142 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.142 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.142 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:53:40.143 DEBUG:boto:Data: 2024-04-12T03:53:40.143 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.143 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.143 DEBUG:boto:Params: {} 2024-04-12T03:53:40.143 DEBUG:boto:Token: None 2024-04-12T03:53:40.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34 2024-04-12T03:53:40.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K8fIHiFis56uvtUv9A2Az802010= 2024-04-12T03:53:40.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fIHiFis56uvtUv9A2Az802010=', 'Content-Length': '0'} 2024-04-12T03:53:40.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1efcf7ba680efe7-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:53:40.145 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.145 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.146 DEBUG:boto:Method: GET 2024-04-12T03:53:40.146 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:53:40.146 DEBUG:boto:Data: 2024-04-12T03:53:40.146 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.146 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.146 DEBUG:boto:Params: {} 2024-04-12T03:53:40.146 DEBUG:boto:Token: None 2024-04-12T03:53:40.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34 2024-04-12T03:53:40.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9J/Ev1cLg3Y1kMucvGndbDLw6jo= 2024-04-12T03:53:40.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9J/Ev1cLg3Y1kMucvGndbDLw6jo=', 'Content-Length': '0'} 2024-04-12T03:53:40.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d131c164ee5bfd43-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.149 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:53:40.149 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:53:40.149 DEBUG:boto:Method: GET 2024-04-12T03:53:40.149 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:53:40.149 DEBUG:boto:Data: 2024-04-12T03:53:40.149 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.149 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.149 DEBUG:boto:Params: {} 2024-04-12T03:53:40.150 DEBUG:boto:Token: None 2024-04-12T03:53:40.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj34 2024-04-12T03:53:40.150 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9J/Ev1cLg3Y1kMucvGndbDLw6jo= 2024-04-12T03:53:40.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9J/Ev1cLg3Y1kMucvGndbDLw6jo=', 'Content-Length': '0'} 2024-04-12T03:53:40.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f39872136fd4f3-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.152 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.152 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.152 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.152 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.152 DEBUG:boto:Method: GET 2024-04-12T03:53:40.152 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.152 DEBUG:boto:Data: 2024-04-12T03:53:40.153 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.153 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.153 DEBUG:boto:Params: {} 2024-04-12T03:53:40.153 DEBUG:boto:Token: None 2024-04-12T03:53:40.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099bae3f212ad232e-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.156 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.156 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.156 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.156 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.156 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.156 DEBUG:boto:Method: GET 2024-04-12T03:53:40.156 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:53:40.157 DEBUG:boto:Data: 2024-04-12T03:53:40.157 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.157 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.157 DEBUG:boto:Params: {} 2024-04-12T03:53:40.157 DEBUG:boto:Token: None 2024-04-12T03:53:40.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d34ceae773f50e-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.161 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:53:40.162 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.162 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.162 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.162 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.162 DEBUG:boto:Method: GET 2024-04-12T03:53:40.162 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.162 DEBUG:boto:Data: 2024-04-12T03:53:40.162 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.163 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.163 DEBUG:boto:Params: {} 2024-04-12T03:53:40.163 DEBUG:boto:Token: None 2024-04-12T03:53:40.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4= 2024-04-12T03:53:40.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4=', 'Content-Length': '0'} 2024-04-12T03:53:40.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f0e34e39f5ab15-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.206 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.207 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.207 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.207 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.207 DEBUG:boto:Method: GET 2024-04-12T03:53:40.207 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.207 DEBUG:boto:Data: 2024-04-12T03:53:40.207 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.208 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.208 DEBUG:boto:Params: {} 2024-04-12T03:53:40.208 DEBUG:boto:Token: None 2024-04-12T03:53:40.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4= 2024-04-12T03:53:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4=', 'Content-Length': '0'} 2024-04-12T03:53:40.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008bc39ac57ebfa53-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.210 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.210 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.210 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.211 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.211 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.211 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.211 DEBUG:boto:Data: 2024-04-12T03:53:40.211 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.211 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.211 DEBUG:boto:Params: {} 2024-04-12T03:53:40.211 DEBUG:boto:Token: None 2024-04-12T03:53:40.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YdLMimx5Vbcqg4MEdqdExirRdac= 2024-04-12T03:53:40.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YdLMimx5Vbcqg4MEdqdExirRdac=', 'Content-Length': '0'} 2024-04-12T03:53:40.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec52eba0bebc248-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.213 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.214 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.214 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.214 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.214 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.214 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.214 DEBUG:boto:Data: 2024-04-12T03:53:40.214 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.214 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.214 DEBUG:boto:Params: {} 2024-04-12T03:53:40.215 DEBUG:boto:Token: None 2024-04-12T03:53:40.215 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YdLMimx5Vbcqg4MEdqdExirRdac= 2024-04-12T03:53:40.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YdLMimx5Vbcqg4MEdqdExirRdac=', 'Content-Length': '0'} 2024-04-12T03:53:40.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005754fc5bbbb566c5-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:53:40.217 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.217 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.217 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.217 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.217 DEBUG:boto:Method: GET 2024-04-12T03:53:40.217 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.218 DEBUG:boto:Data: 2024-04-12T03:53:40.218 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.218 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.218 DEBUG:boto:Params: {} 2024-04-12T03:53:40.218 DEBUG:boto:Token: None 2024-04-12T03:53:40.218 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.218 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4= 2024-04-12T03:53:40.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4=', 'Content-Length': '0'} 2024-04-12T03:53:40.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be73399e03cd5236-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.220 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.221 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.221 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.221 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.221 DEBUG:boto:Method: GET 2024-04-12T03:53:40.221 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.221 DEBUG:boto:Data: 2024-04-12T03:53:40.221 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.221 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.221 DEBUG:boto:Params: {} 2024-04-12T03:53:40.222 DEBUG:boto:Token: None 2024-04-12T03:53:40.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:53:40.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4= 2024-04-12T03:53:40.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KKOFbd4Xg/hC05z6Butl477r2z4=', 'Content-Length': '0'} 2024-04-12T03:53:40.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4b304a6773dcc5-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.224 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.224 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.224 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.224 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.224 DEBUG:boto:Method: GET 2024-04-12T03:53:40.225 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.225 DEBUG:boto:Data: 2024-04-12T03:53:40.225 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.225 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.225 DEBUG:boto:Params: {} 2024-04-12T03:53:40.225 DEBUG:boto:Token: None 2024-04-12T03:53:40.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b0965aaffc186d-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.228 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.228 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.228 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.228 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.228 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.229 DEBUG:boto:Method: GET 2024-04-12T03:53:40.229 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.229 DEBUG:boto:Data: 2024-04-12T03:53:40.229 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.229 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.229 DEBUG:boto:Params: {} 2024-04-12T03:53:40.229 DEBUG:boto:Token: None 2024-04-12T03:53:40.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da7fc0d3089d269a-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.233 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.234 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.234 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.234 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.234 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:53:40.234 DEBUG:boto:Data: 2024-04-12T03:53:40.234 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.235 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.235 DEBUG:boto:Params: {} 2024-04-12T03:53:40.235 DEBUG:boto:Token: None 2024-04-12T03:53:40.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37 2024-04-12T03:53:40.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UpUOvF+ohVH6u3zVJ7DWheQCVZ4= 2024-04-12T03:53:40.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UpUOvF+ohVH6u3zVJ7DWheQCVZ4=', 'Content-Length': '0'} 2024-04-12T03:53:40.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8623fec932050c6-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.237 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.237 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.237 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.237 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:53:40.238 DEBUG:boto:Data: 2024-04-12T03:53:40.238 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.238 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.238 DEBUG:boto:Params: {} 2024-04-12T03:53:40.238 DEBUG:boto:Token: None 2024-04-12T03:53:40.238 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37 2024-04-12T03:53:40.238 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UpUOvF+ohVH6u3zVJ7DWheQCVZ4= 2024-04-12T03:53:40.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UpUOvF+ohVH6u3zVJ7DWheQCVZ4=', 'Content-Length': '0'} 2024-04-12T03:53:40.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051870445a83826cf-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:53:40.241 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.241 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.241 DEBUG:boto:Method: GET 2024-04-12T03:53:40.241 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:53:40.241 DEBUG:boto:Data: 2024-04-12T03:53:40.241 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.241 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.241 DEBUG:boto:Params: {} 2024-04-12T03:53:40.241 DEBUG:boto:Token: None 2024-04-12T03:53:40.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37 2024-04-12T03:53:40.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e6zs3UTXxaDIAWEOQJVqCCnjELI= 2024-04-12T03:53:40.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e6zs3UTXxaDIAWEOQJVqCCnjELI=', 'Content-Length': '0'} 2024-04-12T03:53:40.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971ed1bdf43f4179-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.244 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:53:40.244 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:53:40.244 DEBUG:boto:Method: GET 2024-04-12T03:53:40.244 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:53:40.244 DEBUG:boto:Data: 2024-04-12T03:53:40.244 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.244 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.244 DEBUG:boto:Params: {} 2024-04-12T03:53:40.245 DEBUG:boto:Token: None 2024-04-12T03:53:40.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj37 2024-04-12T03:53:40.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e6zs3UTXxaDIAWEOQJVqCCnjELI= 2024-04-12T03:53:40.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e6zs3UTXxaDIAWEOQJVqCCnjELI=', 'Content-Length': '0'} 2024-04-12T03:53:40.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001716c51d53736c5c-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.247 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.247 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.247 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.247 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.247 DEBUG:boto:Method: GET 2024-04-12T03:53:40.247 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.248 DEBUG:boto:Data: 2024-04-12T03:53:40.248 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.248 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.248 DEBUG:boto:Params: {} 2024-04-12T03:53:40.248 DEBUG:boto:Token: None 2024-04-12T03:53:40.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000031b3dc8567d78a5-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.251 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.251 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.251 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.252 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.252 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.252 DEBUG:boto:Method: GET 2024-04-12T03:53:40.252 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:53:40.252 DEBUG:boto:Data: 2024-04-12T03:53:40.252 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.252 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.252 DEBUG:boto:Params: {} 2024-04-12T03:53:40.252 DEBUG:boto:Token: None 2024-04-12T03:53:40.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c87de4396030c343-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.256 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:53:40.257 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.257 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.257 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.257 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.257 DEBUG:boto:Method: GET 2024-04-12T03:53:40.257 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.257 DEBUG:boto:Data: 2024-04-12T03:53:40.258 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.258 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.258 DEBUG:boto:Params: {} 2024-04-12T03:53:40.258 DEBUG:boto:Token: None 2024-04-12T03:53:40.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.258 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w= 2024-04-12T03:53:40.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w=', 'Content-Length': '0'} 2024-04-12T03:53:40.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cffb6f79c8de636-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.302 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.302 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.303 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.303 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.303 DEBUG:boto:Method: GET 2024-04-12T03:53:40.303 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.303 DEBUG:boto:Data: 2024-04-12T03:53:40.303 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.303 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.303 DEBUG:boto:Params: {} 2024-04-12T03:53:40.303 DEBUG:boto:Token: None 2024-04-12T03:53:40.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w= 2024-04-12T03:53:40.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w=', 'Content-Length': '0'} 2024-04-12T03:53:40.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc83e36b7b828e21-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.306 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.306 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.306 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.306 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.307 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.307 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.307 DEBUG:boto:Data: 2024-04-12T03:53:40.307 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.307 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.307 DEBUG:boto:Params: {} 2024-04-12T03:53:40.307 DEBUG:boto:Token: None 2024-04-12T03:53:40.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+dGFwdqiW/l8Jtxewi7WC4tkmA= 2024-04-12T03:53:40.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+dGFwdqiW/l8Jtxewi7WC4tkmA=', 'Content-Length': '0'} 2024-04-12T03:53:40.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b9dcb6fbaf3bfb-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.310 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.310 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.310 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.310 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.310 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.311 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.311 DEBUG:boto:Data: 2024-04-12T03:53:40.311 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.311 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.311 DEBUG:boto:Params: {} 2024-04-12T03:53:40.311 DEBUG:boto:Token: None 2024-04-12T03:53:40.311 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.311 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5+dGFwdqiW/l8Jtxewi7WC4tkmA= 2024-04-12T03:53:40.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5+dGFwdqiW/l8Jtxewi7WC4tkmA=', 'Content-Length': '0'} 2024-04-12T03:53:40.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d45d2a110fb2be-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:53:40.313 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.314 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.314 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.314 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.314 DEBUG:boto:Method: GET 2024-04-12T03:53:40.314 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.314 DEBUG:boto:Data: 2024-04-12T03:53:40.314 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.314 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.315 DEBUG:boto:Params: {} 2024-04-12T03:53:40.315 DEBUG:boto:Token: None 2024-04-12T03:53:40.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w= 2024-04-12T03:53:40.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w=', 'Content-Length': '0'} 2024-04-12T03:53:40.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058819082e268303f-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.317 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.317 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.317 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.317 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.317 DEBUG:boto:Method: GET 2024-04-12T03:53:40.317 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.318 DEBUG:boto:Data: 2024-04-12T03:53:40.318 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.318 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.318 DEBUG:boto:Params: {} 2024-04-12T03:53:40.318 DEBUG:boto:Token: None 2024-04-12T03:53:40.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:53:40.318 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w= 2024-04-12T03:53:40.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:remawlfKXRRbpfA6wxyhAufT68w=', 'Content-Length': '0'} 2024-04-12T03:53:40.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ad07bd8efaefa1-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.320 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.320 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.320 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.320 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.321 DEBUG:boto:Method: GET 2024-04-12T03:53:40.321 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.321 DEBUG:boto:Data: 2024-04-12T03:53:40.321 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.321 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.321 DEBUG:boto:Params: {} 2024-04-12T03:53:40.321 DEBUG:boto:Token: None 2024-04-12T03:53:40.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000936aa898311ce6b0-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.324 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.324 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.324 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.324 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.325 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.325 DEBUG:boto:Method: GET 2024-04-12T03:53:40.325 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.325 DEBUG:boto:Data: 2024-04-12T03:53:40.325 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.325 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.325 DEBUG:boto:Params: {} 2024-04-12T03:53:40.325 DEBUG:boto:Token: None 2024-04-12T03:53:40.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842fdfdeb567e4f2-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.329 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.330 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.330 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.330 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.330 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:53:40.330 DEBUG:boto:Data: 2024-04-12T03:53:40.330 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.330 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.331 DEBUG:boto:Params: {} 2024-04-12T03:53:40.331 DEBUG:boto:Token: None 2024-04-12T03:53:40.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38 2024-04-12T03:53:40.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZPIm320Oi+ZPFPGF43Mb4+CPwb4= 2024-04-12T03:53:40.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZPIm320Oi+ZPFPGF43Mb4+CPwb4=', 'Content-Length': '0'} 2024-04-12T03:53:40.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6eb1b05eecd4eac-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.333 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.333 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.333 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.333 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:53:40.333 DEBUG:boto:Data: 2024-04-12T03:53:40.333 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.334 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.334 DEBUG:boto:Params: {} 2024-04-12T03:53:40.334 DEBUG:boto:Token: None 2024-04-12T03:53:40.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38 2024-04-12T03:53:40.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZPIm320Oi+ZPFPGF43Mb4+CPwb4= 2024-04-12T03:53:40.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZPIm320Oi+ZPFPGF43Mb4+CPwb4=', 'Content-Length': '0'} 2024-04-12T03:53:40.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a17b6fd8d7ff05f-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:53:40.336 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.336 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.336 DEBUG:boto:Method: GET 2024-04-12T03:53:40.336 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:53:40.337 DEBUG:boto:Data: 2024-04-12T03:53:40.337 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.337 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.337 DEBUG:boto:Params: {} 2024-04-12T03:53:40.337 DEBUG:boto:Token: None 2024-04-12T03:53:40.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38 2024-04-12T03:53:40.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0ti9rDQQ7qBepXX6pXvxEbQs0OE= 2024-04-12T03:53:40.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0ti9rDQQ7qBepXX6pXvxEbQs0OE=', 'Content-Length': '0'} 2024-04-12T03:53:40.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9cee6bdf618f89-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.339 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:53:40.339 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:53:40.339 DEBUG:boto:Method: GET 2024-04-12T03:53:40.340 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:53:40.340 DEBUG:boto:Data: 2024-04-12T03:53:40.340 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.340 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.340 DEBUG:boto:Params: {} 2024-04-12T03:53:40.340 DEBUG:boto:Token: None 2024-04-12T03:53:40.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj38 2024-04-12T03:53:40.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0ti9rDQQ7qBepXX6pXvxEbQs0OE= 2024-04-12T03:53:40.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0ti9rDQQ7qBepXX6pXvxEbQs0OE=', 'Content-Length': '0'} 2024-04-12T03:53:40.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015c7b0ecfb42c37f-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.342 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.343 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.343 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.343 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.343 DEBUG:boto:Method: GET 2024-04-12T03:53:40.343 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.343 DEBUG:boto:Data: 2024-04-12T03:53:40.343 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.343 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.344 DEBUG:boto:Params: {} 2024-04-12T03:53:40.344 DEBUG:boto:Token: None 2024-04-12T03:53:40.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ccb1aace19558e-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.386 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.387 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.387 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.387 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.387 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.387 DEBUG:boto:Method: GET 2024-04-12T03:53:40.387 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:53:40.387 DEBUG:boto:Data: 2024-04-12T03:53:40.387 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.388 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.388 DEBUG:boto:Params: {} 2024-04-12T03:53:40.388 DEBUG:boto:Token: None 2024-04-12T03:53:40.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a4460d3f562fa8-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.392 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:53:40.392 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.392 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.392 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.393 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.393 DEBUG:boto:Method: GET 2024-04-12T03:53:40.393 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.393 DEBUG:boto:Data: 2024-04-12T03:53:40.393 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.393 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.393 DEBUG:boto:Params: {} 2024-04-12T03:53:40.393 DEBUG:boto:Token: None 2024-04-12T03:53:40.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.393 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg= 2024-04-12T03:53:40.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg=', 'Content-Length': '0'} 2024-04-12T03:53:40.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004059d64e12a32526-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.395 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.395 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.396 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.396 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.396 DEBUG:boto:Method: GET 2024-04-12T03:53:40.396 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.396 DEBUG:boto:Data: 2024-04-12T03:53:40.396 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.396 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.396 DEBUG:boto:Params: {} 2024-04-12T03:53:40.396 DEBUG:boto:Token: None 2024-04-12T03:53:40.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg= 2024-04-12T03:53:40.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg=', 'Content-Length': '0'} 2024-04-12T03:53:40.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c62450c9dbe0bef-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.442 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.442 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.443 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.443 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.443 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.443 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.443 DEBUG:boto:Data: 2024-04-12T03:53:40.443 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.443 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.443 DEBUG:boto:Params: {} 2024-04-12T03:53:40.443 DEBUG:boto:Token: None 2024-04-12T03:53:40.443 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BCzPjvSznfIITO0BFZXu6AIxB5c= 2024-04-12T03:53:40.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BCzPjvSznfIITO0BFZXu6AIxB5c=', 'Content-Length': '0'} 2024-04-12T03:53:40.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75991a2b56dffe9-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.446 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.446 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.446 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.446 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.446 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.446 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.446 DEBUG:boto:Data: 2024-04-12T03:53:40.446 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.449 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.449 DEBUG:boto:Params: {} 2024-04-12T03:53:40.449 DEBUG:boto:Token: None 2024-04-12T03:53:40.450 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.450 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BCzPjvSznfIITO0BFZXu6AIxB5c= 2024-04-12T03:53:40.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BCzPjvSznfIITO0BFZXu6AIxB5c=', 'Content-Length': '0'} 2024-04-12T03:53:40.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dbba56b1de433b9-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:53:40.452 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.452 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.452 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.452 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.452 DEBUG:boto:Method: GET 2024-04-12T03:53:40.452 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.452 DEBUG:boto:Data: 2024-04-12T03:53:40.452 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.453 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.453 DEBUG:boto:Params: {} 2024-04-12T03:53:40.453 DEBUG:boto:Token: None 2024-04-12T03:53:40.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.453 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg= 2024-04-12T03:53:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg=', 'Content-Length': '0'} 2024-04-12T03:53:40.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d228ab018f5cea6-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.455 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.455 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.455 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.455 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.456 DEBUG:boto:Method: GET 2024-04-12T03:53:40.456 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.456 DEBUG:boto:Data: 2024-04-12T03:53:40.456 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.456 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.456 DEBUG:boto:Params: {} 2024-04-12T03:53:40.456 DEBUG:boto:Token: None 2024-04-12T03:53:40.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:53:40.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg= 2024-04-12T03:53:40.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zu/sXdZtcHLo7OJYglthCreKksg=', 'Content-Length': '0'} 2024-04-12T03:53:40.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be3f27ff4fd6052-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.458 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.458 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.459 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.459 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.459 DEBUG:boto:Method: GET 2024-04-12T03:53:40.459 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.459 DEBUG:boto:Data: 2024-04-12T03:53:40.459 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.459 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.459 DEBUG:boto:Params: {} 2024-04-12T03:53:40.459 DEBUG:boto:Token: None 2024-04-12T03:53:40.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cddd67e34e2cd5ea-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.462 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.462 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.463 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.463 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.463 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.463 DEBUG:boto:Method: GET 2024-04-12T03:53:40.463 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.463 DEBUG:boto:Data: 2024-04-12T03:53:40.463 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.463 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.463 DEBUG:boto:Params: {} 2024-04-12T03:53:40.463 DEBUG:boto:Token: None 2024-04-12T03:53:40.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a23270dcb2632fa-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.467 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.468 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.468 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.468 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.468 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:53:40.468 DEBUG:boto:Data: 2024-04-12T03:53:40.468 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.468 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.468 DEBUG:boto:Params: {} 2024-04-12T03:53:40.468 DEBUG:boto:Token: None 2024-04-12T03:53:40.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44 2024-04-12T03:53:40.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:evQhfk6dgdott8E0sOv/p9COZBs= 2024-04-12T03:53:40.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:evQhfk6dgdott8E0sOv/p9COZBs=', 'Content-Length': '0'} 2024-04-12T03:53:40.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac48186ab18a4da-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.471 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.471 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.471 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.471 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:53:40.471 DEBUG:boto:Data: 2024-04-12T03:53:40.471 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.471 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.471 DEBUG:boto:Params: {} 2024-04-12T03:53:40.472 DEBUG:boto:Token: None 2024-04-12T03:53:40.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44 2024-04-12T03:53:40.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:evQhfk6dgdott8E0sOv/p9COZBs= 2024-04-12T03:53:40.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:evQhfk6dgdott8E0sOv/p9COZBs=', 'Content-Length': '0'} 2024-04-12T03:53:40.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999355994966b3f1-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:53:40.473 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.474 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.474 DEBUG:boto:Method: GET 2024-04-12T03:53:40.474 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:53:40.474 DEBUG:boto:Data: 2024-04-12T03:53:40.474 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.474 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.474 DEBUG:boto:Params: {} 2024-04-12T03:53:40.474 DEBUG:boto:Token: None 2024-04-12T03:53:40.474 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44 2024-04-12T03:53:40.474 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jpSguBiKtql5NuyUGBFZ5ByygB4= 2024-04-12T03:53:40.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jpSguBiKtql5NuyUGBFZ5ByygB4=', 'Content-Length': '0'} 2024-04-12T03:53:40.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278f606d4dca5261-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.518 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:53:40.518 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:53:40.519 DEBUG:boto:Method: GET 2024-04-12T03:53:40.519 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:53:40.519 DEBUG:boto:Data: 2024-04-12T03:53:40.519 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.519 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.519 DEBUG:boto:Params: {} 2024-04-12T03:53:40.519 DEBUG:boto:Token: None 2024-04-12T03:53:40.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj44 2024-04-12T03:53:40.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jpSguBiKtql5NuyUGBFZ5ByygB4= 2024-04-12T03:53:40.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jpSguBiKtql5NuyUGBFZ5ByygB4=', 'Content-Length': '0'} 2024-04-12T03:53:40.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430592c039b5f6d7-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.521 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.522 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.522 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.522 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.522 DEBUG:boto:Method: GET 2024-04-12T03:53:40.522 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.522 DEBUG:boto:Data: 2024-04-12T03:53:40.522 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.522 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.522 DEBUG:boto:Params: {} 2024-04-12T03:53:40.522 DEBUG:boto:Token: None 2024-04-12T03:53:40.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.523 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4053a5816609ade-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.525 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.526 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.526 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.526 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.526 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.526 DEBUG:boto:Method: GET 2024-04-12T03:53:40.526 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:53:40.526 DEBUG:boto:Data: 2024-04-12T03:53:40.526 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.526 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.526 DEBUG:boto:Params: {} 2024-04-12T03:53:40.527 DEBUG:boto:Token: None 2024-04-12T03:53:40.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090abfcc854140b65-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.530 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:53:40.531 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.531 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.531 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.531 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.531 DEBUG:boto:Method: GET 2024-04-12T03:53:40.531 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.531 DEBUG:boto:Data: 2024-04-12T03:53:40.531 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.531 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.531 DEBUG:boto:Params: {} 2024-04-12T03:53:40.532 DEBUG:boto:Token: None 2024-04-12T03:53:40.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg= 2024-04-12T03:53:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg=', 'Content-Length': '0'} 2024-04-12T03:53:40.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0111658236594a-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.534 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.534 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.534 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.534 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.534 DEBUG:boto:Method: GET 2024-04-12T03:53:40.534 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.534 DEBUG:boto:Data: 2024-04-12T03:53:40.534 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.534 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.534 DEBUG:boto:Params: {} 2024-04-12T03:53:40.535 DEBUG:boto:Token: None 2024-04-12T03:53:40.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg= 2024-04-12T03:53:40.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg=', 'Content-Length': '0'} 2024-04-12T03:53:40.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ac8271c8a127a2-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.578 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.578 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.578 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.578 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.579 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.579 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.579 DEBUG:boto:Data: 2024-04-12T03:53:40.579 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.579 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.579 DEBUG:boto:Params: {} 2024-04-12T03:53:40.579 DEBUG:boto:Token: None 2024-04-12T03:53:40.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U3S8TnWptI6mwLMTvebPdSGqbAo= 2024-04-12T03:53:40.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U3S8TnWptI6mwLMTvebPdSGqbAo=', 'Content-Length': '0'} 2024-04-12T03:53:40.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c819a2a4f7e74a-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.581 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.581 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.581 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.581 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.581 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.581 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.581 DEBUG:boto:Data: 2024-04-12T03:53:40.582 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.582 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.582 DEBUG:boto:Params: {} 2024-04-12T03:53:40.582 DEBUG:boto:Token: None 2024-04-12T03:53:40.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U3S8TnWptI6mwLMTvebPdSGqbAo= 2024-04-12T03:53:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U3S8TnWptI6mwLMTvebPdSGqbAo=', 'Content-Length': '0'} 2024-04-12T03:53:40.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aab2375e89f57c4-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:53:40.584 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.584 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.584 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.584 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.584 DEBUG:boto:Method: GET 2024-04-12T03:53:40.584 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.584 DEBUG:boto:Data: 2024-04-12T03:53:40.585 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.585 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.585 DEBUG:boto:Params: {} 2024-04-12T03:53:40.585 DEBUG:boto:Token: None 2024-04-12T03:53:40.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.585 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg= 2024-04-12T03:53:40.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg=', 'Content-Length': '0'} 2024-04-12T03:53:40.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfc0a5a5a29ba3a-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.587 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.587 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.587 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.587 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.587 DEBUG:boto:Method: GET 2024-04-12T03:53:40.587 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.588 DEBUG:boto:Data: 2024-04-12T03:53:40.588 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.588 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.588 DEBUG:boto:Params: {} 2024-04-12T03:53:40.588 DEBUG:boto:Token: None 2024-04-12T03:53:40.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:53:40.588 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg= 2024-04-12T03:53:40.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K3un5P0mT1tcDOWjKsQiNDxOqJg=', 'Content-Length': '0'} 2024-04-12T03:53:40.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009495aa1d09526755-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.590 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.591 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.591 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.591 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.591 DEBUG:boto:Method: GET 2024-04-12T03:53:40.591 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.591 DEBUG:boto:Data: 2024-04-12T03:53:40.591 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.591 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.591 DEBUG:boto:Params: {} 2024-04-12T03:53:40.591 DEBUG:boto:Token: None 2024-04-12T03:53:40.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5b42e5039cc62ef-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.594 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.594 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.594 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.594 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.594 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.594 DEBUG:boto:Method: GET 2024-04-12T03:53:40.594 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.595 DEBUG:boto:Data: 2024-04-12T03:53:40.595 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.595 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.595 DEBUG:boto:Params: {} 2024-04-12T03:53:40.595 DEBUG:boto:Token: None 2024-04-12T03:53:40.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007abd836af5098412-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.599 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.599 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.599 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.599 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.599 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:53:40.599 DEBUG:boto:Data: 2024-04-12T03:53:40.599 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.600 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.600 DEBUG:boto:Params: {} 2024-04-12T03:53:40.600 DEBUG:boto:Token: None 2024-04-12T03:53:40.600 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46 2024-04-12T03:53:40.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GrLBE1eu85Sx9WAN+ud4vmOsJPI= 2024-04-12T03:53:40.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrLBE1eu85Sx9WAN+ud4vmOsJPI=', 'Content-Length': '0'} 2024-04-12T03:53:40.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93b9d7e0cd5cb9c-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.602 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.602 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.602 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.602 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:53:40.602 DEBUG:boto:Data: 2024-04-12T03:53:40.602 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.602 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.602 DEBUG:boto:Params: {} 2024-04-12T03:53:40.603 DEBUG:boto:Token: None 2024-04-12T03:53:40.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46 2024-04-12T03:53:40.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GrLBE1eu85Sx9WAN+ud4vmOsJPI= 2024-04-12T03:53:40.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrLBE1eu85Sx9WAN+ud4vmOsJPI=', 'Content-Length': '0'} 2024-04-12T03:53:40.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b384b5714d384ca3-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:53:40.605 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.605 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.605 DEBUG:boto:Method: GET 2024-04-12T03:53:40.605 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:53:40.605 DEBUG:boto:Data: 2024-04-12T03:53:40.605 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.605 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.605 DEBUG:boto:Params: {} 2024-04-12T03:53:40.605 DEBUG:boto:Token: None 2024-04-12T03:53:40.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46 2024-04-12T03:53:40.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RWkrsJ46Bo1h/ICOBcbzAjmNXXQ= 2024-04-12T03:53:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RWkrsJ46Bo1h/ICOBcbzAjmNXXQ=', 'Content-Length': '0'} 2024-04-12T03:53:40.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054be00ddb95f37b1-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.607 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:53:40.607 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:53:40.607 DEBUG:boto:Method: GET 2024-04-12T03:53:40.608 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:53:40.608 DEBUG:boto:Data: 2024-04-12T03:53:40.608 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.608 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.608 DEBUG:boto:Params: {} 2024-04-12T03:53:40.608 DEBUG:boto:Token: None 2024-04-12T03:53:40.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj46 2024-04-12T03:53:40.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RWkrsJ46Bo1h/ICOBcbzAjmNXXQ= 2024-04-12T03:53:40.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RWkrsJ46Bo1h/ICOBcbzAjmNXXQ=', 'Content-Length': '0'} 2024-04-12T03:53:40.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005641f671b02ef207-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.610 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.610 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.610 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.610 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.610 DEBUG:boto:Method: GET 2024-04-12T03:53:40.610 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.611 DEBUG:boto:Data: 2024-04-12T03:53:40.611 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.611 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.611 DEBUG:boto:Params: {} 2024-04-12T03:53:40.611 DEBUG:boto:Token: None 2024-04-12T03:53:40.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc153dcedcc82ca9-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.654 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.655 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.655 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.655 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.655 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.655 DEBUG:boto:Method: GET 2024-04-12T03:53:40.655 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:53:40.655 DEBUG:boto:Data: 2024-04-12T03:53:40.656 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.656 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.656 DEBUG:boto:Params: {} 2024-04-12T03:53:40.656 DEBUG:boto:Token: None 2024-04-12T03:53:40.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a1b16548869de4-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.660 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:53:40.660 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.660 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.660 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.660 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.661 DEBUG:boto:Method: GET 2024-04-12T03:53:40.661 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.661 DEBUG:boto:Data: 2024-04-12T03:53:40.661 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.661 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.661 DEBUG:boto:Params: {} 2024-04-12T03:53:40.661 DEBUG:boto:Token: None 2024-04-12T03:53:40.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms= 2024-04-12T03:53:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms=', 'Content-Length': '0'} 2024-04-12T03:53:40.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef14b04d323624a0-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.663 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.663 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.663 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.663 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.663 DEBUG:boto:Method: GET 2024-04-12T03:53:40.664 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.664 DEBUG:boto:Data: 2024-04-12T03:53:40.664 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.664 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.664 DEBUG:boto:Params: {} 2024-04-12T03:53:40.664 DEBUG:boto:Token: None 2024-04-12T03:53:40.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms= 2024-04-12T03:53:40.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms=', 'Content-Length': '0'} 2024-04-12T03:53:40.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de288874f5a9709-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.706 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.707 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.707 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.707 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.707 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.707 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.707 DEBUG:boto:Data: 2024-04-12T03:53:40.707 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.707 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.707 DEBUG:boto:Params: {} 2024-04-12T03:53:40.707 DEBUG:boto:Token: None 2024-04-12T03:53:40.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dINu7HDoe7qbQXV8xMBYVoTXXXg= 2024-04-12T03:53:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dINu7HDoe7qbQXV8xMBYVoTXXXg=', 'Content-Length': '0'} 2024-04-12T03:53:40.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a49dfdec6facad-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.710 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.710 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.710 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.710 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.710 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.710 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.710 DEBUG:boto:Data: 2024-04-12T03:53:40.710 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.710 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.710 DEBUG:boto:Params: {} 2024-04-12T03:53:40.711 DEBUG:boto:Token: None 2024-04-12T03:53:40.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dINu7HDoe7qbQXV8xMBYVoTXXXg= 2024-04-12T03:53:40.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dINu7HDoe7qbQXV8xMBYVoTXXXg=', 'Content-Length': '0'} 2024-04-12T03:53:40.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a814af492bc8e13-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:53:40.713 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.713 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.713 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.713 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.713 DEBUG:boto:Method: GET 2024-04-12T03:53:40.713 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.713 DEBUG:boto:Data: 2024-04-12T03:53:40.714 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.714 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.714 DEBUG:boto:Params: {} 2024-04-12T03:53:40.714 DEBUG:boto:Token: None 2024-04-12T03:53:40.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms= 2024-04-12T03:53:40.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms=', 'Content-Length': '0'} 2024-04-12T03:53:40.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5af0c1017c8742c-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.716 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.716 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.716 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.716 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.717 DEBUG:boto:Method: GET 2024-04-12T03:53:40.717 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.717 DEBUG:boto:Data: 2024-04-12T03:53:40.717 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.717 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.717 DEBUG:boto:Params: {} 2024-04-12T03:53:40.717 DEBUG:boto:Token: None 2024-04-12T03:53:40.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:53:40.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms= 2024-04-12T03:53:40.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cOVfjnGR0UNtC0AI4+B4R/H+Nms=', 'Content-Length': '0'} 2024-04-12T03:53:40.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddaa7a364fd4ab1d-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.720 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.720 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.720 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.720 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.720 DEBUG:boto:Method: GET 2024-04-12T03:53:40.720 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.720 DEBUG:boto:Data: 2024-04-12T03:53:40.720 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.721 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.721 DEBUG:boto:Params: {} 2024-04-12T03:53:40.721 DEBUG:boto:Token: None 2024-04-12T03:53:40.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009238a2493f6ce746-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.723 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.724 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.724 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.724 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.724 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.724 DEBUG:boto:Method: GET 2024-04-12T03:53:40.724 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.724 DEBUG:boto:Data: 2024-04-12T03:53:40.724 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.724 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.725 DEBUG:boto:Params: {} 2024-04-12T03:53:40.725 DEBUG:boto:Token: None 2024-04-12T03:53:40.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46214dc550aaa0c-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.728 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.728 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.729 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.729 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.729 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:53:40.729 DEBUG:boto:Data: 2024-04-12T03:53:40.729 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.729 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.729 DEBUG:boto:Params: {} 2024-04-12T03:53:40.729 DEBUG:boto:Token: None 2024-04-12T03:53:40.729 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51 2024-04-12T03:53:40.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u3hFhwkbE+OR37kh8x8PRESfHj8= 2024-04-12T03:53:40.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u3hFhwkbE+OR37kh8x8PRESfHj8=', 'Content-Length': '0'} 2024-04-12T03:53:40.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7e6e2a1aed5dad-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.732 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.732 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.732 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.732 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:53:40.732 DEBUG:boto:Data: 2024-04-12T03:53:40.732 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.733 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.733 DEBUG:boto:Params: {} 2024-04-12T03:53:40.733 DEBUG:boto:Token: None 2024-04-12T03:53:40.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51 2024-04-12T03:53:40.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u3hFhwkbE+OR37kh8x8PRESfHj8= 2024-04-12T03:53:40.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u3hFhwkbE+OR37kh8x8PRESfHj8=', 'Content-Length': '0'} 2024-04-12T03:53:40.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db859f56d52d05cf-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:53:40.735 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.735 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.735 DEBUG:boto:Method: GET 2024-04-12T03:53:40.735 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:53:40.735 DEBUG:boto:Data: 2024-04-12T03:53:40.735 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.736 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.736 DEBUG:boto:Params: {} 2024-04-12T03:53:40.736 DEBUG:boto:Token: None 2024-04-12T03:53:40.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51 2024-04-12T03:53:40.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o0wXt/YFsACqhEqiglPJJbxmTeA= 2024-04-12T03:53:40.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o0wXt/YFsACqhEqiglPJJbxmTeA=', 'Content-Length': '0'} 2024-04-12T03:53:40.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1635d757f8fddca-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.738 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:53:40.738 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:53:40.738 DEBUG:boto:Method: GET 2024-04-12T03:53:40.738 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:53:40.738 DEBUG:boto:Data: 2024-04-12T03:53:40.738 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.739 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.739 DEBUG:boto:Params: {} 2024-04-12T03:53:40.739 DEBUG:boto:Token: None 2024-04-12T03:53:40.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj51 2024-04-12T03:53:40.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o0wXt/YFsACqhEqiglPJJbxmTeA= 2024-04-12T03:53:40.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o0wXt/YFsACqhEqiglPJJbxmTeA=', 'Content-Length': '0'} 2024-04-12T03:53:40.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c2a0c36289eec3-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.782 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.783 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.783 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.783 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.783 DEBUG:boto:Method: GET 2024-04-12T03:53:40.783 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.783 DEBUG:boto:Data: 2024-04-12T03:53:40.783 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.783 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.783 DEBUG:boto:Params: {} 2024-04-12T03:53:40.784 DEBUG:boto:Token: None 2024-04-12T03:53:40.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597a368be73ecad8-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.786 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.787 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.787 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.787 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.787 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.787 DEBUG:boto:Method: GET 2024-04-12T03:53:40.787 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:53:40.787 DEBUG:boto:Data: 2024-04-12T03:53:40.787 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.788 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.788 DEBUG:boto:Params: {} 2024-04-12T03:53:40.788 DEBUG:boto:Token: None 2024-04-12T03:53:40.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bd706bd94d3f5c-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.792 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:53:40.792 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.792 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.792 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.792 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.793 DEBUG:boto:Method: GET 2024-04-12T03:53:40.793 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.793 DEBUG:boto:Data: 2024-04-12T03:53:40.793 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.793 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.793 DEBUG:boto:Params: {} 2024-04-12T03:53:40.793 DEBUG:boto:Token: None 2024-04-12T03:53:40.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8= 2024-04-12T03:53:40.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8=', 'Content-Length': '0'} 2024-04-12T03:53:40.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13d68fcc87ce530-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.838 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.839 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.839 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.839 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.839 DEBUG:boto:Method: GET 2024-04-12T03:53:40.839 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.839 DEBUG:boto:Data: 2024-04-12T03:53:40.839 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.839 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.839 DEBUG:boto:Params: {} 2024-04-12T03:53:40.840 DEBUG:boto:Token: None 2024-04-12T03:53:40.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8= 2024-04-12T03:53:40.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8=', 'Content-Length': '0'} 2024-04-12T03:53:40.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861217a083544af5-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.842 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.842 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.842 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.842 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.842 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.843 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.843 DEBUG:boto:Data: 2024-04-12T03:53:40.843 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.843 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.843 DEBUG:boto:Params: {} 2024-04-12T03:53:40.843 DEBUG:boto:Token: None 2024-04-12T03:53:40.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UDMK8cuWbsPkhgcO3oM8LaHMSLM= 2024-04-12T03:53:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UDMK8cuWbsPkhgcO3oM8LaHMSLM=', 'Content-Length': '0'} 2024-04-12T03:53:40.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000827a2e8e3d26706c-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.845 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.846 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.846 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.846 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.846 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.846 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.846 DEBUG:boto:Data: 2024-04-12T03:53:40.846 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.846 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.846 DEBUG:boto:Params: {} 2024-04-12T03:53:40.847 DEBUG:boto:Token: None 2024-04-12T03:53:40.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UDMK8cuWbsPkhgcO3oM8LaHMSLM= 2024-04-12T03:53:40.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UDMK8cuWbsPkhgcO3oM8LaHMSLM=', 'Content-Length': '0'} 2024-04-12T03:53:40.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072845c22556a072e-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:53:40.849 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.849 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.849 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.849 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.850 DEBUG:boto:Method: GET 2024-04-12T03:53:40.850 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.850 DEBUG:boto:Data: 2024-04-12T03:53:40.850 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.850 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.850 DEBUG:boto:Params: {} 2024-04-12T03:53:40.850 DEBUG:boto:Token: None 2024-04-12T03:53:40.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.850 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8= 2024-04-12T03:53:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8=', 'Content-Length': '0'} 2024-04-12T03:53:40.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b44a648daef8812b-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.852 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.852 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.853 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.853 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.853 DEBUG:boto:Method: GET 2024-04-12T03:53:40.853 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.853 DEBUG:boto:Data: 2024-04-12T03:53:40.853 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.853 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.853 DEBUG:boto:Params: {} 2024-04-12T03:53:40.853 DEBUG:boto:Token: None 2024-04-12T03:53:40.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:53:40.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8= 2024-04-12T03:53:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wMqIc2lq1QgG2ksDSGoO9Gcbxd8=', 'Content-Length': '0'} 2024-04-12T03:53:40.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7fd91ff3108328-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.856 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.856 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.856 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.856 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.856 DEBUG:boto:Method: GET 2024-04-12T03:53:40.856 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.856 DEBUG:boto:Data: 2024-04-12T03:53:40.856 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.857 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.857 DEBUG:boto:Params: {} 2024-04-12T03:53:40.857 DEBUG:boto:Token: None 2024-04-12T03:53:40.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096cd8a24c51d43ec-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.859 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.860 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.860 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.860 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.860 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.860 DEBUG:boto:Method: GET 2024-04-12T03:53:40.860 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.860 DEBUG:boto:Data: 2024-04-12T03:53:40.860 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.861 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.861 DEBUG:boto:Params: {} 2024-04-12T03:53:40.861 DEBUG:boto:Token: None 2024-04-12T03:53:40.861 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000951fe5aaedc148ef-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.865 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.865 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.865 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.865 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.865 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:53:40.866 DEBUG:boto:Data: 2024-04-12T03:53:40.866 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.866 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.866 DEBUG:boto:Params: {} 2024-04-12T03:53:40.866 DEBUG:boto:Token: None 2024-04-12T03:53:40.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55 2024-04-12T03:53:40.866 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjKlZ9ngtBwnPKeZGbdde8OQ/sU= 2024-04-12T03:53:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjKlZ9ngtBwnPKeZGbdde8OQ/sU=', 'Content-Length': '0'} 2024-04-12T03:53:40.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff26b8fb11de030e-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.868 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.868 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.868 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.868 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:53:40.869 DEBUG:boto:Data: 2024-04-12T03:53:40.869 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.869 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.869 DEBUG:boto:Params: {} 2024-04-12T03:53:40.869 DEBUG:boto:Token: None 2024-04-12T03:53:40.869 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55 2024-04-12T03:53:40.869 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vjKlZ9ngtBwnPKeZGbdde8OQ/sU= 2024-04-12T03:53:40.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vjKlZ9ngtBwnPKeZGbdde8OQ/sU=', 'Content-Length': '0'} 2024-04-12T03:53:40.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051bbeaaac96a3ac-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:53:40.871 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.871 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.871 DEBUG:boto:Method: GET 2024-04-12T03:53:40.871 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:53:40.872 DEBUG:boto:Data: 2024-04-12T03:53:40.872 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.872 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.872 DEBUG:boto:Params: {} 2024-04-12T03:53:40.872 DEBUG:boto:Token: None 2024-04-12T03:53:40.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55 2024-04-12T03:53:40.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FrR/AfXf9tCcmQ/dS4LbsHXbx18= 2024-04-12T03:53:40.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FrR/AfXf9tCcmQ/dS4LbsHXbx18=', 'Content-Length': '0'} 2024-04-12T03:53:40.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ae44ae4788a0ff-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.874 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:53:40.874 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:53:40.874 DEBUG:boto:Method: GET 2024-04-12T03:53:40.874 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:53:40.874 DEBUG:boto:Data: 2024-04-12T03:53:40.874 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.875 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.875 DEBUG:boto:Params: {} 2024-04-12T03:53:40.875 DEBUG:boto:Token: None 2024-04-12T03:53:40.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj55 2024-04-12T03:53:40.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FrR/AfXf9tCcmQ/dS4LbsHXbx18= 2024-04-12T03:53:40.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FrR/AfXf9tCcmQ/dS4LbsHXbx18=', 'Content-Length': '0'} 2024-04-12T03:53:40.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd0765160d0440a-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.918 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.918 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.919 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.919 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.919 DEBUG:boto:Method: GET 2024-04-12T03:53:40.919 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.919 DEBUG:boto:Data: 2024-04-12T03:53:40.919 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.919 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.919 DEBUG:boto:Params: {} 2024-04-12T03:53:40.919 DEBUG:boto:Token: None 2024-04-12T03:53:40.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000988eb1641e081c94-006618b044-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.922 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.922 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.922 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.922 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.923 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.923 DEBUG:boto:Method: GET 2024-04-12T03:53:40.923 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:53:40.923 DEBUG:boto:Data: 2024-04-12T03:53:40.923 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.923 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.923 DEBUG:boto:Params: {} 2024-04-12T03:53:40.923 DEBUG:boto:Token: None 2024-04-12T03:53:40.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dcac71ee083d2c3-006618b044-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.927 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:53:40.928 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.928 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.928 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.928 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.928 DEBUG:boto:Method: GET 2024-04-12T03:53:40.928 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.928 DEBUG:boto:Data: 2024-04-12T03:53:40.928 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.929 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.929 DEBUG:boto:Params: {} 2024-04-12T03:53:40.929 DEBUG:boto:Token: None 2024-04-12T03:53:40.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.929 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0= 2024-04-12T03:53:40.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0=', 'Content-Length': '0'} 2024-04-12T03:53:40.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df7ac15c6a81548-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.931 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.931 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.931 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.931 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.931 DEBUG:boto:Method: GET 2024-04-12T03:53:40.931 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.931 DEBUG:boto:Data: 2024-04-12T03:53:40.932 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.932 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.932 DEBUG:boto:Params: {} 2024-04-12T03:53:40.932 DEBUG:boto:Token: None 2024-04-12T03:53:40.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0= 2024-04-12T03:53:40.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0=', 'Content-Length': '0'} 2024-04-12T03:53:40.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66909c0da5e119e-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.934 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.934 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.934 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.934 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.935 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.935 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.935 DEBUG:boto:Data: 2024-04-12T03:53:40.935 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.935 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.935 DEBUG:boto:Params: {} 2024-04-12T03:53:40.935 DEBUG:boto:Token: None 2024-04-12T03:53:40.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPf47/3iQLKCjGChzCodzWUSl4c= 2024-04-12T03:53:40.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPf47/3iQLKCjGChzCodzWUSl4c=', 'Content-Length': '0'} 2024-04-12T03:53:40.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15728f25d29b749-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.937 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.938 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.938 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.938 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.938 DEBUG:boto:Method: HEAD 2024-04-12T03:53:40.938 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.938 DEBUG:boto:Data: 2024-04-12T03:53:40.938 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.938 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.938 DEBUG:boto:Params: {} 2024-04-12T03:53:40.939 DEBUG:boto:Token: None 2024-04-12T03:53:40.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pPf47/3iQLKCjGChzCodzWUSl4c= 2024-04-12T03:53:40.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pPf47/3iQLKCjGChzCodzWUSl4c=', 'Content-Length': '0'} 2024-04-12T03:53:40.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a33b023464bfcc-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:53:40.941 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.941 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.941 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.941 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.941 DEBUG:boto:Method: GET 2024-04-12T03:53:40.941 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.941 DEBUG:boto:Data: 2024-04-12T03:53:40.942 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.942 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.942 DEBUG:boto:Params: {} 2024-04-12T03:53:40.942 DEBUG:boto:Token: None 2024-04-12T03:53:40.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0= 2024-04-12T03:53:40.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0=', 'Content-Length': '0'} 2024-04-12T03:53:40.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8e19ccb64a7fff-006618b044-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.986 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:40.986 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:40.987 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.987 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.987 DEBUG:boto:Method: GET 2024-04-12T03:53:40.987 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.987 DEBUG:boto:Data: 2024-04-12T03:53:40.987 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.987 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.987 DEBUG:boto:Params: {} 2024-04-12T03:53:40.987 DEBUG:boto:Token: None 2024-04-12T03:53:40.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:53:40.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0= 2024-04-12T03:53:40.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBmO6rJkFjGwEvXEOjK/jyBd0I0=', 'Content-Length': '0'} 2024-04-12T03:53:40.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f75a5facbf18fedd-006618b044-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.990 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.990 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.990 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.990 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.990 DEBUG:boto:Method: GET 2024-04-12T03:53:40.990 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.990 DEBUG:boto:Data: 2024-04-12T03:53:40.991 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:40.991 DEBUG:boto:Port: 8001 2024-04-12T03:53:40.991 DEBUG:boto:Params: {} 2024-04-12T03:53:40.991 DEBUG:boto:Token: None 2024-04-12T03:53:40.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.991 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adecca69f209803b-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.994 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.994 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:40.994 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:40.994 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.994 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.995 DEBUG:boto:Method: GET 2024-04-12T03:53:40.995 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:53:40.995 DEBUG:boto:Data: 2024-04-12T03:53:40.995 DEBUG:boto:Headers: {} 2024-04-12T03:53:40.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:40.995 DEBUG:boto:Port: 8000 2024-04-12T03:53:40.995 DEBUG:boto:Params: {} 2024-04-12T03:53:40.995 DEBUG:boto:Token: None 2024-04-12T03:53:40.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:40 GMT /dgbpxg-61/ 2024-04-12T03:53:40.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s= 2024-04-12T03:53:40.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V9NuIh78BzLj9FRogWEQhfYPw1s=', 'Content-Length': '0'} 2024-04-12T03:53:40.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deabb553bb6fdb2c-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:40.999 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:40.999 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:41.000 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:41.000 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.000 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:53:41.000 DEBUG:boto:Data: 2024-04-12T03:53:41.000 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.000 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.000 DEBUG:boto:Params: {} 2024-04-12T03:53:41.000 DEBUG:boto:Token: None 2024-04-12T03:53:41.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj57 2024-04-12T03:53:41.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ncC9jTEn6zF4PEOrsRBcYJ3ZVsY= 2024-04-12T03:53:41.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ncC9jTEn6zF4PEOrsRBcYJ3ZVsY=', 'Content-Length': '0'} 2024-04-12T03:53:41.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002312531903d2777f-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.003 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:41.003 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:41.003 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.003 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:53:41.003 DEBUG:boto:Data: 2024-04-12T03:53:41.003 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.004 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.004 DEBUG:boto:Params: {} 2024-04-12T03:53:41.004 DEBUG:boto:Token: None 2024-04-12T03:53:41.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj57 2024-04-12T03:53:41.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ncC9jTEn6zF4PEOrsRBcYJ3ZVsY= 2024-04-12T03:53:41.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ncC9jTEn6zF4PEOrsRBcYJ3ZVsY=', 'Content-Length': '0'} 2024-04-12T03:53:41.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008930480c5cd59160-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:53:41.006 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:41.006 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:41.006 DEBUG:boto:Method: GET 2024-04-12T03:53:41.007 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:53:41.007 DEBUG:boto:Data: 2024-04-12T03:53:41.007 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.007 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.007 DEBUG:boto:Params: {} 2024-04-12T03:53:41.007 DEBUG:boto:Token: None 2024-04-12T03:53:41.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj57 2024-04-12T03:53:41.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E1tARDUbqCA0L7HTckAOYNmGGx4= 2024-04-12T03:53:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E1tARDUbqCA0L7HTckAOYNmGGx4=', 'Content-Length': '0'} 2024-04-12T03:53:41.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219167301bd6099a-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.009 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:53:41.010 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:53:41.010 DEBUG:boto:Method: GET 2024-04-12T03:53:41.010 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:53:41.010 DEBUG:boto:Data: 2024-04-12T03:53:41.010 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.010 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.010 DEBUG:boto:Params: {} 2024-04-12T03:53:41.011 DEBUG:boto:Token: None 2024-04-12T03:53:41.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj57 2024-04-12T03:53:41.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E1tARDUbqCA0L7HTckAOYNmGGx4= 2024-04-12T03:53:41.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E1tARDUbqCA0L7HTckAOYNmGGx4=', 'Content-Length': '0'} 2024-04-12T03:53:41.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4eb4e7b95319f25-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.054 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.054 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.055 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.055 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.055 DEBUG:boto:Method: GET 2024-04-12T03:53:41.055 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.055 DEBUG:boto:Data: 2024-04-12T03:53:41.055 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.055 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.055 DEBUG:boto:Params: {} 2024-04-12T03:53:41.056 DEBUG:boto:Token: None 2024-04-12T03:53:41.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cf650efea9f9b53-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.059 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.059 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.059 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.059 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.059 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.059 DEBUG:boto:Method: GET 2024-04-12T03:53:41.059 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:53:41.059 DEBUG:boto:Data: 2024-04-12T03:53:41.060 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.060 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.060 DEBUG:boto:Params: {} 2024-04-12T03:53:41.060 DEBUG:boto:Token: None 2024-04-12T03:53:41.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6641f8915179496-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.064 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:53:41.064 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.064 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.065 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.065 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.065 DEBUG:boto:Method: GET 2024-04-12T03:53:41.065 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.065 DEBUG:boto:Data: 2024-04-12T03:53:41.065 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.065 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.065 DEBUG:boto:Params: {} 2024-04-12T03:53:41.065 DEBUG:boto:Token: None 2024-04-12T03:53:41.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA= 2024-04-12T03:53:41.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA=', 'Content-Length': '0'} 2024-04-12T03:53:41.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235fe48a4bab5578-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.110 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.111 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.111 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.111 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.111 DEBUG:boto:Method: GET 2024-04-12T03:53:41.111 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.111 DEBUG:boto:Data: 2024-04-12T03:53:41.111 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.111 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.111 DEBUG:boto:Params: {} 2024-04-12T03:53:41.112 DEBUG:boto:Token: None 2024-04-12T03:53:41.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA= 2024-04-12T03:53:41.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA=', 'Content-Length': '0'} 2024-04-12T03:53:41.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021af7682865daf12-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.114 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.114 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.114 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.114 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.114 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.114 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.114 DEBUG:boto:Data: 2024-04-12T03:53:41.114 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.115 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.115 DEBUG:boto:Params: {} 2024-04-12T03:53:41.115 DEBUG:boto:Token: None 2024-04-12T03:53:41.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.115 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NQK6T+pVp77fwuqbbcjH4o96m5g= 2024-04-12T03:53:41.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NQK6T+pVp77fwuqbbcjH4o96m5g=', 'Content-Length': '0'} 2024-04-12T03:53:41.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d778e1892822f6-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.117 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.117 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.117 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.117 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.118 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.118 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.118 DEBUG:boto:Data: 2024-04-12T03:53:41.118 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.118 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.118 DEBUG:boto:Params: {} 2024-04-12T03:53:41.118 DEBUG:boto:Token: None 2024-04-12T03:53:41.118 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NQK6T+pVp77fwuqbbcjH4o96m5g= 2024-04-12T03:53:41.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NQK6T+pVp77fwuqbbcjH4o96m5g=', 'Content-Length': '0'} 2024-04-12T03:53:41.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dafddf013699fb7-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:53:41.121 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.121 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.121 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.121 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.121 DEBUG:boto:Method: GET 2024-04-12T03:53:41.121 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.121 DEBUG:boto:Data: 2024-04-12T03:53:41.121 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.121 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.122 DEBUG:boto:Params: {} 2024-04-12T03:53:41.122 DEBUG:boto:Token: None 2024-04-12T03:53:41.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA= 2024-04-12T03:53:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA=', 'Content-Length': '0'} 2024-04-12T03:53:41.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6cd61213047a67-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.124 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.124 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.124 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.124 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.124 DEBUG:boto:Method: GET 2024-04-12T03:53:41.125 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.125 DEBUG:boto:Data: 2024-04-12T03:53:41.125 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.125 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.125 DEBUG:boto:Params: {} 2024-04-12T03:53:41.125 DEBUG:boto:Token: None 2024-04-12T03:53:41.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:53:41.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA= 2024-04-12T03:53:41.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dQasxad2LXwLTxDsADWUSDnmpoA=', 'Content-Length': '0'} 2024-04-12T03:53:41.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7e1ace9a2f2ee1-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.127 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.127 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.127 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.127 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.127 DEBUG:boto:Method: GET 2024-04-12T03:53:41.128 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.128 DEBUG:boto:Data: 2024-04-12T03:53:41.128 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.128 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.128 DEBUG:boto:Params: {} 2024-04-12T03:53:41.128 DEBUG:boto:Token: None 2024-04-12T03:53:41.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4b48d274088c15-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.131 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.131 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.131 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.131 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.131 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.131 DEBUG:boto:Method: GET 2024-04-12T03:53:41.132 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.132 DEBUG:boto:Data: 2024-04-12T03:53:41.132 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.132 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.132 DEBUG:boto:Params: {} 2024-04-12T03:53:41.132 DEBUG:boto:Token: None 2024-04-12T03:53:41.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c0ac3453d0a7802-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.136 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.136 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.136 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.137 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.137 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:53:41.137 DEBUG:boto:Data: 2024-04-12T03:53:41.137 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.137 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.137 DEBUG:boto:Params: {} 2024-04-12T03:53:41.137 DEBUG:boto:Token: None 2024-04-12T03:53:41.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58 2024-04-12T03:53:41.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wJMLJmfzA0Vk+Z2fVYDhkNYG9Hw= 2024-04-12T03:53:41.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wJMLJmfzA0Vk+Z2fVYDhkNYG9Hw=', 'Content-Length': '0'} 2024-04-12T03:53:41.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be314fcc362ecfb-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.139 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.139 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.139 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.139 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:53:41.140 DEBUG:boto:Data: 2024-04-12T03:53:41.140 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.140 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.140 DEBUG:boto:Params: {} 2024-04-12T03:53:41.140 DEBUG:boto:Token: None 2024-04-12T03:53:41.140 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58 2024-04-12T03:53:41.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wJMLJmfzA0Vk+Z2fVYDhkNYG9Hw= 2024-04-12T03:53:41.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wJMLJmfzA0Vk+Z2fVYDhkNYG9Hw=', 'Content-Length': '0'} 2024-04-12T03:53:41.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc8d7a68e6b576f-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:53:41.142 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.142 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.142 DEBUG:boto:Method: GET 2024-04-12T03:53:41.142 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:53:41.142 DEBUG:boto:Data: 2024-04-12T03:53:41.142 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.143 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.143 DEBUG:boto:Params: {} 2024-04-12T03:53:41.143 DEBUG:boto:Token: None 2024-04-12T03:53:41.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58 2024-04-12T03:53:41.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+CcGKLCJ5jKYeGXM6BjkHsVu6n0= 2024-04-12T03:53:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+CcGKLCJ5jKYeGXM6BjkHsVu6n0=', 'Content-Length': '0'} 2024-04-12T03:53:41.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a092aaab77bd8852-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.186 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:53:41.186 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:53:41.187 DEBUG:boto:Method: GET 2024-04-12T03:53:41.187 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:53:41.187 DEBUG:boto:Data: 2024-04-12T03:53:41.187 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.187 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.187 DEBUG:boto:Params: {} 2024-04-12T03:53:41.187 DEBUG:boto:Token: None 2024-04-12T03:53:41.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj58 2024-04-12T03:53:41.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+CcGKLCJ5jKYeGXM6BjkHsVu6n0= 2024-04-12T03:53:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+CcGKLCJ5jKYeGXM6BjkHsVu6n0=', 'Content-Length': '0'} 2024-04-12T03:53:41.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043760138e2c061c-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.189 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.190 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.190 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.190 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.190 DEBUG:boto:Method: GET 2024-04-12T03:53:41.190 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.190 DEBUG:boto:Data: 2024-04-12T03:53:41.190 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.190 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.190 DEBUG:boto:Params: {} 2024-04-12T03:53:41.190 DEBUG:boto:Token: None 2024-04-12T03:53:41.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.191 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007019ae400b98d9ba-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.193 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.194 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.194 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.194 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.194 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.194 DEBUG:boto:Method: GET 2024-04-12T03:53:41.194 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:53:41.194 DEBUG:boto:Data: 2024-04-12T03:53:41.194 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.194 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.194 DEBUG:boto:Params: {} 2024-04-12T03:53:41.195 DEBUG:boto:Token: None 2024-04-12T03:53:41.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085f318c69df511d1-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.198 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:53:41.199 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.199 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.199 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.199 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.199 DEBUG:boto:Method: GET 2024-04-12T03:53:41.199 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.199 DEBUG:boto:Data: 2024-04-12T03:53:41.199 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.200 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.200 DEBUG:boto:Params: {} 2024-04-12T03:53:41.200 DEBUG:boto:Token: None 2024-04-12T03:53:41.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs= 2024-04-12T03:53:41.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs=', 'Content-Length': '0'} 2024-04-12T03:53:41.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3c32dd036f8018-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.202 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.202 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.202 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.202 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.202 DEBUG:boto:Method: GET 2024-04-12T03:53:41.203 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.203 DEBUG:boto:Data: 2024-04-12T03:53:41.203 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.203 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.203 DEBUG:boto:Params: {} 2024-04-12T03:53:41.203 DEBUG:boto:Token: None 2024-04-12T03:53:41.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs= 2024-04-12T03:53:41.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs=', 'Content-Length': '0'} 2024-04-12T03:53:41.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f635d7bdb8fa659b-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.205 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.205 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.205 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.206 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.206 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.206 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.206 DEBUG:boto:Data: 2024-04-12T03:53:41.206 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.206 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.206 DEBUG:boto:Params: {} 2024-04-12T03:53:41.206 DEBUG:boto:Token: None 2024-04-12T03:53:41.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.207 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ff8HEaqnB1osD+4Uz4jVwVny/1o= 2024-04-12T03:53:41.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ff8HEaqnB1osD+4Uz4jVwVny/1o=', 'Content-Length': '0'} 2024-04-12T03:53:41.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de732e47bdfdad7a-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.208 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.208 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.208 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.209 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.209 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.209 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.209 DEBUG:boto:Data: 2024-04-12T03:53:41.209 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.209 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.209 DEBUG:boto:Params: {} 2024-04-12T03:53:41.209 DEBUG:boto:Token: None 2024-04-12T03:53:41.209 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.210 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ff8HEaqnB1osD+4Uz4jVwVny/1o= 2024-04-12T03:53:41.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ff8HEaqnB1osD+4Uz4jVwVny/1o=', 'Content-Length': '0'} 2024-04-12T03:53:41.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd39e7b7f453862f-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:53:41.211 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.211 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.211 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.212 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.212 DEBUG:boto:Method: GET 2024-04-12T03:53:41.212 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.212 DEBUG:boto:Data: 2024-04-12T03:53:41.212 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.212 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.212 DEBUG:boto:Params: {} 2024-04-12T03:53:41.212 DEBUG:boto:Token: None 2024-04-12T03:53:41.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs= 2024-04-12T03:53:41.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs=', 'Content-Length': '0'} 2024-04-12T03:53:41.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d13bf9402c08971-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.214 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.215 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.215 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.215 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.215 DEBUG:boto:Method: GET 2024-04-12T03:53:41.215 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.215 DEBUG:boto:Data: 2024-04-12T03:53:41.215 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.215 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.215 DEBUG:boto:Params: {} 2024-04-12T03:53:41.216 DEBUG:boto:Token: None 2024-04-12T03:53:41.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:53:41.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs= 2024-04-12T03:53:41.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CePhAh0ccffxeHIQvchMGgHysKs=', 'Content-Length': '0'} 2024-04-12T03:53:41.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9d015c871672ea-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.218 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.218 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.218 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.218 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.218 DEBUG:boto:Method: GET 2024-04-12T03:53:41.218 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.218 DEBUG:boto:Data: 2024-04-12T03:53:41.218 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.219 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.219 DEBUG:boto:Params: {} 2024-04-12T03:53:41.219 DEBUG:boto:Token: None 2024-04-12T03:53:41.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede0425604c8f4fe-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.221 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.222 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.222 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.222 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.222 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.222 DEBUG:boto:Method: GET 2024-04-12T03:53:41.222 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.222 DEBUG:boto:Data: 2024-04-12T03:53:41.222 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.223 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.223 DEBUG:boto:Params: {} 2024-04-12T03:53:41.223 DEBUG:boto:Token: None 2024-04-12T03:53:41.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834f19196eb75e74-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.227 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.227 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.227 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.227 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.227 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:53:41.227 DEBUG:boto:Data: 2024-04-12T03:53:41.227 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.228 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.228 DEBUG:boto:Params: {} 2024-04-12T03:53:41.228 DEBUG:boto:Token: None 2024-04-12T03:53:41.228 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62 2024-04-12T03:53:41.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Oq7VVhN7lFX+8C3g6RDenAsYn7U= 2024-04-12T03:53:41.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Oq7VVhN7lFX+8C3g6RDenAsYn7U=', 'Content-Length': '0'} 2024-04-12T03:53:41.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde75eb3ea5adfb9-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.230 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.230 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.230 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.230 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:53:41.230 DEBUG:boto:Data: 2024-04-12T03:53:41.230 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.230 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.231 DEBUG:boto:Params: {} 2024-04-12T03:53:41.231 DEBUG:boto:Token: None 2024-04-12T03:53:41.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62 2024-04-12T03:53:41.231 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Oq7VVhN7lFX+8C3g6RDenAsYn7U= 2024-04-12T03:53:41.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Oq7VVhN7lFX+8C3g6RDenAsYn7U=', 'Content-Length': '0'} 2024-04-12T03:53:41.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d374c5a3e0e776cb-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:53:41.232 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.233 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.233 DEBUG:boto:Method: GET 2024-04-12T03:53:41.233 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:53:41.233 DEBUG:boto:Data: 2024-04-12T03:53:41.233 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.233 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.233 DEBUG:boto:Params: {} 2024-04-12T03:53:41.233 DEBUG:boto:Token: None 2024-04-12T03:53:41.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62 2024-04-12T03:53:41.233 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YEqQ207H5yOy+NfPC4SEDxMrth8= 2024-04-12T03:53:41.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YEqQ207H5yOy+NfPC4SEDxMrth8=', 'Content-Length': '0'} 2024-04-12T03:53:41.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047daba38b2860e21-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.235 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:53:41.235 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:53:41.235 DEBUG:boto:Method: GET 2024-04-12T03:53:41.235 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:53:41.235 DEBUG:boto:Data: 2024-04-12T03:53:41.235 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.236 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.236 DEBUG:boto:Params: {} 2024-04-12T03:53:41.236 DEBUG:boto:Token: None 2024-04-12T03:53:41.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj62 2024-04-12T03:53:41.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YEqQ207H5yOy+NfPC4SEDxMrth8= 2024-04-12T03:53:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YEqQ207H5yOy+NfPC4SEDxMrth8=', 'Content-Length': '0'} 2024-04-12T03:53:41.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac3d065303ff402-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.278 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.278 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.278 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.279 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.279 DEBUG:boto:Method: GET 2024-04-12T03:53:41.279 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.279 DEBUG:boto:Data: 2024-04-12T03:53:41.279 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.279 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.279 DEBUG:boto:Params: {} 2024-04-12T03:53:41.279 DEBUG:boto:Token: None 2024-04-12T03:53:41.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e038844dbe3a18-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.282 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.282 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.282 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.282 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.282 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.282 DEBUG:boto:Method: GET 2024-04-12T03:53:41.282 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:53:41.283 DEBUG:boto:Data: 2024-04-12T03:53:41.283 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.283 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.283 DEBUG:boto:Params: {} 2024-04-12T03:53:41.283 DEBUG:boto:Token: None 2024-04-12T03:53:41.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256a81675101c672-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.287 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:53:41.287 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.287 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.287 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.287 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.287 DEBUG:boto:Method: GET 2024-04-12T03:53:41.288 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.288 DEBUG:boto:Data: 2024-04-12T03:53:41.288 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.288 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.288 DEBUG:boto:Params: {} 2024-04-12T03:53:41.288 DEBUG:boto:Token: None 2024-04-12T03:53:41.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw= 2024-04-12T03:53:41.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw=', 'Content-Length': '0'} 2024-04-12T03:53:41.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ced3d9ec31d92e0-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.290 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.290 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.291 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.291 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.291 DEBUG:boto:Method: GET 2024-04-12T03:53:41.291 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.291 DEBUG:boto:Data: 2024-04-12T03:53:41.291 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.291 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.291 DEBUG:boto:Params: {} 2024-04-12T03:53:41.291 DEBUG:boto:Token: None 2024-04-12T03:53:41.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw= 2024-04-12T03:53:41.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw=', 'Content-Length': '0'} 2024-04-12T03:53:41.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223b14f1a0cc48b7-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.294 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.294 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.294 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.294 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.294 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.294 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.294 DEBUG:boto:Data: 2024-04-12T03:53:41.294 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.295 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.295 DEBUG:boto:Params: {} 2024-04-12T03:53:41.295 DEBUG:boto:Token: None 2024-04-12T03:53:41.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WcG/kY8nETSALlWqLCtNsI3UwqM= 2024-04-12T03:53:41.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WcG/kY8nETSALlWqLCtNsI3UwqM=', 'Content-Length': '0'} 2024-04-12T03:53:41.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb93426e73bd918-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.297 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.297 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.297 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.297 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.297 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.297 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.297 DEBUG:boto:Data: 2024-04-12T03:53:41.297 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.298 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.298 DEBUG:boto:Params: {} 2024-04-12T03:53:41.298 DEBUG:boto:Token: None 2024-04-12T03:53:41.298 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.298 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WcG/kY8nETSALlWqLCtNsI3UwqM= 2024-04-12T03:53:41.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WcG/kY8nETSALlWqLCtNsI3UwqM=', 'Content-Length': '0'} 2024-04-12T03:53:41.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b6dc176f7cf25f-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:53:41.300 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.300 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.300 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.300 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.300 DEBUG:boto:Method: GET 2024-04-12T03:53:41.300 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.300 DEBUG:boto:Data: 2024-04-12T03:53:41.300 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.301 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.301 DEBUG:boto:Params: {} 2024-04-12T03:53:41.301 DEBUG:boto:Token: None 2024-04-12T03:53:41.301 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.301 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw= 2024-04-12T03:53:41.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw=', 'Content-Length': '0'} 2024-04-12T03:53:41.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5bfe9b05e11f833-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.346 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.346 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.346 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.347 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.347 DEBUG:boto:Method: GET 2024-04-12T03:53:41.347 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.347 DEBUG:boto:Data: 2024-04-12T03:53:41.347 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.347 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.347 DEBUG:boto:Params: {} 2024-04-12T03:53:41.347 DEBUG:boto:Token: None 2024-04-12T03:53:41.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:53:41.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw= 2024-04-12T03:53:41.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eI3hD9dUYjeOe973UJFNwk/K8zw=', 'Content-Length': '0'} 2024-04-12T03:53:41.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5fe95f282560cd-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.349 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.350 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.350 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.350 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.350 DEBUG:boto:Method: GET 2024-04-12T03:53:41.350 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.350 DEBUG:boto:Data: 2024-04-12T03:53:41.350 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.350 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.350 DEBUG:boto:Params: {} 2024-04-12T03:53:41.351 DEBUG:boto:Token: None 2024-04-12T03:53:41.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.351 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f187f30b62aeedb5-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.353 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.354 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.354 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.354 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.354 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.354 DEBUG:boto:Method: GET 2024-04-12T03:53:41.354 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.354 DEBUG:boto:Data: 2024-04-12T03:53:41.354 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.354 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.354 DEBUG:boto:Params: {} 2024-04-12T03:53:41.355 DEBUG:boto:Token: None 2024-04-12T03:53:41.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b44fd09a76d4224d-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.358 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.359 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.359 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.359 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.359 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:53:41.359 DEBUG:boto:Data: 2024-04-12T03:53:41.359 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.359 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.359 DEBUG:boto:Params: {} 2024-04-12T03:53:41.360 DEBUG:boto:Token: None 2024-04-12T03:53:41.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68 2024-04-12T03:53:41.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UWSjL9cB3kpag8M0/vqTuiYLdVI= 2024-04-12T03:53:41.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UWSjL9cB3kpag8M0/vqTuiYLdVI=', 'Content-Length': '0'} 2024-04-12T03:53:41.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ef91e318e9aa91-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.362 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.362 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.362 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.362 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:53:41.362 DEBUG:boto:Data: 2024-04-12T03:53:41.362 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.363 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.363 DEBUG:boto:Params: {} 2024-04-12T03:53:41.363 DEBUG:boto:Token: None 2024-04-12T03:53:41.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68 2024-04-12T03:53:41.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UWSjL9cB3kpag8M0/vqTuiYLdVI= 2024-04-12T03:53:41.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UWSjL9cB3kpag8M0/vqTuiYLdVI=', 'Content-Length': '0'} 2024-04-12T03:53:41.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e14a0a8002ca4239-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:53:41.365 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.365 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.365 DEBUG:boto:Method: GET 2024-04-12T03:53:41.365 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:53:41.365 DEBUG:boto:Data: 2024-04-12T03:53:41.365 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.365 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.365 DEBUG:boto:Params: {} 2024-04-12T03:53:41.366 DEBUG:boto:Token: None 2024-04-12T03:53:41.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68 2024-04-12T03:53:41.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N6x14osGyErwdJE9QnhkgjB0gCo= 2024-04-12T03:53:41.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6x14osGyErwdJE9QnhkgjB0gCo=', 'Content-Length': '0'} 2024-04-12T03:53:41.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003475b657665634e6-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.368 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:53:41.368 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:53:41.368 DEBUG:boto:Method: GET 2024-04-12T03:53:41.368 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:53:41.368 DEBUG:boto:Data: 2024-04-12T03:53:41.368 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.368 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.368 DEBUG:boto:Params: {} 2024-04-12T03:53:41.369 DEBUG:boto:Token: None 2024-04-12T03:53:41.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj68 2024-04-12T03:53:41.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:N6x14osGyErwdJE9QnhkgjB0gCo= 2024-04-12T03:53:41.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6x14osGyErwdJE9QnhkgjB0gCo=', 'Content-Length': '0'} 2024-04-12T03:53:41.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058f859f0cdd4450-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.371 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.371 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.371 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.371 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.371 DEBUG:boto:Method: GET 2024-04-12T03:53:41.371 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.371 DEBUG:boto:Data: 2024-04-12T03:53:41.371 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.371 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.372 DEBUG:boto:Params: {} 2024-04-12T03:53:41.372 DEBUG:boto:Token: None 2024-04-12T03:53:41.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000703a5abb4d4de80f-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.374 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.374 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.375 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.375 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.375 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.375 DEBUG:boto:Method: GET 2024-04-12T03:53:41.375 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:53:41.375 DEBUG:boto:Data: 2024-04-12T03:53:41.375 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.375 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.375 DEBUG:boto:Params: {} 2024-04-12T03:53:41.375 DEBUG:boto:Token: None 2024-04-12T03:53:41.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb282b2a47bbfd4-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.422 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:53:41.423 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.423 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.423 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.423 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.423 DEBUG:boto:Method: GET 2024-04-12T03:53:41.423 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.423 DEBUG:boto:Data: 2024-04-12T03:53:41.423 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.423 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.424 DEBUG:boto:Params: {} 2024-04-12T03:53:41.424 DEBUG:boto:Token: None 2024-04-12T03:53:41.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos= 2024-04-12T03:53:41.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos=', 'Content-Length': '0'} 2024-04-12T03:53:41.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1fcdde058d2f0d-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.426 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.426 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.426 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.426 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.426 DEBUG:boto:Method: GET 2024-04-12T03:53:41.427 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.427 DEBUG:boto:Data: 2024-04-12T03:53:41.427 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.427 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.427 DEBUG:boto:Params: {} 2024-04-12T03:53:41.427 DEBUG:boto:Token: None 2024-04-12T03:53:41.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos= 2024-04-12T03:53:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos=', 'Content-Length': '0'} 2024-04-12T03:53:41.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094632ebea5996250-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.429 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.429 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.429 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.429 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.430 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.430 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.430 DEBUG:boto:Data: 2024-04-12T03:53:41.430 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.430 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.430 DEBUG:boto:Params: {} 2024-04-12T03:53:41.430 DEBUG:boto:Token: None 2024-04-12T03:53:41.430 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.430 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kUtG7iBNl4zeDWrbTM1OneuZ7qE= 2024-04-12T03:53:41.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kUtG7iBNl4zeDWrbTM1OneuZ7qE=', 'Content-Length': '0'} 2024-04-12T03:53:41.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec122ae589718fa-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.432 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.432 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.432 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.432 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.432 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.433 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.433 DEBUG:boto:Data: 2024-04-12T03:53:41.433 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.433 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.433 DEBUG:boto:Params: {} 2024-04-12T03:53:41.433 DEBUG:boto:Token: None 2024-04-12T03:53:41.433 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kUtG7iBNl4zeDWrbTM1OneuZ7qE= 2024-04-12T03:53:41.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kUtG7iBNl4zeDWrbTM1OneuZ7qE=', 'Content-Length': '0'} 2024-04-12T03:53:41.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8f6c83f9feb437-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:53:41.435 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.436 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.436 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.436 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.436 DEBUG:boto:Method: GET 2024-04-12T03:53:41.436 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.436 DEBUG:boto:Data: 2024-04-12T03:53:41.436 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.436 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.436 DEBUG:boto:Params: {} 2024-04-12T03:53:41.437 DEBUG:boto:Token: None 2024-04-12T03:53:41.437 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos= 2024-04-12T03:53:41.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos=', 'Content-Length': '0'} 2024-04-12T03:53:41.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644a7f359ca2d514-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.439 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.439 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.439 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.439 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.439 DEBUG:boto:Method: GET 2024-04-12T03:53:41.440 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.440 DEBUG:boto:Data: 2024-04-12T03:53:41.440 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.440 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.440 DEBUG:boto:Params: {} 2024-04-12T03:53:41.440 DEBUG:boto:Token: None 2024-04-12T03:53:41.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:53:41.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos= 2024-04-12T03:53:41.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T9rSiLU1MGxZXiyRcmUnRk/YNos=', 'Content-Length': '0'} 2024-04-12T03:53:41.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951b8abd31f777c4-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.442 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.442 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.442 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.443 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.443 DEBUG:boto:Method: GET 2024-04-12T03:53:41.443 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.443 DEBUG:boto:Data: 2024-04-12T03:53:41.443 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.443 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.443 DEBUG:boto:Params: {} 2024-04-12T03:53:41.443 DEBUG:boto:Token: None 2024-04-12T03:53:41.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4bc5c90a1d379d8-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.446 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.446 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.446 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.446 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.446 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.446 DEBUG:boto:Method: GET 2024-04-12T03:53:41.447 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.447 DEBUG:boto:Data: 2024-04-12T03:53:41.447 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.447 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.447 DEBUG:boto:Params: {} 2024-04-12T03:53:41.447 DEBUG:boto:Token: None 2024-04-12T03:53:41.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.447 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebefbb51e040abf-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.451 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.452 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.452 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.452 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.452 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:53:41.452 DEBUG:boto:Data: 2024-04-12T03:53:41.452 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.452 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.452 DEBUG:boto:Params: {} 2024-04-12T03:53:41.453 DEBUG:boto:Token: None 2024-04-12T03:53:41.453 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69 2024-04-12T03:53:41.453 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fVyt1s8fYAa2IScKWafKS7kRI7U= 2024-04-12T03:53:41.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fVyt1s8fYAa2IScKWafKS7kRI7U=', 'Content-Length': '0'} 2024-04-12T03:53:41.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1bdfec57a4e6f47-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.455 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.455 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.455 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.455 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:53:41.455 DEBUG:boto:Data: 2024-04-12T03:53:41.455 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.455 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.455 DEBUG:boto:Params: {} 2024-04-12T03:53:41.456 DEBUG:boto:Token: None 2024-04-12T03:53:41.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69 2024-04-12T03:53:41.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fVyt1s8fYAa2IScKWafKS7kRI7U= 2024-04-12T03:53:41.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fVyt1s8fYAa2IScKWafKS7kRI7U=', 'Content-Length': '0'} 2024-04-12T03:53:41.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a601306f1823848d-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:53:41.458 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.458 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.458 DEBUG:boto:Method: GET 2024-04-12T03:53:41.458 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:53:41.458 DEBUG:boto:Data: 2024-04-12T03:53:41.458 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.458 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.459 DEBUG:boto:Params: {} 2024-04-12T03:53:41.459 DEBUG:boto:Token: None 2024-04-12T03:53:41.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69 2024-04-12T03:53:41.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:klFkjWQbhevqWTUQOR/HcRQGfXI= 2024-04-12T03:53:41.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:klFkjWQbhevqWTUQOR/HcRQGfXI=', 'Content-Length': '0'} 2024-04-12T03:53:41.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012949b6aea7d02a3-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.506 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:53:41.506 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:53:41.507 DEBUG:boto:Method: GET 2024-04-12T03:53:41.507 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:53:41.507 DEBUG:boto:Data: 2024-04-12T03:53:41.507 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.507 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.507 DEBUG:boto:Params: {} 2024-04-12T03:53:41.507 DEBUG:boto:Token: None 2024-04-12T03:53:41.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj69 2024-04-12T03:53:41.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:klFkjWQbhevqWTUQOR/HcRQGfXI= 2024-04-12T03:53:41.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:klFkjWQbhevqWTUQOR/HcRQGfXI=', 'Content-Length': '0'} 2024-04-12T03:53:41.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f355032cd736bdbd-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.509 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.509 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.509 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.510 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.510 DEBUG:boto:Method: GET 2024-04-12T03:53:41.510 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.510 DEBUG:boto:Data: 2024-04-12T03:53:41.510 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.510 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.510 DEBUG:boto:Params: {} 2024-04-12T03:53:41.510 DEBUG:boto:Token: None 2024-04-12T03:53:41.510 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd967acace7985a-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.513 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.513 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.513 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.513 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.513 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.513 DEBUG:boto:Method: GET 2024-04-12T03:53:41.513 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:53:41.514 DEBUG:boto:Data: 2024-04-12T03:53:41.514 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.514 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.514 DEBUG:boto:Params: {} 2024-04-12T03:53:41.514 DEBUG:boto:Token: None 2024-04-12T03:53:41.514 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.514 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ee3248bc95dee74-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.518 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:53:41.518 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.518 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.518 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.518 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.518 DEBUG:boto:Method: GET 2024-04-12T03:53:41.518 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.518 DEBUG:boto:Data: 2024-04-12T03:53:41.519 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.519 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.519 DEBUG:boto:Params: {} 2024-04-12T03:53:41.519 DEBUG:boto:Token: None 2024-04-12T03:53:41.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM= 2024-04-12T03:53:41.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM=', 'Content-Length': '0'} 2024-04-12T03:53:41.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ce5aace4107df1-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.521 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.521 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.521 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.522 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.522 DEBUG:boto:Method: GET 2024-04-12T03:53:41.522 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.522 DEBUG:boto:Data: 2024-04-12T03:53:41.522 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.522 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.522 DEBUG:boto:Params: {} 2024-04-12T03:53:41.522 DEBUG:boto:Token: None 2024-04-12T03:53:41.522 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM= 2024-04-12T03:53:41.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM=', 'Content-Length': '0'} 2024-04-12T03:53:41.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b37b683a08214ca-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.524 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.524 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.524 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.524 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.524 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.524 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.525 DEBUG:boto:Data: 2024-04-12T03:53:41.525 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.525 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.525 DEBUG:boto:Params: {} 2024-04-12T03:53:41.525 DEBUG:boto:Token: None 2024-04-12T03:53:41.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlaG6Z6GoHCR5VtTdSjWwwH9FRA= 2024-04-12T03:53:41.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlaG6Z6GoHCR5VtTdSjWwwH9FRA=', 'Content-Length': '0'} 2024-04-12T03:53:41.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebb9a83924c38f67-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.527 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.527 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.527 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.528 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.528 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.528 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.528 DEBUG:boto:Data: 2024-04-12T03:53:41.528 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.528 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.528 DEBUG:boto:Params: {} 2024-04-12T03:53:41.528 DEBUG:boto:Token: None 2024-04-12T03:53:41.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlaG6Z6GoHCR5VtTdSjWwwH9FRA= 2024-04-12T03:53:41.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlaG6Z6GoHCR5VtTdSjWwwH9FRA=', 'Content-Length': '0'} 2024-04-12T03:53:41.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c929961a6f2906-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:53:41.530 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.530 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.530 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.530 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.530 DEBUG:boto:Method: GET 2024-04-12T03:53:41.531 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.531 DEBUG:boto:Data: 2024-04-12T03:53:41.531 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.531 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.531 DEBUG:boto:Params: {} 2024-04-12T03:53:41.531 DEBUG:boto:Token: None 2024-04-12T03:53:41.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM= 2024-04-12T03:53:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM=', 'Content-Length': '0'} 2024-04-12T03:53:41.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007958ba117ade8245-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.533 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.533 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.534 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.534 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.534 DEBUG:boto:Method: GET 2024-04-12T03:53:41.534 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.534 DEBUG:boto:Data: 2024-04-12T03:53:41.534 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.534 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.534 DEBUG:boto:Params: {} 2024-04-12T03:53:41.534 DEBUG:boto:Token: None 2024-04-12T03:53:41.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:53:41.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM= 2024-04-12T03:53:41.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s/7rWq1/SVY9zpD0cwSujCED0xM=', 'Content-Length': '0'} 2024-04-12T03:53:41.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003beea212efeb5343-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.536 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.537 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.537 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.537 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.537 DEBUG:boto:Method: GET 2024-04-12T03:53:41.537 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.537 DEBUG:boto:Data: 2024-04-12T03:53:41.537 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.537 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.537 DEBUG:boto:Params: {} 2024-04-12T03:53:41.537 DEBUG:boto:Token: None 2024-04-12T03:53:41.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.538 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084822363e5bc78b8-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.540 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.540 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.540 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.541 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.541 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.541 DEBUG:boto:Method: GET 2024-04-12T03:53:41.541 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.541 DEBUG:boto:Data: 2024-04-12T03:53:41.541 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.541 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.541 DEBUG:boto:Params: {} 2024-04-12T03:53:41.541 DEBUG:boto:Token: None 2024-04-12T03:53:41.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb3b16613bc9286-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.546 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.546 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.546 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.546 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.546 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:53:41.546 DEBUG:boto:Data: 2024-04-12T03:53:41.546 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.547 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.547 DEBUG:boto:Params: {} 2024-04-12T03:53:41.547 DEBUG:boto:Token: None 2024-04-12T03:53:41.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74 2024-04-12T03:53:41.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c2DmMv77O6dGJPBnzNj/yziY3BA= 2024-04-12T03:53:41.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c2DmMv77O6dGJPBnzNj/yziY3BA=', 'Content-Length': '0'} 2024-04-12T03:53:41.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9da122e5074428-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.549 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.549 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.549 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.549 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:53:41.549 DEBUG:boto:Data: 2024-04-12T03:53:41.549 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.549 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.549 DEBUG:boto:Params: {} 2024-04-12T03:53:41.549 DEBUG:boto:Token: None 2024-04-12T03:53:41.550 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74 2024-04-12T03:53:41.550 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c2DmMv77O6dGJPBnzNj/yziY3BA= 2024-04-12T03:53:41.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c2DmMv77O6dGJPBnzNj/yziY3BA=', 'Content-Length': '0'} 2024-04-12T03:53:41.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000afddd2315ba93-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:53:41.551 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.552 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.552 DEBUG:boto:Method: GET 2024-04-12T03:53:41.552 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:53:41.552 DEBUG:boto:Data: 2024-04-12T03:53:41.552 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.552 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.552 DEBUG:boto:Params: {} 2024-04-12T03:53:41.552 DEBUG:boto:Token: None 2024-04-12T03:53:41.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74 2024-04-12T03:53:41.552 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z6h9pvkUrL47ku8XjoGXO2CPbAE= 2024-04-12T03:53:41.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z6h9pvkUrL47ku8XjoGXO2CPbAE=', 'Content-Length': '0'} 2024-04-12T03:53:41.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c9faf9a5d3f989-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.554 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:53:41.554 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:53:41.554 DEBUG:boto:Method: GET 2024-04-12T03:53:41.555 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:53:41.555 DEBUG:boto:Data: 2024-04-12T03:53:41.555 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.555 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.555 DEBUG:boto:Params: {} 2024-04-12T03:53:41.555 DEBUG:boto:Token: None 2024-04-12T03:53:41.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj74 2024-04-12T03:53:41.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z6h9pvkUrL47ku8XjoGXO2CPbAE= 2024-04-12T03:53:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z6h9pvkUrL47ku8XjoGXO2CPbAE=', 'Content-Length': '0'} 2024-04-12T03:53:41.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9234b73d3d3177-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.557 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.557 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.557 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.557 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.558 DEBUG:boto:Method: GET 2024-04-12T03:53:41.558 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.558 DEBUG:boto:Data: 2024-04-12T03:53:41.558 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.558 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.558 DEBUG:boto:Params: {} 2024-04-12T03:53:41.558 DEBUG:boto:Token: None 2024-04-12T03:53:41.558 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.558 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adfcd14a2a301d0b-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.561 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.561 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.561 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.561 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.561 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.561 DEBUG:boto:Method: GET 2024-04-12T03:53:41.561 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:53:41.561 DEBUG:boto:Data: 2024-04-12T03:53:41.561 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.562 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.562 DEBUG:boto:Params: {} 2024-04-12T03:53:41.562 DEBUG:boto:Token: None 2024-04-12T03:53:41.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.562 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000192f4862f9b5f119-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.565 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:53:41.566 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.566 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.566 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.566 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.566 DEBUG:boto:Method: GET 2024-04-12T03:53:41.566 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.566 DEBUG:boto:Data: 2024-04-12T03:53:41.566 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.566 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.567 DEBUG:boto:Params: {} 2024-04-12T03:53:41.567 DEBUG:boto:Token: None 2024-04-12T03:53:41.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4= 2024-04-12T03:53:41.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4=', 'Content-Length': '0'} 2024-04-12T03:53:41.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594ad0a5186620e1-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.569 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.569 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.569 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.569 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.569 DEBUG:boto:Method: GET 2024-04-12T03:53:41.569 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.569 DEBUG:boto:Data: 2024-04-12T03:53:41.569 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.570 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.570 DEBUG:boto:Params: {} 2024-04-12T03:53:41.570 DEBUG:boto:Token: None 2024-04-12T03:53:41.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.570 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4= 2024-04-12T03:53:41.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4=', 'Content-Length': '0'} 2024-04-12T03:53:41.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000843d2a2186412f4-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.572 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.572 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.572 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.572 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.572 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.572 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.572 DEBUG:boto:Data: 2024-04-12T03:53:41.572 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.572 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.572 DEBUG:boto:Params: {} 2024-04-12T03:53:41.573 DEBUG:boto:Token: None 2024-04-12T03:53:41.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.573 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:g9GxE7WmdXDVa8wa/UdagpDcZ+c= 2024-04-12T03:53:41.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:g9GxE7WmdXDVa8wa/UdagpDcZ+c=', 'Content-Length': '0'} 2024-04-12T03:53:41.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a889dc83c3b3c293-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.574 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.575 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.575 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.575 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.575 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.575 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.575 DEBUG:boto:Data: 2024-04-12T03:53:41.575 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.575 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.575 DEBUG:boto:Params: {} 2024-04-12T03:53:41.575 DEBUG:boto:Token: None 2024-04-12T03:53:41.576 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:g9GxE7WmdXDVa8wa/UdagpDcZ+c= 2024-04-12T03:53:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:g9GxE7WmdXDVa8wa/UdagpDcZ+c=', 'Content-Length': '0'} 2024-04-12T03:53:41.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ae63c4017887bb-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:53:41.578 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.578 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.578 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.578 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.578 DEBUG:boto:Method: GET 2024-04-12T03:53:41.578 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.578 DEBUG:boto:Data: 2024-04-12T03:53:41.578 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.578 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.578 DEBUG:boto:Params: {} 2024-04-12T03:53:41.578 DEBUG:boto:Token: None 2024-04-12T03:53:41.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4= 2024-04-12T03:53:41.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4=', 'Content-Length': '0'} 2024-04-12T03:53:41.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34d45309ec13abf-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.622 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.622 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.623 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.623 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.623 DEBUG:boto:Method: GET 2024-04-12T03:53:41.623 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.623 DEBUG:boto:Data: 2024-04-12T03:53:41.623 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.623 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.623 DEBUG:boto:Params: {} 2024-04-12T03:53:41.623 DEBUG:boto:Token: None 2024-04-12T03:53:41.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:53:41.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4= 2024-04-12T03:53:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tX3bMaWzxsa7Z9iGGb+i32yhri4=', 'Content-Length': '0'} 2024-04-12T03:53:41.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c23639dfe3566a7-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.626 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.626 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.626 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.626 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.626 DEBUG:boto:Method: GET 2024-04-12T03:53:41.626 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.626 DEBUG:boto:Data: 2024-04-12T03:53:41.626 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.626 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.626 DEBUG:boto:Params: {} 2024-04-12T03:53:41.627 DEBUG:boto:Token: None 2024-04-12T03:53:41.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043fbeff5129f0826-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.629 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.629 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.629 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.629 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.629 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.630 DEBUG:boto:Method: GET 2024-04-12T03:53:41.630 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.630 DEBUG:boto:Data: 2024-04-12T03:53:41.630 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.630 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.630 DEBUG:boto:Params: {} 2024-04-12T03:53:41.630 DEBUG:boto:Token: None 2024-04-12T03:53:41.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.630 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352c6caa9a73629f-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.635 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.635 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.635 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.635 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.635 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:53:41.635 DEBUG:boto:Data: 2024-04-12T03:53:41.635 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.635 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.636 DEBUG:boto:Params: {} 2024-04-12T03:53:41.636 DEBUG:boto:Token: None 2024-04-12T03:53:41.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76 2024-04-12T03:53:41.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7FewcLRifahYoe+uupBxya8R/XU= 2024-04-12T03:53:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7FewcLRifahYoe+uupBxya8R/XU=', 'Content-Length': '0'} 2024-04-12T03:53:41.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005adfc00cc3f58e91-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.638 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.638 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.638 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.638 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:53:41.638 DEBUG:boto:Data: 2024-04-12T03:53:41.638 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.638 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.639 DEBUG:boto:Params: {} 2024-04-12T03:53:41.639 DEBUG:boto:Token: None 2024-04-12T03:53:41.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76 2024-04-12T03:53:41.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7FewcLRifahYoe+uupBxya8R/XU= 2024-04-12T03:53:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7FewcLRifahYoe+uupBxya8R/XU=', 'Content-Length': '0'} 2024-04-12T03:53:41.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1bc2023c4d6cbf8-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:53:41.641 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.641 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.641 DEBUG:boto:Method: GET 2024-04-12T03:53:41.641 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:53:41.641 DEBUG:boto:Data: 2024-04-12T03:53:41.641 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.641 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.641 DEBUG:boto:Params: {} 2024-04-12T03:53:41.641 DEBUG:boto:Token: None 2024-04-12T03:53:41.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76 2024-04-12T03:53:41.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cQx7oTjBzg4baZrjl6/sAEVrUJM= 2024-04-12T03:53:41.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cQx7oTjBzg4baZrjl6/sAEVrUJM=', 'Content-Length': '0'} 2024-04-12T03:53:41.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ec731dc942af65-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.643 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:53:41.644 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:53:41.644 DEBUG:boto:Method: GET 2024-04-12T03:53:41.644 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:53:41.644 DEBUG:boto:Data: 2024-04-12T03:53:41.644 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.644 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.644 DEBUG:boto:Params: {} 2024-04-12T03:53:41.644 DEBUG:boto:Token: None 2024-04-12T03:53:41.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj76 2024-04-12T03:53:41.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cQx7oTjBzg4baZrjl6/sAEVrUJM= 2024-04-12T03:53:41.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cQx7oTjBzg4baZrjl6/sAEVrUJM=', 'Content-Length': '0'} 2024-04-12T03:53:41.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d280ccb2c20975a-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.646 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.647 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.647 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.647 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.647 DEBUG:boto:Method: GET 2024-04-12T03:53:41.647 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.647 DEBUG:boto:Data: 2024-04-12T03:53:41.647 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.647 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.647 DEBUG:boto:Params: {} 2024-04-12T03:53:41.647 DEBUG:boto:Token: None 2024-04-12T03:53:41.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.648 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b25965474fe48b-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.650 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.650 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.650 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.650 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.650 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.650 DEBUG:boto:Method: GET 2024-04-12T03:53:41.651 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:53:41.651 DEBUG:boto:Data: 2024-04-12T03:53:41.651 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.651 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.651 DEBUG:boto:Params: {} 2024-04-12T03:53:41.651 DEBUG:boto:Token: None 2024-04-12T03:53:41.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.651 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025256cd563831df7-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.698 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:53:41.699 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.699 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.699 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.699 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.699 DEBUG:boto:Method: GET 2024-04-12T03:53:41.699 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.699 DEBUG:boto:Data: 2024-04-12T03:53:41.699 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.699 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.700 DEBUG:boto:Params: {} 2024-04-12T03:53:41.700 DEBUG:boto:Token: None 2024-04-12T03:53:41.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y= 2024-04-12T03:53:41.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y=', 'Content-Length': '0'} 2024-04-12T03:53:41.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539d4b5ba908f74f-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.701 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.702 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.702 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.702 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.702 DEBUG:boto:Method: GET 2024-04-12T03:53:41.702 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.702 DEBUG:boto:Data: 2024-04-12T03:53:41.702 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.702 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.702 DEBUG:boto:Params: {} 2024-04-12T03:53:41.702 DEBUG:boto:Token: None 2024-04-12T03:53:41.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y= 2024-04-12T03:53:41.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y=', 'Content-Length': '0'} 2024-04-12T03:53:41.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd835e00c11722fe-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.705 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.705 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.705 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.705 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.705 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.705 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.705 DEBUG:boto:Data: 2024-04-12T03:53:41.705 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.705 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.705 DEBUG:boto:Params: {} 2024-04-12T03:53:41.706 DEBUG:boto:Token: None 2024-04-12T03:53:41.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wvmz23IHpz5wUuGAPAvG2x9ssZE= 2024-04-12T03:53:41.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wvmz23IHpz5wUuGAPAvG2x9ssZE=', 'Content-Length': '0'} 2024-04-12T03:53:41.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005672b89db1a396f-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.710 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.710 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.710 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.710 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.711 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.711 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.711 DEBUG:boto:Data: 2024-04-12T03:53:41.711 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.711 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.711 DEBUG:boto:Params: {} 2024-04-12T03:53:41.711 DEBUG:boto:Token: None 2024-04-12T03:53:41.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wvmz23IHpz5wUuGAPAvG2x9ssZE= 2024-04-12T03:53:41.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wvmz23IHpz5wUuGAPAvG2x9ssZE=', 'Content-Length': '0'} 2024-04-12T03:53:41.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000458d2cdb58c5e02-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:53:41.713 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.713 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.713 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.714 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.714 DEBUG:boto:Method: GET 2024-04-12T03:53:41.714 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.714 DEBUG:boto:Data: 2024-04-12T03:53:41.714 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.714 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.714 DEBUG:boto:Params: {} 2024-04-12T03:53:41.714 DEBUG:boto:Token: None 2024-04-12T03:53:41.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y= 2024-04-12T03:53:41.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y=', 'Content-Length': '0'} 2024-04-12T03:53:41.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe5bfe6419ee7fe-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.717 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.717 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.717 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.717 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.717 DEBUG:boto:Method: GET 2024-04-12T03:53:41.717 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.717 DEBUG:boto:Data: 2024-04-12T03:53:41.717 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.718 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.718 DEBUG:boto:Params: {} 2024-04-12T03:53:41.718 DEBUG:boto:Token: None 2024-04-12T03:53:41.718 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:53:41.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y= 2024-04-12T03:53:41.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mVi5LRvm0V60c1yc0y1tPqH3S8Y=', 'Content-Length': '0'} 2024-04-12T03:53:41.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1e54d3522e8524-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.720 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.720 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.720 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.720 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.721 DEBUG:boto:Method: GET 2024-04-12T03:53:41.721 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.721 DEBUG:boto:Data: 2024-04-12T03:53:41.721 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.721 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.721 DEBUG:boto:Params: {} 2024-04-12T03:53:41.721 DEBUG:boto:Token: None 2024-04-12T03:53:41.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b64ab6b36d19728f-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.724 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.724 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.724 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.725 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.725 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.725 DEBUG:boto:Method: GET 2024-04-12T03:53:41.725 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.725 DEBUG:boto:Data: 2024-04-12T03:53:41.725 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.725 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.725 DEBUG:boto:Params: {} 2024-04-12T03:53:41.725 DEBUG:boto:Token: None 2024-04-12T03:53:41.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003318277a212f8ef8-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.729 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.729 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.729 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.730 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.730 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:53:41.730 DEBUG:boto:Data: 2024-04-12T03:53:41.730 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.730 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.730 DEBUG:boto:Params: {} 2024-04-12T03:53:41.730 DEBUG:boto:Token: None 2024-04-12T03:53:41.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85 2024-04-12T03:53:41.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n+NrrrIYbtkSjM3zRuatLftQEVM= 2024-04-12T03:53:41.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n+NrrrIYbtkSjM3zRuatLftQEVM=', 'Content-Length': '0'} 2024-04-12T03:53:41.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c68c68393fd2cf-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.732 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.732 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.733 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.733 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:53:41.733 DEBUG:boto:Data: 2024-04-12T03:53:41.733 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.733 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.733 DEBUG:boto:Params: {} 2024-04-12T03:53:41.733 DEBUG:boto:Token: None 2024-04-12T03:53:41.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85 2024-04-12T03:53:41.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n+NrrrIYbtkSjM3zRuatLftQEVM= 2024-04-12T03:53:41.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n+NrrrIYbtkSjM3zRuatLftQEVM=', 'Content-Length': '0'} 2024-04-12T03:53:41.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad00f8d581ca3cd-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:53:41.736 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.736 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.736 DEBUG:boto:Method: GET 2024-04-12T03:53:41.736 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:53:41.736 DEBUG:boto:Data: 2024-04-12T03:53:41.736 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.736 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.736 DEBUG:boto:Params: {} 2024-04-12T03:53:41.736 DEBUG:boto:Token: None 2024-04-12T03:53:41.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85 2024-04-12T03:53:41.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I20x9FcXVRZlgypXXswTVYsvWew= 2024-04-12T03:53:41.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I20x9FcXVRZlgypXXswTVYsvWew=', 'Content-Length': '0'} 2024-04-12T03:53:41.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d9deba35f7a3d4-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.738 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:53:41.738 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:53:41.739 DEBUG:boto:Method: GET 2024-04-12T03:53:41.739 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:53:41.739 DEBUG:boto:Data: 2024-04-12T03:53:41.739 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.739 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.739 DEBUG:boto:Params: {} 2024-04-12T03:53:41.739 DEBUG:boto:Token: None 2024-04-12T03:53:41.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj85 2024-04-12T03:53:41.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I20x9FcXVRZlgypXXswTVYsvWew= 2024-04-12T03:53:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I20x9FcXVRZlgypXXswTVYsvWew=', 'Content-Length': '0'} 2024-04-12T03:53:41.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d45bc1025673b22-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.782 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.782 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.783 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.783 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.783 DEBUG:boto:Method: GET 2024-04-12T03:53:41.783 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.783 DEBUG:boto:Data: 2024-04-12T03:53:41.783 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.783 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.783 DEBUG:boto:Params: {} 2024-04-12T03:53:41.783 DEBUG:boto:Token: None 2024-04-12T03:53:41.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ded9f4e673563334-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.786 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.786 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.786 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.786 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.786 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.786 DEBUG:boto:Method: GET 2024-04-12T03:53:41.786 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:53:41.787 DEBUG:boto:Data: 2024-04-12T03:53:41.787 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.787 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.787 DEBUG:boto:Params: {} 2024-04-12T03:53:41.787 DEBUG:boto:Token: None 2024-04-12T03:53:41.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.787 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d83999c9fa5c40ab-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.791 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:53:41.791 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.792 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.792 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.792 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.792 DEBUG:boto:Method: GET 2024-04-12T03:53:41.792 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.792 DEBUG:boto:Data: 2024-04-12T03:53:41.792 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.792 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.792 DEBUG:boto:Params: {} 2024-04-12T03:53:41.792 DEBUG:boto:Token: None 2024-04-12T03:53:41.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68= 2024-04-12T03:53:41.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68=', 'Content-Length': '0'} 2024-04-12T03:53:41.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064869bdcf0acd43d-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.795 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.795 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.795 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.795 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.795 DEBUG:boto:Method: GET 2024-04-12T03:53:41.795 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.795 DEBUG:boto:Data: 2024-04-12T03:53:41.795 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.795 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.795 DEBUG:boto:Params: {} 2024-04-12T03:53:41.796 DEBUG:boto:Token: None 2024-04-12T03:53:41.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68= 2024-04-12T03:53:41.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68=', 'Content-Length': '0'} 2024-04-12T03:53:41.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b9517d7c44dd0f9-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.798 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.798 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.798 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.798 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.798 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.798 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.799 DEBUG:boto:Data: 2024-04-12T03:53:41.799 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.799 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.799 DEBUG:boto:Params: {} 2024-04-12T03:53:41.799 DEBUG:boto:Token: None 2024-04-12T03:53:41.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zjwym5ZcUHalxHfcExdvA3RLdOM= 2024-04-12T03:53:41.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zjwym5ZcUHalxHfcExdvA3RLdOM=', 'Content-Length': '0'} 2024-04-12T03:53:41.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4c21fbb968e637-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.801 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.801 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.801 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.801 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.801 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.802 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.802 DEBUG:boto:Data: 2024-04-12T03:53:41.802 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.802 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.802 DEBUG:boto:Params: {} 2024-04-12T03:53:41.802 DEBUG:boto:Token: None 2024-04-12T03:53:41.802 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.802 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zjwym5ZcUHalxHfcExdvA3RLdOM= 2024-04-12T03:53:41.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zjwym5ZcUHalxHfcExdvA3RLdOM=', 'Content-Length': '0'} 2024-04-12T03:53:41.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077792aee09fff596-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:53:41.804 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.804 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.805 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.805 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.805 DEBUG:boto:Method: GET 2024-04-12T03:53:41.805 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.805 DEBUG:boto:Data: 2024-04-12T03:53:41.805 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.805 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.805 DEBUG:boto:Params: {} 2024-04-12T03:53:41.805 DEBUG:boto:Token: None 2024-04-12T03:53:41.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68= 2024-04-12T03:53:41.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68=', 'Content-Length': '0'} 2024-04-12T03:53:41.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a392ff48012257c9-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.850 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.850 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.851 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.851 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.851 DEBUG:boto:Method: GET 2024-04-12T03:53:41.851 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.851 DEBUG:boto:Data: 2024-04-12T03:53:41.851 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.851 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.851 DEBUG:boto:Params: {} 2024-04-12T03:53:41.851 DEBUG:boto:Token: None 2024-04-12T03:53:41.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:53:41.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68= 2024-04-12T03:53:41.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Xw3RKaUi8GBk4t9X7zbDfmTG68=', 'Content-Length': '0'} 2024-04-12T03:53:41.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97942f54a5418f7-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.854 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.854 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.854 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.854 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.854 DEBUG:boto:Method: GET 2024-04-12T03:53:41.854 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.855 DEBUG:boto:Data: 2024-04-12T03:53:41.855 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.855 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.855 DEBUG:boto:Params: {} 2024-04-12T03:53:41.855 DEBUG:boto:Token: None 2024-04-12T03:53:41.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0caf72ed7666f7-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.858 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.858 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.858 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.858 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.858 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.858 DEBUG:boto:Method: GET 2024-04-12T03:53:41.859 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.859 DEBUG:boto:Data: 2024-04-12T03:53:41.859 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.859 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.859 DEBUG:boto:Params: {} 2024-04-12T03:53:41.859 DEBUG:boto:Token: None 2024-04-12T03:53:41.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfaa6020d6742273-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.863 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.863 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.863 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.863 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.863 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:53:41.863 DEBUG:boto:Data: 2024-04-12T03:53:41.863 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.864 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.864 DEBUG:boto:Params: {} 2024-04-12T03:53:41.864 DEBUG:boto:Token: None 2024-04-12T03:53:41.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87 2024-04-12T03:53:41.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gNf8moHjazZbOAmqt105z2uF58I= 2024-04-12T03:53:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gNf8moHjazZbOAmqt105z2uF58I=', 'Content-Length': '0'} 2024-04-12T03:53:41.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f7cd633fbb4f62a-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.866 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.866 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.866 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.866 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:53:41.867 DEBUG:boto:Data: 2024-04-12T03:53:41.867 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.867 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.867 DEBUG:boto:Params: {} 2024-04-12T03:53:41.867 DEBUG:boto:Token: None 2024-04-12T03:53:41.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87 2024-04-12T03:53:41.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gNf8moHjazZbOAmqt105z2uF58I= 2024-04-12T03:53:41.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gNf8moHjazZbOAmqt105z2uF58I=', 'Content-Length': '0'} 2024-04-12T03:53:41.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547bee12d6a7e7bf-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:53:41.869 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.869 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.869 DEBUG:boto:Method: GET 2024-04-12T03:53:41.869 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:53:41.869 DEBUG:boto:Data: 2024-04-12T03:53:41.870 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.870 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.870 DEBUG:boto:Params: {} 2024-04-12T03:53:41.870 DEBUG:boto:Token: None 2024-04-12T03:53:41.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87 2024-04-12T03:53:41.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vNqaAd1vH2ZPI3IlevRsn7FIIiI= 2024-04-12T03:53:41.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vNqaAd1vH2ZPI3IlevRsn7FIIiI=', 'Content-Length': '0'} 2024-04-12T03:53:41.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a04debb0380f38-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.872 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:53:41.872 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:53:41.872 DEBUG:boto:Method: GET 2024-04-12T03:53:41.872 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:53:41.872 DEBUG:boto:Data: 2024-04-12T03:53:41.873 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.873 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.873 DEBUG:boto:Params: {} 2024-04-12T03:53:41.873 DEBUG:boto:Token: None 2024-04-12T03:53:41.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj87 2024-04-12T03:53:41.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vNqaAd1vH2ZPI3IlevRsn7FIIiI= 2024-04-12T03:53:41.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vNqaAd1vH2ZPI3IlevRsn7FIIiI=', 'Content-Length': '0'} 2024-04-12T03:53:41.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5323fc117517c4-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.918 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.918 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.919 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.919 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.919 DEBUG:boto:Method: GET 2024-04-12T03:53:41.919 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.919 DEBUG:boto:Data: 2024-04-12T03:53:41.919 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.919 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.919 DEBUG:boto:Params: {} 2024-04-12T03:53:41.919 DEBUG:boto:Token: None 2024-04-12T03:53:41.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748938b7e7b24cf7-006618b045-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.922 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.922 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.922 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.923 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.923 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.923 DEBUG:boto:Method: GET 2024-04-12T03:53:41.923 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:53:41.923 DEBUG:boto:Data: 2024-04-12T03:53:41.923 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.923 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.923 DEBUG:boto:Params: {} 2024-04-12T03:53:41.923 DEBUG:boto:Token: None 2024-04-12T03:53:41.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dbd83faf352ede4-006618b045-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.927 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:53:41.928 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.928 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.928 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.928 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.928 DEBUG:boto:Method: GET 2024-04-12T03:53:41.929 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.929 DEBUG:boto:Data: 2024-04-12T03:53:41.929 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.929 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.929 DEBUG:boto:Params: {} 2024-04-12T03:53:41.929 DEBUG:boto:Token: None 2024-04-12T03:53:41.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.929 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM= 2024-04-12T03:53:41.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM=', 'Content-Length': '0'} 2024-04-12T03:53:41.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000765ac9b2ab9cde-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.932 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.932 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.932 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.932 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.932 DEBUG:boto:Method: GET 2024-04-12T03:53:41.932 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.932 DEBUG:boto:Data: 2024-04-12T03:53:41.932 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.933 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.933 DEBUG:boto:Params: {} 2024-04-12T03:53:41.933 DEBUG:boto:Token: None 2024-04-12T03:53:41.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM= 2024-04-12T03:53:41.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM=', 'Content-Length': '0'} 2024-04-12T03:53:41.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c3e4c97eb6055a-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.935 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.935 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.935 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.935 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.936 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.936 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.936 DEBUG:boto:Data: 2024-04-12T03:53:41.936 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.936 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.936 DEBUG:boto:Params: {} 2024-04-12T03:53:41.936 DEBUG:boto:Token: None 2024-04-12T03:53:41.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flhoRZ4+UQTXDDgZp/p0EoEK6bk= 2024-04-12T03:53:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flhoRZ4+UQTXDDgZp/p0EoEK6bk=', 'Content-Length': '0'} 2024-04-12T03:53:41.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9f4d48289adfbb-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.939 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.939 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.939 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.939 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.939 DEBUG:boto:Method: HEAD 2024-04-12T03:53:41.939 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.939 DEBUG:boto:Data: 2024-04-12T03:53:41.939 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.940 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.940 DEBUG:boto:Params: {} 2024-04-12T03:53:41.940 DEBUG:boto:Token: None 2024-04-12T03:53:41.940 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:flhoRZ4+UQTXDDgZp/p0EoEK6bk= 2024-04-12T03:53:41.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:flhoRZ4+UQTXDDgZp/p0EoEK6bk=', 'Content-Length': '0'} 2024-04-12T03:53:41.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b8d524e88305c4-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:53:41.942 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.942 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.942 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.943 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.943 DEBUG:boto:Method: GET 2024-04-12T03:53:41.943 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.943 DEBUG:boto:Data: 2024-04-12T03:53:41.943 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.943 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.943 DEBUG:boto:Params: {} 2024-04-12T03:53:41.943 DEBUG:boto:Token: None 2024-04-12T03:53:41.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM= 2024-04-12T03:53:41.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM=', 'Content-Length': '0'} 2024-04-12T03:53:41.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021add5de30b11203-006618b045-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.946 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:41.946 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:41.946 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.946 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.946 DEBUG:boto:Method: GET 2024-04-12T03:53:41.946 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.947 DEBUG:boto:Data: 2024-04-12T03:53:41.947 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.947 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.947 DEBUG:boto:Params: {} 2024-04-12T03:53:41.947 DEBUG:boto:Token: None 2024-04-12T03:53:41.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:53:41.947 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM= 2024-04-12T03:53:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B0+Whv5fYvh3ZfZeO54TLf6plsM=', 'Content-Length': '0'} 2024-04-12T03:53:41.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2651345ca182f01-006618b045-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.990 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.990 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.991 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.991 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.991 DEBUG:boto:Method: GET 2024-04-12T03:53:41.991 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.991 DEBUG:boto:Data: 2024-04-12T03:53:41.991 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:41.991 DEBUG:boto:Port: 8001 2024-04-12T03:53:41.991 DEBUG:boto:Params: {} 2024-04-12T03:53:41.992 DEBUG:boto:Token: None 2024-04-12T03:53:41.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071207d01d35de4dc-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:41.994 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:41.995 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:41.995 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:41.995 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.995 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.995 DEBUG:boto:Method: GET 2024-04-12T03:53:41.995 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:53:41.995 DEBUG:boto:Data: 2024-04-12T03:53:41.995 DEBUG:boto:Headers: {} 2024-04-12T03:53:41.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:41.996 DEBUG:boto:Port: 8000 2024-04-12T03:53:41.996 DEBUG:boto:Params: {} 2024-04-12T03:53:41.996 DEBUG:boto:Token: None 2024-04-12T03:53:41.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:41 GMT /dgbpxg-61/ 2024-04-12T03:53:41.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM= 2024-04-12T03:53:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gPim+DJZNnYQIGMPw7j2JWIEllM=', 'Content-Length': '0'} 2024-04-12T03:53:41.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122c12564e1fe4cb-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.000 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.000 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:42.000 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:42.000 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.000 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:53:42.000 DEBUG:boto:Data: 2024-04-12T03:53:42.000 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.001 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.001 DEBUG:boto:Params: {} 2024-04-12T03:53:42.001 DEBUG:boto:Token: None 2024-04-12T03:53:42.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj92 2024-04-12T03:53:42.001 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QeUlvJ94GX0aHc3X886lopqcJDw= 2024-04-12T03:53:42.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QeUlvJ94GX0aHc3X886lopqcJDw=', 'Content-Length': '0'} 2024-04-12T03:53:42.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037853fbf3c24281a-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.003 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:42.003 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:42.003 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.003 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:53:42.004 DEBUG:boto:Data: 2024-04-12T03:53:42.004 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.004 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.004 DEBUG:boto:Params: {} 2024-04-12T03:53:42.004 DEBUG:boto:Token: None 2024-04-12T03:53:42.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj92 2024-04-12T03:53:42.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QeUlvJ94GX0aHc3X886lopqcJDw= 2024-04-12T03:53:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QeUlvJ94GX0aHc3X886lopqcJDw=', 'Content-Length': '0'} 2024-04-12T03:53:42.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091841df0ab87270-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:53:42.006 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:42.007 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:42.007 DEBUG:boto:Method: GET 2024-04-12T03:53:42.007 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:53:42.007 DEBUG:boto:Data: 2024-04-12T03:53:42.007 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.007 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.007 DEBUG:boto:Params: {} 2024-04-12T03:53:42.007 DEBUG:boto:Token: None 2024-04-12T03:53:42.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj92 2024-04-12T03:53:42.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dic0k3XLaW6lhYBjjzILR9oWN8k= 2024-04-12T03:53:42.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dic0k3XLaW6lhYBjjzILR9oWN8k=', 'Content-Length': '0'} 2024-04-12T03:53:42.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008851e17e4e10ffca-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.050 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:53:42.050 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:53:42.051 DEBUG:boto:Method: GET 2024-04-12T03:53:42.051 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:53:42.051 DEBUG:boto:Data: 2024-04-12T03:53:42.051 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.051 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.051 DEBUG:boto:Params: {} 2024-04-12T03:53:42.051 DEBUG:boto:Token: None 2024-04-12T03:53:42.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj92 2024-04-12T03:53:42.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dic0k3XLaW6lhYBjjzILR9oWN8k= 2024-04-12T03:53:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dic0k3XLaW6lhYBjjzILR9oWN8k=', 'Content-Length': '0'} 2024-04-12T03:53:42.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d681db5474c91071-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.054 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.054 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.054 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.054 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.054 DEBUG:boto:Method: GET 2024-04-12T03:53:42.054 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.054 DEBUG:boto:Data: 2024-04-12T03:53:42.054 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.055 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.055 DEBUG:boto:Params: {} 2024-04-12T03:53:42.055 DEBUG:boto:Token: None 2024-04-12T03:53:42.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.055 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea77de7d9d844b6a-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.058 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.058 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.058 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.058 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.058 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.058 DEBUG:boto:Method: GET 2024-04-12T03:53:42.058 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:53:42.059 DEBUG:boto:Data: 2024-04-12T03:53:42.059 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.059 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.059 DEBUG:boto:Params: {} 2024-04-12T03:53:42.059 DEBUG:boto:Token: None 2024-04-12T03:53:42.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c8e361a66df824-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.063 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:53:42.064 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.064 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.064 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.064 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.064 DEBUG:boto:Method: GET 2024-04-12T03:53:42.064 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.064 DEBUG:boto:Data: 2024-04-12T03:53:42.064 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.064 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.065 DEBUG:boto:Params: {} 2024-04-12T03:53:42.065 DEBUG:boto:Token: None 2024-04-12T03:53:42.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.065 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM= 2024-04-12T03:53:42.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM=', 'Content-Length': '0'} 2024-04-12T03:53:42.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004482cda8bd0df913-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.067 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.067 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.067 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.067 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.067 DEBUG:boto:Method: GET 2024-04-12T03:53:42.068 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.068 DEBUG:boto:Data: 2024-04-12T03:53:42.068 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.068 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.068 DEBUG:boto:Params: {} 2024-04-12T03:53:42.068 DEBUG:boto:Token: None 2024-04-12T03:53:42.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM= 2024-04-12T03:53:42.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM=', 'Content-Length': '0'} 2024-04-12T03:53:42.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008690c8a07c867577-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.114 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.114 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.115 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.115 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.115 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.115 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.115 DEBUG:boto:Data: 2024-04-12T03:53:42.115 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.115 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.115 DEBUG:boto:Params: {} 2024-04-12T03:53:42.116 DEBUG:boto:Token: None 2024-04-12T03:53:42.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cqEY9sgQtuxEWBLvoMFP1KTFDg4= 2024-04-12T03:53:42.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cqEY9sgQtuxEWBLvoMFP1KTFDg4=', 'Content-Length': '0'} 2024-04-12T03:53:42.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b7b513bec760ef-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.118 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.118 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.118 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.118 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.118 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.118 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.118 DEBUG:boto:Data: 2024-04-12T03:53:42.118 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.119 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.119 DEBUG:boto:Params: {} 2024-04-12T03:53:42.119 DEBUG:boto:Token: None 2024-04-12T03:53:42.119 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cqEY9sgQtuxEWBLvoMFP1KTFDg4= 2024-04-12T03:53:42.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cqEY9sgQtuxEWBLvoMFP1KTFDg4=', 'Content-Length': '0'} 2024-04-12T03:53:42.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078cab5f2a5b5b563-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:53:42.121 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.122 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.122 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.122 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.122 DEBUG:boto:Method: GET 2024-04-12T03:53:42.122 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.122 DEBUG:boto:Data: 2024-04-12T03:53:42.122 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.122 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.122 DEBUG:boto:Params: {} 2024-04-12T03:53:42.123 DEBUG:boto:Token: None 2024-04-12T03:53:42.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM= 2024-04-12T03:53:42.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM=', 'Content-Length': '0'} 2024-04-12T03:53:42.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a0e9aa4815c4405-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.125 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.125 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.125 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.125 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.125 DEBUG:boto:Method: GET 2024-04-12T03:53:42.125 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.125 DEBUG:boto:Data: 2024-04-12T03:53:42.125 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.126 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.126 DEBUG:boto:Params: {} 2024-04-12T03:53:42.126 DEBUG:boto:Token: None 2024-04-12T03:53:42.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:53:42.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM= 2024-04-12T03:53:42.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpMHk6ltypBxWyS0VHAFdkoM8iM=', 'Content-Length': '0'} 2024-04-12T03:53:42.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa0ba5863d3965c-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.128 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.128 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.129 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.129 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.129 DEBUG:boto:Method: GET 2024-04-12T03:53:42.129 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.129 DEBUG:boto:Data: 2024-04-12T03:53:42.129 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.129 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.129 DEBUG:boto:Params: {} 2024-04-12T03:53:42.130 DEBUG:boto:Token: None 2024-04-12T03:53:42.130 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa2867f6fcfda1b-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.132 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.133 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.133 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.133 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.133 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.133 DEBUG:boto:Method: GET 2024-04-12T03:53:42.133 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.133 DEBUG:boto:Data: 2024-04-12T03:53:42.133 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.134 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.134 DEBUG:boto:Params: {} 2024-04-12T03:53:42.134 DEBUG:boto:Token: None 2024-04-12T03:53:42.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc1cc9bd1b9a946-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.138 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.138 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.138 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.138 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.139 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:53:42.139 DEBUG:boto:Data: 2024-04-12T03:53:42.139 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.139 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.139 DEBUG:boto:Params: {} 2024-04-12T03:53:42.139 DEBUG:boto:Token: None 2024-04-12T03:53:42.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93 2024-04-12T03:53:42.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qpAbhF7nyNFBPE8LXO7l4c02EKc= 2024-04-12T03:53:42.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qpAbhF7nyNFBPE8LXO7l4c02EKc=', 'Content-Length': '0'} 2024-04-12T03:53:42.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139ec7ef6c35af81-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.141 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.142 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.142 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.142 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:53:42.142 DEBUG:boto:Data: 2024-04-12T03:53:42.142 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.142 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.142 DEBUG:boto:Params: {} 2024-04-12T03:53:42.142 DEBUG:boto:Token: None 2024-04-12T03:53:42.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93 2024-04-12T03:53:42.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qpAbhF7nyNFBPE8LXO7l4c02EKc= 2024-04-12T03:53:42.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qpAbhF7nyNFBPE8LXO7l4c02EKc=', 'Content-Length': '0'} 2024-04-12T03:53:42.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e94bb1812d158d-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:53:42.144 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.145 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.145 DEBUG:boto:Method: GET 2024-04-12T03:53:42.145 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:53:42.145 DEBUG:boto:Data: 2024-04-12T03:53:42.145 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.145 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.145 DEBUG:boto:Params: {} 2024-04-12T03:53:42.145 DEBUG:boto:Token: None 2024-04-12T03:53:42.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93 2024-04-12T03:53:42.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qJwwgeqzZDsVPStCzvtJbHL49uU= 2024-04-12T03:53:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qJwwgeqzZDsVPStCzvtJbHL49uU=', 'Content-Length': '0'} 2024-04-12T03:53:42.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ebadadba009775-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.148 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:53:42.148 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:53:42.148 DEBUG:boto:Method: GET 2024-04-12T03:53:42.148 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:53:42.148 DEBUG:boto:Data: 2024-04-12T03:53:42.148 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.148 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.148 DEBUG:boto:Params: {} 2024-04-12T03:53:42.149 DEBUG:boto:Token: None 2024-04-12T03:53:42.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj93 2024-04-12T03:53:42.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qJwwgeqzZDsVPStCzvtJbHL49uU= 2024-04-12T03:53:42.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qJwwgeqzZDsVPStCzvtJbHL49uU=', 'Content-Length': '0'} 2024-04-12T03:53:42.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c6bca9a000ae2f-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.151 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.151 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.151 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.151 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.151 DEBUG:boto:Method: GET 2024-04-12T03:53:42.151 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.151 DEBUG:boto:Data: 2024-04-12T03:53:42.151 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.152 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.152 DEBUG:boto:Params: {} 2024-04-12T03:53:42.152 DEBUG:boto:Token: None 2024-04-12T03:53:42.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81acf1d4fbf7598-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.154 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.154 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.154 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.155 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.155 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.155 DEBUG:boto:Method: GET 2024-04-12T03:53:42.155 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:53:42.155 DEBUG:boto:Data: 2024-04-12T03:53:42.155 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.155 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.155 DEBUG:boto:Params: {} 2024-04-12T03:53:42.155 DEBUG:boto:Token: None 2024-04-12T03:53:42.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7af54df311f5ea6-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.160 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:53:42.160 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.160 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.160 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.161 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.161 DEBUG:boto:Method: GET 2024-04-12T03:53:42.161 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.161 DEBUG:boto:Data: 2024-04-12T03:53:42.161 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.161 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.161 DEBUG:boto:Params: {} 2024-04-12T03:53:42.161 DEBUG:boto:Token: None 2024-04-12T03:53:42.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY= 2024-04-12T03:53:42.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY=', 'Content-Length': '0'} 2024-04-12T03:53:42.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003de6506d3fbc0af1-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.164 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.164 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.164 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.164 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.165 DEBUG:boto:Method: GET 2024-04-12T03:53:42.165 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.165 DEBUG:boto:Data: 2024-04-12T03:53:42.165 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.165 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.165 DEBUG:boto:Params: {} 2024-04-12T03:53:42.165 DEBUG:boto:Token: None 2024-04-12T03:53:42.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY= 2024-04-12T03:53:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY=', 'Content-Length': '0'} 2024-04-12T03:53:42.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca13880868450bb9-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.210 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.210 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.211 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.211 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.211 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.211 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.211 DEBUG:boto:Data: 2024-04-12T03:53:42.211 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.211 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.211 DEBUG:boto:Params: {} 2024-04-12T03:53:42.211 DEBUG:boto:Token: None 2024-04-12T03:53:42.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/HaERyxKpHhw7On1S6W99ru2Ls= 2024-04-12T03:53:42.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/HaERyxKpHhw7On1S6W99ru2Ls=', 'Content-Length': '0'} 2024-04-12T03:53:42.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667d8dc9dfdaca1d-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.214 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.214 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.214 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.215 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.215 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.215 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.215 DEBUG:boto:Data: 2024-04-12T03:53:42.215 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.215 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.215 DEBUG:boto:Params: {} 2024-04-12T03:53:42.215 DEBUG:boto:Token: None 2024-04-12T03:53:42.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6/HaERyxKpHhw7On1S6W99ru2Ls= 2024-04-12T03:53:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6/HaERyxKpHhw7On1S6W99ru2Ls=', 'Content-Length': '0'} 2024-04-12T03:53:42.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67ded1db0bfad0d-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:53:42.218 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.218 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.218 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.218 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.218 DEBUG:boto:Method: GET 2024-04-12T03:53:42.219 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.219 DEBUG:boto:Data: 2024-04-12T03:53:42.219 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.219 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.219 DEBUG:boto:Params: {} 2024-04-12T03:53:42.219 DEBUG:boto:Token: None 2024-04-12T03:53:42.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY= 2024-04-12T03:53:42.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY=', 'Content-Length': '0'} 2024-04-12T03:53:42.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a3bb4d14305a61-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.221 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.221 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.221 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.222 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.222 DEBUG:boto:Method: GET 2024-04-12T03:53:42.222 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.222 DEBUG:boto:Data: 2024-04-12T03:53:42.222 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.222 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.222 DEBUG:boto:Params: {} 2024-04-12T03:53:42.223 DEBUG:boto:Token: None 2024-04-12T03:53:42.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:53:42.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY= 2024-04-12T03:53:42.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ST1jyHxlRdNiJbNyHfQ/Mu2o7QY=', 'Content-Length': '0'} 2024-04-12T03:53:42.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca97935f203d96c1-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.225 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.225 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.225 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.225 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.225 DEBUG:boto:Method: GET 2024-04-12T03:53:42.225 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.225 DEBUG:boto:Data: 2024-04-12T03:53:42.225 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.226 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.226 DEBUG:boto:Params: {} 2024-04-12T03:53:42.226 DEBUG:boto:Token: None 2024-04-12T03:53:42.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0b93a5dbe6b6993-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.228 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.229 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.229 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.229 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.229 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.229 DEBUG:boto:Method: GET 2024-04-12T03:53:42.229 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.229 DEBUG:boto:Data: 2024-04-12T03:53:42.229 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.230 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.230 DEBUG:boto:Params: {} 2024-04-12T03:53:42.230 DEBUG:boto:Token: None 2024-04-12T03:53:42.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a092b25ffbb11fff-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.234 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.234 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.234 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.235 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.235 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:53:42.235 DEBUG:boto:Data: 2024-04-12T03:53:42.235 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.235 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.235 DEBUG:boto:Params: {} 2024-04-12T03:53:42.235 DEBUG:boto:Token: None 2024-04-12T03:53:42.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95 2024-04-12T03:53:42.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K+LbO0Zl0wLN6BH275qzVmTFpaY= 2024-04-12T03:53:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K+LbO0Zl0wLN6BH275qzVmTFpaY=', 'Content-Length': '0'} 2024-04-12T03:53:42.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005657dac4ea596b1b-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.237 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.238 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.238 DEBUG:boto:Method: HEAD 2024-04-12T03:53:42.238 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:53:42.238 DEBUG:boto:Data: 2024-04-12T03:53:42.238 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.238 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.238 DEBUG:boto:Params: {} 2024-04-12T03:53:42.238 DEBUG:boto:Token: None 2024-04-12T03:53:42.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95 2024-04-12T03:53:42.239 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K+LbO0Zl0wLN6BH275qzVmTFpaY= 2024-04-12T03:53:42.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K+LbO0Zl0wLN6BH275qzVmTFpaY=', 'Content-Length': '0'} 2024-04-12T03:53:42.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e27eabc44d6a80-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:53:42.241 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.241 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.241 DEBUG:boto:Method: GET 2024-04-12T03:53:42.241 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:53:42.241 DEBUG:boto:Data: 2024-04-12T03:53:42.242 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.242 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.242 DEBUG:boto:Params: {} 2024-04-12T03:53:42.242 DEBUG:boto:Token: None 2024-04-12T03:53:42.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95 2024-04-12T03:53:42.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q0zaQcjzNdnhoLKQO8vxSut0pUc= 2024-04-12T03:53:42.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q0zaQcjzNdnhoLKQO8vxSut0pUc=', 'Content-Length': '0'} 2024-04-12T03:53:42.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025020336ba3a7a82-006618b046-1e69-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.244 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:53:42.244 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:53:42.244 DEBUG:boto:Method: GET 2024-04-12T03:53:42.245 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:53:42.245 DEBUG:boto:Data: 2024-04-12T03:53:42.245 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.245 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.245 DEBUG:boto:Params: {} 2024-04-12T03:53:42.245 DEBUG:boto:Token: None 2024-04-12T03:53:42.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/obj95 2024-04-12T03:53:42.245 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q0zaQcjzNdnhoLKQO8vxSut0pUc= 2024-04-12T03:53:42.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q0zaQcjzNdnhoLKQO8vxSut0pUc=', 'Content-Length': '0'} 2024-04-12T03:53:42.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e9ea71aa983b8b-006618b046-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.247 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.247 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.248 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.248 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.248 DEBUG:boto:Method: GET 2024-04-12T03:53:42.248 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.248 DEBUG:boto:Data: 2024-04-12T03:53:42.248 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:42.248 DEBUG:boto:Port: 8001 2024-04-12T03:53:42.248 DEBUG:boto:Params: {} 2024-04-12T03:53:42.248 DEBUG:boto:Token: None 2024-04-12T03:53:42.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.249 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc34ec78bc57dfe-006618b046-1e69-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.251 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.251 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:53:42.251 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:53:42.252 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.252 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.252 DEBUG:boto:Method: GET 2024-04-12T03:53:42.252 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:53:42.252 DEBUG:boto:Data: 2024-04-12T03:53:42.252 DEBUG:boto:Headers: {} 2024-04-12T03:53:42.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:53:42.252 DEBUG:boto:Port: 8000 2024-04-12T03:53:42.252 DEBUG:boto:Params: {} 2024-04-12T03:53:42.252 DEBUG:boto:Token: None 2024-04-12T03:53:42.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:53:42 GMT /dgbpxg-61/ 2024-04-12T03:53:42.253 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw= 2024-04-12T03:53:42.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:53:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIdc7IxLx+srLOlJ9/V7CHSlIUw=', 'Content-Length': '0'} 2024-04-12T03:53:42.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017be81d8d983edbb-006618b046-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:53:42.256 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:53:42.257 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:53:42.257 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:53:42.258 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:53:42.259 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:53:42.268+0000 7fd707979700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 96303) UID: 0 2024-04-12T03:53:42.259 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:53:42.268+0000 7fd70cb4e7c0 -1 shutting down 2024-04-12T03:53:43.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:53:48.357 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:53:48.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:48.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:48.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:53:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:48.417 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:48.452 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:53:48.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.452 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2024-04-12T03:53:48.452 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.473626Z", 2024-04-12T03:53:48.453 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_sBwlytARB_7ir9I31SYdW6AMFT0OttR", 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.454 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.455 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.770710Z", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.456 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_paqsyPETZRigWQovc_o0mZ0QkA4W4Br", 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.742039Z", 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.458 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_r7Xaij1NYXE9WlAj-owWK6CL2q7r8Ka", 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.460 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.709165Z", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.461 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_107K4tYJlvjqMY8-2cxP9VsT1DwRbm9", 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:48.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.544771Z", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.463 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Scx30NKoMQfwKoFORyelnFXfLstFjHT", 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.464 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.761554Z", 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QJ9YbiXLyxU49bJjd6FmwNIkecJyLLt", 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.781521Z", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Lft1H9M14mYJ7nBdf0cunOkeyxr7zqB", 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.593478Z", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6XcuTvmMiDr5oBOpAdHuxXjL3D3D3rr", 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2024-04-12T03:53:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.529275Z", 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.472 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CuACATy9SFYzXlYc0KvanMsEd4yiPu5", 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.473 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.474 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.830619Z", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.475 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8ihvuCkDYUeMWYsxzLlJHbgqEZAyrFK", 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.476 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.404399Z", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.477 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_YZv6OVpO4Iott2AuA7_FrEcuVX0JIzB", 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.479 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.483912Z", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.480 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3rd2J76ZniJ4BCGJhYjc_BD6cQ7-E6k", 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.481 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.620794Z", 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.482 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6tENwjvAcT30YxLJYt9qI_fDdJrnwo_", 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.484 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.638506Z", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uTD15Epta9qKa3P83ObYwZ-eX-oG5iI", 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.486 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.469990Z", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.487 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_eAcZ_TL5pRbfmS_m98CzHX3nsWCbRYY", 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.491639Z", 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.489 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_sPGdz32xBzYCw7gvZ2ZMtLzEt3vkmXD", 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.491 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.539684Z", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rsKv3cbixQXpPRwy_DP1R0IwZ22jtzX", 2024-04-12T03:53:48.492 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 50 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.493 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.712363Z", 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_WHo3HP81uYxWt4VJEv-4JrR0MkIliSl", 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.495 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.415260Z", 2024-04-12T03:53:48.496 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_slR_QkSZS5P66OVu3T5fSikjnXkldiT", 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.497 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 48 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.498 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.500835Z", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.499 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_k53wgSI1QmBkbZEgPVK9ZFbCIEGiZ0M", 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.500 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.672236Z", 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6II9MDvQ02SUI843cvgbke3efIX8wXS", 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.502 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.503 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.590168Z", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bvKR9DnNMmzuu9-qpO95US51rsN3Yxb", 2024-04-12T03:53:48.504 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 47 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.548708Z", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.506 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Cm3dI3UmFNGfhFFnj7stlpUgdh0Rhc1", 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.507 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.469997Z", 2024-04-12T03:53:48.508 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_i1LmwGkefxC6EORUxegEccLEQHZ2jb3", 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.509 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 49 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.510 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.650218Z", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.511 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nCX3m-2KPd1cUMfeNi5ZLh5ENnSTxIc", 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.512 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.481252Z", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.513 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_jvYUl_8rIGR4JjooibZ6VBeq7Q9V7KO", 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.601182Z", 2024-04-12T03:53:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_K8VziXOZVsN0rQRH5usFXFYaUm2__g3", 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.517 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.682821Z", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zaE7t71MZGd6nAuWnhHWTb7OWCZAYx3", 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.519 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.723084Z", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.520 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_B_CjwUSEsLN-SHfDvWNHNF76sc_CPcL", 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.521 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.478755Z", 2024-04-12T03:53:48.522 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_dPKPsbFGEpDd_4AC-p2kHkkM0T1Kt0k", 2024-04-12T03:53:48.523 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.524 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.649780Z", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.525 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UeXYngTH0JFCMZCslxQPoDkgw-vGyIN", 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.527 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:48.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.527 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.527 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.698268Z", 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8pEb-dJd6mLZQl34KFN2t02Hl5rjJ6A", 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.530 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.731924Z", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.531 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Trprd31ZwfEM612LhwW_DMEhGjbiVxJ", 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.531093Z", 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.533 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ScXaEApgYozY55iXxwJ0NDo-QmnWtzy", 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:48.535 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.525811Z", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_5u7fDvK-MOZAbJ12wohcLH6aos1550b", 2024-04-12T03:53:48.536 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:48.537 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:48.537 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:48.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:53:48.537 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:53:48.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.538 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:48.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:48.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:48.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.679 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:48.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:48.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:48.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.819 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:48.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:48.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:48.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:48.959 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:53:49.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.097 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.235 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.372 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.510 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.559 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.559 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.652 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.792 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:49.842 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:49.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:49.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:49.963 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.012 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.013 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.105 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.155 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.155 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.243 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.521 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.658 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.795 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:50.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:50.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.072 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.211 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.351 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.491 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.628 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.767 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:51.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:51.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.046 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.324 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.462 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.600 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.743 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:52.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:52.881 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:52.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:52.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.017 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.069 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.069 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.159 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.297 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.435 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.483 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.483 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.534 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:53:53.631 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:53:53.684 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:53.684 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:53.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.764+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-04-12T03:53:53.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.764+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:53.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.770+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: init 2024-04-12T03:53:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.782+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:53.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:53.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:53.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj116[0] 2024-04-12T03:53:53.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj124[0] 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj145[0] 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj148[0] 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj17[0] 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:53.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:53.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.827+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:53.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.828+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.829+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.829+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.831+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.831+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.831+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.831+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.832+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.832+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.832+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.832+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj55[0] 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj57[0] 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:53.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:53.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.833+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj62[0] 2024-04-12T03:53:53.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj68[0] 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:53.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.852+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: success 2024-04-12T03:53:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj76[0] 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj85[0] 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: success 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj87[0] 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.854+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: success 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: success 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj93[0] 2024-04-12T03:53:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.856+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.857+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: success 2024-04-12T03:53:53.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.861+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:53.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.862+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: success 2024-04-12T03:53:53.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.884+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: success 2024-04-12T03:53:53.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.885+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:53.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.885+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: success 2024-04-12T03:53:53.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.885+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: success 2024-04-12T03:53:53.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.885+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:53.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.886+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:53.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.888+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:53.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.897+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:53.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.901+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: success 2024-04-12T03:53:53.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.901+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: success 2024-04-12T03:53:53.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.903+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: success 2024-04-12T03:53:53.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.903+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:53.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.903+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:53.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.904+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: success 2024-04-12T03:53:53.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.904+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj11 2024-04-12T03:53:53.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.906+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: backing out with sync_status=-5 2024-04-12T03:53:53.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.906+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: ERROR: full sync failed. error: -5 2024-04-12T03:53:53.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.907+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T03:53:53.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.907+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:53.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.911+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: full 2024-04-12T03:53:53.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.924+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:53.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:53.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:53.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj111[0] 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj116[0] 2024-04-12T03:53:53.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj124[0] 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj145[0] 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj148[0] 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:53.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj155[0] 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj17[0] 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:53.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj22[0] 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj29[0] 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:53.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj34[0] 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:53.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.969+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:53.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.970+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:53.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:53.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:53.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:53.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:53.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.974+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj55[0] 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: success 2024-04-12T03:53:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:53.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: success 2024-04-12T03:53:53.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:53.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj57[0] 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.975+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj62[0] 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:53.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:53.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj85[0] 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:53.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj93[0] 2024-04-12T03:53:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.976+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.981+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.982+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.983+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.983+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj148 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.983+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: success 2024-04-12T03:53:53.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.983+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: success 2024-04-12T03:53:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.986+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: success 2024-04-12T03:53:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.986+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: success 2024-04-12T03:53:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.986+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj62 2024-04-12T03:53:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.988+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: backing out with sync_status=-5 2024-04-12T03:53:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.988+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: ERROR: full sync failed. error: -5 2024-04-12T03:53:53.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.989+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T03:53:53.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:53.989+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:54.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.034+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: full 2024-04-12T03:53:54.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.045+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj111[0] 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj124[0] 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj145[0] 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj148[0] 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj17[0] 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.090+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj22[0] 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj29[0] 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:54.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj34[0] 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:54.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.091+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:54.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: success 2024-04-12T03:53:54.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj115 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.096+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj55[0] 2024-04-12T03:53:54.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj57[0] 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj68[0] 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj76[0] 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:54.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.097+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.099+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.100+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.100+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: success 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.101+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.101+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: success 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: success 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: success 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: backing out with sync_status=-5 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.102+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: ERROR: full sync failed. error: -5 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.104+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.104+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:54.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.149+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: full 2024-04-12T03:53:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.161+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj116[0] 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj145[0] 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj155[0] 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:54.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj22[0] 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.207+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj29[0] 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:54.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj34[0] 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:54.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.208+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:54.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.210+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:54.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.210+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:54.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.210+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.210+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:54.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.212+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj62[0] 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj68[0] 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.213+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: success 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: success 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: success 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj76[0] 2024-04-12T03:53:54.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj85[0] 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj87[0] 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj93[0] 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: success 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.214+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: success 2024-04-12T03:53:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.215+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.215+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.215+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.215+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.215+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.216+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.217+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:54.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: success 2024-04-12T03:53:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.218+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj11 2024-04-12T03:53:54.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.219+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: backing out with sync_status=-5 2024-04-12T03:53:54.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.219+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: ERROR: full sync failed. error: -5 2024-04-12T03:53:54.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.221+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.221+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:54.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.266+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: full 2024-04-12T03:53:54.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.277+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:54.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:54.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj111[0] 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj116[0] 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj124[0] 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:54.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj148[0] 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj155[0] 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj17[0] 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:54.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.322+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj22[0] 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj34[0] 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.323+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: success 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: success 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.326+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj55[0] 2024-04-12T03:53:54.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj57[0] 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: success 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.327+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:54.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj62[0] 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: success 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj68[0] 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: success 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj124 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj76[0] 2024-04-12T03:53:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 0 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: ERROR: failed to sync object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.328+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj87[0] 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj93[0] 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.329+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.331+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: success 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.331+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.331+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.331+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: success 2024-04-12T03:53:54.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.332+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: a sync operation returned error 2024-04-12T03:53:54.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.332+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: success 2024-04-12T03:53:54.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.332+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: success 2024-04-12T03:53:54.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.332+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:54.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.333+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:54.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.333+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: success 2024-04-12T03:53:54.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.333+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: success 2024-04-12T03:53:54.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: success 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: success 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: backing out with sync_status=-5 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.334+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: ERROR: full sync failed. error: -5 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.336+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.336+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: waiting on bucket lease 2024-04-12T03:53:54.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.380+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: status after acquiring the lock is: full 2024-04-12T03:53:54.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.391+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bucket for full sync 2024-04-12T03:53:54.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj11 2024-04-12T03:53:54.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj11 op=0 op_state=1 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj11[0] 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj111 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj111 op=0 op_state=1 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj111[0] 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj115 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj115 op=0 op_state=1 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj115[0] 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj116 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj116 op=0 op_state=1 2024-04-12T03:53:54.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj116[0] 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj124 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj124 op=0 op_state=1 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.436+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj124[0] 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj145 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj145 op=0 op_state=1 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj145[0] 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj148 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj148 op=0 op_state=1 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj148[0] 2024-04-12T03:53:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj155 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj155 op=0 op_state=1 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj155[0] 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj17 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj17 op=0 op_state=1 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj17[0] 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj185 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj185 op=0 op_state=1 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj185[0] 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj19 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj19 op=0 op_state=1 2024-04-12T03:53:54.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj19[0] 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj22 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj22 op=0 op_state=1 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj22[0] 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj23 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj23 op=0 op_state=1 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj23[0] 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj29 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj29 op=0 op_state=1 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj29[0] 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj31 2024-04-12T03:53:54.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj31 op=0 op_state=1 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj31[0] 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj33 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj33 op=0 op_state=1 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj33[0] 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj34 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj34 op=0 op_state=1 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj34[0] 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj37 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj37 op=0 op_state=1 2024-04-12T03:53:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj37[0] 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj38 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj38 op=0 op_state=1 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj38[0] 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj44 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj44 op=0 op_state=1 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj44[0] 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj46 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj46 op=0 op_state=1 2024-04-12T03:53:54.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.437+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj46[0] 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.440+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj11]: success 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.440+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj51 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.440+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj51 op=0 op_state=1 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.440+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj51[0] 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj116]: success 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj145]: success 2024-04-12T03:53:54.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj111]: success 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj22]: success 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj55 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj55 op=0 op_state=1 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj55[0] 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj57 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj57 op=0 op_state=1 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj57[0] 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj58 2024-04-12T03:53:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj58 op=0 op_state=1 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj58[0] 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj62 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj62 op=0 op_state=1 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.441+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj62[0] 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj23]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj31]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj124]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj155]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj19]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj37]: success 2024-04-12T03:53:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj148]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj29]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj34]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj38]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj115]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj155 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj68 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj68 op=0 op_state=1 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj68[0] 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj17]: success 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj69 2024-04-12T03:53:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj69 op=0 op_state=1 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj69[0] 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj74 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj74 op=0 op_state=1 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj46]: success 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj74[0] 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj44]: success 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj185]: success 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj76 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj76 op=0 op_state=1 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.443+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj76[0] 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj85 2024-04-12T03:53:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj85 op=0 op_state=1 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj85[0] 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj87 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj87 op=0 op_state=1 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj87[0] 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj92 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj92 op=0 op_state=1 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj92[0] 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj33]: success 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj93 2024-04-12T03:53:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj93 op=0 op_state=1 2024-04-12T03:53:54.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj93[0] 2024-04-12T03:53:54.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: [full sync] syncing object: dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0/obj95 2024-04-12T03:53:54.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b=bucket sync single entry (source_zone=b7d2416c-9d6f-4780-9e14-1f66614caba8) b= log_entry=obj95 op=0 op_state=1 2024-04-12T03:53:54.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.444+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: bucket sync: sync obj: b7d2416c-9d6f-4780-9e14-1f66614caba8/:dgbpxg-61[b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24])/obj95[0] 2024-04-12T03:53:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.445+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj51]: success 2024-04-12T03:53:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.445+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj55]: success 2024-04-12T03:53:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj62]: success 2024-04-12T03:53:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj57]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj74]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj68]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj85]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj93]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj76]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj87]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj92]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.447+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj69]: success 2024-04-12T03:53:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.448+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj58]: success 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.448+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj93 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.448+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:entry[obj95]: success 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.448+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: updating marker oid=bucket.full-sync-status.b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24 marker=obj95 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.450+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: set bucket state=incremental 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.451+0000 7f692f258780 5 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:full_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: bucket status objv={r=_rhhwJDbxj3Ck9uu1H0iBaGH:150,w=:0} 2024-04-12T03:53:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.452+0000 7f692f258780 10 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.454+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.454+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.454+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.454+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:1]: sync status for source bucket shard: 2 2024-04-12T03:53:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:1]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:1]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: sync status for source bucket shard: 2 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:0]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:4]: sync status for source bucket shard: 2 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:2]: sync status for source bucket shard: 2 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:4]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:4]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:2]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:2]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.455+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:3]: sync status for source bucket shard: 2 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:3]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:3]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:5]: sync status for source bucket shard: 2 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:5]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:5]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:6]: sync status for source bucket shard: 2 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:6]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:6]: listing bilog for incremental sync; position=00000000002.112.6 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:8]: sync status for source bucket shard: 2 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.456+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:8]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:8]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:7]: sync status for source bucket shard: 2 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:7]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:7]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:9]: sync status for source bucket shard: 2 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:10]: sync status for source bucket shard: 2 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:9]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:9]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:10]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:10]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:11]: sync status for source bucket shard: 2 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:11]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:11]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.457+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:13]: sync status for source bucket shard: 2 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:13]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:13]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:12]: sync status for source bucket shard: 2 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:14]: sync status for source bucket shard: 2 2024-04-12T03:53:54.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:12]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:12]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.458+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:14]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:14]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:21]: sync status for source bucket shard: 2 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:19]: sync status for source bucket shard: 2 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:21]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:21]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:19]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:19]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:23]: sync status for source bucket shard: 2 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.459+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:23]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:23]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:24]: sync status for source bucket shard: 2 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:24]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:24]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:27]: sync status for source bucket shard: 2 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.460+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:27]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:27]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:29]: sync status for source bucket shard: 2 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:29]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:29]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:31]: sync status for source bucket shard: 2 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:31]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:31]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.461+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:16]: sync status for source bucket shard: 2 2024-04-12T03:53:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:16]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:16]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:20]: sync status for source bucket shard: 2 2024-04-12T03:53:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:22]: sync status for source bucket shard: 2 2024-04-12T03:53:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:33]: sync status for source bucket shard: 2 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:20]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:20]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:22]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:22]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:33]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:33]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:34]: sync status for source bucket shard: 2 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:18]: sync status for source bucket shard: 2 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:17]: sync status for source bucket shard: 2 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:25]: sync status for source bucket shard: 2 2024-04-12T03:53:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:28]: sync status for source bucket shard: 2 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.462+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:34]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:34]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.463+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:18]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:18]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.463+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:17]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:17]: listing bilog for incremental sync; position=00000000002.61.6 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.463+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:25]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:25]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.463+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:28]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:28]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:26]: sync status for source bucket shard: 2 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:26]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:26]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:32]: sync status for source bucket shard: 2 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:15]: sync status for source bucket shard: 2 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:35]: sync status for source bucket shard: 2 2024-04-12T03:53:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:36]: sync status for source bucket shard: 2 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:30]: sync status for source bucket shard: 2 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:32]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:32]: listing bilog for incremental sync; position=00000000001.75.6 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:15]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:15]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:35]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:35]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:36]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:36]: listing bilog for incremental sync; position=00000000001.99.6 2024-04-12T03:53:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:54.465+0000 7f692f258780 20 RGW-SYNC:bucket[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24<-dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:30]:inc_sync[dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24:30]: listing bilog for incremental sync; position= 2024-04-12T03:53:54.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:54.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:54.520 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:53:54.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:54.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.473626Z", 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.17311443982319269538", 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2024-04-12T03:53:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.770710Z", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.9876502991265771055", 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.742039Z", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15044618720932212595", 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.709165Z", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3048303128727154563", 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2024-04-12T03:53:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.544771Z", 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.12496463252935782756", 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.761554Z", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1240661724997294327", 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.781521Z", 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4345025821208541068", 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2024-04-12T03:53:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 18 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.593478Z", 2024-04-12T03:53:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1396374685028368619", 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.621 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.622 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.529275Z", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.623 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.12416343696140222752", 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.830619Z", 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.625 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.13559126115052071446", 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.626 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.627 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.404399Z", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.18259525660668444708", 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.628 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.629 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.483912Z", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.630 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.8405780414609670411", 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.620794Z", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16090202877019810517", 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 26 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.638506Z", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10145254304165908788", 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.469990Z", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.6558746615492843501", 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.638 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.491639Z", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.639 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4174951181312081480", 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 23 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.539684Z", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.641 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10207189132069419922", 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2024-04-12T03:53:54.642 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 42 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.712363Z", 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.643 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3714451509324739036", 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.644 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.415260Z", 2024-04-12T03:53:54.645 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.8369147918085052901", 2024-04-12T03:53:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 39 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.647 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.500835Z", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.648 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3760839505550298275", 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.649 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.672236Z", 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.650 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2375768318007011382", 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.651 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.652 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.590168Z", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.3128782847118704909", 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 40 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.548708Z", 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.655 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.10696544926560521518", 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.657 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.469997Z", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.658 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.12716904440739784146", 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 41 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.659 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.650218Z", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.660 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.11851406009016018142", 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.481252Z", 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.662 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.4597458724845434471", 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.664 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.601182Z", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2054852750209983851", 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.666 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.682821Z", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.667 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.13683965889156283711", 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.668 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.723084Z", 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.669 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.15103071078348402784", 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.671 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.478755Z", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.602436322121578310", 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.672 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.673 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.674 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.649780Z", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.675 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14667789499887167780", 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.676 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.698268Z", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.677 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.5811448462002331822", 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.678 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.731924Z", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.679 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.2503706896610079359", 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.531093Z", 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.681 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16041827042704813280", 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.683 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.525811Z", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.11810253346162634686", 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.685 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:53:54.685 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:53:54.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:54.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:53:54.686 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:53:54.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:54.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.768 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.473626Z", 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.769 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_A0E3egzioy9JzE67klR-J0gP0uCeY07", 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.771 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.770710Z", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.772 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SuedP1hRACxOE-E-ShfJwXGxA1NLC8T", 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.742039Z", 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.774 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_5a7W3gD242mKuGf0UmwPA1vSlT3KNat", 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.776 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.709165Z", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.777 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QO79-34EtB4JgojHCDXRZeGA29BXsam", 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.778 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.544771Z", 2024-04-12T03:53:54.779 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_P0ivOSjy64CO2b9xkIA-lPli6QQ4Tya", 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.780 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.781 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.761554Z", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BGxFfddnwO3K04Oyp4khFf_DCuKLw1H", 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.783 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.781521Z", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_HA0lFR6WdBqCmb55VlVM1IJzW0k8Bpx", 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.593478Z", 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VxqFnnNMiSHvKCyRGMBeT3MyLIAvQLk", 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.529275Z", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GdRy0Dk4AS8kTX82226ZEviU9p4CydT", 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.791 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.830619Z", 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.792 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NlGkN-5RtJbBZ3YcgtXFuguJRQ6SOqN", 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2024-04-12T03:53:54.793 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.794 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.404399Z", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SnivFh5Y2prAO2-XtJHmEITgVQALpLR", 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.796 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.483912Z", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.797 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_cTo0L6xtLKuxGQwMLIzw6Ptmor-JV3e", 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.798 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.799 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.620794Z", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_mnGWYyKwPn6QWBnnpGbEgBzKJaitRA9", 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.801 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.638506Z", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_m5ljV0uFLGmjZh_nu04IkHEioqYY1_k", 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.803 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.469990Z", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.804 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PaHOUCSv0XBrlFeP8ZDHziEEIS_FZKE", 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.491639Z", 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.806 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tMq3dnrpZVuWj2uZc5xk6-RyPa28s0Y", 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.539684Z", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_HYsKQ-kYtpisf5bVhQI7Iq0tUbtiWbT", 2024-04-12T03:53:54.809 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 55 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.712363Z", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MBUDtOBTQ_RGwkgYnMLgi9ndrX5gK_S", 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.415260Z", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.813 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1vDDtlMbnmXxmbQyospEWCHp05Oup9t", 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 56 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.815 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.500835Z", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.816 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RQfHf91OWHAhWcevozz9o9oLZnOuzaP", 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.672236Z", 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.818 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npW7GxzKnQXo6hXNE-FLEY7AvhbJhdQ", 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.819 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.590168Z", 2024-04-12T03:53:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qq2hSfv57vnbEYnNpylNdFzsi3D8jqr", 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.821 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 57 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.822 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:40:51.548708Z", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_W5C6pLxY96PwajIyL_Ml7Z0TMnx82DD", 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.824 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.469997Z", 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.825 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Bh07se4L9yRRwapJa2V12MvQAQtW2Yb", 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.826 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 58 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.827 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.650218Z", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6yWoOS-EjvLAYS6yNMJ532kOB-T1gfw", 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2024-04-12T03:53:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.481252Z", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_mxTDm9ewtGIIR5PXYmIUZkKQft-8CaR", 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.601182Z", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kbFFm33d0pmej53J6xgLBtexEeBaORM", 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.682821Z", 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hwmmAuX_zY9Jzlyvo0ZJpbLrSqCmRmW", 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:51:42.723084Z", 2024-04-12T03:53:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NIdpiqSmKk4LFMJmwRQ7fqRT1PAZdZ7", 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.478755Z", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ssOGXAvtg-W9i70oKUg3-S8NBIvmDhr", 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:41:32.649780Z", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ItVGDcMAETWI964CgrUKj0ftOZV5mb0", 2024-04-12T03:53:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.844 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.844 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:47:19.698268Z", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_cXlFhbx6Fl5m6Toq4fEgE_Lp3Y0QLcX", 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:46:16.731924Z", 2024-04-12T03:53:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_2EXpZdSsmkVGAV8ZFyAjEh5QvfJc789", 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:50:18.531093Z", 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-A1mn4CPm3dJ4e2Vpaz3Ck0X3iK-XWb", 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2024-04-12T03:53:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2024-04-12T03:43:54.525811Z", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:53:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1qhKQOeXEiTCkjyRnGs4MWhp7Vro5Ax", 2024-04-12T03:53:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2024-04-12T03:53:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2024-04-12T03:53:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2024-04-12T03:53:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:53:54.854 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:53:54.854 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:53:54.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:53:54.857 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:53:54.857 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:53:54.877 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:53:54.877 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:53:54.878 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:53:55.879 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:53:55.905 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:53:55.905 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:53:55.905 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:53:55.906 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T03:53:55.906 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-61 2024-04-12T03:53:55.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:55.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:55.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:53:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:56.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:56.056+0000 7efdebfb2780 1 bucket sync caught up with source: 2024-04-12T03:53:56.048 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:56.048 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.112.6, , , , , , 00000000002.73.6, , , , , 00000000002.61.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.99.6] 2024-04-12T03:53:56.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:56.056+0000 7efdebfb2780 0 bucket checkpoint complete 2024-04-12T03:53:56.063 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-61 2024-04-12T03:53:56.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:56.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:53:56.064 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:53:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:53:56.119 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:53:56.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:53:56.161+0000 7fb02d6ee780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:56.153 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1083.6, 00000000002.2097.6, 00000000006.1948.6, 00000000005.166.6, 00000000004.448.6, 00000000001.445.6, 00000000003.870.6, 00000000004.440.6, 00000000007.2253.6, 00000000006.1504.6] 2024-04-12T03:53:56.153 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000009.1099.6, 00000000006.2116.6, 00000000006.1948.6, 00000000015.188.6, 00000000012.473.6, 00000000003.458.6, 00000000003.870.6, 00000000012.458.6, 00000000021.2283.6, 00000000018.1530.6] 2024-04-12T03:54:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:01.163+0000 7fb02d6ee780 1 bucket sync caught up with source: 2024-04-12T03:54:01.155 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000009.1099.6, 00000000006.2116.6, 00000000006.1948.6, 00000000015.188.6, 00000000012.473.6, 00000000003.458.6, 00000000003.870.6, 00000000012.458.6, 00000000021.2283.6, 00000000018.1530.6] 2024-04-12T03:54:01.155 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000009.1099.6, 00000000006.2116.6, 00000000006.1948.6, 00000000015.188.6, 00000000012.473.6, 00000000003.458.6, 00000000003.870.6, 00000000012.458.6, 00000000021.2283.6, 00000000018.1530.6] 2024-04-12T03:54:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:01.163+0000 7fb02d6ee780 0 bucket checkpoint complete 2024-04-12T03:54:01.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:54:01.172 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.172 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.172 DEBUG:boto:Method: HEAD 2024-04-12T03:54:01.172 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:54:01.172 DEBUG:boto:Data: 2024-04-12T03:54:01.172 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:01.172 DEBUG:boto:Port: 8001 2024-04-12T03:54:01.172 DEBUG:boto:Params: {} 2024-04-12T03:54:01.173 DEBUG:boto:Token: None 2024-04-12T03:54:01.173 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/ 2024-04-12T03:54:01.173 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU= 2024-04-12T03:54:01.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU=', 'Content-Length': '0'} 2024-04-12T03:54:01.174 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-12T03:54:01.174 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:54:01.877 DEBUG:boto:Token: None 2024-04-12T03:54:01.877 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/ 2024-04-12T03:54:01.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU= 2024-04-12T03:54:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU=', 'Content-Length': '0'} 2024-04-12T03:54:01.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e76c54aad4a7784c-006618b059-20c9-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.909 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.909 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.909 DEBUG:boto:Method: HEAD 2024-04-12T03:54:01.910 DEBUG:boto:Path: /dgbpxg-61/ 2024-04-12T03:54:01.910 DEBUG:boto:Data: 2024-04-12T03:54:01.910 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:01.910 DEBUG:boto:Port: 8000 2024-04-12T03:54:01.910 DEBUG:boto:Params: {} 2024-04-12T03:54:01.910 DEBUG:boto:Token: None 2024-04-12T03:54:01.910 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/ 2024-04-12T03:54:01.910 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU= 2024-04-12T03:54:01.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kMOgff7K4/lZBCoGvTonTXnANVU=', 'Content-Length': '0'} 2024-04-12T03:54:01.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fb366fdf7922ec6-006618b059-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:54:01.915 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.915 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.915 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.915 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.915 DEBUG:boto:Method: GET 2024-04-12T03:54:01.915 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.915 DEBUG:boto:Data: 2024-04-12T03:54:01.915 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:01.916 DEBUG:boto:Port: 8001 2024-04-12T03:54:01.916 DEBUG:boto:Params: {} 2024-04-12T03:54:01.916 DEBUG:boto:Token: None 2024-04-12T03:54:01.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/?versions 2024-04-12T03:54:01.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs= 2024-04-12T03:54:01.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs=', 'Content-Length': '0'} 2024-04-12T03:54:01.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061bd6a1bae0e6af2-006618b059-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.919 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T03:54:01.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:54:01.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:54:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:54:01.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:54:01.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:54:01.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:54:01.924 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:54:01.924 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.924 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.924 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.924 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.924 DEBUG:boto:Method: GET 2024-04-12T03:54:01.924 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.925 DEBUG:boto:Data: 2024-04-12T03:54:01.925 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:01.925 DEBUG:boto:Port: 8000 2024-04-12T03:54:01.925 DEBUG:boto:Params: {} 2024-04-12T03:54:01.925 DEBUG:boto:Token: None 2024-04-12T03:54:01.925 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/?versions 2024-04-12T03:54:01.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs= 2024-04-12T03:54:01.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs=', 'Content-Length': '0'} 2024-04-12T03:54:01.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000adaeda02ae1dd08-006618b059-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.930 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T03:54:01.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:54:01.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T03:54:01.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T03:54:01.935 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.935 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.935 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.935 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.935 DEBUG:boto:Method: GET 2024-04-12T03:54:01.935 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.935 DEBUG:boto:Data: 2024-04-12T03:54:01.935 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:01.935 DEBUG:boto:Port: 8001 2024-04-12T03:54:01.935 DEBUG:boto:Params: {} 2024-04-12T03:54:01.935 DEBUG:boto:Token: None 2024-04-12T03:54:01.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/?versions 2024-04-12T03:54:01.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs= 2024-04-12T03:54:01.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs=', 'Content-Length': '0'} 2024-04-12T03:54:01.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d3d5632f6d7b571-006618b059-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.939 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:01.940 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:01.941 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:01.941 DEBUG:boto:path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.941 DEBUG:boto:auth_path=/dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.941 DEBUG:boto:Method: GET 2024-04-12T03:54:01.941 DEBUG:boto:Path: /dgbpxg-61/?versions&max-keys=999 2024-04-12T03:54:01.941 DEBUG:boto:Data: 2024-04-12T03:54:01.941 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:01.941 DEBUG:boto:Port: 8000 2024-04-12T03:54:01.941 DEBUG:boto:Params: {} 2024-04-12T03:54:01.941 DEBUG:boto:Token: None 2024-04-12T03:54:01.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/?versions 2024-04-12T03:54:01.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs= 2024-04-12T03:54:01.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qlihbKjuLGI4OACLDWxdzy3ojTs=', 'Content-Length': '0'} 2024-04-12T03:54:01.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144c5e49f25375dc-006618b059-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.950 DEBUG:boto:b'dgbpxg-61999falseobj11nulltrue2024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:01.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:54:01.952 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:01.952 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:01.952 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.952 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.952 DEBUG:boto:Method: GET 2024-04-12T03:54:01.952 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.952 DEBUG:boto:Data: 2024-04-12T03:54:01.952 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:01.953 DEBUG:boto:Port: 8001 2024-04-12T03:54:01.953 DEBUG:boto:Params: {} 2024-04-12T03:54:01.953 DEBUG:boto:Token: None 2024-04-12T03:54:01.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.953 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ysWENvLJhzpxjWiKxOTrYwrIAbs= 2024-04-12T03:54:01.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ysWENvLJhzpxjWiKxOTrYwrIAbs=', 'Content-Length': '0'} 2024-04-12T03:54:01.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b604e80b202b7c12-006618b059-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:01.998 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:01.998 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:01.999 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.999 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.999 DEBUG:boto:Method: GET 2024-04-12T03:54:01.999 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:01.999 DEBUG:boto:Data: 2024-04-12T03:54:01.999 DEBUG:boto:Headers: {} 2024-04-12T03:54:01.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:01.999 DEBUG:boto:Port: 8000 2024-04-12T03:54:01.999 DEBUG:boto:Params: {} 2024-04-12T03:54:01.999 DEBUG:boto:Token: None 2024-04-12T03:54:01.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:01 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ysWENvLJhzpxjWiKxOTrYwrIAbs= 2024-04-12T03:54:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:01 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ysWENvLJhzpxjWiKxOTrYwrIAbs=', 'Content-Length': '0'} 2024-04-12T03:54:02.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed86ef80359380c6-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.002 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.002 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.002 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.002 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.002 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.003 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.003 DEBUG:boto:Data: 2024-04-12T03:54:02.003 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.003 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.003 DEBUG:boto:Params: {} 2024-04-12T03:54:02.003 DEBUG:boto:Token: None 2024-04-12T03:54:02.003 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k+Fh4MibCuHNqof4ithB8FynB0c= 2024-04-12T03:54:02.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k+Fh4MibCuHNqof4ithB8FynB0c=', 'Content-Length': '0'} 2024-04-12T03:54:02.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b6b2b96248675f-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.005 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.006 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.006 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.006 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.006 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.006 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.006 DEBUG:boto:Data: 2024-04-12T03:54:02.006 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.006 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.006 DEBUG:boto:Params: {} 2024-04-12T03:54:02.007 DEBUG:boto:Token: None 2024-04-12T03:54:02.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k+Fh4MibCuHNqof4ithB8FynB0c= 2024-04-12T03:54:02.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k+Fh4MibCuHNqof4ithB8FynB0c=', 'Content-Length': '0'} 2024-04-12T03:54:02.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c868584753593793-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:54:02.009 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.009 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.009 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.009 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.009 DEBUG:boto:Method: GET 2024-04-12T03:54:02.009 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.009 DEBUG:boto:Data: 2024-04-12T03:54:02.010 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.010 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.010 DEBUG:boto:Params: {} 2024-04-12T03:54:02.010 DEBUG:boto:Token: None 2024-04-12T03:54:02.010 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.010 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxZV8hQcsePz/IgJhIzR5lFh3Yw= 2024-04-12T03:54:02.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxZV8hQcsePz/IgJhIzR5lFh3Yw=', 'Content-Length': '0'} 2024-04-12T03:54:02.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056cd0dc40552a649-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.012 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.013 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.013 DEBUG:boto:path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.013 DEBUG:boto:auth_path=/dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.013 DEBUG:boto:Method: GET 2024-04-12T03:54:02.013 DEBUG:boto:Path: /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.013 DEBUG:boto:Data: 2024-04-12T03:54:02.013 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.013 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.013 DEBUG:boto:Params: {} 2024-04-12T03:54:02.014 DEBUG:boto:Token: None 2024-04-12T03:54:02.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11?versionId=null 2024-04-12T03:54:02.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PxZV8hQcsePz/IgJhIzR5lFh3Yw= 2024-04-12T03:54:02.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PxZV8hQcsePz/IgJhIzR5lFh3Yw=', 'Content-Length': '0'} 2024-04-12T03:54:02.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ab1627dbce5994-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.016 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.016 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.017 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.017 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.017 DEBUG:boto:Method: GET 2024-04-12T03:54:02.017 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.017 DEBUG:boto:Data: 2024-04-12T03:54:02.017 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.017 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.017 DEBUG:boto:Params: {} 2024-04-12T03:54:02.017 DEBUG:boto:Token: None 2024-04-12T03:54:02.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.018 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565e2cada817537c-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.020 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.020 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.021 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.021 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.021 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.021 DEBUG:boto:Method: GET 2024-04-12T03:54:02.021 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.021 DEBUG:boto:Data: 2024-04-12T03:54:02.021 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.021 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.021 DEBUG:boto:Params: {} 2024-04-12T03:54:02.021 DEBUG:boto:Token: None 2024-04-12T03:54:02.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.022 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a053b2cb2d2293-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.026 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.026 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.027 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.027 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.027 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:54:02.027 DEBUG:boto:Data: 2024-04-12T03:54:02.027 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.027 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.027 DEBUG:boto:Params: {} 2024-04-12T03:54:02.027 DEBUG:boto:Token: None 2024-04-12T03:54:02.027 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11 2024-04-12T03:54:02.028 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NlSV+XgUz5vZqJzUrpXURZ+VUPo= 2024-04-12T03:54:02.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NlSV+XgUz5vZqJzUrpXURZ+VUPo=', 'Content-Length': '0'} 2024-04-12T03:54:02.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd443b3ac00b7172-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.030 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.030 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.030 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.030 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:54:02.030 DEBUG:boto:Data: 2024-04-12T03:54:02.030 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.030 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.030 DEBUG:boto:Params: {} 2024-04-12T03:54:02.030 DEBUG:boto:Token: None 2024-04-12T03:54:02.031 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11 2024-04-12T03:54:02.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NlSV+XgUz5vZqJzUrpXURZ+VUPo= 2024-04-12T03:54:02.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NlSV+XgUz5vZqJzUrpXURZ+VUPo=', 'Content-Length': '0'} 2024-04-12T03:54:02.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a8418c682f44ee-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:54:02.033 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.033 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.033 DEBUG:boto:Method: GET 2024-04-12T03:54:02.033 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:54:02.033 DEBUG:boto:Data: 2024-04-12T03:54:02.033 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.034 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.034 DEBUG:boto:Params: {} 2024-04-12T03:54:02.034 DEBUG:boto:Token: None 2024-04-12T03:54:02.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11 2024-04-12T03:54:02.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SmyTtZ9imjmr6TXcQCqgM8cAt0s= 2024-04-12T03:54:02.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SmyTtZ9imjmr6TXcQCqgM8cAt0s=', 'Content-Length': '0'} 2024-04-12T03:54:02.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2186cb89ace0e4-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.036 DEBUG:boto:path=/dgbpxg-61/obj11 2024-04-12T03:54:02.036 DEBUG:boto:auth_path=/dgbpxg-61/obj11 2024-04-12T03:54:02.036 DEBUG:boto:Method: GET 2024-04-12T03:54:02.036 DEBUG:boto:Path: /dgbpxg-61/obj11 2024-04-12T03:54:02.037 DEBUG:boto:Data: 2024-04-12T03:54:02.037 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.037 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.037 DEBUG:boto:Params: {} 2024-04-12T03:54:02.037 DEBUG:boto:Token: None 2024-04-12T03:54:02.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj11 2024-04-12T03:54:02.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SmyTtZ9imjmr6TXcQCqgM8cAt0s= 2024-04-12T03:54:02.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SmyTtZ9imjmr6TXcQCqgM8cAt0s=', 'Content-Length': '0'} 2024-04-12T03:54:02.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3284b75e84a531-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.039 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.040 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.040 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.040 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.040 DEBUG:boto:Method: GET 2024-04-12T03:54:02.040 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.040 DEBUG:boto:Data: 2024-04-12T03:54:02.040 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.040 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.040 DEBUG:boto:Params: {} 2024-04-12T03:54:02.040 DEBUG:boto:Token: None 2024-04-12T03:54:02.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030ddb10511ab6fdd-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.043 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.043 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.044 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.044 DEBUG:boto:path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.044 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.044 DEBUG:boto:Method: GET 2024-04-12T03:54:02.044 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj11 2024-04-12T03:54:02.044 DEBUG:boto:Data: 2024-04-12T03:54:02.044 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.044 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.044 DEBUG:boto:Params: {} 2024-04-12T03:54:02.044 DEBUG:boto:Token: None 2024-04-12T03:54:02.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d8e9bfd252e19c-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.049 DEBUG:boto:b'dgbpxg-61obj111000falseobj112024-04-12T03:40:51.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:54:02.049 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.049 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.050 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.050 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.050 DEBUG:boto:Method: GET 2024-04-12T03:54:02.050 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.050 DEBUG:boto:Data: 2024-04-12T03:54:02.050 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.050 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.051 DEBUG:boto:Params: {} 2024-04-12T03:54:02.051 DEBUG:boto:Token: None 2024-04-12T03:54:02.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g= 2024-04-12T03:54:02.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g=', 'Content-Length': '0'} 2024-04-12T03:54:02.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7054a0ddc93629-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.098 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.099 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.099 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.099 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.099 DEBUG:boto:Method: GET 2024-04-12T03:54:02.099 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.099 DEBUG:boto:Data: 2024-04-12T03:54:02.099 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.099 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.099 DEBUG:boto:Params: {} 2024-04-12T03:54:02.100 DEBUG:boto:Token: None 2024-04-12T03:54:02.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g= 2024-04-12T03:54:02.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g=', 'Content-Length': '0'} 2024-04-12T03:54:02.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a49b9313338517-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.103 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.103 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.103 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.103 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.103 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.103 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.103 DEBUG:boto:Data: 2024-04-12T03:54:02.103 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.103 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.104 DEBUG:boto:Params: {} 2024-04-12T03:54:02.104 DEBUG:boto:Token: None 2024-04-12T03:54:02.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vy2aRmxwzVbidMpkSNoFcXCwNi4= 2024-04-12T03:54:02.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vy2aRmxwzVbidMpkSNoFcXCwNi4=', 'Content-Length': '0'} 2024-04-12T03:54:02.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa71c448fc56d41f-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.106 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.106 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.106 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.106 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.106 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.106 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.107 DEBUG:boto:Data: 2024-04-12T03:54:02.107 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.107 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.107 DEBUG:boto:Params: {} 2024-04-12T03:54:02.107 DEBUG:boto:Token: None 2024-04-12T03:54:02.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.107 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Vy2aRmxwzVbidMpkSNoFcXCwNi4= 2024-04-12T03:54:02.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Vy2aRmxwzVbidMpkSNoFcXCwNi4=', 'Content-Length': '0'} 2024-04-12T03:54:02.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3af4be90486e170-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:54:02.110 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.110 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.110 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.110 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.110 DEBUG:boto:Method: GET 2024-04-12T03:54:02.110 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.110 DEBUG:boto:Data: 2024-04-12T03:54:02.110 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.110 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.111 DEBUG:boto:Params: {} 2024-04-12T03:54:02.111 DEBUG:boto:Token: None 2024-04-12T03:54:02.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g= 2024-04-12T03:54:02.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g=', 'Content-Length': '0'} 2024-04-12T03:54:02.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ced0e470ad4c46e-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.113 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.113 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.113 DEBUG:boto:path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.113 DEBUG:boto:auth_path=/dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.113 DEBUG:boto:Method: GET 2024-04-12T03:54:02.114 DEBUG:boto:Path: /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.114 DEBUG:boto:Data: 2024-04-12T03:54:02.114 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.114 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.114 DEBUG:boto:Params: {} 2024-04-12T03:54:02.114 DEBUG:boto:Token: None 2024-04-12T03:54:02.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111?versionId=null 2024-04-12T03:54:02.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g= 2024-04-12T03:54:02.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6+ieWysRbcYUe3GNgDlDUgSRH+g=', 'Content-Length': '0'} 2024-04-12T03:54:02.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9c5db4bd26c44f-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.117 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.117 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.117 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.117 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.117 DEBUG:boto:Method: GET 2024-04-12T03:54:02.117 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.117 DEBUG:boto:Data: 2024-04-12T03:54:02.117 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.118 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.118 DEBUG:boto:Params: {} 2024-04-12T03:54:02.118 DEBUG:boto:Token: None 2024-04-12T03:54:02.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.118 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a563d000624cd6d5-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.120 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.121 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.121 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.121 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.121 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.121 DEBUG:boto:Method: GET 2024-04-12T03:54:02.121 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.121 DEBUG:boto:Data: 2024-04-12T03:54:02.121 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.122 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.122 DEBUG:boto:Params: {} 2024-04-12T03:54:02.122 DEBUG:boto:Token: None 2024-04-12T03:54:02.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.122 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a499ee725a43bc-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.126 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.127 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.127 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.127 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.127 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:54:02.127 DEBUG:boto:Data: 2024-04-12T03:54:02.127 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.127 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.127 DEBUG:boto:Params: {} 2024-04-12T03:54:02.128 DEBUG:boto:Token: None 2024-04-12T03:54:02.128 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111 2024-04-12T03:54:02.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uy6HVk8J2+X0M8RlN+vwacqIdWw= 2024-04-12T03:54:02.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uy6HVk8J2+X0M8RlN+vwacqIdWw=', 'Content-Length': '0'} 2024-04-12T03:54:02.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0ecf85107b66a6-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.130 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.130 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.130 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.130 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:54:02.130 DEBUG:boto:Data: 2024-04-12T03:54:02.130 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.130 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.131 DEBUG:boto:Params: {} 2024-04-12T03:54:02.131 DEBUG:boto:Token: None 2024-04-12T03:54:02.131 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111 2024-04-12T03:54:02.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Uy6HVk8J2+X0M8RlN+vwacqIdWw= 2024-04-12T03:54:02.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Uy6HVk8J2+X0M8RlN+vwacqIdWw=', 'Content-Length': '0'} 2024-04-12T03:54:02.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012910e05181af123-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T03:54:02.133 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.133 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.133 DEBUG:boto:Method: GET 2024-04-12T03:54:02.133 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:54:02.133 DEBUG:boto:Data: 2024-04-12T03:54:02.133 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.134 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.134 DEBUG:boto:Params: {} 2024-04-12T03:54:02.134 DEBUG:boto:Token: None 2024-04-12T03:54:02.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111 2024-04-12T03:54:02.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LTVWIhstbqn0LdPFdvCDMmbPLz4= 2024-04-12T03:54:02.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LTVWIhstbqn0LdPFdvCDMmbPLz4=', 'Content-Length': '0'} 2024-04-12T03:54:02.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047dd019eddb7e39f-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.136 DEBUG:boto:path=/dgbpxg-61/obj111 2024-04-12T03:54:02.136 DEBUG:boto:auth_path=/dgbpxg-61/obj111 2024-04-12T03:54:02.136 DEBUG:boto:Method: GET 2024-04-12T03:54:02.136 DEBUG:boto:Path: /dgbpxg-61/obj111 2024-04-12T03:54:02.137 DEBUG:boto:Data: 2024-04-12T03:54:02.137 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.137 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.137 DEBUG:boto:Params: {} 2024-04-12T03:54:02.137 DEBUG:boto:Token: None 2024-04-12T03:54:02.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj111 2024-04-12T03:54:02.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LTVWIhstbqn0LdPFdvCDMmbPLz4= 2024-04-12T03:54:02.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LTVWIhstbqn0LdPFdvCDMmbPLz4=', 'Content-Length': '0'} 2024-04-12T03:54:02.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096316a05b38c1d56-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.182 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.183 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.183 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.183 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.183 DEBUG:boto:Method: GET 2024-04-12T03:54:02.183 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.183 DEBUG:boto:Data: 2024-04-12T03:54:02.183 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.183 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.183 DEBUG:boto:Params: {} 2024-04-12T03:54:02.184 DEBUG:boto:Token: None 2024-04-12T03:54:02.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db4b73f50033972-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.187 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.187 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.187 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.187 DEBUG:boto:path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.187 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.188 DEBUG:boto:Method: GET 2024-04-12T03:54:02.188 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj111 2024-04-12T03:54:02.188 DEBUG:boto:Data: 2024-04-12T03:54:02.188 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.188 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.188 DEBUG:boto:Params: {} 2024-04-12T03:54:02.188 DEBUG:boto:Token: None 2024-04-12T03:54:02.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd7c64f0add7586-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.193 DEBUG:boto:b'dgbpxg-61obj1111000falseobj1112024-04-12T03:51:42.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:54:02.194 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.194 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.194 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.194 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.194 DEBUG:boto:Method: GET 2024-04-12T03:54:02.194 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.194 DEBUG:boto:Data: 2024-04-12T03:54:02.194 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.195 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.195 DEBUG:boto:Params: {} 2024-04-12T03:54:02.195 DEBUG:boto:Token: None 2024-04-12T03:54:02.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA= 2024-04-12T03:54:02.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA=', 'Content-Length': '0'} 2024-04-12T03:54:02.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003782a49265fb4615-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.198 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.198 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.198 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.198 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.198 DEBUG:boto:Method: GET 2024-04-12T03:54:02.198 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.198 DEBUG:boto:Data: 2024-04-12T03:54:02.198 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.199 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.199 DEBUG:boto:Params: {} 2024-04-12T03:54:02.199 DEBUG:boto:Token: None 2024-04-12T03:54:02.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA= 2024-04-12T03:54:02.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA=', 'Content-Length': '0'} 2024-04-12T03:54:02.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7bc21d2e1d0009-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.201 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.201 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.201 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.202 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.202 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.202 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.202 DEBUG:boto:Data: 2024-04-12T03:54:02.202 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.202 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.202 DEBUG:boto:Params: {} 2024-04-12T03:54:02.202 DEBUG:boto:Token: None 2024-04-12T03:54:02.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dcWxdTc5Ws352il1a6JoLgww1fc= 2024-04-12T03:54:02.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dcWxdTc5Ws352il1a6JoLgww1fc=', 'Content-Length': '0'} 2024-04-12T03:54:02.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c790dc4f509f7906-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.204 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.205 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.205 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.205 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.205 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.205 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.205 DEBUG:boto:Data: 2024-04-12T03:54:02.205 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.205 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.205 DEBUG:boto:Params: {} 2024-04-12T03:54:02.206 DEBUG:boto:Token: None 2024-04-12T03:54:02.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.206 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dcWxdTc5Ws352il1a6JoLgww1fc= 2024-04-12T03:54:02.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dcWxdTc5Ws352il1a6JoLgww1fc=', 'Content-Length': '0'} 2024-04-12T03:54:02.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c4ed4f5a5fb444-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:54:02.208 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.208 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.208 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.208 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.208 DEBUG:boto:Method: GET 2024-04-12T03:54:02.209 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.209 DEBUG:boto:Data: 2024-04-12T03:54:02.209 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.209 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.209 DEBUG:boto:Params: {} 2024-04-12T03:54:02.209 DEBUG:boto:Token: None 2024-04-12T03:54:02.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.209 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA= 2024-04-12T03:54:02.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA=', 'Content-Length': '0'} 2024-04-12T03:54:02.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f95804581dbfcd1-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.211 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.211 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.212 DEBUG:boto:path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.212 DEBUG:boto:auth_path=/dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.212 DEBUG:boto:Method: GET 2024-04-12T03:54:02.212 DEBUG:boto:Path: /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.212 DEBUG:boto:Data: 2024-04-12T03:54:02.212 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.212 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.212 DEBUG:boto:Params: {} 2024-04-12T03:54:02.212 DEBUG:boto:Token: None 2024-04-12T03:54:02.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115?versionId=null 2024-04-12T03:54:02.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA= 2024-04-12T03:54:02.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RJOd4Y8YSrmkiFX/aREE9B+j7dA=', 'Content-Length': '0'} 2024-04-12T03:54:02.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029647d6be33b32f9-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.215 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.215 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.215 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.215 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.215 DEBUG:boto:Method: GET 2024-04-12T03:54:02.215 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.215 DEBUG:boto:Data: 2024-04-12T03:54:02.215 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.215 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.215 DEBUG:boto:Params: {} 2024-04-12T03:54:02.216 DEBUG:boto:Token: None 2024-04-12T03:54:02.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008087830bff3d54d9-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.262 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.263 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.263 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.263 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.263 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.263 DEBUG:boto:Method: GET 2024-04-12T03:54:02.263 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.263 DEBUG:boto:Data: 2024-04-12T03:54:02.263 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.264 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.264 DEBUG:boto:Params: {} 2024-04-12T03:54:02.264 DEBUG:boto:Token: None 2024-04-12T03:54:02.264 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b895cbf502ae1e58-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.268 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.269 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.269 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.269 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.269 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:54:02.269 DEBUG:boto:Data: 2024-04-12T03:54:02.269 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.269 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.269 DEBUG:boto:Params: {} 2024-04-12T03:54:02.269 DEBUG:boto:Token: None 2024-04-12T03:54:02.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115 2024-04-12T03:54:02.270 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HfmmDg02s3UEt9bWPXg6KWgmGnQ= 2024-04-12T03:54:02.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HfmmDg02s3UEt9bWPXg6KWgmGnQ=', 'Content-Length': '0'} 2024-04-12T03:54:02.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196fc21f8aa49758-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.272 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.272 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.272 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.272 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:54:02.272 DEBUG:boto:Data: 2024-04-12T03:54:02.272 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.272 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.272 DEBUG:boto:Params: {} 2024-04-12T03:54:02.272 DEBUG:boto:Token: None 2024-04-12T03:54:02.272 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115 2024-04-12T03:54:02.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HfmmDg02s3UEt9bWPXg6KWgmGnQ= 2024-04-12T03:54:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HfmmDg02s3UEt9bWPXg6KWgmGnQ=', 'Content-Length': '0'} 2024-04-12T03:54:02.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cff8944051dd5d3-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T03:54:02.275 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.275 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.275 DEBUG:boto:Method: GET 2024-04-12T03:54:02.275 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:54:02.275 DEBUG:boto:Data: 2024-04-12T03:54:02.275 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.275 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.275 DEBUG:boto:Params: {} 2024-04-12T03:54:02.275 DEBUG:boto:Token: None 2024-04-12T03:54:02.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115 2024-04-12T03:54:02.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eBswUqKtp6HxHv48sEUL0qZHCeQ= 2024-04-12T03:54:02.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eBswUqKtp6HxHv48sEUL0qZHCeQ=', 'Content-Length': '0'} 2024-04-12T03:54:02.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f4975b10275e97e-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.278 DEBUG:boto:path=/dgbpxg-61/obj115 2024-04-12T03:54:02.278 DEBUG:boto:auth_path=/dgbpxg-61/obj115 2024-04-12T03:54:02.278 DEBUG:boto:Method: GET 2024-04-12T03:54:02.278 DEBUG:boto:Path: /dgbpxg-61/obj115 2024-04-12T03:54:02.278 DEBUG:boto:Data: 2024-04-12T03:54:02.278 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.278 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.278 DEBUG:boto:Params: {} 2024-04-12T03:54:02.278 DEBUG:boto:Token: None 2024-04-12T03:54:02.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj115 2024-04-12T03:54:02.279 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eBswUqKtp6HxHv48sEUL0qZHCeQ= 2024-04-12T03:54:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eBswUqKtp6HxHv48sEUL0qZHCeQ=', 'Content-Length': '0'} 2024-04-12T03:54:02.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8a30ab42d9dee3-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.322 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.322 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.323 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.323 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.323 DEBUG:boto:Method: GET 2024-04-12T03:54:02.323 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.323 DEBUG:boto:Data: 2024-04-12T03:54:02.323 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.323 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.323 DEBUG:boto:Params: {} 2024-04-12T03:54:02.323 DEBUG:boto:Token: None 2024-04-12T03:54:02.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007870b99b7f91fd3b-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.326 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.327 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.327 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.327 DEBUG:boto:path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.327 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.327 DEBUG:boto:Method: GET 2024-04-12T03:54:02.327 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj115 2024-04-12T03:54:02.327 DEBUG:boto:Data: 2024-04-12T03:54:02.327 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.327 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.327 DEBUG:boto:Params: {} 2024-04-12T03:54:02.328 DEBUG:boto:Token: None 2024-04-12T03:54:02.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d41c494ec9b407e5-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.332 DEBUG:boto:b'dgbpxg-61obj1151000falseobj1152024-04-12T03:46:16.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:54:02.332 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.332 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.333 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.333 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.333 DEBUG:boto:Method: GET 2024-04-12T03:54:02.333 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.333 DEBUG:boto:Data: 2024-04-12T03:54:02.333 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.333 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.333 DEBUG:boto:Params: {} 2024-04-12T03:54:02.333 DEBUG:boto:Token: None 2024-04-12T03:54:02.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.334 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA= 2024-04-12T03:54:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA=', 'Content-Length': '0'} 2024-04-12T03:54:02.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7794f269cbe6c5-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.336 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.336 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.336 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.337 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.337 DEBUG:boto:Method: GET 2024-04-12T03:54:02.337 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.337 DEBUG:boto:Data: 2024-04-12T03:54:02.337 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.337 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.337 DEBUG:boto:Params: {} 2024-04-12T03:54:02.337 DEBUG:boto:Token: None 2024-04-12T03:54:02.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.338 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA= 2024-04-12T03:54:02.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA=', 'Content-Length': '0'} 2024-04-12T03:54:02.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d4748cb10dca403-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.340 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.340 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.340 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.340 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.340 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.340 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.340 DEBUG:boto:Data: 2024-04-12T03:54:02.340 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.341 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.341 DEBUG:boto:Params: {} 2024-04-12T03:54:02.341 DEBUG:boto:Token: None 2024-04-12T03:54:02.341 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaUDcHb689332V5DuwUSzsLlNn4= 2024-04-12T03:54:02.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaUDcHb689332V5DuwUSzsLlNn4=', 'Content-Length': '0'} 2024-04-12T03:54:02.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f572ce30b2c7a98-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.343 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.343 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.343 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.343 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.343 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.344 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.344 DEBUG:boto:Data: 2024-04-12T03:54:02.344 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.344 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.344 DEBUG:boto:Params: {} 2024-04-12T03:54:02.344 DEBUG:boto:Token: None 2024-04-12T03:54:02.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kaUDcHb689332V5DuwUSzsLlNn4= 2024-04-12T03:54:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kaUDcHb689332V5DuwUSzsLlNn4=', 'Content-Length': '0'} 2024-04-12T03:54:02.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0a1005611c228f-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:54:02.347 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.347 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.347 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.347 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.347 DEBUG:boto:Method: GET 2024-04-12T03:54:02.347 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.347 DEBUG:boto:Data: 2024-04-12T03:54:02.347 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.347 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.347 DEBUG:boto:Params: {} 2024-04-12T03:54:02.348 DEBUG:boto:Token: None 2024-04-12T03:54:02.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA= 2024-04-12T03:54:02.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA=', 'Content-Length': '0'} 2024-04-12T03:54:02.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc0cc5fce53b60f-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.390 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.391 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.391 DEBUG:boto:path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.391 DEBUG:boto:auth_path=/dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.391 DEBUG:boto:Method: GET 2024-04-12T03:54:02.391 DEBUG:boto:Path: /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.391 DEBUG:boto:Data: 2024-04-12T03:54:02.391 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.391 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.391 DEBUG:boto:Params: {} 2024-04-12T03:54:02.392 DEBUG:boto:Token: None 2024-04-12T03:54:02.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116?versionId=null 2024-04-12T03:54:02.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA= 2024-04-12T03:54:02.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+W1SED4vMFc24jk2g8dP8S8S0GA=', 'Content-Length': '0'} 2024-04-12T03:54:02.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6af3f76fc2dffa-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.394 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.395 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.395 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.395 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.395 DEBUG:boto:Method: GET 2024-04-12T03:54:02.395 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.395 DEBUG:boto:Data: 2024-04-12T03:54:02.395 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.395 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.395 DEBUG:boto:Params: {} 2024-04-12T03:54:02.396 DEBUG:boto:Token: None 2024-04-12T03:54:02.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16592fb2195a04b-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.398 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.398 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.399 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.399 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.399 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.399 DEBUG:boto:Method: GET 2024-04-12T03:54:02.399 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.399 DEBUG:boto:Data: 2024-04-12T03:54:02.399 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.399 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.399 DEBUG:boto:Params: {} 2024-04-12T03:54:02.399 DEBUG:boto:Token: None 2024-04-12T03:54:02.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042bb23780d3ff3bb-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.404 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.404 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.404 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.404 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.404 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:54:02.404 DEBUG:boto:Data: 2024-04-12T03:54:02.404 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.405 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.405 DEBUG:boto:Params: {} 2024-04-12T03:54:02.405 DEBUG:boto:Token: None 2024-04-12T03:54:02.405 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116 2024-04-12T03:54:02.405 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ReGu0UYzp493yY43B09lFRILKCY= 2024-04-12T03:54:02.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ReGu0UYzp493yY43B09lFRILKCY=', 'Content-Length': '0'} 2024-04-12T03:54:02.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3bc7e0c89002e60-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.407 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.407 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.407 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.408 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:54:02.408 DEBUG:boto:Data: 2024-04-12T03:54:02.408 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.408 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.408 DEBUG:boto:Params: {} 2024-04-12T03:54:02.408 DEBUG:boto:Token: None 2024-04-12T03:54:02.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116 2024-04-12T03:54:02.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ReGu0UYzp493yY43B09lFRILKCY= 2024-04-12T03:54:02.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ReGu0UYzp493yY43B09lFRILKCY=', 'Content-Length': '0'} 2024-04-12T03:54:02.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ab50905bbf3bf5-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T03:54:02.411 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.411 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.411 DEBUG:boto:Method: GET 2024-04-12T03:54:02.411 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:54:02.411 DEBUG:boto:Data: 2024-04-12T03:54:02.411 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.411 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.411 DEBUG:boto:Params: {} 2024-04-12T03:54:02.411 DEBUG:boto:Token: None 2024-04-12T03:54:02.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116 2024-04-12T03:54:02.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VI9nBHPI3eVcfjFdTszITtCOqwM= 2024-04-12T03:54:02.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VI9nBHPI3eVcfjFdTszITtCOqwM=', 'Content-Length': '0'} 2024-04-12T03:54:02.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80b674609287ccc-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.413 DEBUG:boto:path=/dgbpxg-61/obj116 2024-04-12T03:54:02.413 DEBUG:boto:auth_path=/dgbpxg-61/obj116 2024-04-12T03:54:02.414 DEBUG:boto:Method: GET 2024-04-12T03:54:02.414 DEBUG:boto:Path: /dgbpxg-61/obj116 2024-04-12T03:54:02.414 DEBUG:boto:Data: 2024-04-12T03:54:02.414 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.414 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.414 DEBUG:boto:Params: {} 2024-04-12T03:54:02.414 DEBUG:boto:Token: None 2024-04-12T03:54:02.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj116 2024-04-12T03:54:02.415 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VI9nBHPI3eVcfjFdTszITtCOqwM= 2024-04-12T03:54:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VI9nBHPI3eVcfjFdTszITtCOqwM=', 'Content-Length': '0'} 2024-04-12T03:54:02.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe623725caa288e-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.417 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.417 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.417 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.417 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.417 DEBUG:boto:Method: GET 2024-04-12T03:54:02.417 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.417 DEBUG:boto:Data: 2024-04-12T03:54:02.418 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.418 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.418 DEBUG:boto:Params: {} 2024-04-12T03:54:02.418 DEBUG:boto:Token: None 2024-04-12T03:54:02.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.418 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ae771833625fc3-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.420 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.421 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.421 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.421 DEBUG:boto:path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.421 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.421 DEBUG:boto:Method: GET 2024-04-12T03:54:02.421 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj116 2024-04-12T03:54:02.421 DEBUG:boto:Data: 2024-04-12T03:54:02.421 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.421 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.422 DEBUG:boto:Params: {} 2024-04-12T03:54:02.422 DEBUG:boto:Token: None 2024-04-12T03:54:02.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.422 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdfebbd765c4f694-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.426 DEBUG:boto:b'dgbpxg-61obj1161000falseobj1162024-04-12T03:47:19.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:54:02.426 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.426 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.426 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.427 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.427 DEBUG:boto:Method: GET 2024-04-12T03:54:02.427 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.427 DEBUG:boto:Data: 2024-04-12T03:54:02.427 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.427 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.427 DEBUG:boto:Params: {} 2024-04-12T03:54:02.427 DEBUG:boto:Token: None 2024-04-12T03:54:02.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.428 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U= 2024-04-12T03:54:02.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U=', 'Content-Length': '0'} 2024-04-12T03:54:02.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24935149f562d33-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.430 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.430 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.430 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.430 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.430 DEBUG:boto:Method: GET 2024-04-12T03:54:02.430 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.430 DEBUG:boto:Data: 2024-04-12T03:54:02.431 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.431 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.431 DEBUG:boto:Params: {} 2024-04-12T03:54:02.431 DEBUG:boto:Token: None 2024-04-12T03:54:02.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U= 2024-04-12T03:54:02.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U=', 'Content-Length': '0'} 2024-04-12T03:54:02.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071d99358bf4cdfc0-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.433 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.433 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.433 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.433 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.434 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.434 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.434 DEBUG:boto:Data: 2024-04-12T03:54:02.434 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.434 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.434 DEBUG:boto:Params: {} 2024-04-12T03:54:02.434 DEBUG:boto:Token: None 2024-04-12T03:54:02.434 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.434 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NQzxPTgK1WT9ak2OpCGn9uFeVvE= 2024-04-12T03:54:02.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NQzxPTgK1WT9ak2OpCGn9uFeVvE=', 'Content-Length': '0'} 2024-04-12T03:54:02.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004675b54799cb4c09-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.436 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.436 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.437 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.437 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.437 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.437 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.437 DEBUG:boto:Data: 2024-04-12T03:54:02.437 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.437 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.437 DEBUG:boto:Params: {} 2024-04-12T03:54:02.437 DEBUG:boto:Token: None 2024-04-12T03:54:02.437 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.437 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NQzxPTgK1WT9ak2OpCGn9uFeVvE= 2024-04-12T03:54:02.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NQzxPTgK1WT9ak2OpCGn9uFeVvE=', 'Content-Length': '0'} 2024-04-12T03:54:02.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c428fbf9acee744-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:54:02.439 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.439 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.440 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.440 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.440 DEBUG:boto:Method: GET 2024-04-12T03:54:02.440 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.440 DEBUG:boto:Data: 2024-04-12T03:54:02.440 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.440 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.440 DEBUG:boto:Params: {} 2024-04-12T03:54:02.440 DEBUG:boto:Token: None 2024-04-12T03:54:02.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.441 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U= 2024-04-12T03:54:02.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U=', 'Content-Length': '0'} 2024-04-12T03:54:02.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aedba7e17269ac3b-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.442 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.443 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.443 DEBUG:boto:path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.443 DEBUG:boto:auth_path=/dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.443 DEBUG:boto:Method: GET 2024-04-12T03:54:02.443 DEBUG:boto:Path: /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.443 DEBUG:boto:Data: 2024-04-12T03:54:02.443 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.443 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.443 DEBUG:boto:Params: {} 2024-04-12T03:54:02.444 DEBUG:boto:Token: None 2024-04-12T03:54:02.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124?versionId=null 2024-04-12T03:54:02.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U= 2024-04-12T03:54:02.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LZQPaaWtbTSWcNJBxcuQeBO/13U=', 'Content-Length': '0'} 2024-04-12T03:54:02.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a4d09510633d10-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.486 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.486 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.486 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.487 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.487 DEBUG:boto:Method: GET 2024-04-12T03:54:02.487 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.487 DEBUG:boto:Data: 2024-04-12T03:54:02.487 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.487 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.487 DEBUG:boto:Params: {} 2024-04-12T03:54:02.487 DEBUG:boto:Token: None 2024-04-12T03:54:02.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7a3eba9f7147e5-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.490 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.490 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.490 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.490 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.490 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.491 DEBUG:boto:Method: GET 2024-04-12T03:54:02.491 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.491 DEBUG:boto:Data: 2024-04-12T03:54:02.491 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.491 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.491 DEBUG:boto:Params: {} 2024-04-12T03:54:02.491 DEBUG:boto:Token: None 2024-04-12T03:54:02.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.492 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000468af50a7f90837d-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.495 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.496 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.496 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.496 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.496 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:54:02.496 DEBUG:boto:Data: 2024-04-12T03:54:02.496 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.496 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.497 DEBUG:boto:Params: {} 2024-04-12T03:54:02.497 DEBUG:boto:Token: None 2024-04-12T03:54:02.497 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124 2024-04-12T03:54:02.497 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMF/NKVDxmw1ugU5QXCoHANgXCg= 2024-04-12T03:54:02.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMF/NKVDxmw1ugU5QXCoHANgXCg=', 'Content-Length': '0'} 2024-04-12T03:54:02.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ac48cf0110595c-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.499 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.499 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.499 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.499 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:54:02.499 DEBUG:boto:Data: 2024-04-12T03:54:02.499 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.499 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.499 DEBUG:boto:Params: {} 2024-04-12T03:54:02.500 DEBUG:boto:Token: None 2024-04-12T03:54:02.500 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124 2024-04-12T03:54:02.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RMF/NKVDxmw1ugU5QXCoHANgXCg= 2024-04-12T03:54:02.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RMF/NKVDxmw1ugU5QXCoHANgXCg=', 'Content-Length': '0'} 2024-04-12T03:54:02.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b35efcdb75f686-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T03:54:02.502 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.502 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.502 DEBUG:boto:Method: GET 2024-04-12T03:54:02.502 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:54:02.502 DEBUG:boto:Data: 2024-04-12T03:54:02.502 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.503 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.503 DEBUG:boto:Params: {} 2024-04-12T03:54:02.503 DEBUG:boto:Token: None 2024-04-12T03:54:02.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124 2024-04-12T03:54:02.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:frAU1WAGQ2i0mT65Gdg0z3fOkMI= 2024-04-12T03:54:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:frAU1WAGQ2i0mT65Gdg0z3fOkMI=', 'Content-Length': '0'} 2024-04-12T03:54:02.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da03f25b0a3236f-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.546 DEBUG:boto:path=/dgbpxg-61/obj124 2024-04-12T03:54:02.546 DEBUG:boto:auth_path=/dgbpxg-61/obj124 2024-04-12T03:54:02.547 DEBUG:boto:Method: GET 2024-04-12T03:54:02.547 DEBUG:boto:Path: /dgbpxg-61/obj124 2024-04-12T03:54:02.547 DEBUG:boto:Data: 2024-04-12T03:54:02.547 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.547 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.547 DEBUG:boto:Params: {} 2024-04-12T03:54:02.547 DEBUG:boto:Token: None 2024-04-12T03:54:02.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj124 2024-04-12T03:54:02.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:frAU1WAGQ2i0mT65Gdg0z3fOkMI= 2024-04-12T03:54:02.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:frAU1WAGQ2i0mT65Gdg0z3fOkMI=', 'Content-Length': '0'} 2024-04-12T03:54:02.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95b63dd58890870-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.550 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.550 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.550 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.550 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.550 DEBUG:boto:Method: GET 2024-04-12T03:54:02.550 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.550 DEBUG:boto:Data: 2024-04-12T03:54:02.551 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.551 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.551 DEBUG:boto:Params: {} 2024-04-12T03:54:02.551 DEBUG:boto:Token: None 2024-04-12T03:54:02.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf4c08fb34b7314-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.553 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.554 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.554 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.554 DEBUG:boto:path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.554 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.554 DEBUG:boto:Method: GET 2024-04-12T03:54:02.554 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj124 2024-04-12T03:54:02.554 DEBUG:boto:Data: 2024-04-12T03:54:02.554 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.554 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.555 DEBUG:boto:Params: {} 2024-04-12T03:54:02.555 DEBUG:boto:Token: None 2024-04-12T03:54:02.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.555 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fa71e57aaf0995-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.558 DEBUG:boto:b'dgbpxg-61obj1241000falseobj1242024-04-12T03:50:18.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:54:02.559 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.559 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.559 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.559 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.559 DEBUG:boto:Method: GET 2024-04-12T03:54:02.559 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.559 DEBUG:boto:Data: 2024-04-12T03:54:02.560 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.560 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.560 DEBUG:boto:Params: {} 2024-04-12T03:54:02.560 DEBUG:boto:Token: None 2024-04-12T03:54:02.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc= 2024-04-12T03:54:02.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc=', 'Content-Length': '0'} 2024-04-12T03:54:02.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dade0afc63d1d37a-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.562 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.562 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.563 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.563 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.563 DEBUG:boto:Method: GET 2024-04-12T03:54:02.563 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.563 DEBUG:boto:Data: 2024-04-12T03:54:02.563 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.563 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.563 DEBUG:boto:Params: {} 2024-04-12T03:54:02.563 DEBUG:boto:Token: None 2024-04-12T03:54:02.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.563 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc= 2024-04-12T03:54:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc=', 'Content-Length': '0'} 2024-04-12T03:54:02.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb940a2efa03cc2e-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.606 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.606 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.607 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.607 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.607 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.607 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.607 DEBUG:boto:Data: 2024-04-12T03:54:02.607 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.607 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.607 DEBUG:boto:Params: {} 2024-04-12T03:54:02.607 DEBUG:boto:Token: None 2024-04-12T03:54:02.608 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m5KzZuDYkiW7FXuYFs4FXQTf4Aw= 2024-04-12T03:54:02.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m5KzZuDYkiW7FXuYFs4FXQTf4Aw=', 'Content-Length': '0'} 2024-04-12T03:54:02.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000485a773d352bff71-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.610 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.610 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.610 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.610 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.610 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.610 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.610 DEBUG:boto:Data: 2024-04-12T03:54:02.610 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.610 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.611 DEBUG:boto:Params: {} 2024-04-12T03:54:02.611 DEBUG:boto:Token: None 2024-04-12T03:54:02.611 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m5KzZuDYkiW7FXuYFs4FXQTf4Aw= 2024-04-12T03:54:02.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m5KzZuDYkiW7FXuYFs4FXQTf4Aw=', 'Content-Length': '0'} 2024-04-12T03:54:02.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002defd99103aac954-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:54:02.613 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.613 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.613 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.613 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.613 DEBUG:boto:Method: GET 2024-04-12T03:54:02.613 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.614 DEBUG:boto:Data: 2024-04-12T03:54:02.614 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.614 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.614 DEBUG:boto:Params: {} 2024-04-12T03:54:02.614 DEBUG:boto:Token: None 2024-04-12T03:54:02.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc= 2024-04-12T03:54:02.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc=', 'Content-Length': '0'} 2024-04-12T03:54:02.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5927952906fa47e-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.617 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.617 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.617 DEBUG:boto:path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.617 DEBUG:boto:auth_path=/dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.617 DEBUG:boto:Method: GET 2024-04-12T03:54:02.617 DEBUG:boto:Path: /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.618 DEBUG:boto:Data: 2024-04-12T03:54:02.618 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.618 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.618 DEBUG:boto:Params: {} 2024-04-12T03:54:02.618 DEBUG:boto:Token: None 2024-04-12T03:54:02.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145?versionId=null 2024-04-12T03:54:02.619 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc= 2024-04-12T03:54:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zHzVBtUEoi4ZpTqe8mW6mIcOdWc=', 'Content-Length': '0'} 2024-04-12T03:54:02.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40a980e96e2b944-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.621 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.621 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.621 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.621 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.622 DEBUG:boto:Method: GET 2024-04-12T03:54:02.622 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.622 DEBUG:boto:Data: 2024-04-12T03:54:02.622 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.622 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.622 DEBUG:boto:Params: {} 2024-04-12T03:54:02.622 DEBUG:boto:Token: None 2024-04-12T03:54:02.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ca0be904816f77-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.625 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.626 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.626 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.626 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.626 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.627 DEBUG:boto:Method: GET 2024-04-12T03:54:02.627 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.627 DEBUG:boto:Data: 2024-04-12T03:54:02.627 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.627 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.627 DEBUG:boto:Params: {} 2024-04-12T03:54:02.628 DEBUG:boto:Token: None 2024-04-12T03:54:02.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3ce4f350c2dc9f1-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.632 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.632 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.632 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.632 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.632 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:54:02.632 DEBUG:boto:Data: 2024-04-12T03:54:02.632 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.633 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.633 DEBUG:boto:Params: {} 2024-04-12T03:54:02.633 DEBUG:boto:Token: None 2024-04-12T03:54:02.633 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145 2024-04-12T03:54:02.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M5t/ZgHTe1G1+jEizem2XJ/oy10= 2024-04-12T03:54:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M5t/ZgHTe1G1+jEizem2XJ/oy10=', 'Content-Length': '0'} 2024-04-12T03:54:02.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80647b06883db10-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.635 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.635 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.635 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.635 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:54:02.635 DEBUG:boto:Data: 2024-04-12T03:54:02.635 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.635 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.635 DEBUG:boto:Params: {} 2024-04-12T03:54:02.636 DEBUG:boto:Token: None 2024-04-12T03:54:02.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145 2024-04-12T03:54:02.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M5t/ZgHTe1G1+jEizem2XJ/oy10= 2024-04-12T03:54:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M5t/ZgHTe1G1+jEizem2XJ/oy10=', 'Content-Length': '0'} 2024-04-12T03:54:02.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d8664284e1b13bc-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T03:54:02.638 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.638 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.638 DEBUG:boto:Method: GET 2024-04-12T03:54:02.638 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:54:02.638 DEBUG:boto:Data: 2024-04-12T03:54:02.638 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.638 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.638 DEBUG:boto:Params: {} 2024-04-12T03:54:02.639 DEBUG:boto:Token: None 2024-04-12T03:54:02.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145 2024-04-12T03:54:02.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nRbFlDgmya2CeDlkzz5P5D2n83w= 2024-04-12T03:54:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nRbFlDgmya2CeDlkzz5P5D2n83w=', 'Content-Length': '0'} 2024-04-12T03:54:02.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc72a40d6319c703-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.641 DEBUG:boto:path=/dgbpxg-61/obj145 2024-04-12T03:54:02.641 DEBUG:boto:auth_path=/dgbpxg-61/obj145 2024-04-12T03:54:02.641 DEBUG:boto:Method: GET 2024-04-12T03:54:02.641 DEBUG:boto:Path: /dgbpxg-61/obj145 2024-04-12T03:54:02.641 DEBUG:boto:Data: 2024-04-12T03:54:02.641 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.641 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.641 DEBUG:boto:Params: {} 2024-04-12T03:54:02.641 DEBUG:boto:Token: None 2024-04-12T03:54:02.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj145 2024-04-12T03:54:02.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nRbFlDgmya2CeDlkzz5P5D2n83w= 2024-04-12T03:54:02.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nRbFlDgmya2CeDlkzz5P5D2n83w=', 'Content-Length': '0'} 2024-04-12T03:54:02.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b884ef455cfa6b8-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.643 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.644 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.644 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.644 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.644 DEBUG:boto:Method: GET 2024-04-12T03:54:02.644 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.644 DEBUG:boto:Data: 2024-04-12T03:54:02.644 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.644 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.644 DEBUG:boto:Params: {} 2024-04-12T03:54:02.644 DEBUG:boto:Token: None 2024-04-12T03:54:02.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006942de712529d3e0-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.648 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.648 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.648 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.648 DEBUG:boto:path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.648 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.648 DEBUG:boto:Method: GET 2024-04-12T03:54:02.648 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj145 2024-04-12T03:54:02.649 DEBUG:boto:Data: 2024-04-12T03:54:02.649 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.649 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.649 DEBUG:boto:Params: {} 2024-04-12T03:54:02.649 DEBUG:boto:Token: None 2024-04-12T03:54:02.649 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057fc792aa0fd6847-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.653 DEBUG:boto:b'dgbpxg-61obj1451000falseobj1452024-04-12T03:47:19.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:54:02.653 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.654 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.654 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.654 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.654 DEBUG:boto:Method: GET 2024-04-12T03:54:02.654 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.654 DEBUG:boto:Data: 2024-04-12T03:54:02.654 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.654 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.654 DEBUG:boto:Params: {} 2024-04-12T03:54:02.655 DEBUG:boto:Token: None 2024-04-12T03:54:02.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4= 2024-04-12T03:54:02.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4=', 'Content-Length': '0'} 2024-04-12T03:54:02.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c416736e2b9bc00-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.698 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.698 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.698 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.699 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.699 DEBUG:boto:Method: GET 2024-04-12T03:54:02.699 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.699 DEBUG:boto:Data: 2024-04-12T03:54:02.699 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.699 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.699 DEBUG:boto:Params: {} 2024-04-12T03:54:02.699 DEBUG:boto:Token: None 2024-04-12T03:54:02.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4= 2024-04-12T03:54:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4=', 'Content-Length': '0'} 2024-04-12T03:54:02.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b089288cb7686061-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.702 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.702 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.702 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.702 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.702 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.702 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.702 DEBUG:boto:Data: 2024-04-12T03:54:02.702 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.703 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.703 DEBUG:boto:Params: {} 2024-04-12T03:54:02.703 DEBUG:boto:Token: None 2024-04-12T03:54:02.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YFdNIDGbLPFESbfuJnaI61MJEvM= 2024-04-12T03:54:02.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YFdNIDGbLPFESbfuJnaI61MJEvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ce3e3ca579b588-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.705 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.705 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.705 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.705 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.705 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.705 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.705 DEBUG:boto:Data: 2024-04-12T03:54:02.705 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.706 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.706 DEBUG:boto:Params: {} 2024-04-12T03:54:02.706 DEBUG:boto:Token: None 2024-04-12T03:54:02.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YFdNIDGbLPFESbfuJnaI61MJEvM= 2024-04-12T03:54:02.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YFdNIDGbLPFESbfuJnaI61MJEvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f2176a45f8b7fe-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:54:02.708 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.708 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.708 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.708 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.708 DEBUG:boto:Method: GET 2024-04-12T03:54:02.708 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.708 DEBUG:boto:Data: 2024-04-12T03:54:02.709 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.709 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.709 DEBUG:boto:Params: {} 2024-04-12T03:54:02.709 DEBUG:boto:Token: None 2024-04-12T03:54:02.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4= 2024-04-12T03:54:02.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4=', 'Content-Length': '0'} 2024-04-12T03:54:02.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ed10bc2617627b-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.711 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.711 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.711 DEBUG:boto:path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.711 DEBUG:boto:auth_path=/dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.711 DEBUG:boto:Method: GET 2024-04-12T03:54:02.711 DEBUG:boto:Path: /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.711 DEBUG:boto:Data: 2024-04-12T03:54:02.712 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.712 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.712 DEBUG:boto:Params: {} 2024-04-12T03:54:02.712 DEBUG:boto:Token: None 2024-04-12T03:54:02.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148?versionId=null 2024-04-12T03:54:02.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4= 2024-04-12T03:54:02.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tII66xD5+bc5slEYqazQSgydt+4=', 'Content-Length': '0'} 2024-04-12T03:54:02.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cad558cbce3a9e3-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.714 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.714 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.714 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.714 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.714 DEBUG:boto:Method: GET 2024-04-12T03:54:02.714 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.715 DEBUG:boto:Data: 2024-04-12T03:54:02.715 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.715 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.715 DEBUG:boto:Params: {} 2024-04-12T03:54:02.715 DEBUG:boto:Token: None 2024-04-12T03:54:02.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d231caee3d53db7-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.717 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.718 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.718 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.718 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.718 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.718 DEBUG:boto:Method: GET 2024-04-12T03:54:02.718 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.718 DEBUG:boto:Data: 2024-04-12T03:54:02.718 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.719 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.719 DEBUG:boto:Params: {} 2024-04-12T03:54:02.719 DEBUG:boto:Token: None 2024-04-12T03:54:02.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001b2d3df001c9eda-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.722 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.723 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.723 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.723 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.723 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:54:02.723 DEBUG:boto:Data: 2024-04-12T03:54:02.723 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.723 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.724 DEBUG:boto:Params: {} 2024-04-12T03:54:02.724 DEBUG:boto:Token: None 2024-04-12T03:54:02.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148 2024-04-12T03:54:02.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bQB2LFvF67IuYRChUqWkHcY4oOo= 2024-04-12T03:54:02.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bQB2LFvF67IuYRChUqWkHcY4oOo=', 'Content-Length': '0'} 2024-04-12T03:54:02.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf6b8f1d5f7b779-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.726 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.726 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.726 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.726 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:54:02.726 DEBUG:boto:Data: 2024-04-12T03:54:02.726 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.726 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.726 DEBUG:boto:Params: {} 2024-04-12T03:54:02.727 DEBUG:boto:Token: None 2024-04-12T03:54:02.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148 2024-04-12T03:54:02.727 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bQB2LFvF67IuYRChUqWkHcY4oOo= 2024-04-12T03:54:02.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bQB2LFvF67IuYRChUqWkHcY4oOo=', 'Content-Length': '0'} 2024-04-12T03:54:02.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d655a4f0482c2b5c-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T03:54:02.729 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.729 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.729 DEBUG:boto:Method: GET 2024-04-12T03:54:02.729 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:54:02.729 DEBUG:boto:Data: 2024-04-12T03:54:02.729 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.729 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.729 DEBUG:boto:Params: {} 2024-04-12T03:54:02.729 DEBUG:boto:Token: None 2024-04-12T03:54:02.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148 2024-04-12T03:54:02.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/T91UCCLcFPI9kIIFvAQdnlva0s= 2024-04-12T03:54:02.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/T91UCCLcFPI9kIIFvAQdnlva0s=', 'Content-Length': '0'} 2024-04-12T03:54:02.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5048d83a7ea568-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.732 DEBUG:boto:path=/dgbpxg-61/obj148 2024-04-12T03:54:02.732 DEBUG:boto:auth_path=/dgbpxg-61/obj148 2024-04-12T03:54:02.732 DEBUG:boto:Method: GET 2024-04-12T03:54:02.732 DEBUG:boto:Path: /dgbpxg-61/obj148 2024-04-12T03:54:02.732 DEBUG:boto:Data: 2024-04-12T03:54:02.732 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.732 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.732 DEBUG:boto:Params: {} 2024-04-12T03:54:02.733 DEBUG:boto:Token: None 2024-04-12T03:54:02.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj148 2024-04-12T03:54:02.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/T91UCCLcFPI9kIIFvAQdnlva0s= 2024-04-12T03:54:02.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/T91UCCLcFPI9kIIFvAQdnlva0s=', 'Content-Length': '0'} 2024-04-12T03:54:02.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a1a2a46af4811f-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.782 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.783 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.783 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.783 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.783 DEBUG:boto:Method: GET 2024-04-12T03:54:02.783 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.783 DEBUG:boto:Data: 2024-04-12T03:54:02.783 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.783 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.783 DEBUG:boto:Params: {} 2024-04-12T03:54:02.784 DEBUG:boto:Token: None 2024-04-12T03:54:02.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d1f792286171196-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.786 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.787 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.787 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.787 DEBUG:boto:path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.787 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.787 DEBUG:boto:Method: GET 2024-04-12T03:54:02.787 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj148 2024-04-12T03:54:02.787 DEBUG:boto:Data: 2024-04-12T03:54:02.788 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.788 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.788 DEBUG:boto:Params: {} 2024-04-12T03:54:02.788 DEBUG:boto:Token: None 2024-04-12T03:54:02.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c2861136d59baba-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.792 DEBUG:boto:b'dgbpxg-61obj1481000falseobj1482024-04-12T03:51:42.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:54:02.792 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.792 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.792 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.792 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.792 DEBUG:boto:Method: GET 2024-04-12T03:54:02.792 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.793 DEBUG:boto:Data: 2024-04-12T03:54:02.793 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.793 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.793 DEBUG:boto:Params: {} 2024-04-12T03:54:02.793 DEBUG:boto:Token: None 2024-04-12T03:54:02.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo= 2024-04-12T03:54:02.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de10f03adbd7165-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.838 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.838 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.839 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.839 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.839 DEBUG:boto:Method: GET 2024-04-12T03:54:02.839 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.839 DEBUG:boto:Data: 2024-04-12T03:54:02.839 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.839 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.839 DEBUG:boto:Params: {} 2024-04-12T03:54:02.839 DEBUG:boto:Token: None 2024-04-12T03:54:02.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo= 2024-04-12T03:54:02.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baef28876ae1da9a-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.841 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.842 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.842 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.842 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.842 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.842 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.842 DEBUG:boto:Data: 2024-04-12T03:54:02.842 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.842 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.843 DEBUG:boto:Params: {} 2024-04-12T03:54:02.843 DEBUG:boto:Token: None 2024-04-12T03:54:02.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zZm9lKLmMt3wBAo8esGC7+mkeIU= 2024-04-12T03:54:02.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zZm9lKLmMt3wBAo8esGC7+mkeIU=', 'Content-Length': '0'} 2024-04-12T03:54:02.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014afd50174d3adcd-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.845 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.845 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.845 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.845 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.845 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.845 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.845 DEBUG:boto:Data: 2024-04-12T03:54:02.845 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.845 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.846 DEBUG:boto:Params: {} 2024-04-12T03:54:02.846 DEBUG:boto:Token: None 2024-04-12T03:54:02.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zZm9lKLmMt3wBAo8esGC7+mkeIU= 2024-04-12T03:54:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zZm9lKLmMt3wBAo8esGC7+mkeIU=', 'Content-Length': '0'} 2024-04-12T03:54:02.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a81990f3ae55ca-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:54:02.848 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.848 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.848 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.848 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.848 DEBUG:boto:Method: GET 2024-04-12T03:54:02.848 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.848 DEBUG:boto:Data: 2024-04-12T03:54:02.848 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.849 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.849 DEBUG:boto:Params: {} 2024-04-12T03:54:02.849 DEBUG:boto:Token: None 2024-04-12T03:54:02.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo= 2024-04-12T03:54:02.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a0af143e0a14f15-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.851 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.851 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.851 DEBUG:boto:path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.851 DEBUG:boto:auth_path=/dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.851 DEBUG:boto:Method: GET 2024-04-12T03:54:02.851 DEBUG:boto:Path: /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.851 DEBUG:boto:Data: 2024-04-12T03:54:02.851 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.852 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.852 DEBUG:boto:Params: {} 2024-04-12T03:54:02.852 DEBUG:boto:Token: None 2024-04-12T03:54:02.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155?versionId=null 2024-04-12T03:54:02.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo= 2024-04-12T03:54:02.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wsK7IqvAqMC3E3j9bLQgvY5XJFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074431f1674513f80-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.854 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.854 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.854 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.854 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.854 DEBUG:boto:Method: GET 2024-04-12T03:54:02.854 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.854 DEBUG:boto:Data: 2024-04-12T03:54:02.854 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.855 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.855 DEBUG:boto:Params: {} 2024-04-12T03:54:02.855 DEBUG:boto:Token: None 2024-04-12T03:54:02.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900da3e01ecf5e0d-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.857 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.858 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.858 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.858 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.858 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.858 DEBUG:boto:Method: GET 2024-04-12T03:54:02.858 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.858 DEBUG:boto:Data: 2024-04-12T03:54:02.858 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.859 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.859 DEBUG:boto:Params: {} 2024-04-12T03:54:02.859 DEBUG:boto:Token: None 2024-04-12T03:54:02.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8089d377b275f1-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.863 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.863 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.863 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.863 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.863 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:54:02.864 DEBUG:boto:Data: 2024-04-12T03:54:02.864 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.864 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.864 DEBUG:boto:Params: {} 2024-04-12T03:54:02.864 DEBUG:boto:Token: None 2024-04-12T03:54:02.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155 2024-04-12T03:54:02.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o3Bgz95OJuD3BbRfb1tg0xlNXFo= 2024-04-12T03:54:02.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o3Bgz95OJuD3BbRfb1tg0xlNXFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c8e0c405189ae4-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.866 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.866 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.866 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.866 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:54:02.866 DEBUG:boto:Data: 2024-04-12T03:54:02.866 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.867 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.867 DEBUG:boto:Params: {} 2024-04-12T03:54:02.867 DEBUG:boto:Token: None 2024-04-12T03:54:02.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155 2024-04-12T03:54:02.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o3Bgz95OJuD3BbRfb1tg0xlNXFo= 2024-04-12T03:54:02.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o3Bgz95OJuD3BbRfb1tg0xlNXFo=', 'Content-Length': '0'} 2024-04-12T03:54:02.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ab2894a541119e-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T03:54:02.869 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.869 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.869 DEBUG:boto:Method: GET 2024-04-12T03:54:02.869 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:54:02.869 DEBUG:boto:Data: 2024-04-12T03:54:02.869 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.870 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.870 DEBUG:boto:Params: {} 2024-04-12T03:54:02.870 DEBUG:boto:Token: None 2024-04-12T03:54:02.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155 2024-04-12T03:54:02.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gFrD7SwGcXLfW+CL2WlCUdZ23cw= 2024-04-12T03:54:02.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gFrD7SwGcXLfW+CL2WlCUdZ23cw=', 'Content-Length': '0'} 2024-04-12T03:54:02.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3be697fc5097280-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.872 DEBUG:boto:path=/dgbpxg-61/obj155 2024-04-12T03:54:02.872 DEBUG:boto:auth_path=/dgbpxg-61/obj155 2024-04-12T03:54:02.872 DEBUG:boto:Method: GET 2024-04-12T03:54:02.872 DEBUG:boto:Path: /dgbpxg-61/obj155 2024-04-12T03:54:02.872 DEBUG:boto:Data: 2024-04-12T03:54:02.872 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.872 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.873 DEBUG:boto:Params: {} 2024-04-12T03:54:02.873 DEBUG:boto:Token: None 2024-04-12T03:54:02.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj155 2024-04-12T03:54:02.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gFrD7SwGcXLfW+CL2WlCUdZ23cw= 2024-04-12T03:54:02.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gFrD7SwGcXLfW+CL2WlCUdZ23cw=', 'Content-Length': '0'} 2024-04-12T03:54:02.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e2cf050380109a-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.875 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.875 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.875 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.875 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.875 DEBUG:boto:Method: GET 2024-04-12T03:54:02.875 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.875 DEBUG:boto:Data: 2024-04-12T03:54:02.875 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.875 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.875 DEBUG:boto:Params: {} 2024-04-12T03:54:02.876 DEBUG:boto:Token: None 2024-04-12T03:54:02.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e47a3ce418820266-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.878 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.878 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.878 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.878 DEBUG:boto:path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.878 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.879 DEBUG:boto:Method: GET 2024-04-12T03:54:02.879 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj155 2024-04-12T03:54:02.879 DEBUG:boto:Data: 2024-04-12T03:54:02.879 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.879 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.879 DEBUG:boto:Params: {} 2024-04-12T03:54:02.879 DEBUG:boto:Token: None 2024-04-12T03:54:02.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.879 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005eb15f982334e57-006618b05a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.926 DEBUG:boto:b'dgbpxg-61obj1551000falseobj1552024-04-12T03:50:18.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:54:02.927 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.927 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.927 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.927 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.927 DEBUG:boto:Method: GET 2024-04-12T03:54:02.927 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.927 DEBUG:boto:Data: 2024-04-12T03:54:02.927 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.927 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.927 DEBUG:boto:Params: {} 2024-04-12T03:54:02.928 DEBUG:boto:Token: None 2024-04-12T03:54:02.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc= 2024-04-12T03:54:02.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc=', 'Content-Length': '0'} 2024-04-12T03:54:02.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f66b0045fdf2f0-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.930 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.930 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.930 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.930 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.931 DEBUG:boto:Method: GET 2024-04-12T03:54:02.931 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.931 DEBUG:boto:Data: 2024-04-12T03:54:02.931 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.931 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.931 DEBUG:boto:Params: {} 2024-04-12T03:54:02.931 DEBUG:boto:Token: None 2024-04-12T03:54:02.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc= 2024-04-12T03:54:02.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc=', 'Content-Length': '0'} 2024-04-12T03:54:02.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082cb5455daf072ea-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.933 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.933 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.934 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.934 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.934 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.934 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.934 DEBUG:boto:Data: 2024-04-12T03:54:02.934 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.934 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.934 DEBUG:boto:Params: {} 2024-04-12T03:54:02.935 DEBUG:boto:Token: None 2024-04-12T03:54:02.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KQTzEzCBrDSj+S/17FR+rQ7ag78= 2024-04-12T03:54:02.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KQTzEzCBrDSj+S/17FR+rQ7ag78=', 'Content-Length': '0'} 2024-04-12T03:54:02.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5938eb7beb2d4c0-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.937 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.937 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.937 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.937 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.937 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.937 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.937 DEBUG:boto:Data: 2024-04-12T03:54:02.937 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.937 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.937 DEBUG:boto:Params: {} 2024-04-12T03:54:02.938 DEBUG:boto:Token: None 2024-04-12T03:54:02.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.938 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KQTzEzCBrDSj+S/17FR+rQ7ag78= 2024-04-12T03:54:02.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KQTzEzCBrDSj+S/17FR+rQ7ag78=', 'Content-Length': '0'} 2024-04-12T03:54:02.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d862ba4c91e26e0e-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:54:02.940 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.940 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.940 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.940 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.941 DEBUG:boto:Method: GET 2024-04-12T03:54:02.941 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.941 DEBUG:boto:Data: 2024-04-12T03:54:02.941 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.941 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.941 DEBUG:boto:Params: {} 2024-04-12T03:54:02.941 DEBUG:boto:Token: None 2024-04-12T03:54:02.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.941 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc= 2024-04-12T03:54:02.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc=', 'Content-Length': '0'} 2024-04-12T03:54:02.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008226d24cfcd8824d-006618b05a-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.943 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.943 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.944 DEBUG:boto:path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.944 DEBUG:boto:auth_path=/dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.944 DEBUG:boto:Method: GET 2024-04-12T03:54:02.944 DEBUG:boto:Path: /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.944 DEBUG:boto:Data: 2024-04-12T03:54:02.944 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.944 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.944 DEBUG:boto:Params: {} 2024-04-12T03:54:02.944 DEBUG:boto:Token: None 2024-04-12T03:54:02.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17?versionId=null 2024-04-12T03:54:02.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc= 2024-04-12T03:54:02.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yDi61hmH/7hRPb9drvk4I75F1Mc=', 'Content-Length': '0'} 2024-04-12T03:54:02.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ee47faa3b7a15f-006618b05a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.947 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.947 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.947 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.947 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.947 DEBUG:boto:Method: GET 2024-04-12T03:54:02.947 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.947 DEBUG:boto:Data: 2024-04-12T03:54:02.947 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.947 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.947 DEBUG:boto:Params: {} 2024-04-12T03:54:02.948 DEBUG:boto:Token: None 2024-04-12T03:54:02.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d97fcc3c44776e93-006618b05a-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.990 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.991 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:02.991 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:02.991 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.991 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.991 DEBUG:boto:Method: GET 2024-04-12T03:54:02.991 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:54:02.991 DEBUG:boto:Data: 2024-04-12T03:54:02.991 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:02.991 DEBUG:boto:Port: 8000 2024-04-12T03:54:02.992 DEBUG:boto:Params: {} 2024-04-12T03:54:02.992 DEBUG:boto:Token: None 2024-04-12T03:54:02.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/ 2024-04-12T03:54:02.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM= 2024-04-12T03:54:02.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpewPsE3Ci3kNzmlk6Pyr2HLhvM=', 'Content-Length': '0'} 2024-04-12T03:54:02.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620a496d1a8dad50-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.996 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:02.996 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.997 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:02.997 DEBUG:boto:Method: HEAD 2024-04-12T03:54:02.997 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:54:02.997 DEBUG:boto:Data: 2024-04-12T03:54:02.997 DEBUG:boto:Headers: {} 2024-04-12T03:54:02.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:02.997 DEBUG:boto:Port: 8001 2024-04-12T03:54:02.997 DEBUG:boto:Params: {} 2024-04-12T03:54:02.997 DEBUG:boto:Token: None 2024-04-12T03:54:02.997 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:02 GMT /dgbpxg-61/obj17 2024-04-12T03:54:02.998 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i4BBF6iJxulbWsVbC/tT1NXMd9M= 2024-04-12T03:54:02.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:02 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i4BBF6iJxulbWsVbC/tT1NXMd9M=', 'Content-Length': '0'} 2024-04-12T03:54:02.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170a40cde8207ed4-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:02.999 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:02.999 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:03.000 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.000 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:54:03.000 DEBUG:boto:Data: 2024-04-12T03:54:03.000 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.000 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.000 DEBUG:boto:Params: {} 2024-04-12T03:54:03.000 DEBUG:boto:Token: None 2024-04-12T03:54:03.000 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj17 2024-04-12T03:54:03.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/VjrzG00HO2reSy2QtmZe48xNTQ= 2024-04-12T03:54:03.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VjrzG00HO2reSy2QtmZe48xNTQ=', 'Content-Length': '0'} 2024-04-12T03:54:03.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916859076dbaf35c-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T03:54:03.002 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:03.003 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:03.003 DEBUG:boto:Method: GET 2024-04-12T03:54:03.003 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:54:03.003 DEBUG:boto:Data: 2024-04-12T03:54:03.003 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.003 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.003 DEBUG:boto:Params: {} 2024-04-12T03:54:03.003 DEBUG:boto:Token: None 2024-04-12T03:54:03.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj17 2024-04-12T03:54:03.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SvF7nRAkwaFYvGDy1Ct+NUrMGUI= 2024-04-12T03:54:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SvF7nRAkwaFYvGDy1Ct+NUrMGUI=', 'Content-Length': '0'} 2024-04-12T03:54:03.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309b511d8768402e-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.005 DEBUG:boto:path=/dgbpxg-61/obj17 2024-04-12T03:54:03.005 DEBUG:boto:auth_path=/dgbpxg-61/obj17 2024-04-12T03:54:03.006 DEBUG:boto:Method: GET 2024-04-12T03:54:03.006 DEBUG:boto:Path: /dgbpxg-61/obj17 2024-04-12T03:54:03.006 DEBUG:boto:Data: 2024-04-12T03:54:03.006 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.006 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.006 DEBUG:boto:Params: {} 2024-04-12T03:54:03.006 DEBUG:boto:Token: None 2024-04-12T03:54:03.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj17 2024-04-12T03:54:03.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SvF7nRAkwaFYvGDy1Ct+NUrMGUI= 2024-04-12T03:54:03.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SvF7nRAkwaFYvGDy1Ct+NUrMGUI=', 'Content-Length': '0'} 2024-04-12T03:54:03.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030896ac3595b9c18-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.008 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.008 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.009 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.009 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.009 DEBUG:boto:Method: GET 2024-04-12T03:54:03.009 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.009 DEBUG:boto:Data: 2024-04-12T03:54:03.009 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.009 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.009 DEBUG:boto:Params: {} 2024-04-12T03:54:03.009 DEBUG:boto:Token: None 2024-04-12T03:54:03.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.010 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb76673ee7127666-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.012 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.012 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.012 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.012 DEBUG:boto:path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.012 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.012 DEBUG:boto:Method: GET 2024-04-12T03:54:03.013 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj17 2024-04-12T03:54:03.013 DEBUG:boto:Data: 2024-04-12T03:54:03.013 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.013 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.013 DEBUG:boto:Params: {} 2024-04-12T03:54:03.013 DEBUG:boto:Token: None 2024-04-12T03:54:03.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026de66280a90aaba-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.017 DEBUG:boto:b'dgbpxg-61obj171000falseobj172024-04-12T03:41:32.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:54:03.017 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.017 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.018 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.018 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.018 DEBUG:boto:Method: GET 2024-04-12T03:54:03.018 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.018 DEBUG:boto:Data: 2024-04-12T03:54:03.018 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.018 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.018 DEBUG:boto:Params: {} 2024-04-12T03:54:03.018 DEBUG:boto:Token: None 2024-04-12T03:54:03.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.019 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM= 2024-04-12T03:54:03.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM=', 'Content-Length': '0'} 2024-04-12T03:54:03.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009013db6d4f08d57b-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.066 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.067 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.067 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.067 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.067 DEBUG:boto:Method: GET 2024-04-12T03:54:03.067 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.067 DEBUG:boto:Data: 2024-04-12T03:54:03.067 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.067 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.067 DEBUG:boto:Params: {} 2024-04-12T03:54:03.067 DEBUG:boto:Token: None 2024-04-12T03:54:03.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM= 2024-04-12T03:54:03.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM=', 'Content-Length': '0'} 2024-04-12T03:54:03.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9892304ce93808a-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.070 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.070 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.070 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.070 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.070 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.070 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.070 DEBUG:boto:Data: 2024-04-12T03:54:03.070 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.070 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.071 DEBUG:boto:Params: {} 2024-04-12T03:54:03.071 DEBUG:boto:Token: None 2024-04-12T03:54:03.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YU8j7sZ+Qip/TXS0nLDnb6Hfdeg= 2024-04-12T03:54:03.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YU8j7sZ+Qip/TXS0nLDnb6Hfdeg=', 'Content-Length': '0'} 2024-04-12T03:54:03.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b500c58937513d19-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.073 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.073 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.073 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.073 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.073 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.073 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.073 DEBUG:boto:Data: 2024-04-12T03:54:03.073 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.074 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.074 DEBUG:boto:Params: {} 2024-04-12T03:54:03.074 DEBUG:boto:Token: None 2024-04-12T03:54:03.074 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.074 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YU8j7sZ+Qip/TXS0nLDnb6Hfdeg= 2024-04-12T03:54:03.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YU8j7sZ+Qip/TXS0nLDnb6Hfdeg=', 'Content-Length': '0'} 2024-04-12T03:54:03.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0cfc3f2ad8ec7f-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:54:03.076 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.076 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.076 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.076 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.076 DEBUG:boto:Method: GET 2024-04-12T03:54:03.077 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.077 DEBUG:boto:Data: 2024-04-12T03:54:03.077 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.077 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.077 DEBUG:boto:Params: {} 2024-04-12T03:54:03.077 DEBUG:boto:Token: None 2024-04-12T03:54:03.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.077 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM= 2024-04-12T03:54:03.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM=', 'Content-Length': '0'} 2024-04-12T03:54:03.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900edd3fbf85d229-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.079 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.080 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.080 DEBUG:boto:path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.080 DEBUG:boto:auth_path=/dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.080 DEBUG:boto:Method: GET 2024-04-12T03:54:03.080 DEBUG:boto:Path: /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.080 DEBUG:boto:Data: 2024-04-12T03:54:03.080 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.080 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.080 DEBUG:boto:Params: {} 2024-04-12T03:54:03.080 DEBUG:boto:Token: None 2024-04-12T03:54:03.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185?versionId=null 2024-04-12T03:54:03.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM= 2024-04-12T03:54:03.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MPdDZ7lGq1g0wIzKpDVcTaFnDEM=', 'Content-Length': '0'} 2024-04-12T03:54:03.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5dc705f1f3c8450-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.083 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.083 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.083 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.083 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.083 DEBUG:boto:Method: GET 2024-04-12T03:54:03.083 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.083 DEBUG:boto:Data: 2024-04-12T03:54:03.083 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.083 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.084 DEBUG:boto:Params: {} 2024-04-12T03:54:03.084 DEBUG:boto:Token: None 2024-04-12T03:54:03.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002297296b59e460b4-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.086 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.086 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.087 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.087 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.087 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.087 DEBUG:boto:Method: GET 2024-04-12T03:54:03.087 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.087 DEBUG:boto:Data: 2024-04-12T03:54:03.087 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.087 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.087 DEBUG:boto:Params: {} 2024-04-12T03:54:03.087 DEBUG:boto:Token: None 2024-04-12T03:54:03.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033312abd75160b15-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.134 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.135 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.135 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.135 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.135 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:54:03.135 DEBUG:boto:Data: 2024-04-12T03:54:03.135 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.135 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.136 DEBUG:boto:Params: {} 2024-04-12T03:54:03.136 DEBUG:boto:Token: None 2024-04-12T03:54:03.136 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185 2024-04-12T03:54:03.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BR25uBoQlSQ9jAiJeF1vEhZzkuI= 2024-04-12T03:54:03.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BR25uBoQlSQ9jAiJeF1vEhZzkuI=', 'Content-Length': '0'} 2024-04-12T03:54:03.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004976dd318f8d96c2-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.138 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.138 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.138 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.138 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:54:03.138 DEBUG:boto:Data: 2024-04-12T03:54:03.138 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.138 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.138 DEBUG:boto:Params: {} 2024-04-12T03:54:03.139 DEBUG:boto:Token: None 2024-04-12T03:54:03.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185 2024-04-12T03:54:03.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BR25uBoQlSQ9jAiJeF1vEhZzkuI= 2024-04-12T03:54:03.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BR25uBoQlSQ9jAiJeF1vEhZzkuI=', 'Content-Length': '0'} 2024-04-12T03:54:03.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049125ba48434fc99-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T03:54:03.141 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.141 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.141 DEBUG:boto:Method: GET 2024-04-12T03:54:03.141 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:54:03.141 DEBUG:boto:Data: 2024-04-12T03:54:03.141 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.142 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.142 DEBUG:boto:Params: {} 2024-04-12T03:54:03.142 DEBUG:boto:Token: None 2024-04-12T03:54:03.142 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185 2024-04-12T03:54:03.142 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1fq0v620dyH5OJVByPVd6IokLhA= 2024-04-12T03:54:03.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1fq0v620dyH5OJVByPVd6IokLhA=', 'Content-Length': '0'} 2024-04-12T03:54:03.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193d6d6e3ac188fb-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.144 DEBUG:boto:path=/dgbpxg-61/obj185 2024-04-12T03:54:03.144 DEBUG:boto:auth_path=/dgbpxg-61/obj185 2024-04-12T03:54:03.144 DEBUG:boto:Method: GET 2024-04-12T03:54:03.144 DEBUG:boto:Path: /dgbpxg-61/obj185 2024-04-12T03:54:03.144 DEBUG:boto:Data: 2024-04-12T03:54:03.144 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.144 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.144 DEBUG:boto:Params: {} 2024-04-12T03:54:03.145 DEBUG:boto:Token: None 2024-04-12T03:54:03.145 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj185 2024-04-12T03:54:03.145 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1fq0v620dyH5OJVByPVd6IokLhA= 2024-04-12T03:54:03.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1fq0v620dyH5OJVByPVd6IokLhA=', 'Content-Length': '0'} 2024-04-12T03:54:03.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406e97d4b7aae99c-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.147 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.147 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.147 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.147 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.147 DEBUG:boto:Method: GET 2024-04-12T03:54:03.147 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.147 DEBUG:boto:Data: 2024-04-12T03:54:03.147 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.148 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.148 DEBUG:boto:Params: {} 2024-04-12T03:54:03.148 DEBUG:boto:Token: None 2024-04-12T03:54:03.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375aafafe81e94f3-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.150 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.150 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.151 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.151 DEBUG:boto:path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.151 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.151 DEBUG:boto:Method: GET 2024-04-12T03:54:03.151 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj185 2024-04-12T03:54:03.151 DEBUG:boto:Data: 2024-04-12T03:54:03.151 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.151 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.151 DEBUG:boto:Params: {} 2024-04-12T03:54:03.151 DEBUG:boto:Token: None 2024-04-12T03:54:03.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa65bfa8527f1a88-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.155 DEBUG:boto:b'dgbpxg-61obj1851000falseobj1852024-04-12T03:51:42.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:54:03.156 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.156 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.156 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.156 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.156 DEBUG:boto:Method: GET 2024-04-12T03:54:03.156 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.156 DEBUG:boto:Data: 2024-04-12T03:54:03.156 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.156 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.156 DEBUG:boto:Params: {} 2024-04-12T03:54:03.157 DEBUG:boto:Token: None 2024-04-12T03:54:03.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8= 2024-04-12T03:54:03.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8=', 'Content-Length': '0'} 2024-04-12T03:54:03.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca6a9b3b106a6ab-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.159 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.159 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.159 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.159 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.160 DEBUG:boto:Method: GET 2024-04-12T03:54:03.160 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.160 DEBUG:boto:Data: 2024-04-12T03:54:03.160 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.160 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.160 DEBUG:boto:Params: {} 2024-04-12T03:54:03.160 DEBUG:boto:Token: None 2024-04-12T03:54:03.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8= 2024-04-12T03:54:03.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8=', 'Content-Length': '0'} 2024-04-12T03:54:03.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000312c38ffd7f7c6d-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.202 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.202 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.203 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.203 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.203 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.203 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.203 DEBUG:boto:Data: 2024-04-12T03:54:03.203 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.203 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.203 DEBUG:boto:Params: {} 2024-04-12T03:54:03.203 DEBUG:boto:Token: None 2024-04-12T03:54:03.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.204 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Hqx27rgo4i7ejsl5qlZidInCa0= 2024-04-12T03:54:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Hqx27rgo4i7ejsl5qlZidInCa0=', 'Content-Length': '0'} 2024-04-12T03:54:03.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fc799db5386492-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.205 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.205 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.205 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.206 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.206 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.206 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.206 DEBUG:boto:Data: 2024-04-12T03:54:03.206 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.206 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.206 DEBUG:boto:Params: {} 2024-04-12T03:54:03.206 DEBUG:boto:Token: None 2024-04-12T03:54:03.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.207 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5Hqx27rgo4i7ejsl5qlZidInCa0= 2024-04-12T03:54:03.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5Hqx27rgo4i7ejsl5qlZidInCa0=', 'Content-Length': '0'} 2024-04-12T03:54:03.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd030b3871a29b05-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:54:03.208 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.209 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.209 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.209 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.209 DEBUG:boto:Method: GET 2024-04-12T03:54:03.209 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.209 DEBUG:boto:Data: 2024-04-12T03:54:03.209 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.209 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.209 DEBUG:boto:Params: {} 2024-04-12T03:54:03.209 DEBUG:boto:Token: None 2024-04-12T03:54:03.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.210 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8= 2024-04-12T03:54:03.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8=', 'Content-Length': '0'} 2024-04-12T03:54:03.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042aca0911c11e7e6-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.211 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.211 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.212 DEBUG:boto:path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.212 DEBUG:boto:auth_path=/dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.212 DEBUG:boto:Method: GET 2024-04-12T03:54:03.212 DEBUG:boto:Path: /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.212 DEBUG:boto:Data: 2024-04-12T03:54:03.212 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.212 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.212 DEBUG:boto:Params: {} 2024-04-12T03:54:03.212 DEBUG:boto:Token: None 2024-04-12T03:54:03.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19?versionId=null 2024-04-12T03:54:03.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8= 2024-04-12T03:54:03.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yvl4UzGACV+k+TAasqdlkN5yYw8=', 'Content-Length': '0'} 2024-04-12T03:54:03.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5490b2ec7f337d-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.215 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.215 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.215 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.215 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.215 DEBUG:boto:Method: GET 2024-04-12T03:54:03.215 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.215 DEBUG:boto:Data: 2024-04-12T03:54:03.215 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.215 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.215 DEBUG:boto:Params: {} 2024-04-12T03:54:03.215 DEBUG:boto:Token: None 2024-04-12T03:54:03.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe61a66d8067fb37-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.218 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.218 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.219 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.219 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.219 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.219 DEBUG:boto:Method: GET 2024-04-12T03:54:03.219 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.219 DEBUG:boto:Data: 2024-04-12T03:54:03.219 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.219 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.219 DEBUG:boto:Params: {} 2024-04-12T03:54:03.219 DEBUG:boto:Token: None 2024-04-12T03:54:03.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322d9bc657e651a5-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.223 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.224 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.224 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.224 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.224 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:54:03.224 DEBUG:boto:Data: 2024-04-12T03:54:03.224 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.224 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.224 DEBUG:boto:Params: {} 2024-04-12T03:54:03.224 DEBUG:boto:Token: None 2024-04-12T03:54:03.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19 2024-04-12T03:54:03.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFZGwX+yOy7n2y0MlwH43p04qFo= 2024-04-12T03:54:03.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFZGwX+yOy7n2y0MlwH43p04qFo=', 'Content-Length': '0'} 2024-04-12T03:54:03.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005409b5bcc2cf9c6e-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.226 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.227 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.227 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.227 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:54:03.227 DEBUG:boto:Data: 2024-04-12T03:54:03.227 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.227 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.227 DEBUG:boto:Params: {} 2024-04-12T03:54:03.227 DEBUG:boto:Token: None 2024-04-12T03:54:03.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19 2024-04-12T03:54:03.228 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oFZGwX+yOy7n2y0MlwH43p04qFo= 2024-04-12T03:54:03.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oFZGwX+yOy7n2y0MlwH43p04qFo=', 'Content-Length': '0'} 2024-04-12T03:54:03.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb1a7eb324b92ed-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T03:54:03.229 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.229 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.230 DEBUG:boto:Method: GET 2024-04-12T03:54:03.230 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:54:03.230 DEBUG:boto:Data: 2024-04-12T03:54:03.230 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.230 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.230 DEBUG:boto:Params: {} 2024-04-12T03:54:03.230 DEBUG:boto:Token: None 2024-04-12T03:54:03.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19 2024-04-12T03:54:03.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dxlHAHAXhP7KE8sKi0JsysV6UgE= 2024-04-12T03:54:03.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dxlHAHAXhP7KE8sKi0JsysV6UgE=', 'Content-Length': '0'} 2024-04-12T03:54:03.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec5bf0ac5ebf9bf1-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.274 DEBUG:boto:path=/dgbpxg-61/obj19 2024-04-12T03:54:03.274 DEBUG:boto:auth_path=/dgbpxg-61/obj19 2024-04-12T03:54:03.275 DEBUG:boto:Method: GET 2024-04-12T03:54:03.275 DEBUG:boto:Path: /dgbpxg-61/obj19 2024-04-12T03:54:03.275 DEBUG:boto:Data: 2024-04-12T03:54:03.275 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.275 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.275 DEBUG:boto:Params: {} 2024-04-12T03:54:03.275 DEBUG:boto:Token: None 2024-04-12T03:54:03.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj19 2024-04-12T03:54:03.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dxlHAHAXhP7KE8sKi0JsysV6UgE= 2024-04-12T03:54:03.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dxlHAHAXhP7KE8sKi0JsysV6UgE=', 'Content-Length': '0'} 2024-04-12T03:54:03.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005616813ae6a3eaf4-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.277 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.277 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.277 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.277 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.277 DEBUG:boto:Method: GET 2024-04-12T03:54:03.278 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.278 DEBUG:boto:Data: 2024-04-12T03:54:03.278 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.278 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.278 DEBUG:boto:Params: {} 2024-04-12T03:54:03.278 DEBUG:boto:Token: None 2024-04-12T03:54:03.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.278 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e62dab98e806a377-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.281 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.281 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.281 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.281 DEBUG:boto:path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.281 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.281 DEBUG:boto:Method: GET 2024-04-12T03:54:03.282 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj19 2024-04-12T03:54:03.282 DEBUG:boto:Data: 2024-04-12T03:54:03.282 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.282 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.282 DEBUG:boto:Params: {} 2024-04-12T03:54:03.282 DEBUG:boto:Token: None 2024-04-12T03:54:03.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.282 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168e25edc2801995-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.286 DEBUG:boto:b'dgbpxg-61obj191000falseobj192024-04-12T03:43:54.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:54:03.286 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.287 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.287 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.287 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.287 DEBUG:boto:Method: GET 2024-04-12T03:54:03.287 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.287 DEBUG:boto:Data: 2024-04-12T03:54:03.287 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.287 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.287 DEBUG:boto:Params: {} 2024-04-12T03:54:03.287 DEBUG:boto:Token: None 2024-04-12T03:54:03.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI= 2024-04-12T03:54:03.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI=', 'Content-Length': '0'} 2024-04-12T03:54:03.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be325416bb9f9bf-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.290 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.290 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.290 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.290 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.290 DEBUG:boto:Method: GET 2024-04-12T03:54:03.290 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.290 DEBUG:boto:Data: 2024-04-12T03:54:03.290 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.291 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.291 DEBUG:boto:Params: {} 2024-04-12T03:54:03.291 DEBUG:boto:Token: None 2024-04-12T03:54:03.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI= 2024-04-12T03:54:03.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI=', 'Content-Length': '0'} 2024-04-12T03:54:03.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d223bded0888958b-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.293 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.293 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.293 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.293 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.293 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.293 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.293 DEBUG:boto:Data: 2024-04-12T03:54:03.294 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.294 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.294 DEBUG:boto:Params: {} 2024-04-12T03:54:03.294 DEBUG:boto:Token: None 2024-04-12T03:54:03.294 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Co1scwAV8wAAY1qq+ss1vvPfK2E= 2024-04-12T03:54:03.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Co1scwAV8wAAY1qq+ss1vvPfK2E=', 'Content-Length': '0'} 2024-04-12T03:54:03.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a2f5d192e68bf4-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.296 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.296 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.296 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.296 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.296 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.296 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.296 DEBUG:boto:Data: 2024-04-12T03:54:03.297 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.297 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.297 DEBUG:boto:Params: {} 2024-04-12T03:54:03.297 DEBUG:boto:Token: None 2024-04-12T03:54:03.297 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.297 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Co1scwAV8wAAY1qq+ss1vvPfK2E= 2024-04-12T03:54:03.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Co1scwAV8wAAY1qq+ss1vvPfK2E=', 'Content-Length': '0'} 2024-04-12T03:54:03.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68bf4dae2c3a0bd-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:54:03.299 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.299 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.299 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.299 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.299 DEBUG:boto:Method: GET 2024-04-12T03:54:03.299 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.299 DEBUG:boto:Data: 2024-04-12T03:54:03.300 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.300 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.300 DEBUG:boto:Params: {} 2024-04-12T03:54:03.300 DEBUG:boto:Token: None 2024-04-12T03:54:03.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI= 2024-04-12T03:54:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI=', 'Content-Length': '0'} 2024-04-12T03:54:03.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e7217a92f4102b-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.302 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.302 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.302 DEBUG:boto:path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.302 DEBUG:boto:auth_path=/dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.302 DEBUG:boto:Method: GET 2024-04-12T03:54:03.302 DEBUG:boto:Path: /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.303 DEBUG:boto:Data: 2024-04-12T03:54:03.303 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.303 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.303 DEBUG:boto:Params: {} 2024-04-12T03:54:03.303 DEBUG:boto:Token: None 2024-04-12T03:54:03.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22?versionId=null 2024-04-12T03:54:03.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI= 2024-04-12T03:54:03.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xflij+aJYZuvap5TIkw20HLj4cI=', 'Content-Length': '0'} 2024-04-12T03:54:03.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b7327b93cc2a2e-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.305 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.305 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.305 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.305 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.305 DEBUG:boto:Method: GET 2024-04-12T03:54:03.305 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.306 DEBUG:boto:Data: 2024-04-12T03:54:03.306 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.306 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.306 DEBUG:boto:Params: {} 2024-04-12T03:54:03.306 DEBUG:boto:Token: None 2024-04-12T03:54:03.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.306 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc2f5b8f418e0643-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.308 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.309 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.309 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.309 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.309 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.309 DEBUG:boto:Method: GET 2024-04-12T03:54:03.309 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.309 DEBUG:boto:Data: 2024-04-12T03:54:03.309 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.310 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.310 DEBUG:boto:Params: {} 2024-04-12T03:54:03.310 DEBUG:boto:Token: None 2024-04-12T03:54:03.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.310 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001202466ddeb2f9eb-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.314 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.314 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.314 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.314 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.314 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:54:03.315 DEBUG:boto:Data: 2024-04-12T03:54:03.315 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.315 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.315 DEBUG:boto:Params: {} 2024-04-12T03:54:03.315 DEBUG:boto:Token: None 2024-04-12T03:54:03.315 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22 2024-04-12T03:54:03.315 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zR7ea4wGC18NugJ8tXAFQTr/8K0= 2024-04-12T03:54:03.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zR7ea4wGC18NugJ8tXAFQTr/8K0=', 'Content-Length': '0'} 2024-04-12T03:54:03.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515de1fe0fc9a77a-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.317 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.317 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.317 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.317 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:54:03.317 DEBUG:boto:Data: 2024-04-12T03:54:03.317 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.318 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.318 DEBUG:boto:Params: {} 2024-04-12T03:54:03.318 DEBUG:boto:Token: None 2024-04-12T03:54:03.318 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22 2024-04-12T03:54:03.318 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zR7ea4wGC18NugJ8tXAFQTr/8K0= 2024-04-12T03:54:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zR7ea4wGC18NugJ8tXAFQTr/8K0=', 'Content-Length': '0'} 2024-04-12T03:54:03.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007540d0ea0fa5e227-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:54:03.320 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.320 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.320 DEBUG:boto:Method: GET 2024-04-12T03:54:03.320 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:54:03.320 DEBUG:boto:Data: 2024-04-12T03:54:03.320 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.320 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.320 DEBUG:boto:Params: {} 2024-04-12T03:54:03.321 DEBUG:boto:Token: None 2024-04-12T03:54:03.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22 2024-04-12T03:54:03.321 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:psvzIVNFutXWVc25cV279cPA5b4= 2024-04-12T03:54:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:psvzIVNFutXWVc25cV279cPA5b4=', 'Content-Length': '0'} 2024-04-12T03:54:03.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318214dfe0495ed4-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.323 DEBUG:boto:path=/dgbpxg-61/obj22 2024-04-12T03:54:03.323 DEBUG:boto:auth_path=/dgbpxg-61/obj22 2024-04-12T03:54:03.323 DEBUG:boto:Method: GET 2024-04-12T03:54:03.323 DEBUG:boto:Path: /dgbpxg-61/obj22 2024-04-12T03:54:03.323 DEBUG:boto:Data: 2024-04-12T03:54:03.323 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.324 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.324 DEBUG:boto:Params: {} 2024-04-12T03:54:03.324 DEBUG:boto:Token: None 2024-04-12T03:54:03.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj22 2024-04-12T03:54:03.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:psvzIVNFutXWVc25cV279cPA5b4= 2024-04-12T03:54:03.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:psvzIVNFutXWVc25cV279cPA5b4=', 'Content-Length': '0'} 2024-04-12T03:54:03.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f2585e93462a76-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.326 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.326 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.326 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.326 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.326 DEBUG:boto:Method: GET 2024-04-12T03:54:03.326 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.326 DEBUG:boto:Data: 2024-04-12T03:54:03.326 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.327 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.327 DEBUG:boto:Params: {} 2024-04-12T03:54:03.327 DEBUG:boto:Token: None 2024-04-12T03:54:03.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.327 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c2ec4a5342c608-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.329 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.330 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.330 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.330 DEBUG:boto:path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.330 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.330 DEBUG:boto:Method: GET 2024-04-12T03:54:03.330 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj22 2024-04-12T03:54:03.330 DEBUG:boto:Data: 2024-04-12T03:54:03.330 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.330 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.330 DEBUG:boto:Params: {} 2024-04-12T03:54:03.331 DEBUG:boto:Token: None 2024-04-12T03:54:03.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.331 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8138fc5383d7030-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.334 DEBUG:boto:b'dgbpxg-61obj221000falseobj222024-04-12T03:40:51.483Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:54:03.335 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.335 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.335 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.335 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.335 DEBUG:boto:Method: GET 2024-04-12T03:54:03.335 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.335 DEBUG:boto:Data: 2024-04-12T03:54:03.336 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.336 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.336 DEBUG:boto:Params: {} 2024-04-12T03:54:03.336 DEBUG:boto:Token: None 2024-04-12T03:54:03.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc= 2024-04-12T03:54:03.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc=', 'Content-Length': '0'} 2024-04-12T03:54:03.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fdeda4a0020c2cb-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.338 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.338 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.338 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.338 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.338 DEBUG:boto:Method: GET 2024-04-12T03:54:03.338 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.339 DEBUG:boto:Data: 2024-04-12T03:54:03.339 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.339 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.339 DEBUG:boto:Params: {} 2024-04-12T03:54:03.339 DEBUG:boto:Token: None 2024-04-12T03:54:03.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc= 2024-04-12T03:54:03.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc=', 'Content-Length': '0'} 2024-04-12T03:54:03.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a8d31152d1bd08-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.341 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.341 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.341 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.341 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.341 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.342 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.342 DEBUG:boto:Data: 2024-04-12T03:54:03.342 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.342 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.342 DEBUG:boto:Params: {} 2024-04-12T03:54:03.342 DEBUG:boto:Token: None 2024-04-12T03:54:03.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.342 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mDqd0vl99kVkNtxPktrz7UOZ5jM= 2024-04-12T03:54:03.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mDqd0vl99kVkNtxPktrz7UOZ5jM=', 'Content-Length': '0'} 2024-04-12T03:54:03.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044a41d33d26460a-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.344 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.344 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.345 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.345 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.345 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.345 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.345 DEBUG:boto:Data: 2024-04-12T03:54:03.345 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.345 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.345 DEBUG:boto:Params: {} 2024-04-12T03:54:03.345 DEBUG:boto:Token: None 2024-04-12T03:54:03.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.346 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mDqd0vl99kVkNtxPktrz7UOZ5jM= 2024-04-12T03:54:03.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mDqd0vl99kVkNtxPktrz7UOZ5jM=', 'Content-Length': '0'} 2024-04-12T03:54:03.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002370d0f0f9886050-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:54:03.348 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.348 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.348 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.348 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.348 DEBUG:boto:Method: GET 2024-04-12T03:54:03.348 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.348 DEBUG:boto:Data: 2024-04-12T03:54:03.348 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.349 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.349 DEBUG:boto:Params: {} 2024-04-12T03:54:03.349 DEBUG:boto:Token: None 2024-04-12T03:54:03.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc= 2024-04-12T03:54:03.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc=', 'Content-Length': '0'} 2024-04-12T03:54:03.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb3b7dea10bac39-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.351 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.351 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.351 DEBUG:boto:path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.351 DEBUG:boto:auth_path=/dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.351 DEBUG:boto:Method: GET 2024-04-12T03:54:03.351 DEBUG:boto:Path: /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.351 DEBUG:boto:Data: 2024-04-12T03:54:03.352 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.352 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.352 DEBUG:boto:Params: {} 2024-04-12T03:54:03.352 DEBUG:boto:Token: None 2024-04-12T03:54:03.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23?versionId=null 2024-04-12T03:54:03.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc= 2024-04-12T03:54:03.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AJ0+sYcqFF3pPBjriReW851rcLc=', 'Content-Length': '0'} 2024-04-12T03:54:03.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4e8b240bea70b2-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.354 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.354 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.354 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.354 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.354 DEBUG:boto:Method: GET 2024-04-12T03:54:03.354 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.355 DEBUG:boto:Data: 2024-04-12T03:54:03.355 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.355 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.355 DEBUG:boto:Params: {} 2024-04-12T03:54:03.355 DEBUG:boto:Token: None 2024-04-12T03:54:03.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b71dc289293b130-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.398 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.399 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.399 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.399 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.399 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.399 DEBUG:boto:Method: GET 2024-04-12T03:54:03.399 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.399 DEBUG:boto:Data: 2024-04-12T03:54:03.399 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.400 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.400 DEBUG:boto:Params: {} 2024-04-12T03:54:03.400 DEBUG:boto:Token: None 2024-04-12T03:54:03.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025618d0db961580a-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.404 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.404 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.404 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.405 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.405 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:54:03.405 DEBUG:boto:Data: 2024-04-12T03:54:03.405 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.405 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.405 DEBUG:boto:Params: {} 2024-04-12T03:54:03.405 DEBUG:boto:Token: None 2024-04-12T03:54:03.405 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23 2024-04-12T03:54:03.406 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LOVx+h10qUddapETSL0DzJ1chfA= 2024-04-12T03:54:03.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LOVx+h10qUddapETSL0DzJ1chfA=', 'Content-Length': '0'} 2024-04-12T03:54:03.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c63de1bcf87ed75-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.407 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.407 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.407 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.408 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:54:03.408 DEBUG:boto:Data: 2024-04-12T03:54:03.408 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.408 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.408 DEBUG:boto:Params: {} 2024-04-12T03:54:03.408 DEBUG:boto:Token: None 2024-04-12T03:54:03.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23 2024-04-12T03:54:03.408 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LOVx+h10qUddapETSL0DzJ1chfA= 2024-04-12T03:54:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LOVx+h10qUddapETSL0DzJ1chfA=', 'Content-Length': '0'} 2024-04-12T03:54:03.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a414c6de0fcadf7-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T03:54:03.410 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.410 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.410 DEBUG:boto:Method: GET 2024-04-12T03:54:03.410 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:54:03.411 DEBUG:boto:Data: 2024-04-12T03:54:03.411 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.411 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.411 DEBUG:boto:Params: {} 2024-04-12T03:54:03.411 DEBUG:boto:Token: None 2024-04-12T03:54:03.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23 2024-04-12T03:54:03.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BqiBGBFwOUWoUT5GusWp8qV3EQA= 2024-04-12T03:54:03.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BqiBGBFwOUWoUT5GusWp8qV3EQA=', 'Content-Length': '0'} 2024-04-12T03:54:03.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8710b7fa9929d61-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.413 DEBUG:boto:path=/dgbpxg-61/obj23 2024-04-12T03:54:03.413 DEBUG:boto:auth_path=/dgbpxg-61/obj23 2024-04-12T03:54:03.413 DEBUG:boto:Method: GET 2024-04-12T03:54:03.413 DEBUG:boto:Path: /dgbpxg-61/obj23 2024-04-12T03:54:03.413 DEBUG:boto:Data: 2024-04-12T03:54:03.414 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.414 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.414 DEBUG:boto:Params: {} 2024-04-12T03:54:03.414 DEBUG:boto:Token: None 2024-04-12T03:54:03.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj23 2024-04-12T03:54:03.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BqiBGBFwOUWoUT5GusWp8qV3EQA= 2024-04-12T03:54:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BqiBGBFwOUWoUT5GusWp8qV3EQA=', 'Content-Length': '0'} 2024-04-12T03:54:03.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006318f594eae3a850-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.458 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.459 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.459 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.459 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.459 DEBUG:boto:Method: GET 2024-04-12T03:54:03.459 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.459 DEBUG:boto:Data: 2024-04-12T03:54:03.459 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.459 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.459 DEBUG:boto:Params: {} 2024-04-12T03:54:03.460 DEBUG:boto:Token: None 2024-04-12T03:54:03.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da18a279a657c24-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.462 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.463 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.463 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.463 DEBUG:boto:path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.463 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.463 DEBUG:boto:Method: GET 2024-04-12T03:54:03.463 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj23 2024-04-12T03:54:03.463 DEBUG:boto:Data: 2024-04-12T03:54:03.463 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.463 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.464 DEBUG:boto:Params: {} 2024-04-12T03:54:03.464 DEBUG:boto:Token: None 2024-04-12T03:54:03.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf9e01be3e93ce4e-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.468 DEBUG:boto:b'dgbpxg-61obj231000falseobj232024-04-12T03:46:16.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:54:03.468 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.468 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.468 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.468 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.469 DEBUG:boto:Method: GET 2024-04-12T03:54:03.469 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.469 DEBUG:boto:Data: 2024-04-12T03:54:03.469 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.469 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.469 DEBUG:boto:Params: {} 2024-04-12T03:54:03.469 DEBUG:boto:Token: None 2024-04-12T03:54:03.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q= 2024-04-12T03:54:03.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q=', 'Content-Length': '0'} 2024-04-12T03:54:03.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be217ce7989ba347-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.472 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.472 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.472 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.472 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.472 DEBUG:boto:Method: GET 2024-04-12T03:54:03.472 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.472 DEBUG:boto:Data: 2024-04-12T03:54:03.472 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.472 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.472 DEBUG:boto:Params: {} 2024-04-12T03:54:03.473 DEBUG:boto:Token: None 2024-04-12T03:54:03.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q= 2024-04-12T03:54:03.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q=', 'Content-Length': '0'} 2024-04-12T03:54:03.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8151b3f633761d-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.475 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.475 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.475 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.475 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.475 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.475 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.475 DEBUG:boto:Data: 2024-04-12T03:54:03.475 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.476 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.476 DEBUG:boto:Params: {} 2024-04-12T03:54:03.476 DEBUG:boto:Token: None 2024-04-12T03:54:03.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vPQmlBFL+ERirFagZk3mlOLZ0mw= 2024-04-12T03:54:03.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vPQmlBFL+ERirFagZk3mlOLZ0mw=', 'Content-Length': '0'} 2024-04-12T03:54:03.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000706f00c14d7af801-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.478 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.478 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.478 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.478 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.478 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.478 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.478 DEBUG:boto:Data: 2024-04-12T03:54:03.478 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.478 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.479 DEBUG:boto:Params: {} 2024-04-12T03:54:03.479 DEBUG:boto:Token: None 2024-04-12T03:54:03.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vPQmlBFL+ERirFagZk3mlOLZ0mw= 2024-04-12T03:54:03.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vPQmlBFL+ERirFagZk3mlOLZ0mw=', 'Content-Length': '0'} 2024-04-12T03:54:03.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc374573b0873dc8-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:54:03.481 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.481 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.481 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.481 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.481 DEBUG:boto:Method: GET 2024-04-12T03:54:03.481 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.481 DEBUG:boto:Data: 2024-04-12T03:54:03.481 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.481 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.481 DEBUG:boto:Params: {} 2024-04-12T03:54:03.482 DEBUG:boto:Token: None 2024-04-12T03:54:03.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.482 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q= 2024-04-12T03:54:03.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q=', 'Content-Length': '0'} 2024-04-12T03:54:03.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e376e7bf98fb413-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.526 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.527 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.527 DEBUG:boto:path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.527 DEBUG:boto:auth_path=/dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.527 DEBUG:boto:Method: GET 2024-04-12T03:54:03.527 DEBUG:boto:Path: /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.527 DEBUG:boto:Data: 2024-04-12T03:54:03.527 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.527 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.527 DEBUG:boto:Params: {} 2024-04-12T03:54:03.528 DEBUG:boto:Token: None 2024-04-12T03:54:03.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29?versionId=null 2024-04-12T03:54:03.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q= 2024-04-12T03:54:03.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AzTtOsSVMxsVlO4O5woOVGndg7Q=', 'Content-Length': '0'} 2024-04-12T03:54:03.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f0317b56c29274-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.530 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.530 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.530 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.531 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.531 DEBUG:boto:Method: GET 2024-04-12T03:54:03.531 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.531 DEBUG:boto:Data: 2024-04-12T03:54:03.531 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.531 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.531 DEBUG:boto:Params: {} 2024-04-12T03:54:03.531 DEBUG:boto:Token: None 2024-04-12T03:54:03.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d532937f021c0ae2-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.534 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.534 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.534 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.534 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.534 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.535 DEBUG:boto:Method: GET 2024-04-12T03:54:03.535 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.535 DEBUG:boto:Data: 2024-04-12T03:54:03.535 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.535 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.535 DEBUG:boto:Params: {} 2024-04-12T03:54:03.535 DEBUG:boto:Token: None 2024-04-12T03:54:03.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596b89286046c186-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.539 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.539 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.539 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.540 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.540 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:54:03.540 DEBUG:boto:Data: 2024-04-12T03:54:03.540 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.540 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.540 DEBUG:boto:Params: {} 2024-04-12T03:54:03.540 DEBUG:boto:Token: None 2024-04-12T03:54:03.540 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29 2024-04-12T03:54:03.540 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zc5cjvQ7S1t4f30br1tX6eTdasA= 2024-04-12T03:54:03.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zc5cjvQ7S1t4f30br1tX6eTdasA=', 'Content-Length': '0'} 2024-04-12T03:54:03.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018076f123ffced25-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.543 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.543 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.543 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.543 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:54:03.543 DEBUG:boto:Data: 2024-04-12T03:54:03.543 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.543 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.543 DEBUG:boto:Params: {} 2024-04-12T03:54:03.543 DEBUG:boto:Token: None 2024-04-12T03:54:03.543 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29 2024-04-12T03:54:03.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zc5cjvQ7S1t4f30br1tX6eTdasA= 2024-04-12T03:54:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zc5cjvQ7S1t4f30br1tX6eTdasA=', 'Content-Length': '0'} 2024-04-12T03:54:03.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cdd6bc2265b223-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T03:54:03.546 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.546 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.546 DEBUG:boto:Method: GET 2024-04-12T03:54:03.546 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:54:03.546 DEBUG:boto:Data: 2024-04-12T03:54:03.546 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.546 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.546 DEBUG:boto:Params: {} 2024-04-12T03:54:03.547 DEBUG:boto:Token: None 2024-04-12T03:54:03.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29 2024-04-12T03:54:03.547 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0ScE+gQ8eiZat1u9XjV1zB/zNL0= 2024-04-12T03:54:03.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0ScE+gQ8eiZat1u9XjV1zB/zNL0=', 'Content-Length': '0'} 2024-04-12T03:54:03.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002328a53b764ddd0f-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.549 DEBUG:boto:path=/dgbpxg-61/obj29 2024-04-12T03:54:03.549 DEBUG:boto:auth_path=/dgbpxg-61/obj29 2024-04-12T03:54:03.549 DEBUG:boto:Method: GET 2024-04-12T03:54:03.549 DEBUG:boto:Path: /dgbpxg-61/obj29 2024-04-12T03:54:03.549 DEBUG:boto:Data: 2024-04-12T03:54:03.549 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.549 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.549 DEBUG:boto:Params: {} 2024-04-12T03:54:03.549 DEBUG:boto:Token: None 2024-04-12T03:54:03.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj29 2024-04-12T03:54:03.550 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0ScE+gQ8eiZat1u9XjV1zB/zNL0= 2024-04-12T03:54:03.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0ScE+gQ8eiZat1u9XjV1zB/zNL0=', 'Content-Length': '0'} 2024-04-12T03:54:03.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003607d1db1ea67505-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.552 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.552 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.552 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.552 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.552 DEBUG:boto:Method: GET 2024-04-12T03:54:03.552 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.552 DEBUG:boto:Data: 2024-04-12T03:54:03.552 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.552 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.552 DEBUG:boto:Params: {} 2024-04-12T03:54:03.553 DEBUG:boto:Token: None 2024-04-12T03:54:03.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2a8943b68981025-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.555 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.555 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.555 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.555 DEBUG:boto:path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.555 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.555 DEBUG:boto:Method: GET 2024-04-12T03:54:03.556 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj29 2024-04-12T03:54:03.556 DEBUG:boto:Data: 2024-04-12T03:54:03.556 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.556 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.556 DEBUG:boto:Params: {} 2024-04-12T03:54:03.556 DEBUG:boto:Token: None 2024-04-12T03:54:03.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6310abbc2c4a77-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.560 DEBUG:boto:b'dgbpxg-61obj291000falseobj292024-04-12T03:47:19.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:54:03.560 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.560 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.560 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.560 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.561 DEBUG:boto:Method: GET 2024-04-12T03:54:03.561 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.561 DEBUG:boto:Data: 2024-04-12T03:54:03.561 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.561 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.561 DEBUG:boto:Params: {} 2024-04-12T03:54:03.561 DEBUG:boto:Token: None 2024-04-12T03:54:03.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.561 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg= 2024-04-12T03:54:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg=', 'Content-Length': '0'} 2024-04-12T03:54:03.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1179c5cc5fe387-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.563 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.564 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.564 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.564 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.564 DEBUG:boto:Method: GET 2024-04-12T03:54:03.564 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.564 DEBUG:boto:Data: 2024-04-12T03:54:03.564 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.564 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.564 DEBUG:boto:Params: {} 2024-04-12T03:54:03.564 DEBUG:boto:Token: None 2024-04-12T03:54:03.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.565 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg= 2024-04-12T03:54:03.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg=', 'Content-Length': '0'} 2024-04-12T03:54:03.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2903413a858709b-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.567 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.567 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.567 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.567 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.567 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.567 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.567 DEBUG:boto:Data: 2024-04-12T03:54:03.567 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.567 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.568 DEBUG:boto:Params: {} 2024-04-12T03:54:03.568 DEBUG:boto:Token: None 2024-04-12T03:54:03.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GNmO50zIK+fx2/I0MAx/yf9u0fQ= 2024-04-12T03:54:03.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GNmO50zIK+fx2/I0MAx/yf9u0fQ=', 'Content-Length': '0'} 2024-04-12T03:54:03.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000969fbaf746dda89-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.570 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.570 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.570 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.570 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.570 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.570 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.570 DEBUG:boto:Data: 2024-04-12T03:54:03.570 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.570 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.570 DEBUG:boto:Params: {} 2024-04-12T03:54:03.570 DEBUG:boto:Token: None 2024-04-12T03:54:03.571 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GNmO50zIK+fx2/I0MAx/yf9u0fQ= 2024-04-12T03:54:03.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GNmO50zIK+fx2/I0MAx/yf9u0fQ=', 'Content-Length': '0'} 2024-04-12T03:54:03.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bbcb8f497611af-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:54:03.573 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.573 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.573 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.573 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.573 DEBUG:boto:Method: GET 2024-04-12T03:54:03.573 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.573 DEBUG:boto:Data: 2024-04-12T03:54:03.573 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.573 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.573 DEBUG:boto:Params: {} 2024-04-12T03:54:03.573 DEBUG:boto:Token: None 2024-04-12T03:54:03.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.574 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg= 2024-04-12T03:54:03.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg=', 'Content-Length': '0'} 2024-04-12T03:54:03.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691797c4737d2057-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.618 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.618 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.618 DEBUG:boto:path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.619 DEBUG:boto:auth_path=/dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.619 DEBUG:boto:Method: GET 2024-04-12T03:54:03.619 DEBUG:boto:Path: /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.619 DEBUG:boto:Data: 2024-04-12T03:54:03.619 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.619 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.619 DEBUG:boto:Params: {} 2024-04-12T03:54:03.619 DEBUG:boto:Token: None 2024-04-12T03:54:03.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31?versionId=null 2024-04-12T03:54:03.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg= 2024-04-12T03:54:03.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1yaN8x16DKaSwVYxQ8DGP07cqHg=', 'Content-Length': '0'} 2024-04-12T03:54:03.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbbd20468c333c3b-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.621 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.621 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.622 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.622 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.622 DEBUG:boto:Method: GET 2024-04-12T03:54:03.622 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.622 DEBUG:boto:Data: 2024-04-12T03:54:03.622 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.622 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.622 DEBUG:boto:Params: {} 2024-04-12T03:54:03.622 DEBUG:boto:Token: None 2024-04-12T03:54:03.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018be69e6973beb28-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.625 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.625 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.625 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.625 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.625 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.625 DEBUG:boto:Method: GET 2024-04-12T03:54:03.626 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.626 DEBUG:boto:Data: 2024-04-12T03:54:03.626 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.626 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.626 DEBUG:boto:Params: {} 2024-04-12T03:54:03.626 DEBUG:boto:Token: None 2024-04-12T03:54:03.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005340bdfa31ef3441-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.630 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.631 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.631 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.631 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.631 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:54:03.631 DEBUG:boto:Data: 2024-04-12T03:54:03.631 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.631 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.631 DEBUG:boto:Params: {} 2024-04-12T03:54:03.631 DEBUG:boto:Token: None 2024-04-12T03:54:03.632 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31 2024-04-12T03:54:03.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:83xABm7JytgysTukAFAz1rl+1Ys= 2024-04-12T03:54:03.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:83xABm7JytgysTukAFAz1rl+1Ys=', 'Content-Length': '0'} 2024-04-12T03:54:03.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b733285e24e437-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.633 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.633 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.634 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.634 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:54:03.634 DEBUG:boto:Data: 2024-04-12T03:54:03.634 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.634 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.634 DEBUG:boto:Params: {} 2024-04-12T03:54:03.634 DEBUG:boto:Token: None 2024-04-12T03:54:03.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31 2024-04-12T03:54:03.635 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:83xABm7JytgysTukAFAz1rl+1Ys= 2024-04-12T03:54:03.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:83xABm7JytgysTukAFAz1rl+1Ys=', 'Content-Length': '0'} 2024-04-12T03:54:03.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5f12ecae2b82c8-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T03:54:03.636 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.636 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.637 DEBUG:boto:Method: GET 2024-04-12T03:54:03.637 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:54:03.637 DEBUG:boto:Data: 2024-04-12T03:54:03.637 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.637 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.637 DEBUG:boto:Params: {} 2024-04-12T03:54:03.637 DEBUG:boto:Token: None 2024-04-12T03:54:03.637 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31 2024-04-12T03:54:03.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kLAbJ+4gyda4mifwh38whwsIgq0= 2024-04-12T03:54:03.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kLAbJ+4gyda4mifwh38whwsIgq0=', 'Content-Length': '0'} 2024-04-12T03:54:03.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d775ec1d9fecc685-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.639 DEBUG:boto:path=/dgbpxg-61/obj31 2024-04-12T03:54:03.639 DEBUG:boto:auth_path=/dgbpxg-61/obj31 2024-04-12T03:54:03.639 DEBUG:boto:Method: GET 2024-04-12T03:54:03.640 DEBUG:boto:Path: /dgbpxg-61/obj31 2024-04-12T03:54:03.640 DEBUG:boto:Data: 2024-04-12T03:54:03.640 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.640 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.640 DEBUG:boto:Params: {} 2024-04-12T03:54:03.640 DEBUG:boto:Token: None 2024-04-12T03:54:03.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj31 2024-04-12T03:54:03.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kLAbJ+4gyda4mifwh38whwsIgq0= 2024-04-12T03:54:03.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kLAbJ+4gyda4mifwh38whwsIgq0=', 'Content-Length': '0'} 2024-04-12T03:54:03.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf167ae25d705c0-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.643 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.643 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.643 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.643 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.643 DEBUG:boto:Method: GET 2024-04-12T03:54:03.643 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.643 DEBUG:boto:Data: 2024-04-12T03:54:03.643 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.643 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.643 DEBUG:boto:Params: {} 2024-04-12T03:54:03.644 DEBUG:boto:Token: None 2024-04-12T03:54:03.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.644 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aa0084e1683f5bf-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.646 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.646 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.646 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.646 DEBUG:boto:path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.647 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.647 DEBUG:boto:Method: GET 2024-04-12T03:54:03.647 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj31 2024-04-12T03:54:03.647 DEBUG:boto:Data: 2024-04-12T03:54:03.647 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.647 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.647 DEBUG:boto:Params: {} 2024-04-12T03:54:03.647 DEBUG:boto:Token: None 2024-04-12T03:54:03.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1dbb892193058b-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.651 DEBUG:boto:b'dgbpxg-61obj311000falseobj312024-04-12T03:50:18.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:54:03.651 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.652 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.652 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.652 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.652 DEBUG:boto:Method: GET 2024-04-12T03:54:03.652 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.652 DEBUG:boto:Data: 2024-04-12T03:54:03.652 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.652 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.652 DEBUG:boto:Params: {} 2024-04-12T03:54:03.652 DEBUG:boto:Token: None 2024-04-12T03:54:03.653 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.653 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4= 2024-04-12T03:54:03.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4=', 'Content-Length': '0'} 2024-04-12T03:54:03.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7a00e6f3bce4d8-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.655 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.655 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.655 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.655 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.655 DEBUG:boto:Method: GET 2024-04-12T03:54:03.655 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.655 DEBUG:boto:Data: 2024-04-12T03:54:03.655 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.656 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.656 DEBUG:boto:Params: {} 2024-04-12T03:54:03.656 DEBUG:boto:Token: None 2024-04-12T03:54:03.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4= 2024-04-12T03:54:03.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4=', 'Content-Length': '0'} 2024-04-12T03:54:03.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db12186015b7de98-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.698 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.698 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.698 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.699 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.699 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.699 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.699 DEBUG:boto:Data: 2024-04-12T03:54:03.699 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.699 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.699 DEBUG:boto:Params: {} 2024-04-12T03:54:03.699 DEBUG:boto:Token: None 2024-04-12T03:54:03.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gyw5sl88ejNibeWhp+MTZq3SYak= 2024-04-12T03:54:03.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gyw5sl88ejNibeWhp+MTZq3SYak=', 'Content-Length': '0'} 2024-04-12T03:54:03.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e437d8128bda7c1-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.701 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.701 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.702 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.702 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.702 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.702 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.702 DEBUG:boto:Data: 2024-04-12T03:54:03.702 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.702 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.702 DEBUG:boto:Params: {} 2024-04-12T03:54:03.702 DEBUG:boto:Token: None 2024-04-12T03:54:03.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gyw5sl88ejNibeWhp+MTZq3SYak= 2024-04-12T03:54:03.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gyw5sl88ejNibeWhp+MTZq3SYak=', 'Content-Length': '0'} 2024-04-12T03:54:03.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d33ae82433d46c4-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:54:03.705 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.705 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.705 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.705 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.705 DEBUG:boto:Method: GET 2024-04-12T03:54:03.705 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.705 DEBUG:boto:Data: 2024-04-12T03:54:03.705 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.705 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.705 DEBUG:boto:Params: {} 2024-04-12T03:54:03.706 DEBUG:boto:Token: None 2024-04-12T03:54:03.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4= 2024-04-12T03:54:03.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4=', 'Content-Length': '0'} 2024-04-12T03:54:03.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086dbddad7e003286-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.708 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.708 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.708 DEBUG:boto:path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.708 DEBUG:boto:auth_path=/dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.708 DEBUG:boto:Method: GET 2024-04-12T03:54:03.708 DEBUG:boto:Path: /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.708 DEBUG:boto:Data: 2024-04-12T03:54:03.708 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.708 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.708 DEBUG:boto:Params: {} 2024-04-12T03:54:03.708 DEBUG:boto:Token: None 2024-04-12T03:54:03.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33?versionId=null 2024-04-12T03:54:03.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4= 2024-04-12T03:54:03.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wHnPQCH2xIvTDVwHKx0ekerNJa4=', 'Content-Length': '0'} 2024-04-12T03:54:03.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94f778a5b27cc84-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.711 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.711 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.711 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.711 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.711 DEBUG:boto:Method: GET 2024-04-12T03:54:03.711 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.711 DEBUG:boto:Data: 2024-04-12T03:54:03.711 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.712 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.712 DEBUG:boto:Params: {} 2024-04-12T03:54:03.712 DEBUG:boto:Token: None 2024-04-12T03:54:03.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd25fe439e1b5c8-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.714 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.714 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.715 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.715 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.715 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.715 DEBUG:boto:Method: GET 2024-04-12T03:54:03.715 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.715 DEBUG:boto:Data: 2024-04-12T03:54:03.715 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.715 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.715 DEBUG:boto:Params: {} 2024-04-12T03:54:03.715 DEBUG:boto:Token: None 2024-04-12T03:54:03.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035eaa94eaa791870-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.719 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.720 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.720 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.720 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.720 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:54:03.720 DEBUG:boto:Data: 2024-04-12T03:54:03.720 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.720 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.720 DEBUG:boto:Params: {} 2024-04-12T03:54:03.720 DEBUG:boto:Token: None 2024-04-12T03:54:03.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33 2024-04-12T03:54:03.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qFUpcdBlDw11MLO1dfJBTWCfgbc= 2024-04-12T03:54:03.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qFUpcdBlDw11MLO1dfJBTWCfgbc=', 'Content-Length': '0'} 2024-04-12T03:54:03.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ed6b90a0ad558a-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.722 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.723 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.723 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.723 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:54:03.723 DEBUG:boto:Data: 2024-04-12T03:54:03.723 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.723 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.723 DEBUG:boto:Params: {} 2024-04-12T03:54:03.723 DEBUG:boto:Token: None 2024-04-12T03:54:03.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33 2024-04-12T03:54:03.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qFUpcdBlDw11MLO1dfJBTWCfgbc= 2024-04-12T03:54:03.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qFUpcdBlDw11MLO1dfJBTWCfgbc=', 'Content-Length': '0'} 2024-04-12T03:54:03.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805a198ab51f90bb-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:54:03.725 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.726 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.726 DEBUG:boto:Method: GET 2024-04-12T03:54:03.726 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:54:03.726 DEBUG:boto:Data: 2024-04-12T03:54:03.726 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.726 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.726 DEBUG:boto:Params: {} 2024-04-12T03:54:03.726 DEBUG:boto:Token: None 2024-04-12T03:54:03.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33 2024-04-12T03:54:03.727 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xplc5UiUFYTtQDGHM41jDT5fWWs= 2024-04-12T03:54:03.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xplc5UiUFYTtQDGHM41jDT5fWWs=', 'Content-Length': '0'} 2024-04-12T03:54:03.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047e4db986487d3ec-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.770 DEBUG:boto:path=/dgbpxg-61/obj33 2024-04-12T03:54:03.770 DEBUG:boto:auth_path=/dgbpxg-61/obj33 2024-04-12T03:54:03.771 DEBUG:boto:Method: GET 2024-04-12T03:54:03.771 DEBUG:boto:Path: /dgbpxg-61/obj33 2024-04-12T03:54:03.771 DEBUG:boto:Data: 2024-04-12T03:54:03.771 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.771 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.771 DEBUG:boto:Params: {} 2024-04-12T03:54:03.771 DEBUG:boto:Token: None 2024-04-12T03:54:03.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj33 2024-04-12T03:54:03.771 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xplc5UiUFYTtQDGHM41jDT5fWWs= 2024-04-12T03:54:03.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xplc5UiUFYTtQDGHM41jDT5fWWs=', 'Content-Length': '0'} 2024-04-12T03:54:03.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd3c14fd125962f-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.773 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.774 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.774 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.774 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.774 DEBUG:boto:Method: GET 2024-04-12T03:54:03.774 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.774 DEBUG:boto:Data: 2024-04-12T03:54:03.774 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.774 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.774 DEBUG:boto:Params: {} 2024-04-12T03:54:03.775 DEBUG:boto:Token: None 2024-04-12T03:54:03.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.775 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f36ddf1196f1dfa-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.777 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.777 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.778 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.778 DEBUG:boto:path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.778 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.778 DEBUG:boto:Method: GET 2024-04-12T03:54:03.778 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj33 2024-04-12T03:54:03.778 DEBUG:boto:Data: 2024-04-12T03:54:03.778 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.778 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.778 DEBUG:boto:Params: {} 2024-04-12T03:54:03.778 DEBUG:boto:Token: None 2024-04-12T03:54:03.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000257959db6e0ff5af-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.786 DEBUG:boto:b'dgbpxg-61obj331000falseobj332024-04-12T03:40:51.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:54:03.787 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.787 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.787 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.787 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.787 DEBUG:boto:Method: GET 2024-04-12T03:54:03.787 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.787 DEBUG:boto:Data: 2024-04-12T03:54:03.787 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.787 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.788 DEBUG:boto:Params: {} 2024-04-12T03:54:03.788 DEBUG:boto:Token: None 2024-04-12T03:54:03.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8= 2024-04-12T03:54:03.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8=', 'Content-Length': '0'} 2024-04-12T03:54:03.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38fd45e61423221-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.790 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.790 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.790 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.790 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.790 DEBUG:boto:Method: GET 2024-04-12T03:54:03.791 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.791 DEBUG:boto:Data: 2024-04-12T03:54:03.791 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.791 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.791 DEBUG:boto:Params: {} 2024-04-12T03:54:03.791 DEBUG:boto:Token: None 2024-04-12T03:54:03.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8= 2024-04-12T03:54:03.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8=', 'Content-Length': '0'} 2024-04-12T03:54:03.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3b41c05bc6d157-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.793 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.794 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.794 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.794 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.794 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.794 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.794 DEBUG:boto:Data: 2024-04-12T03:54:03.794 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.794 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.794 DEBUG:boto:Params: {} 2024-04-12T03:54:03.795 DEBUG:boto:Token: None 2024-04-12T03:54:03.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p6DAT7rkPPE5SaaFczuzj/vV568= 2024-04-12T03:54:03.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p6DAT7rkPPE5SaaFczuzj/vV568=', 'Content-Length': '0'} 2024-04-12T03:54:03.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1b40e37e1588b0-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.797 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.797 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.797 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.797 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.797 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.797 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.797 DEBUG:boto:Data: 2024-04-12T03:54:03.797 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.798 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.798 DEBUG:boto:Params: {} 2024-04-12T03:54:03.798 DEBUG:boto:Token: None 2024-04-12T03:54:03.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.798 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p6DAT7rkPPE5SaaFczuzj/vV568= 2024-04-12T03:54:03.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p6DAT7rkPPE5SaaFczuzj/vV568=', 'Content-Length': '0'} 2024-04-12T03:54:03.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea2c6de06d47a9d-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:54:03.800 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.800 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.800 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.800 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.800 DEBUG:boto:Method: GET 2024-04-12T03:54:03.800 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.800 DEBUG:boto:Data: 2024-04-12T03:54:03.801 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.801 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.801 DEBUG:boto:Params: {} 2024-04-12T03:54:03.801 DEBUG:boto:Token: None 2024-04-12T03:54:03.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.801 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8= 2024-04-12T03:54:03.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8=', 'Content-Length': '0'} 2024-04-12T03:54:03.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001225dd291f1e3d1d-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.803 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.803 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.803 DEBUG:boto:path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.803 DEBUG:boto:auth_path=/dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.803 DEBUG:boto:Method: GET 2024-04-12T03:54:03.803 DEBUG:boto:Path: /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.803 DEBUG:boto:Data: 2024-04-12T03:54:03.804 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.804 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.804 DEBUG:boto:Params: {} 2024-04-12T03:54:03.804 DEBUG:boto:Token: None 2024-04-12T03:54:03.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34?versionId=null 2024-04-12T03:54:03.804 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8= 2024-04-12T03:54:03.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GxHU9GL5N1XcqAtZwl4mtdTB7d8=', 'Content-Length': '0'} 2024-04-12T03:54:03.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcb055f1470d4f6-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.806 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.806 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.806 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.806 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.807 DEBUG:boto:Method: GET 2024-04-12T03:54:03.807 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.807 DEBUG:boto:Data: 2024-04-12T03:54:03.807 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.807 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.807 DEBUG:boto:Params: {} 2024-04-12T03:54:03.807 DEBUG:boto:Token: None 2024-04-12T03:54:03.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070befea253847e5d-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.810 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.810 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.810 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.810 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.810 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.810 DEBUG:boto:Method: GET 2024-04-12T03:54:03.810 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.811 DEBUG:boto:Data: 2024-04-12T03:54:03.811 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.811 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.811 DEBUG:boto:Params: {} 2024-04-12T03:54:03.811 DEBUG:boto:Token: None 2024-04-12T03:54:03.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786c363a2f09602d-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.815 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.815 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.815 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.815 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.815 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:54:03.816 DEBUG:boto:Data: 2024-04-12T03:54:03.816 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.816 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.816 DEBUG:boto:Params: {} 2024-04-12T03:54:03.816 DEBUG:boto:Token: None 2024-04-12T03:54:03.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34 2024-04-12T03:54:03.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoXGJ7AtdwTWVB19vfNoYncqoYg= 2024-04-12T03:54:03.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoXGJ7AtdwTWVB19vfNoYncqoYg=', 'Content-Length': '0'} 2024-04-12T03:54:03.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b87cd4af367318-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.818 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.818 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.818 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.819 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:54:03.819 DEBUG:boto:Data: 2024-04-12T03:54:03.819 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.819 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.819 DEBUG:boto:Params: {} 2024-04-12T03:54:03.819 DEBUG:boto:Token: None 2024-04-12T03:54:03.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34 2024-04-12T03:54:03.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aoXGJ7AtdwTWVB19vfNoYncqoYg= 2024-04-12T03:54:03.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aoXGJ7AtdwTWVB19vfNoYncqoYg=', 'Content-Length': '0'} 2024-04-12T03:54:03.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967d6a73adbc8770-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T03:54:03.821 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.821 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.821 DEBUG:boto:Method: GET 2024-04-12T03:54:03.821 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:54:03.821 DEBUG:boto:Data: 2024-04-12T03:54:03.821 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.822 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.822 DEBUG:boto:Params: {} 2024-04-12T03:54:03.822 DEBUG:boto:Token: None 2024-04-12T03:54:03.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34 2024-04-12T03:54:03.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S+ygLLVkhLx/2TE1a9eULPoCOok= 2024-04-12T03:54:03.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S+ygLLVkhLx/2TE1a9eULPoCOok=', 'Content-Length': '0'} 2024-04-12T03:54:03.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da0030b91d075eb-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.824 DEBUG:boto:path=/dgbpxg-61/obj34 2024-04-12T03:54:03.824 DEBUG:boto:auth_path=/dgbpxg-61/obj34 2024-04-12T03:54:03.824 DEBUG:boto:Method: GET 2024-04-12T03:54:03.824 DEBUG:boto:Path: /dgbpxg-61/obj34 2024-04-12T03:54:03.824 DEBUG:boto:Data: 2024-04-12T03:54:03.825 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.825 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.825 DEBUG:boto:Params: {} 2024-04-12T03:54:03.825 DEBUG:boto:Token: None 2024-04-12T03:54:03.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj34 2024-04-12T03:54:03.825 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S+ygLLVkhLx/2TE1a9eULPoCOok= 2024-04-12T03:54:03.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S+ygLLVkhLx/2TE1a9eULPoCOok=', 'Content-Length': '0'} 2024-04-12T03:54:03.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ffa36764843b10-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.827 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.827 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.827 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.827 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.827 DEBUG:boto:Method: GET 2024-04-12T03:54:03.827 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.828 DEBUG:boto:Data: 2024-04-12T03:54:03.828 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.828 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.828 DEBUG:boto:Params: {} 2024-04-12T03:54:03.828 DEBUG:boto:Token: None 2024-04-12T03:54:03.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a30acd25aaccbb-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.831 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.831 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.831 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.831 DEBUG:boto:path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.831 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.831 DEBUG:boto:Method: GET 2024-04-12T03:54:03.831 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj34 2024-04-12T03:54:03.831 DEBUG:boto:Data: 2024-04-12T03:54:03.831 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.832 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.832 DEBUG:boto:Params: {} 2024-04-12T03:54:03.832 DEBUG:boto:Token: None 2024-04-12T03:54:03.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a057f6335c6c78b-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.878 DEBUG:boto:b'dgbpxg-61obj341000falseobj342024-04-12T03:41:32.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:54:03.879 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.879 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.879 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.879 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.879 DEBUG:boto:Method: GET 2024-04-12T03:54:03.879 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.879 DEBUG:boto:Data: 2024-04-12T03:54:03.879 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.879 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.879 DEBUG:boto:Params: {} 2024-04-12T03:54:03.880 DEBUG:boto:Token: None 2024-04-12T03:54:03.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c= 2024-04-12T03:54:03.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c=', 'Content-Length': '0'} 2024-04-12T03:54:03.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c700cf7af0716eb-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.882 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.882 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.882 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.882 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.882 DEBUG:boto:Method: GET 2024-04-12T03:54:03.882 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.882 DEBUG:boto:Data: 2024-04-12T03:54:03.883 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.883 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.883 DEBUG:boto:Params: {} 2024-04-12T03:54:03.883 DEBUG:boto:Token: None 2024-04-12T03:54:03.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c= 2024-04-12T03:54:03.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c=', 'Content-Length': '0'} 2024-04-12T03:54:03.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c8be246554b094-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.885 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.885 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.885 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.886 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.886 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.886 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.886 DEBUG:boto:Data: 2024-04-12T03:54:03.886 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.886 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.886 DEBUG:boto:Params: {} 2024-04-12T03:54:03.886 DEBUG:boto:Token: None 2024-04-12T03:54:03.886 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FkI5qMOH6x89Ww+/5EwHe4ofnvM= 2024-04-12T03:54:03.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FkI5qMOH6x89Ww+/5EwHe4ofnvM=', 'Content-Length': '0'} 2024-04-12T03:54:03.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db7d46928d7d051-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.888 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.888 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.889 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.889 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.889 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.889 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.889 DEBUG:boto:Data: 2024-04-12T03:54:03.889 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.889 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.889 DEBUG:boto:Params: {} 2024-04-12T03:54:03.889 DEBUG:boto:Token: None 2024-04-12T03:54:03.889 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FkI5qMOH6x89Ww+/5EwHe4ofnvM= 2024-04-12T03:54:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FkI5qMOH6x89Ww+/5EwHe4ofnvM=', 'Content-Length': '0'} 2024-04-12T03:54:03.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ead71b77f90dce-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:54:03.892 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.892 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.892 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.892 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.892 DEBUG:boto:Method: GET 2024-04-12T03:54:03.892 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.892 DEBUG:boto:Data: 2024-04-12T03:54:03.892 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.892 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.892 DEBUG:boto:Params: {} 2024-04-12T03:54:03.893 DEBUG:boto:Token: None 2024-04-12T03:54:03.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c= 2024-04-12T03:54:03.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c=', 'Content-Length': '0'} 2024-04-12T03:54:03.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183f0430c15ca3af-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.895 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.895 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.895 DEBUG:boto:path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.895 DEBUG:boto:auth_path=/dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.895 DEBUG:boto:Method: GET 2024-04-12T03:54:03.895 DEBUG:boto:Path: /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.895 DEBUG:boto:Data: 2024-04-12T03:54:03.895 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.895 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.895 DEBUG:boto:Params: {} 2024-04-12T03:54:03.896 DEBUG:boto:Token: None 2024-04-12T03:54:03.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37?versionId=null 2024-04-12T03:54:03.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c= 2024-04-12T03:54:03.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+IPy5+JE6+hyyvyVp8QuDKrh9/c=', 'Content-Length': '0'} 2024-04-12T03:54:03.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a317c658cf61538f-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.898 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.898 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.898 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.898 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.898 DEBUG:boto:Method: GET 2024-04-12T03:54:03.898 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.898 DEBUG:boto:Data: 2024-04-12T03:54:03.898 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.899 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.899 DEBUG:boto:Params: {} 2024-04-12T03:54:03.899 DEBUG:boto:Token: None 2024-04-12T03:54:03.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.899 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2370ed810868c9c-006618b05b-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.942 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.943 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:03.943 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:03.943 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.943 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.943 DEBUG:boto:Method: GET 2024-04-12T03:54:03.943 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:54:03.943 DEBUG:boto:Data: 2024-04-12T03:54:03.943 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.943 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.944 DEBUG:boto:Params: {} 2024-04-12T03:54:03.944 DEBUG:boto:Token: None 2024-04-12T03:54:03.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/ 2024-04-12T03:54:03.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY= 2024-04-12T03:54:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sQphRt2fA+cyMNOLRzdvO8bdMY=', 'Content-Length': '0'} 2024-04-12T03:54:03.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d82213d76a0470fa-006618b05b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.947 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:03.948 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.948 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.948 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.948 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:54:03.948 DEBUG:boto:Data: 2024-04-12T03:54:03.948 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.948 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.949 DEBUG:boto:Params: {} 2024-04-12T03:54:03.949 DEBUG:boto:Token: None 2024-04-12T03:54:03.949 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37 2024-04-12T03:54:03.949 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YCnXHwg5oXBEu5D54Y8fV6WwiNM= 2024-04-12T03:54:03.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YCnXHwg5oXBEu5D54Y8fV6WwiNM=', 'Content-Length': '0'} 2024-04-12T03:54:03.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004722a19cc8b71c4c-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.950 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.951 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.951 DEBUG:boto:Method: HEAD 2024-04-12T03:54:03.951 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:54:03.951 DEBUG:boto:Data: 2024-04-12T03:54:03.951 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.951 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.951 DEBUG:boto:Params: {} 2024-04-12T03:54:03.951 DEBUG:boto:Token: None 2024-04-12T03:54:03.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37 2024-04-12T03:54:03.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YCnXHwg5oXBEu5D54Y8fV6WwiNM= 2024-04-12T03:54:03.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YCnXHwg5oXBEu5D54Y8fV6WwiNM=', 'Content-Length': '0'} 2024-04-12T03:54:03.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2465dad4a4a44d4-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T03:54:03.953 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.954 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.954 DEBUG:boto:Method: GET 2024-04-12T03:54:03.954 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:54:03.954 DEBUG:boto:Data: 2024-04-12T03:54:03.954 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:03.954 DEBUG:boto:Port: 8001 2024-04-12T03:54:03.954 DEBUG:boto:Params: {} 2024-04-12T03:54:03.955 DEBUG:boto:Token: None 2024-04-12T03:54:03.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37 2024-04-12T03:54:03.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lNyyyfOS7ynVoGbHGlZbxDwHiw0= 2024-04-12T03:54:03.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lNyyyfOS7ynVoGbHGlZbxDwHiw0=', 'Content-Length': '0'} 2024-04-12T03:54:03.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232f8e6c01a1169b-006618b05b-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:03.957 DEBUG:boto:path=/dgbpxg-61/obj37 2024-04-12T03:54:03.957 DEBUG:boto:auth_path=/dgbpxg-61/obj37 2024-04-12T03:54:03.957 DEBUG:boto:Method: GET 2024-04-12T03:54:03.957 DEBUG:boto:Path: /dgbpxg-61/obj37 2024-04-12T03:54:03.957 DEBUG:boto:Data: 2024-04-12T03:54:03.957 DEBUG:boto:Headers: {} 2024-04-12T03:54:03.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:03.957 DEBUG:boto:Port: 8000 2024-04-12T03:54:03.957 DEBUG:boto:Params: {} 2024-04-12T03:54:03.958 DEBUG:boto:Token: None 2024-04-12T03:54:03.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:03 GMT /dgbpxg-61/obj37 2024-04-12T03:54:03.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lNyyyfOS7ynVoGbHGlZbxDwHiw0= 2024-04-12T03:54:03.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:03 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lNyyyfOS7ynVoGbHGlZbxDwHiw0=', 'Content-Length': '0'} 2024-04-12T03:54:03.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065a0d2a6ac49c81d-006618b05b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.003 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.003 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.003 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.003 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.003 DEBUG:boto:Method: GET 2024-04-12T03:54:04.003 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.004 DEBUG:boto:Data: 2024-04-12T03:54:04.004 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.004 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.004 DEBUG:boto:Params: {} 2024-04-12T03:54:04.004 DEBUG:boto:Token: None 2024-04-12T03:54:04.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709aebff538db21a-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.007 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.007 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.007 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.007 DEBUG:boto:path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.007 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.008 DEBUG:boto:Method: GET 2024-04-12T03:54:04.008 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj37 2024-04-12T03:54:04.008 DEBUG:boto:Data: 2024-04-12T03:54:04.008 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.008 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.008 DEBUG:boto:Params: {} 2024-04-12T03:54:04.008 DEBUG:boto:Token: None 2024-04-12T03:54:04.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29114fb3baf357e-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.012 DEBUG:boto:b'dgbpxg-61obj371000falseobj372024-04-12T03:51:42.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:54:04.013 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.013 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.013 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.013 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.013 DEBUG:boto:Method: GET 2024-04-12T03:54:04.013 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.013 DEBUG:boto:Data: 2024-04-12T03:54:04.013 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.013 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.014 DEBUG:boto:Params: {} 2024-04-12T03:54:04.014 DEBUG:boto:Token: None 2024-04-12T03:54:04.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY= 2024-04-12T03:54:04.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY=', 'Content-Length': '0'} 2024-04-12T03:54:04.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8a7931da0dfd3b-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.058 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.058 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.059 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.059 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.059 DEBUG:boto:Method: GET 2024-04-12T03:54:04.059 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.059 DEBUG:boto:Data: 2024-04-12T03:54:04.059 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.059 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.059 DEBUG:boto:Params: {} 2024-04-12T03:54:04.059 DEBUG:boto:Token: None 2024-04-12T03:54:04.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.059 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY= 2024-04-12T03:54:04.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY=', 'Content-Length': '0'} 2024-04-12T03:54:04.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceca6ae7db3b03aa-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.062 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.062 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.062 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.062 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.062 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.062 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.062 DEBUG:boto:Data: 2024-04-12T03:54:04.062 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.063 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.063 DEBUG:boto:Params: {} 2024-04-12T03:54:04.063 DEBUG:boto:Token: None 2024-04-12T03:54:04.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.063 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aIV7P9n/R7byV4ATcTg7/137nTQ= 2024-04-12T03:54:04.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aIV7P9n/R7byV4ATcTg7/137nTQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d4559dc73093581-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.065 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.065 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.065 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.065 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.065 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.065 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.065 DEBUG:boto:Data: 2024-04-12T03:54:04.065 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.065 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.066 DEBUG:boto:Params: {} 2024-04-12T03:54:04.066 DEBUG:boto:Token: None 2024-04-12T03:54:04.066 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aIV7P9n/R7byV4ATcTg7/137nTQ= 2024-04-12T03:54:04.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aIV7P9n/R7byV4ATcTg7/137nTQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edecea43027b36e-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:54:04.068 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.068 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.068 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.068 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.069 DEBUG:boto:Method: GET 2024-04-12T03:54:04.069 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.069 DEBUG:boto:Data: 2024-04-12T03:54:04.069 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.069 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.069 DEBUG:boto:Params: {} 2024-04-12T03:54:04.069 DEBUG:boto:Token: None 2024-04-12T03:54:04.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.069 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY= 2024-04-12T03:54:04.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY=', 'Content-Length': '0'} 2024-04-12T03:54:04.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71224f7542c7114-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.071 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.071 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.071 DEBUG:boto:path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.071 DEBUG:boto:auth_path=/dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.072 DEBUG:boto:Method: GET 2024-04-12T03:54:04.072 DEBUG:boto:Path: /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.072 DEBUG:boto:Data: 2024-04-12T03:54:04.072 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.072 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.072 DEBUG:boto:Params: {} 2024-04-12T03:54:04.072 DEBUG:boto:Token: None 2024-04-12T03:54:04.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38?versionId=null 2024-04-12T03:54:04.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY= 2024-04-12T03:54:04.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7wSQWDuuwjen2zavGVHTVhmxVfY=', 'Content-Length': '0'} 2024-04-12T03:54:04.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335745d0fff4d95f-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.074 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.075 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.075 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.075 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.075 DEBUG:boto:Method: GET 2024-04-12T03:54:04.075 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.075 DEBUG:boto:Data: 2024-04-12T03:54:04.075 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.075 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.075 DEBUG:boto:Params: {} 2024-04-12T03:54:04.075 DEBUG:boto:Token: None 2024-04-12T03:54:04.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b75518c58a6d2b4-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.078 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.078 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.078 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.079 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.079 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.079 DEBUG:boto:Method: GET 2024-04-12T03:54:04.079 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.079 DEBUG:boto:Data: 2024-04-12T03:54:04.079 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.079 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.079 DEBUG:boto:Params: {} 2024-04-12T03:54:04.079 DEBUG:boto:Token: None 2024-04-12T03:54:04.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ebf3bd47ee6d32d-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.083 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.084 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.084 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.084 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.084 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:54:04.084 DEBUG:boto:Data: 2024-04-12T03:54:04.084 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.084 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.084 DEBUG:boto:Params: {} 2024-04-12T03:54:04.084 DEBUG:boto:Token: None 2024-04-12T03:54:04.085 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38 2024-04-12T03:54:04.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8uROcAavVgvtgDK8/41bGnw/cIQ= 2024-04-12T03:54:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8uROcAavVgvtgDK8/41bGnw/cIQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef210bd788a07a36-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.087 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.087 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.087 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.087 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:54:04.087 DEBUG:boto:Data: 2024-04-12T03:54:04.087 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.087 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.087 DEBUG:boto:Params: {} 2024-04-12T03:54:04.087 DEBUG:boto:Token: None 2024-04-12T03:54:04.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38 2024-04-12T03:54:04.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8uROcAavVgvtgDK8/41bGnw/cIQ= 2024-04-12T03:54:04.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8uROcAavVgvtgDK8/41bGnw/cIQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f048c560b229f09-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T03:54:04.090 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.090 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.090 DEBUG:boto:Method: GET 2024-04-12T03:54:04.090 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:54:04.091 DEBUG:boto:Data: 2024-04-12T03:54:04.091 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.091 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.091 DEBUG:boto:Params: {} 2024-04-12T03:54:04.092 DEBUG:boto:Token: None 2024-04-12T03:54:04.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38 2024-04-12T03:54:04.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bbCbO0u+kq/J9X8WaNng7nG81Ag= 2024-04-12T03:54:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bbCbO0u+kq/J9X8WaNng7nG81Ag=', 'Content-Length': '0'} 2024-04-12T03:54:04.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5585c0622a472ef-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.134 DEBUG:boto:path=/dgbpxg-61/obj38 2024-04-12T03:54:04.135 DEBUG:boto:auth_path=/dgbpxg-61/obj38 2024-04-12T03:54:04.135 DEBUG:boto:Method: GET 2024-04-12T03:54:04.135 DEBUG:boto:Path: /dgbpxg-61/obj38 2024-04-12T03:54:04.135 DEBUG:boto:Data: 2024-04-12T03:54:04.135 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.135 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.135 DEBUG:boto:Params: {} 2024-04-12T03:54:04.136 DEBUG:boto:Token: None 2024-04-12T03:54:04.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj38 2024-04-12T03:54:04.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bbCbO0u+kq/J9X8WaNng7nG81Ag= 2024-04-12T03:54:04.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bbCbO0u+kq/J9X8WaNng7nG81Ag=', 'Content-Length': '0'} 2024-04-12T03:54:04.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d14e03576f6c22-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.138 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.138 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.138 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.138 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.138 DEBUG:boto:Method: GET 2024-04-12T03:54:04.138 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.138 DEBUG:boto:Data: 2024-04-12T03:54:04.139 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.139 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.139 DEBUG:boto:Params: {} 2024-04-12T03:54:04.139 DEBUG:boto:Token: None 2024-04-12T03:54:04.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0bbc5072c25038-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.142 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.142 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.142 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.142 DEBUG:boto:path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.143 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.143 DEBUG:boto:Method: GET 2024-04-12T03:54:04.143 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj38 2024-04-12T03:54:04.143 DEBUG:boto:Data: 2024-04-12T03:54:04.143 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.143 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.143 DEBUG:boto:Params: {} 2024-04-12T03:54:04.143 DEBUG:boto:Token: None 2024-04-12T03:54:04.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001115d9789e99d981-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.147 DEBUG:boto:b'dgbpxg-61obj381000falseobj382024-04-12T03:43:54.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:54:04.148 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.148 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.148 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.148 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.148 DEBUG:boto:Method: GET 2024-04-12T03:54:04.148 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.148 DEBUG:boto:Data: 2024-04-12T03:54:04.149 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.149 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.149 DEBUG:boto:Params: {} 2024-04-12T03:54:04.149 DEBUG:boto:Token: None 2024-04-12T03:54:04.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.149 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk= 2024-04-12T03:54:04.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk=', 'Content-Length': '0'} 2024-04-12T03:54:04.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000625a0ff4043425a8-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.151 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.152 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.152 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.152 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.152 DEBUG:boto:Method: GET 2024-04-12T03:54:04.152 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.152 DEBUG:boto:Data: 2024-04-12T03:54:04.152 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.152 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.152 DEBUG:boto:Params: {} 2024-04-12T03:54:04.153 DEBUG:boto:Token: None 2024-04-12T03:54:04.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk= 2024-04-12T03:54:04.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk=', 'Content-Length': '0'} 2024-04-12T03:54:04.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600da467bf14272f-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.155 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.155 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.155 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.156 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.156 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.156 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.156 DEBUG:boto:Data: 2024-04-12T03:54:04.156 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.156 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.156 DEBUG:boto:Params: {} 2024-04-12T03:54:04.156 DEBUG:boto:Token: None 2024-04-12T03:54:04.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.157 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gii035T94m6ix4rLZNEAxhBDSto= 2024-04-12T03:54:04.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gii035T94m6ix4rLZNEAxhBDSto=', 'Content-Length': '0'} 2024-04-12T03:54:04.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2ae6d376886909-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.159 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.159 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.159 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.159 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.159 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.159 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.159 DEBUG:boto:Data: 2024-04-12T03:54:04.159 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.160 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.160 DEBUG:boto:Params: {} 2024-04-12T03:54:04.160 DEBUG:boto:Token: None 2024-04-12T03:54:04.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gii035T94m6ix4rLZNEAxhBDSto= 2024-04-12T03:54:04.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gii035T94m6ix4rLZNEAxhBDSto=', 'Content-Length': '0'} 2024-04-12T03:54:04.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe5727b01e85a82-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:54:04.163 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.163 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.163 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.163 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.163 DEBUG:boto:Method: GET 2024-04-12T03:54:04.163 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.163 DEBUG:boto:Data: 2024-04-12T03:54:04.163 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.164 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.164 DEBUG:boto:Params: {} 2024-04-12T03:54:04.164 DEBUG:boto:Token: None 2024-04-12T03:54:04.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk= 2024-04-12T03:54:04.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk=', 'Content-Length': '0'} 2024-04-12T03:54:04.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f457d9d915e2d07e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.206 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.207 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.207 DEBUG:boto:path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.207 DEBUG:boto:auth_path=/dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.207 DEBUG:boto:Method: GET 2024-04-12T03:54:04.207 DEBUG:boto:Path: /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.207 DEBUG:boto:Data: 2024-04-12T03:54:04.207 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.208 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.208 DEBUG:boto:Params: {} 2024-04-12T03:54:04.208 DEBUG:boto:Token: None 2024-04-12T03:54:04.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44?versionId=null 2024-04-12T03:54:04.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk= 2024-04-12T03:54:04.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOoLnl3Kod4x7dTONxWeCJKd2mk=', 'Content-Length': '0'} 2024-04-12T03:54:04.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd08e1e9b02edabb-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.210 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.211 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.211 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.211 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.211 DEBUG:boto:Method: GET 2024-04-12T03:54:04.211 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.211 DEBUG:boto:Data: 2024-04-12T03:54:04.211 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.211 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.212 DEBUG:boto:Params: {} 2024-04-12T03:54:04.212 DEBUG:boto:Token: None 2024-04-12T03:54:04.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be07e4e4786be7a9-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.214 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.215 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.215 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.215 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.215 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.215 DEBUG:boto:Method: GET 2024-04-12T03:54:04.215 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.215 DEBUG:boto:Data: 2024-04-12T03:54:04.215 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.216 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.216 DEBUG:boto:Params: {} 2024-04-12T03:54:04.216 DEBUG:boto:Token: None 2024-04-12T03:54:04.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.216 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d52bc4af0458077-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.220 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.220 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.220 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.221 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.221 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:54:04.221 DEBUG:boto:Data: 2024-04-12T03:54:04.221 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.221 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.221 DEBUG:boto:Params: {} 2024-04-12T03:54:04.221 DEBUG:boto:Token: None 2024-04-12T03:54:04.221 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44 2024-04-12T03:54:04.222 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MooL6EDOSEWnarNu+7j4S0yCsP8= 2024-04-12T03:54:04.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MooL6EDOSEWnarNu+7j4S0yCsP8=', 'Content-Length': '0'} 2024-04-12T03:54:04.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d13851be4f5774-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.224 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.224 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.224 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.224 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:54:04.224 DEBUG:boto:Data: 2024-04-12T03:54:04.224 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.225 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.225 DEBUG:boto:Params: {} 2024-04-12T03:54:04.225 DEBUG:boto:Token: None 2024-04-12T03:54:04.225 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44 2024-04-12T03:54:04.225 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MooL6EDOSEWnarNu+7j4S0yCsP8= 2024-04-12T03:54:04.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MooL6EDOSEWnarNu+7j4S0yCsP8=', 'Content-Length': '0'} 2024-04-12T03:54:04.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f82fc812347f43-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:54:04.228 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.228 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.228 DEBUG:boto:Method: GET 2024-04-12T03:54:04.228 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:54:04.228 DEBUG:boto:Data: 2024-04-12T03:54:04.228 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.228 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.229 DEBUG:boto:Params: {} 2024-04-12T03:54:04.229 DEBUG:boto:Token: None 2024-04-12T03:54:04.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44 2024-04-12T03:54:04.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Eu//y8WoIw8U9v5Cr5WwYosZe4= 2024-04-12T03:54:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Eu//y8WoIw8U9v5Cr5WwYosZe4=', 'Content-Length': '0'} 2024-04-12T03:54:04.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f5c6d60bcc4516-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.231 DEBUG:boto:path=/dgbpxg-61/obj44 2024-04-12T03:54:04.231 DEBUG:boto:auth_path=/dgbpxg-61/obj44 2024-04-12T03:54:04.231 DEBUG:boto:Method: GET 2024-04-12T03:54:04.231 DEBUG:boto:Path: /dgbpxg-61/obj44 2024-04-12T03:54:04.231 DEBUG:boto:Data: 2024-04-12T03:54:04.231 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.232 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.232 DEBUG:boto:Params: {} 2024-04-12T03:54:04.232 DEBUG:boto:Token: None 2024-04-12T03:54:04.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj44 2024-04-12T03:54:04.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Eu//y8WoIw8U9v5Cr5WwYosZe4= 2024-04-12T03:54:04.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Eu//y8WoIw8U9v5Cr5WwYosZe4=', 'Content-Length': '0'} 2024-04-12T03:54:04.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac424341b39b1c8-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.274 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.275 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.275 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.275 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.275 DEBUG:boto:Method: GET 2024-04-12T03:54:04.275 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.275 DEBUG:boto:Data: 2024-04-12T03:54:04.275 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.275 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.275 DEBUG:boto:Params: {} 2024-04-12T03:54:04.276 DEBUG:boto:Token: None 2024-04-12T03:54:04.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c231bff997a7683-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.278 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.279 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.279 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.279 DEBUG:boto:path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.279 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.279 DEBUG:boto:Method: GET 2024-04-12T03:54:04.279 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj44 2024-04-12T03:54:04.279 DEBUG:boto:Data: 2024-04-12T03:54:04.279 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.280 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.280 DEBUG:boto:Params: {} 2024-04-12T03:54:04.280 DEBUG:boto:Token: None 2024-04-12T03:54:04.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002366c7afd3e9079-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.284 DEBUG:boto:b'dgbpxg-61obj441000falseobj442024-04-12T03:40:51.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:54:04.284 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.284 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.284 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.285 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.285 DEBUG:boto:Method: GET 2024-04-12T03:54:04.285 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.285 DEBUG:boto:Data: 2024-04-12T03:54:04.285 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.285 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.285 DEBUG:boto:Params: {} 2024-04-12T03:54:04.286 DEBUG:boto:Token: None 2024-04-12T03:54:04.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E= 2024-04-12T03:54:04.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E=', 'Content-Length': '0'} 2024-04-12T03:54:04.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411a1e302f00933f-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.330 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.331 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.331 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.331 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.331 DEBUG:boto:Method: GET 2024-04-12T03:54:04.331 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.331 DEBUG:boto:Data: 2024-04-12T03:54:04.331 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.331 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.331 DEBUG:boto:Params: {} 2024-04-12T03:54:04.332 DEBUG:boto:Token: None 2024-04-12T03:54:04.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E= 2024-04-12T03:54:04.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E=', 'Content-Length': '0'} 2024-04-12T03:54:04.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a18c85679aff1fc-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.334 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.334 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.334 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.334 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.334 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.335 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.335 DEBUG:boto:Data: 2024-04-12T03:54:04.335 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.335 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.335 DEBUG:boto:Params: {} 2024-04-12T03:54:04.335 DEBUG:boto:Token: None 2024-04-12T03:54:04.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4vApddtdclpC0iX1niWahlYqv7I= 2024-04-12T03:54:04.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4vApddtdclpC0iX1niWahlYqv7I=', 'Content-Length': '0'} 2024-04-12T03:54:04.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddbd58237cd918b2-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.337 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.337 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.338 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.338 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.338 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.338 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.338 DEBUG:boto:Data: 2024-04-12T03:54:04.338 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.338 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.338 DEBUG:boto:Params: {} 2024-04-12T03:54:04.339 DEBUG:boto:Token: None 2024-04-12T03:54:04.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4vApddtdclpC0iX1niWahlYqv7I= 2024-04-12T03:54:04.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4vApddtdclpC0iX1niWahlYqv7I=', 'Content-Length': '0'} 2024-04-12T03:54:04.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17e531e530f1382-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:54:04.341 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.341 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.341 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.341 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.342 DEBUG:boto:Method: GET 2024-04-12T03:54:04.342 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.342 DEBUG:boto:Data: 2024-04-12T03:54:04.342 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.342 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.342 DEBUG:boto:Params: {} 2024-04-12T03:54:04.342 DEBUG:boto:Token: None 2024-04-12T03:54:04.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E= 2024-04-12T03:54:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E=', 'Content-Length': '0'} 2024-04-12T03:54:04.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3378760f8572e2e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.345 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.345 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.345 DEBUG:boto:path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.345 DEBUG:boto:auth_path=/dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.345 DEBUG:boto:Method: GET 2024-04-12T03:54:04.345 DEBUG:boto:Path: /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.345 DEBUG:boto:Data: 2024-04-12T03:54:04.346 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.346 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.346 DEBUG:boto:Params: {} 2024-04-12T03:54:04.346 DEBUG:boto:Token: None 2024-04-12T03:54:04.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46?versionId=null 2024-04-12T03:54:04.346 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E= 2024-04-12T03:54:04.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uEmYJD3Zpj6NGEcD0KHSRNFQv5E=', 'Content-Length': '0'} 2024-04-12T03:54:04.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b2f11b7d1ac52a9-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.349 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.349 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.349 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.349 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.349 DEBUG:boto:Method: GET 2024-04-12T03:54:04.349 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.349 DEBUG:boto:Data: 2024-04-12T03:54:04.350 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.350 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.350 DEBUG:boto:Params: {} 2024-04-12T03:54:04.350 DEBUG:boto:Token: None 2024-04-12T03:54:04.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.350 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003970f88de9d94143-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.353 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.353 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.353 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.354 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.354 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.354 DEBUG:boto:Method: GET 2024-04-12T03:54:04.354 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.354 DEBUG:boto:Data: 2024-04-12T03:54:04.354 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.354 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.354 DEBUG:boto:Params: {} 2024-04-12T03:54:04.354 DEBUG:boto:Token: None 2024-04-12T03:54:04.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf95c51c252bb382-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.358 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.359 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.359 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.359 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.359 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:54:04.359 DEBUG:boto:Data: 2024-04-12T03:54:04.359 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.359 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.359 DEBUG:boto:Params: {} 2024-04-12T03:54:04.359 DEBUG:boto:Token: None 2024-04-12T03:54:04.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46 2024-04-12T03:54:04.360 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kBWX7wsY2/jBgb6OS8CQ+Y6+bJ4= 2024-04-12T03:54:04.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kBWX7wsY2/jBgb6OS8CQ+Y6+bJ4=', 'Content-Length': '0'} 2024-04-12T03:54:04.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe5dbbacfe259e0e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.361 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.361 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.362 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.362 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:54:04.362 DEBUG:boto:Data: 2024-04-12T03:54:04.362 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.362 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.362 DEBUG:boto:Params: {} 2024-04-12T03:54:04.362 DEBUG:boto:Token: None 2024-04-12T03:54:04.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46 2024-04-12T03:54:04.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kBWX7wsY2/jBgb6OS8CQ+Y6+bJ4= 2024-04-12T03:54:04.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kBWX7wsY2/jBgb6OS8CQ+Y6+bJ4=', 'Content-Length': '0'} 2024-04-12T03:54:04.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000485cea84f6df2a44-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T03:54:04.365 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.365 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.365 DEBUG:boto:Method: GET 2024-04-12T03:54:04.365 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:54:04.365 DEBUG:boto:Data: 2024-04-12T03:54:04.365 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.366 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.366 DEBUG:boto:Params: {} 2024-04-12T03:54:04.366 DEBUG:boto:Token: None 2024-04-12T03:54:04.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46 2024-04-12T03:54:04.366 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SelWEcAWG0BBWLLxfIn2Kfsr3bE= 2024-04-12T03:54:04.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SelWEcAWG0BBWLLxfIn2Kfsr3bE=', 'Content-Length': '0'} 2024-04-12T03:54:04.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870e46f65872bb2e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.368 DEBUG:boto:path=/dgbpxg-61/obj46 2024-04-12T03:54:04.368 DEBUG:boto:auth_path=/dgbpxg-61/obj46 2024-04-12T03:54:04.368 DEBUG:boto:Method: GET 2024-04-12T03:54:04.368 DEBUG:boto:Path: /dgbpxg-61/obj46 2024-04-12T03:54:04.368 DEBUG:boto:Data: 2024-04-12T03:54:04.368 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.369 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.369 DEBUG:boto:Params: {} 2024-04-12T03:54:04.369 DEBUG:boto:Token: None 2024-04-12T03:54:04.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj46 2024-04-12T03:54:04.369 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SelWEcAWG0BBWLLxfIn2Kfsr3bE= 2024-04-12T03:54:04.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SelWEcAWG0BBWLLxfIn2Kfsr3bE=', 'Content-Length': '0'} 2024-04-12T03:54:04.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2752a9eeadaf46-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.371 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.371 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.371 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.371 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.371 DEBUG:boto:Method: GET 2024-04-12T03:54:04.371 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.372 DEBUG:boto:Data: 2024-04-12T03:54:04.372 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.372 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.372 DEBUG:boto:Params: {} 2024-04-12T03:54:04.372 DEBUG:boto:Token: None 2024-04-12T03:54:04.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.372 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e057bc9a3ee30ac4-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.374 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.375 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.375 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.375 DEBUG:boto:path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.375 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.375 DEBUG:boto:Method: GET 2024-04-12T03:54:04.375 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj46 2024-04-12T03:54:04.375 DEBUG:boto:Data: 2024-04-12T03:54:04.376 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.376 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.376 DEBUG:boto:Params: {} 2024-04-12T03:54:04.376 DEBUG:boto:Token: None 2024-04-12T03:54:04.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b9b687a4164a9f1-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.380 DEBUG:boto:b'dgbpxg-61obj461000falseobj462024-04-12T03:46:16.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:54:04.381 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.381 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.381 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.381 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.381 DEBUG:boto:Method: GET 2024-04-12T03:54:04.381 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.381 DEBUG:boto:Data: 2024-04-12T03:54:04.381 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.382 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.382 DEBUG:boto:Params: {} 2024-04-12T03:54:04.382 DEBUG:boto:Token: None 2024-04-12T03:54:04.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.382 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c= 2024-04-12T03:54:04.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c=', 'Content-Length': '0'} 2024-04-12T03:54:04.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6249db3e37cf83-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.384 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.384 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.384 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.385 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.385 DEBUG:boto:Method: GET 2024-04-12T03:54:04.385 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.385 DEBUG:boto:Data: 2024-04-12T03:54:04.385 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.385 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.385 DEBUG:boto:Params: {} 2024-04-12T03:54:04.385 DEBUG:boto:Token: None 2024-04-12T03:54:04.385 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c= 2024-04-12T03:54:04.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c=', 'Content-Length': '0'} 2024-04-12T03:54:04.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045816e2452c7904b-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.430 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.431 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.431 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.431 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.431 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.431 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.431 DEBUG:boto:Data: 2024-04-12T03:54:04.431 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.431 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.431 DEBUG:boto:Params: {} 2024-04-12T03:54:04.432 DEBUG:boto:Token: None 2024-04-12T03:54:04.432 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AXmM1mo6+ssUdSk+pvTG37/Fk/k= 2024-04-12T03:54:04.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AXmM1mo6+ssUdSk+pvTG37/Fk/k=', 'Content-Length': '0'} 2024-04-12T03:54:04.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ecc3c44a9f2eec-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.434 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.434 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.434 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.435 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.435 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.435 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.435 DEBUG:boto:Data: 2024-04-12T03:54:04.435 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.435 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.435 DEBUG:boto:Params: {} 2024-04-12T03:54:04.435 DEBUG:boto:Token: None 2024-04-12T03:54:04.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AXmM1mo6+ssUdSk+pvTG37/Fk/k= 2024-04-12T03:54:04.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AXmM1mo6+ssUdSk+pvTG37/Fk/k=', 'Content-Length': '0'} 2024-04-12T03:54:04.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f463f74fcc7988c-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:54:04.438 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.438 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.438 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.438 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.438 DEBUG:boto:Method: GET 2024-04-12T03:54:04.438 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.438 DEBUG:boto:Data: 2024-04-12T03:54:04.439 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.439 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.439 DEBUG:boto:Params: {} 2024-04-12T03:54:04.439 DEBUG:boto:Token: None 2024-04-12T03:54:04.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.439 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c= 2024-04-12T03:54:04.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c=', 'Content-Length': '0'} 2024-04-12T03:54:04.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ff099a8ba37b4d-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.441 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.441 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.441 DEBUG:boto:path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.441 DEBUG:boto:auth_path=/dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.441 DEBUG:boto:Method: GET 2024-04-12T03:54:04.442 DEBUG:boto:Path: /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.442 DEBUG:boto:Data: 2024-04-12T03:54:04.442 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.442 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.442 DEBUG:boto:Params: {} 2024-04-12T03:54:04.442 DEBUG:boto:Token: None 2024-04-12T03:54:04.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51?versionId=null 2024-04-12T03:54:04.442 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c= 2024-04-12T03:54:04.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F9BHNwbhFx1kwQwaNRgi9mpzQ4c=', 'Content-Length': '0'} 2024-04-12T03:54:04.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d656839b3a5e09cb-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.445 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.445 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.445 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.445 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.445 DEBUG:boto:Method: GET 2024-04-12T03:54:04.445 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.445 DEBUG:boto:Data: 2024-04-12T03:54:04.445 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.446 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.446 DEBUG:boto:Params: {} 2024-04-12T03:54:04.446 DEBUG:boto:Token: None 2024-04-12T03:54:04.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.446 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052ff3932980cbbd3-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.449 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.449 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.449 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.449 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.450 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.450 DEBUG:boto:Method: GET 2024-04-12T03:54:04.450 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.450 DEBUG:boto:Data: 2024-04-12T03:54:04.450 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.450 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.450 DEBUG:boto:Params: {} 2024-04-12T03:54:04.450 DEBUG:boto:Token: None 2024-04-12T03:54:04.450 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d592207002da1bee-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.454 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.455 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.455 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.455 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.455 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:54:04.455 DEBUG:boto:Data: 2024-04-12T03:54:04.455 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.455 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.455 DEBUG:boto:Params: {} 2024-04-12T03:54:04.456 DEBUG:boto:Token: None 2024-04-12T03:54:04.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51 2024-04-12T03:54:04.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mFBEvH8Q0LIStCvz4KRHgqeCCp4= 2024-04-12T03:54:04.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mFBEvH8Q0LIStCvz4KRHgqeCCp4=', 'Content-Length': '0'} 2024-04-12T03:54:04.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2050a57e9cc35c-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.458 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.458 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.458 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.458 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:54:04.458 DEBUG:boto:Data: 2024-04-12T03:54:04.458 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.458 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.459 DEBUG:boto:Params: {} 2024-04-12T03:54:04.459 DEBUG:boto:Token: None 2024-04-12T03:54:04.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51 2024-04-12T03:54:04.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mFBEvH8Q0LIStCvz4KRHgqeCCp4= 2024-04-12T03:54:04.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mFBEvH8Q0LIStCvz4KRHgqeCCp4=', 'Content-Length': '0'} 2024-04-12T03:54:04.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da1af432d6d4897-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T03:54:04.461 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.461 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.461 DEBUG:boto:Method: GET 2024-04-12T03:54:04.461 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:54:04.462 DEBUG:boto:Data: 2024-04-12T03:54:04.462 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.462 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.462 DEBUG:boto:Params: {} 2024-04-12T03:54:04.462 DEBUG:boto:Token: None 2024-04-12T03:54:04.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51 2024-04-12T03:54:04.462 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RwTDOrPpNmHOVDk3AwL4D+9uxpc= 2024-04-12T03:54:04.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RwTDOrPpNmHOVDk3AwL4D+9uxpc=', 'Content-Length': '0'} 2024-04-12T03:54:04.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e415ed5726e73f6-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.464 DEBUG:boto:path=/dgbpxg-61/obj51 2024-04-12T03:54:04.464 DEBUG:boto:auth_path=/dgbpxg-61/obj51 2024-04-12T03:54:04.465 DEBUG:boto:Method: GET 2024-04-12T03:54:04.465 DEBUG:boto:Path: /dgbpxg-61/obj51 2024-04-12T03:54:04.465 DEBUG:boto:Data: 2024-04-12T03:54:04.465 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.465 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.465 DEBUG:boto:Params: {} 2024-04-12T03:54:04.465 DEBUG:boto:Token: None 2024-04-12T03:54:04.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj51 2024-04-12T03:54:04.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RwTDOrPpNmHOVDk3AwL4D+9uxpc= 2024-04-12T03:54:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RwTDOrPpNmHOVDk3AwL4D+9uxpc=', 'Content-Length': '0'} 2024-04-12T03:54:04.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b88cdc9f6dc4578-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.467 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.467 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.467 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.468 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.468 DEBUG:boto:Method: GET 2024-04-12T03:54:04.468 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.468 DEBUG:boto:Data: 2024-04-12T03:54:04.468 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.468 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.468 DEBUG:boto:Params: {} 2024-04-12T03:54:04.468 DEBUG:boto:Token: None 2024-04-12T03:54:04.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f5acfb3289cd3e-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.471 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.471 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.471 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.471 DEBUG:boto:path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.471 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.471 DEBUG:boto:Method: GET 2024-04-12T03:54:04.471 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj51 2024-04-12T03:54:04.471 DEBUG:boto:Data: 2024-04-12T03:54:04.472 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.472 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.472 DEBUG:boto:Params: {} 2024-04-12T03:54:04.472 DEBUG:boto:Token: None 2024-04-12T03:54:04.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be856eeeda592eb3-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.518 DEBUG:boto:b'dgbpxg-61obj511000falseobj512024-04-12T03:41:32.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:54:04.519 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.519 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.519 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.519 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.519 DEBUG:boto:Method: GET 2024-04-12T03:54:04.520 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.520 DEBUG:boto:Data: 2024-04-12T03:54:04.520 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.520 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.520 DEBUG:boto:Params: {} 2024-04-12T03:54:04.520 DEBUG:boto:Token: None 2024-04-12T03:54:04.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg= 2024-04-12T03:54:04.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg=', 'Content-Length': '0'} 2024-04-12T03:54:04.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c34d569c87b0778e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.522 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.523 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.523 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.523 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.523 DEBUG:boto:Method: GET 2024-04-12T03:54:04.523 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.523 DEBUG:boto:Data: 2024-04-12T03:54:04.523 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.523 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.523 DEBUG:boto:Params: {} 2024-04-12T03:54:04.524 DEBUG:boto:Token: None 2024-04-12T03:54:04.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg= 2024-04-12T03:54:04.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg=', 'Content-Length': '0'} 2024-04-12T03:54:04.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b348d99a1caf07-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.526 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.526 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.526 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.527 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.527 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.527 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.527 DEBUG:boto:Data: 2024-04-12T03:54:04.527 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.527 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.527 DEBUG:boto:Params: {} 2024-04-12T03:54:04.527 DEBUG:boto:Token: None 2024-04-12T03:54:04.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+ul8SM1UhyOHIu1NxYL5Fnr6Rc= 2024-04-12T03:54:04.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+ul8SM1UhyOHIu1NxYL5Fnr6Rc=', 'Content-Length': '0'} 2024-04-12T03:54:04.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0cfd79b77879ba-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.529 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.529 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.530 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.530 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.530 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.530 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.530 DEBUG:boto:Data: 2024-04-12T03:54:04.530 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.530 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.530 DEBUG:boto:Params: {} 2024-04-12T03:54:04.530 DEBUG:boto:Token: None 2024-04-12T03:54:04.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C+ul8SM1UhyOHIu1NxYL5Fnr6Rc= 2024-04-12T03:54:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C+ul8SM1UhyOHIu1NxYL5Fnr6Rc=', 'Content-Length': '0'} 2024-04-12T03:54:04.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40c64d1c330151c-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:54:04.533 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.533 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.533 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.533 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.533 DEBUG:boto:Method: GET 2024-04-12T03:54:04.533 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.533 DEBUG:boto:Data: 2024-04-12T03:54:04.533 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.533 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.534 DEBUG:boto:Params: {} 2024-04-12T03:54:04.534 DEBUG:boto:Token: None 2024-04-12T03:54:04.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg= 2024-04-12T03:54:04.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg=', 'Content-Length': '0'} 2024-04-12T03:54:04.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9b01d7eac2e123-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.578 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.578 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.579 DEBUG:boto:path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.579 DEBUG:boto:auth_path=/dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.579 DEBUG:boto:Method: GET 2024-04-12T03:54:04.579 DEBUG:boto:Path: /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.579 DEBUG:boto:Data: 2024-04-12T03:54:04.579 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.579 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.579 DEBUG:boto:Params: {} 2024-04-12T03:54:04.579 DEBUG:boto:Token: None 2024-04-12T03:54:04.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55?versionId=null 2024-04-12T03:54:04.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg= 2024-04-12T03:54:04.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QqVL2WO+O3shCHzLb4mRFoXmjQg=', 'Content-Length': '0'} 2024-04-12T03:54:04.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e03666aca72c69-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.582 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.582 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.582 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.582 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.582 DEBUG:boto:Method: GET 2024-04-12T03:54:04.582 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.582 DEBUG:boto:Data: 2024-04-12T03:54:04.582 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.583 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.583 DEBUG:boto:Params: {} 2024-04-12T03:54:04.583 DEBUG:boto:Token: None 2024-04-12T03:54:04.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08d963d5cd25d19-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.585 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.586 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.586 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.586 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.586 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.586 DEBUG:boto:Method: GET 2024-04-12T03:54:04.586 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.586 DEBUG:boto:Data: 2024-04-12T03:54:04.586 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.586 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.587 DEBUG:boto:Params: {} 2024-04-12T03:54:04.587 DEBUG:boto:Token: None 2024-04-12T03:54:04.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000619ee942894e2119-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.591 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.591 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.591 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.591 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.591 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:54:04.592 DEBUG:boto:Data: 2024-04-12T03:54:04.592 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.592 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.592 DEBUG:boto:Params: {} 2024-04-12T03:54:04.592 DEBUG:boto:Token: None 2024-04-12T03:54:04.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55 2024-04-12T03:54:04.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eZhKopAB+hn0h3tmHF2AMCcZ6N0= 2024-04-12T03:54:04.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eZhKopAB+hn0h3tmHF2AMCcZ6N0=', 'Content-Length': '0'} 2024-04-12T03:54:04.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000260f752a696d481f-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.594 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.594 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.594 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.594 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:54:04.594 DEBUG:boto:Data: 2024-04-12T03:54:04.594 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.595 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.595 DEBUG:boto:Params: {} 2024-04-12T03:54:04.595 DEBUG:boto:Token: None 2024-04-12T03:54:04.595 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55 2024-04-12T03:54:04.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eZhKopAB+hn0h3tmHF2AMCcZ6N0= 2024-04-12T03:54:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eZhKopAB+hn0h3tmHF2AMCcZ6N0=', 'Content-Length': '0'} 2024-04-12T03:54:04.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe37b9c800322af-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:54:04.597 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.597 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.597 DEBUG:boto:Method: GET 2024-04-12T03:54:04.598 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:54:04.598 DEBUG:boto:Data: 2024-04-12T03:54:04.598 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.598 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.598 DEBUG:boto:Params: {} 2024-04-12T03:54:04.598 DEBUG:boto:Token: None 2024-04-12T03:54:04.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55 2024-04-12T03:54:04.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:geAGc5rmm7J4vIm4Bq9re2Lu3vs= 2024-04-12T03:54:04.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:geAGc5rmm7J4vIm4Bq9re2Lu3vs=', 'Content-Length': '0'} 2024-04-12T03:54:04.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ceccc7089110956-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.600 DEBUG:boto:path=/dgbpxg-61/obj55 2024-04-12T03:54:04.600 DEBUG:boto:auth_path=/dgbpxg-61/obj55 2024-04-12T03:54:04.601 DEBUG:boto:Method: GET 2024-04-12T03:54:04.601 DEBUG:boto:Path: /dgbpxg-61/obj55 2024-04-12T03:54:04.601 DEBUG:boto:Data: 2024-04-12T03:54:04.601 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.601 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.601 DEBUG:boto:Params: {} 2024-04-12T03:54:04.601 DEBUG:boto:Token: None 2024-04-12T03:54:04.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj55 2024-04-12T03:54:04.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:geAGc5rmm7J4vIm4Bq9re2Lu3vs= 2024-04-12T03:54:04.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:geAGc5rmm7J4vIm4Bq9re2Lu3vs=', 'Content-Length': '0'} 2024-04-12T03:54:04.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:40:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008651d432333f41c5-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.603 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.603 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.603 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.604 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.604 DEBUG:boto:Method: GET 2024-04-12T03:54:04.604 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.604 DEBUG:boto:Data: 2024-04-12T03:54:04.604 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.604 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.604 DEBUG:boto:Params: {} 2024-04-12T03:54:04.604 DEBUG:boto:Token: None 2024-04-12T03:54:04.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069c485632311ec59-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.607 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.607 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.607 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.607 DEBUG:boto:path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.608 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.608 DEBUG:boto:Method: GET 2024-04-12T03:54:04.608 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj55 2024-04-12T03:54:04.608 DEBUG:boto:Data: 2024-04-12T03:54:04.608 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.608 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.608 DEBUG:boto:Params: {} 2024-04-12T03:54:04.608 DEBUG:boto:Token: None 2024-04-12T03:54:04.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d4065af68a692e2-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.654 DEBUG:boto:b'dgbpxg-61obj551000falseobj552024-04-12T03:40:51.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:54:04.655 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.655 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.655 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.655 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.655 DEBUG:boto:Method: GET 2024-04-12T03:54:04.655 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.655 DEBUG:boto:Data: 2024-04-12T03:54:04.655 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.656 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.656 DEBUG:boto:Params: {} 2024-04-12T03:54:04.656 DEBUG:boto:Token: None 2024-04-12T03:54:04.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y= 2024-04-12T03:54:04.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y=', 'Content-Length': '0'} 2024-04-12T03:54:04.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d7645ba69dee6de-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.658 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.658 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.658 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.658 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.658 DEBUG:boto:Method: GET 2024-04-12T03:54:04.659 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.659 DEBUG:boto:Data: 2024-04-12T03:54:04.659 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.659 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.659 DEBUG:boto:Params: {} 2024-04-12T03:54:04.659 DEBUG:boto:Token: None 2024-04-12T03:54:04.659 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.659 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y= 2024-04-12T03:54:04.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y=', 'Content-Length': '0'} 2024-04-12T03:54:04.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a910efbd8995ffe2-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.661 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.662 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.662 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.662 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.662 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.662 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.662 DEBUG:boto:Data: 2024-04-12T03:54:04.662 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.662 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.662 DEBUG:boto:Params: {} 2024-04-12T03:54:04.662 DEBUG:boto:Token: None 2024-04-12T03:54:04.663 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.663 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jFVxkSkbQPh1MiTI3PN7IeA+m0A= 2024-04-12T03:54:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jFVxkSkbQPh1MiTI3PN7IeA+m0A=', 'Content-Length': '0'} 2024-04-12T03:54:04.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ada638ca44521c-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.664 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.664 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.665 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.665 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.665 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.665 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.665 DEBUG:boto:Data: 2024-04-12T03:54:04.665 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.665 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.665 DEBUG:boto:Params: {} 2024-04-12T03:54:04.665 DEBUG:boto:Token: None 2024-04-12T03:54:04.666 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.666 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jFVxkSkbQPh1MiTI3PN7IeA+m0A= 2024-04-12T03:54:04.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jFVxkSkbQPh1MiTI3PN7IeA+m0A=', 'Content-Length': '0'} 2024-04-12T03:54:04.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d10b37a92e6fbe-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:54:04.668 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.668 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.668 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.668 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.668 DEBUG:boto:Method: GET 2024-04-12T03:54:04.668 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.669 DEBUG:boto:Data: 2024-04-12T03:54:04.669 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.669 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.669 DEBUG:boto:Params: {} 2024-04-12T03:54:04.669 DEBUG:boto:Token: None 2024-04-12T03:54:04.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y= 2024-04-12T03:54:04.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y=', 'Content-Length': '0'} 2024-04-12T03:54:04.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098339470e3e12353-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.671 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.671 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.671 DEBUG:boto:path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.671 DEBUG:boto:auth_path=/dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.671 DEBUG:boto:Method: GET 2024-04-12T03:54:04.671 DEBUG:boto:Path: /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.671 DEBUG:boto:Data: 2024-04-12T03:54:04.672 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.672 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.672 DEBUG:boto:Params: {} 2024-04-12T03:54:04.672 DEBUG:boto:Token: None 2024-04-12T03:54:04.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57?versionId=null 2024-04-12T03:54:04.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y= 2024-04-12T03:54:04.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qq7W5gtL2QgGzkWwWNVSKlUcu5Y=', 'Content-Length': '0'} 2024-04-12T03:54:04.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a961e4478e1f4e19-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.674 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.674 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.674 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.674 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.674 DEBUG:boto:Method: GET 2024-04-12T03:54:04.674 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.674 DEBUG:boto:Data: 2024-04-12T03:54:04.675 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.675 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.675 DEBUG:boto:Params: {} 2024-04-12T03:54:04.675 DEBUG:boto:Token: None 2024-04-12T03:54:04.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000323351e0c5bc84d4-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.677 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.677 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.678 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.678 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.678 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.678 DEBUG:boto:Method: GET 2024-04-12T03:54:04.678 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.678 DEBUG:boto:Data: 2024-04-12T03:54:04.678 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.678 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.678 DEBUG:boto:Params: {} 2024-04-12T03:54:04.678 DEBUG:boto:Token: None 2024-04-12T03:54:04.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.679 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d21cd447802ebb-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.682 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.683 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.683 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.683 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.683 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:54:04.683 DEBUG:boto:Data: 2024-04-12T03:54:04.683 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.683 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.683 DEBUG:boto:Params: {} 2024-04-12T03:54:04.684 DEBUG:boto:Token: None 2024-04-12T03:54:04.684 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57 2024-04-12T03:54:04.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lOkg+ioYm+tVE8XfIbkjGmP4vkE= 2024-04-12T03:54:04.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lOkg+ioYm+tVE8XfIbkjGmP4vkE=', 'Content-Length': '0'} 2024-04-12T03:54:04.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e533cfb1c461d05-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.686 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.686 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.686 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.686 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:54:04.686 DEBUG:boto:Data: 2024-04-12T03:54:04.686 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.687 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.687 DEBUG:boto:Params: {} 2024-04-12T03:54:04.687 DEBUG:boto:Token: None 2024-04-12T03:54:04.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57 2024-04-12T03:54:04.687 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lOkg+ioYm+tVE8XfIbkjGmP4vkE= 2024-04-12T03:54:04.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lOkg+ioYm+tVE8XfIbkjGmP4vkE=', 'Content-Length': '0'} 2024-04-12T03:54:04.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef881bce63967d06-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T03:54:04.689 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.689 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.689 DEBUG:boto:Method: GET 2024-04-12T03:54:04.689 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:54:04.689 DEBUG:boto:Data: 2024-04-12T03:54:04.689 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.689 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.689 DEBUG:boto:Params: {} 2024-04-12T03:54:04.689 DEBUG:boto:Token: None 2024-04-12T03:54:04.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57 2024-04-12T03:54:04.690 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vt19U3Q6ACHxarWUNfpCMnxnIJ8= 2024-04-12T03:54:04.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vt19U3Q6ACHxarWUNfpCMnxnIJ8=', 'Content-Length': '0'} 2024-04-12T03:54:04.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b159aaf326688658-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.692 DEBUG:boto:path=/dgbpxg-61/obj57 2024-04-12T03:54:04.692 DEBUG:boto:auth_path=/dgbpxg-61/obj57 2024-04-12T03:54:04.692 DEBUG:boto:Method: GET 2024-04-12T03:54:04.692 DEBUG:boto:Path: /dgbpxg-61/obj57 2024-04-12T03:54:04.692 DEBUG:boto:Data: 2024-04-12T03:54:04.692 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.692 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.692 DEBUG:boto:Params: {} 2024-04-12T03:54:04.692 DEBUG:boto:Token: None 2024-04-12T03:54:04.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj57 2024-04-12T03:54:04.693 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vt19U3Q6ACHxarWUNfpCMnxnIJ8= 2024-04-12T03:54:04.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vt19U3Q6ACHxarWUNfpCMnxnIJ8=', 'Content-Length': '0'} 2024-04-12T03:54:04.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e517b4cb5a6ce7a-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.695 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.695 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.695 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.695 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.695 DEBUG:boto:Method: GET 2024-04-12T03:54:04.695 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.695 DEBUG:boto:Data: 2024-04-12T03:54:04.695 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.695 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.695 DEBUG:boto:Params: {} 2024-04-12T03:54:04.696 DEBUG:boto:Token: None 2024-04-12T03:54:04.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f59884dc0a1604b-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.698 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.698 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.698 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.698 DEBUG:boto:path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.699 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.699 DEBUG:boto:Method: GET 2024-04-12T03:54:04.699 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj57 2024-04-12T03:54:04.699 DEBUG:boto:Data: 2024-04-12T03:54:04.699 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.699 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.699 DEBUG:boto:Params: {} 2024-04-12T03:54:04.699 DEBUG:boto:Token: None 2024-04-12T03:54:04.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178be17062cc15d2-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.750 DEBUG:boto:b'dgbpxg-61obj571000falseobj572024-04-12T03:43:54.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:54:04.751 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.751 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.751 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.751 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.751 DEBUG:boto:Method: GET 2024-04-12T03:54:04.751 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.751 DEBUG:boto:Data: 2024-04-12T03:54:04.752 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.752 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.752 DEBUG:boto:Params: {} 2024-04-12T03:54:04.752 DEBUG:boto:Token: None 2024-04-12T03:54:04.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.752 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc= 2024-04-12T03:54:04.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc=', 'Content-Length': '0'} 2024-04-12T03:54:04.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d350c70b6b37d511-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.755 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.755 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.755 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.755 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.755 DEBUG:boto:Method: GET 2024-04-12T03:54:04.755 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.756 DEBUG:boto:Data: 2024-04-12T03:54:04.756 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.756 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.756 DEBUG:boto:Params: {} 2024-04-12T03:54:04.756 DEBUG:boto:Token: None 2024-04-12T03:54:04.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.756 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc= 2024-04-12T03:54:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc=', 'Content-Length': '0'} 2024-04-12T03:54:04.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5ed2fae360c383-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.758 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.759 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.759 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.759 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.759 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.759 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.759 DEBUG:boto:Data: 2024-04-12T03:54:04.759 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.759 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.759 DEBUG:boto:Params: {} 2024-04-12T03:54:04.759 DEBUG:boto:Token: None 2024-04-12T03:54:04.760 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zAIYX7a6XgEzfnLansugDYF5sLE= 2024-04-12T03:54:04.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zAIYX7a6XgEzfnLansugDYF5sLE=', 'Content-Length': '0'} 2024-04-12T03:54:04.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011821eb7a5551b26-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.761 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.762 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.762 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.762 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.762 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.762 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.762 DEBUG:boto:Data: 2024-04-12T03:54:04.762 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.762 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.762 DEBUG:boto:Params: {} 2024-04-12T03:54:04.762 DEBUG:boto:Token: None 2024-04-12T03:54:04.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.763 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zAIYX7a6XgEzfnLansugDYF5sLE= 2024-04-12T03:54:04.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zAIYX7a6XgEzfnLansugDYF5sLE=', 'Content-Length': '0'} 2024-04-12T03:54:04.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ec8fc065f11225-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:54:04.764 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.765 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.765 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.765 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.765 DEBUG:boto:Method: GET 2024-04-12T03:54:04.765 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.765 DEBUG:boto:Data: 2024-04-12T03:54:04.765 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.765 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.765 DEBUG:boto:Params: {} 2024-04-12T03:54:04.766 DEBUG:boto:Token: None 2024-04-12T03:54:04.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc= 2024-04-12T03:54:04.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc=', 'Content-Length': '0'} 2024-04-12T03:54:04.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8aafb0ce017bc9-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.810 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.811 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.811 DEBUG:boto:path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.811 DEBUG:boto:auth_path=/dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.811 DEBUG:boto:Method: GET 2024-04-12T03:54:04.811 DEBUG:boto:Path: /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.811 DEBUG:boto:Data: 2024-04-12T03:54:04.811 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.811 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.812 DEBUG:boto:Params: {} 2024-04-12T03:54:04.812 DEBUG:boto:Token: None 2024-04-12T03:54:04.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58?versionId=null 2024-04-12T03:54:04.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc= 2024-04-12T03:54:04.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bNdon9avwQ5F/ww++saG8OhOiNc=', 'Content-Length': '0'} 2024-04-12T03:54:04.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048e0cd8a9a2a9ae2-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.814 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.814 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.814 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.814 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.814 DEBUG:boto:Method: GET 2024-04-12T03:54:04.815 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.815 DEBUG:boto:Data: 2024-04-12T03:54:04.815 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.815 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.815 DEBUG:boto:Params: {} 2024-04-12T03:54:04.815 DEBUG:boto:Token: None 2024-04-12T03:54:04.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701926a9c60fe65e-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.818 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.818 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.818 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.818 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.818 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.818 DEBUG:boto:Method: GET 2024-04-12T03:54:04.819 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.819 DEBUG:boto:Data: 2024-04-12T03:54:04.819 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.819 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.819 DEBUG:boto:Params: {} 2024-04-12T03:54:04.819 DEBUG:boto:Token: None 2024-04-12T03:54:04.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1a2a3bb92f53fb-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.823 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.823 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.823 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.823 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.824 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:54:04.824 DEBUG:boto:Data: 2024-04-12T03:54:04.824 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.824 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.824 DEBUG:boto:Params: {} 2024-04-12T03:54:04.824 DEBUG:boto:Token: None 2024-04-12T03:54:04.824 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58 2024-04-12T03:54:04.824 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sOV3JEL85a+ij9EB8meGQBMGzBs= 2024-04-12T03:54:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sOV3JEL85a+ij9EB8meGQBMGzBs=', 'Content-Length': '0'} 2024-04-12T03:54:04.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a22422ed5c5ed83-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.826 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.826 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.826 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.826 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:54:04.826 DEBUG:boto:Data: 2024-04-12T03:54:04.827 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.827 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.827 DEBUG:boto:Params: {} 2024-04-12T03:54:04.827 DEBUG:boto:Token: None 2024-04-12T03:54:04.827 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58 2024-04-12T03:54:04.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sOV3JEL85a+ij9EB8meGQBMGzBs= 2024-04-12T03:54:04.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sOV3JEL85a+ij9EB8meGQBMGzBs=', 'Content-Length': '0'} 2024-04-12T03:54:04.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094093f0e85f30914-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T03:54:04.829 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.829 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.829 DEBUG:boto:Method: GET 2024-04-12T03:54:04.829 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:54:04.830 DEBUG:boto:Data: 2024-04-12T03:54:04.830 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.830 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.830 DEBUG:boto:Params: {} 2024-04-12T03:54:04.830 DEBUG:boto:Token: None 2024-04-12T03:54:04.830 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58 2024-04-12T03:54:04.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GaXSa0ht1XRszT1jdZAX6ejFFs= 2024-04-12T03:54:04.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GaXSa0ht1XRszT1jdZAX6ejFFs=', 'Content-Length': '0'} 2024-04-12T03:54:04.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c335524db68acb9f-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.832 DEBUG:boto:path=/dgbpxg-61/obj58 2024-04-12T03:54:04.832 DEBUG:boto:auth_path=/dgbpxg-61/obj58 2024-04-12T03:54:04.832 DEBUG:boto:Method: GET 2024-04-12T03:54:04.832 DEBUG:boto:Path: /dgbpxg-61/obj58 2024-04-12T03:54:04.832 DEBUG:boto:Data: 2024-04-12T03:54:04.832 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.833 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.833 DEBUG:boto:Params: {} 2024-04-12T03:54:04.833 DEBUG:boto:Token: None 2024-04-12T03:54:04.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj58 2024-04-12T03:54:04.833 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2GaXSa0ht1XRszT1jdZAX6ejFFs= 2024-04-12T03:54:04.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2GaXSa0ht1XRszT1jdZAX6ejFFs=', 'Content-Length': '0'} 2024-04-12T03:54:04.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0e2f36ca453476-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.882 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.882 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.882 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.883 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.883 DEBUG:boto:Method: GET 2024-04-12T03:54:04.883 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.883 DEBUG:boto:Data: 2024-04-12T03:54:04.883 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.883 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.883 DEBUG:boto:Params: {} 2024-04-12T03:54:04.883 DEBUG:boto:Token: None 2024-04-12T03:54:04.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5771ce4a83ab2ff-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.886 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.886 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.886 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.886 DEBUG:boto:path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.886 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.886 DEBUG:boto:Method: GET 2024-04-12T03:54:04.887 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj58 2024-04-12T03:54:04.887 DEBUG:boto:Data: 2024-04-12T03:54:04.887 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.887 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.887 DEBUG:boto:Params: {} 2024-04-12T03:54:04.887 DEBUG:boto:Token: None 2024-04-12T03:54:04.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030bd643281058363-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.891 DEBUG:boto:b'dgbpxg-61obj581000falseobj582024-04-12T03:47:19.650Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:54:04.891 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.891 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.892 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.892 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.892 DEBUG:boto:Method: GET 2024-04-12T03:54:04.892 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.892 DEBUG:boto:Data: 2024-04-12T03:54:04.892 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.892 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.892 DEBUG:boto:Params: {} 2024-04-12T03:54:04.892 DEBUG:boto:Token: None 2024-04-12T03:54:04.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs= 2024-04-12T03:54:04.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs=', 'Content-Length': '0'} 2024-04-12T03:54:04.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca36b630f30f1d12-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.894 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.895 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.895 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.895 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.895 DEBUG:boto:Method: GET 2024-04-12T03:54:04.895 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.895 DEBUG:boto:Data: 2024-04-12T03:54:04.895 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.895 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.895 DEBUG:boto:Params: {} 2024-04-12T03:54:04.895 DEBUG:boto:Token: None 2024-04-12T03:54:04.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs= 2024-04-12T03:54:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs=', 'Content-Length': '0'} 2024-04-12T03:54:04.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f249bd7fc70873-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.897 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.897 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.898 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.898 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.898 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.898 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.898 DEBUG:boto:Data: 2024-04-12T03:54:04.898 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.898 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.898 DEBUG:boto:Params: {} 2024-04-12T03:54:04.898 DEBUG:boto:Token: None 2024-04-12T03:54:04.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.899 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXwhpkZrS8vlMezEqSFbSrtk6WQ= 2024-04-12T03:54:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXwhpkZrS8vlMezEqSFbSrtk6WQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d506a1ca78b8cf08-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.900 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.900 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.901 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.901 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.901 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.901 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.901 DEBUG:boto:Data: 2024-04-12T03:54:04.901 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.901 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.901 DEBUG:boto:Params: {} 2024-04-12T03:54:04.901 DEBUG:boto:Token: None 2024-04-12T03:54:04.901 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXwhpkZrS8vlMezEqSFbSrtk6WQ= 2024-04-12T03:54:04.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXwhpkZrS8vlMezEqSFbSrtk6WQ=', 'Content-Length': '0'} 2024-04-12T03:54:04.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054cb96ebb115c1cb-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:54:04.903 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.904 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.904 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.904 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.904 DEBUG:boto:Method: GET 2024-04-12T03:54:04.904 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.904 DEBUG:boto:Data: 2024-04-12T03:54:04.904 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.904 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.904 DEBUG:boto:Params: {} 2024-04-12T03:54:04.904 DEBUG:boto:Token: None 2024-04-12T03:54:04.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.905 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs= 2024-04-12T03:54:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs=', 'Content-Length': '0'} 2024-04-12T03:54:04.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042276bcccb901910-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.906 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.907 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.907 DEBUG:boto:path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.907 DEBUG:boto:auth_path=/dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.907 DEBUG:boto:Method: GET 2024-04-12T03:54:04.907 DEBUG:boto:Path: /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.907 DEBUG:boto:Data: 2024-04-12T03:54:04.907 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.907 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.907 DEBUG:boto:Params: {} 2024-04-12T03:54:04.907 DEBUG:boto:Token: None 2024-04-12T03:54:04.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62?versionId=null 2024-04-12T03:54:04.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs= 2024-04-12T03:54:04.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:78fhOMC90uXmJ+royb8JbIfkXUs=', 'Content-Length': '0'} 2024-04-12T03:54:04.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001700e41a9afbd867-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.909 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.909 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.910 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.910 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.910 DEBUG:boto:Method: GET 2024-04-12T03:54:04.910 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.910 DEBUG:boto:Data: 2024-04-12T03:54:04.910 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.910 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.910 DEBUG:boto:Params: {} 2024-04-12T03:54:04.910 DEBUG:boto:Token: None 2024-04-12T03:54:04.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.911 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f2c0e03ee2199f-006618b05c-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.954 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.954 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:04.955 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:04.955 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.955 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.955 DEBUG:boto:Method: GET 2024-04-12T03:54:04.955 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:54:04.955 DEBUG:boto:Data: 2024-04-12T03:54:04.955 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.955 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.955 DEBUG:boto:Params: {} 2024-04-12T03:54:04.955 DEBUG:boto:Token: None 2024-04-12T03:54:04.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/ 2024-04-12T03:54:04.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E= 2024-04-12T03:54:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JI0qwHauUJhwLGU3GeMK9qNCk3E=', 'Content-Length': '0'} 2024-04-12T03:54:04.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de795f466ada1291-006618b05c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.959 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:04.960 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.960 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.960 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.960 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:54:04.960 DEBUG:boto:Data: 2024-04-12T03:54:04.960 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.960 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.960 DEBUG:boto:Params: {} 2024-04-12T03:54:04.961 DEBUG:boto:Token: None 2024-04-12T03:54:04.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62 2024-04-12T03:54:04.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:25te6V4lWEfmT8ZUMDOLz0EZGgM= 2024-04-12T03:54:04.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:25te6V4lWEfmT8ZUMDOLz0EZGgM=', 'Content-Length': '0'} 2024-04-12T03:54:04.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009930936e237e921e-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.963 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.963 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.963 DEBUG:boto:Method: HEAD 2024-04-12T03:54:04.963 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:54:04.963 DEBUG:boto:Data: 2024-04-12T03:54:04.963 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.963 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.963 DEBUG:boto:Params: {} 2024-04-12T03:54:04.963 DEBUG:boto:Token: None 2024-04-12T03:54:04.964 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62 2024-04-12T03:54:04.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:25te6V4lWEfmT8ZUMDOLz0EZGgM= 2024-04-12T03:54:04.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:25te6V4lWEfmT8ZUMDOLz0EZGgM=', 'Content-Length': '0'} 2024-04-12T03:54:04.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c0786991422a93-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T03:54:04.966 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.966 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.966 DEBUG:boto:Method: GET 2024-04-12T03:54:04.966 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:54:04.966 DEBUG:boto:Data: 2024-04-12T03:54:04.966 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:04.966 DEBUG:boto:Port: 8001 2024-04-12T03:54:04.966 DEBUG:boto:Params: {} 2024-04-12T03:54:04.966 DEBUG:boto:Token: None 2024-04-12T03:54:04.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62 2024-04-12T03:54:04.967 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LYybdJAo/oNV2Py+Y53+SnSBOMI= 2024-04-12T03:54:04.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LYybdJAo/oNV2Py+Y53+SnSBOMI=', 'Content-Length': '0'} 2024-04-12T03:54:04.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4475aa3c0053794-006618b05c-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:04.968 DEBUG:boto:path=/dgbpxg-61/obj62 2024-04-12T03:54:04.969 DEBUG:boto:auth_path=/dgbpxg-61/obj62 2024-04-12T03:54:04.969 DEBUG:boto:Method: GET 2024-04-12T03:54:04.969 DEBUG:boto:Path: /dgbpxg-61/obj62 2024-04-12T03:54:04.969 DEBUG:boto:Data: 2024-04-12T03:54:04.969 DEBUG:boto:Headers: {} 2024-04-12T03:54:04.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:04.969 DEBUG:boto:Port: 8000 2024-04-12T03:54:04.969 DEBUG:boto:Params: {} 2024-04-12T03:54:04.969 DEBUG:boto:Token: None 2024-04-12T03:54:04.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:04 GMT /dgbpxg-61/obj62 2024-04-12T03:54:04.970 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LYybdJAo/oNV2Py+Y53+SnSBOMI= 2024-04-12T03:54:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LYybdJAo/oNV2Py+Y53+SnSBOMI=', 'Content-Length': '0'} 2024-04-12T03:54:04.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090932fef9c15c991-006618b05c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.014 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.015 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.015 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.015 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.015 DEBUG:boto:Method: GET 2024-04-12T03:54:05.015 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.015 DEBUG:boto:Data: 2024-04-12T03:54:05.015 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.015 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.015 DEBUG:boto:Params: {} 2024-04-12T03:54:05.016 DEBUG:boto:Token: None 2024-04-12T03:54:05.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3abed380a199362-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.018 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.019 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.019 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.019 DEBUG:boto:path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.019 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.019 DEBUG:boto:Method: GET 2024-04-12T03:54:05.019 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj62 2024-04-12T03:54:05.019 DEBUG:boto:Data: 2024-04-12T03:54:05.019 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.020 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.020 DEBUG:boto:Params: {} 2024-04-12T03:54:05.020 DEBUG:boto:Token: None 2024-04-12T03:54:05.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000969fa684bc5e977d-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.024 DEBUG:boto:b'dgbpxg-61obj621000falseobj622024-04-12T03:50:18.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:54:05.025 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.025 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.025 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.025 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.025 DEBUG:boto:Method: GET 2024-04-12T03:54:05.025 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.025 DEBUG:boto:Data: 2024-04-12T03:54:05.025 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.025 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.025 DEBUG:boto:Params: {} 2024-04-12T03:54:05.026 DEBUG:boto:Token: None 2024-04-12T03:54:05.026 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.026 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE= 2024-04-12T03:54:05.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE=', 'Content-Length': '0'} 2024-04-12T03:54:05.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fdf3cd432dbde2-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.070 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.070 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.071 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.071 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.071 DEBUG:boto:Method: GET 2024-04-12T03:54:05.071 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.071 DEBUG:boto:Data: 2024-04-12T03:54:05.071 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.071 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.071 DEBUG:boto:Params: {} 2024-04-12T03:54:05.071 DEBUG:boto:Token: None 2024-04-12T03:54:05.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE= 2024-04-12T03:54:05.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE=', 'Content-Length': '0'} 2024-04-12T03:54:05.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d3c9003b14c8a3-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.074 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.074 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.074 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.074 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.074 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.074 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.074 DEBUG:boto:Data: 2024-04-12T03:54:05.075 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.075 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.075 DEBUG:boto:Params: {} 2024-04-12T03:54:05.075 DEBUG:boto:Token: None 2024-04-12T03:54:05.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HE5oYNGhSDjRIVbE57tQPevXJRE= 2024-04-12T03:54:05.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HE5oYNGhSDjRIVbE57tQPevXJRE=', 'Content-Length': '0'} 2024-04-12T03:54:05.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003714c00f8a482ec6-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.077 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.077 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.077 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.077 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.077 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.077 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.077 DEBUG:boto:Data: 2024-04-12T03:54:05.077 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.078 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.078 DEBUG:boto:Params: {} 2024-04-12T03:54:05.078 DEBUG:boto:Token: None 2024-04-12T03:54:05.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HE5oYNGhSDjRIVbE57tQPevXJRE= 2024-04-12T03:54:05.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HE5oYNGhSDjRIVbE57tQPevXJRE=', 'Content-Length': '0'} 2024-04-12T03:54:05.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee6705d3a1e7000-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:54:05.080 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.080 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.080 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.080 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.080 DEBUG:boto:Method: GET 2024-04-12T03:54:05.080 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.080 DEBUG:boto:Data: 2024-04-12T03:54:05.081 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.081 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.081 DEBUG:boto:Params: {} 2024-04-12T03:54:05.081 DEBUG:boto:Token: None 2024-04-12T03:54:05.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE= 2024-04-12T03:54:05.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE=', 'Content-Length': '0'} 2024-04-12T03:54:05.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005211046e6e312e37-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.083 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.084 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.084 DEBUG:boto:path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.084 DEBUG:boto:auth_path=/dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.084 DEBUG:boto:Method: GET 2024-04-12T03:54:05.084 DEBUG:boto:Path: /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.084 DEBUG:boto:Data: 2024-04-12T03:54:05.084 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.084 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.084 DEBUG:boto:Params: {} 2024-04-12T03:54:05.084 DEBUG:boto:Token: None 2024-04-12T03:54:05.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68?versionId=null 2024-04-12T03:54:05.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE= 2024-04-12T03:54:05.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iIGd+bG8QCvYym0A5Mj4c5UH7yE=', 'Content-Length': '0'} 2024-04-12T03:54:05.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd799d7280d381c-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.086 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.087 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.087 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.087 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.087 DEBUG:boto:Method: GET 2024-04-12T03:54:05.087 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.087 DEBUG:boto:Data: 2024-04-12T03:54:05.087 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.087 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.087 DEBUG:boto:Params: {} 2024-04-12T03:54:05.087 DEBUG:boto:Token: None 2024-04-12T03:54:05.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ded4a7373b1f66a-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.090 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.090 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.090 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.090 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.091 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.091 DEBUG:boto:Method: GET 2024-04-12T03:54:05.091 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.091 DEBUG:boto:Data: 2024-04-12T03:54:05.091 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.091 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.091 DEBUG:boto:Params: {} 2024-04-12T03:54:05.091 DEBUG:boto:Token: None 2024-04-12T03:54:05.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047b9c9fa4e10a06e-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.095 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.096 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.096 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.096 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.096 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:54:05.096 DEBUG:boto:Data: 2024-04-12T03:54:05.096 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.096 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.096 DEBUG:boto:Params: {} 2024-04-12T03:54:05.097 DEBUG:boto:Token: None 2024-04-12T03:54:05.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68 2024-04-12T03:54:05.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZvMonLZylDOIqPJWeOv8uQweW+8= 2024-04-12T03:54:05.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZvMonLZylDOIqPJWeOv8uQweW+8=', 'Content-Length': '0'} 2024-04-12T03:54:05.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005993a4fc721c48f7-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.099 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.099 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.099 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.099 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:54:05.099 DEBUG:boto:Data: 2024-04-12T03:54:05.099 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.099 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.099 DEBUG:boto:Params: {} 2024-04-12T03:54:05.099 DEBUG:boto:Token: None 2024-04-12T03:54:05.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68 2024-04-12T03:54:05.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZvMonLZylDOIqPJWeOv8uQweW+8= 2024-04-12T03:54:05.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZvMonLZylDOIqPJWeOv8uQweW+8=', 'Content-Length': '0'} 2024-04-12T03:54:05.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304ce3e14ed6f886-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T03:54:05.101 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.102 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.102 DEBUG:boto:Method: GET 2024-04-12T03:54:05.102 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:54:05.102 DEBUG:boto:Data: 2024-04-12T03:54:05.102 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.102 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.102 DEBUG:boto:Params: {} 2024-04-12T03:54:05.102 DEBUG:boto:Token: None 2024-04-12T03:54:05.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68 2024-04-12T03:54:05.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RzIgSqlITnbFjMKQvQrqnp+R/+8= 2024-04-12T03:54:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RzIgSqlITnbFjMKQvQrqnp+R/+8=', 'Content-Length': '0'} 2024-04-12T03:54:05.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac560557d5c65fb-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.146 DEBUG:boto:path=/dgbpxg-61/obj68 2024-04-12T03:54:05.147 DEBUG:boto:auth_path=/dgbpxg-61/obj68 2024-04-12T03:54:05.147 DEBUG:boto:Method: GET 2024-04-12T03:54:05.147 DEBUG:boto:Path: /dgbpxg-61/obj68 2024-04-12T03:54:05.147 DEBUG:boto:Data: 2024-04-12T03:54:05.147 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.147 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.147 DEBUG:boto:Params: {} 2024-04-12T03:54:05.147 DEBUG:boto:Token: None 2024-04-12T03:54:05.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj68 2024-04-12T03:54:05.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RzIgSqlITnbFjMKQvQrqnp+R/+8= 2024-04-12T03:54:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RzIgSqlITnbFjMKQvQrqnp+R/+8=', 'Content-Length': '0'} 2024-04-12T03:54:05.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728edfc5b9d0befc-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.150 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.150 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.150 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.150 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.150 DEBUG:boto:Method: GET 2024-04-12T03:54:05.150 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.150 DEBUG:boto:Data: 2024-04-12T03:54:05.151 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.151 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.151 DEBUG:boto:Params: {} 2024-04-12T03:54:05.151 DEBUG:boto:Token: None 2024-04-12T03:54:05.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.151 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6f714d2731718ec-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.153 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.154 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.154 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.154 DEBUG:boto:path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.154 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.154 DEBUG:boto:Method: GET 2024-04-12T03:54:05.154 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj68 2024-04-12T03:54:05.154 DEBUG:boto:Data: 2024-04-12T03:54:05.154 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.155 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.155 DEBUG:boto:Params: {} 2024-04-12T03:54:05.155 DEBUG:boto:Token: None 2024-04-12T03:54:05.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.155 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ef8b34cd7b13dd-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.159 DEBUG:boto:b'dgbpxg-61obj681000falseobj682024-04-12T03:41:32.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:54:05.159 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.159 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.159 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.160 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.160 DEBUG:boto:Method: GET 2024-04-12T03:54:05.160 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.160 DEBUG:boto:Data: 2024-04-12T03:54:05.160 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.160 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.160 DEBUG:boto:Params: {} 2024-04-12T03:54:05.160 DEBUG:boto:Token: None 2024-04-12T03:54:05.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.161 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk= 2024-04-12T03:54:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk=', 'Content-Length': '0'} 2024-04-12T03:54:05.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f46a14caf17c5e-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.163 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.163 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.163 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.163 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.163 DEBUG:boto:Method: GET 2024-04-12T03:54:05.163 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.163 DEBUG:boto:Data: 2024-04-12T03:54:05.163 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.163 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.163 DEBUG:boto:Params: {} 2024-04-12T03:54:05.164 DEBUG:boto:Token: None 2024-04-12T03:54:05.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.164 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk= 2024-04-12T03:54:05.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk=', 'Content-Length': '0'} 2024-04-12T03:54:05.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c483b932381f89-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.166 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.166 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.166 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.166 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.166 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.166 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.166 DEBUG:boto:Data: 2024-04-12T03:54:05.166 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.167 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.167 DEBUG:boto:Params: {} 2024-04-12T03:54:05.167 DEBUG:boto:Token: None 2024-04-12T03:54:05.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.167 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt1CjzIIi0vlLOGgoIpTK/i5Nas= 2024-04-12T03:54:05.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt1CjzIIi0vlLOGgoIpTK/i5Nas=', 'Content-Length': '0'} 2024-04-12T03:54:05.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f673660c4ad439-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.169 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.169 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.169 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.169 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.169 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.169 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.169 DEBUG:boto:Data: 2024-04-12T03:54:05.169 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.170 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.170 DEBUG:boto:Params: {} 2024-04-12T03:54:05.170 DEBUG:boto:Token: None 2024-04-12T03:54:05.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.170 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dt1CjzIIi0vlLOGgoIpTK/i5Nas= 2024-04-12T03:54:05.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt1CjzIIi0vlLOGgoIpTK/i5Nas=', 'Content-Length': '0'} 2024-04-12T03:54:05.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc72091f53a596af-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:54:05.172 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.172 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.172 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.172 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.172 DEBUG:boto:Method: GET 2024-04-12T03:54:05.172 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.172 DEBUG:boto:Data: 2024-04-12T03:54:05.172 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.173 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.173 DEBUG:boto:Params: {} 2024-04-12T03:54:05.173 DEBUG:boto:Token: None 2024-04-12T03:54:05.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.173 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk= 2024-04-12T03:54:05.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk=', 'Content-Length': '0'} 2024-04-12T03:54:05.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d4f1fd89655ffd-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.175 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.175 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.175 DEBUG:boto:path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.175 DEBUG:boto:auth_path=/dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.175 DEBUG:boto:Method: GET 2024-04-12T03:54:05.175 DEBUG:boto:Path: /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.175 DEBUG:boto:Data: 2024-04-12T03:54:05.176 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.176 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.176 DEBUG:boto:Params: {} 2024-04-12T03:54:05.176 DEBUG:boto:Token: None 2024-04-12T03:54:05.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69?versionId=null 2024-04-12T03:54:05.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk= 2024-04-12T03:54:05.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ImTLKBzbZVuNkpMAie6g+4RfFZk=', 'Content-Length': '0'} 2024-04-12T03:54:05.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5846329b70737c-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.178 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.178 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.178 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.179 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.179 DEBUG:boto:Method: GET 2024-04-12T03:54:05.179 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.179 DEBUG:boto:Data: 2024-04-12T03:54:05.179 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.179 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.179 DEBUG:boto:Params: {} 2024-04-12T03:54:05.179 DEBUG:boto:Token: None 2024-04-12T03:54:05.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283a1e6cde5efd6b-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.222 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.281 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.281 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.281 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.281 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.281 DEBUG:boto:Method: GET 2024-04-12T03:54:05.281 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.282 DEBUG:boto:Data: 2024-04-12T03:54:05.282 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.282 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.282 DEBUG:boto:Params: {} 2024-04-12T03:54:05.282 DEBUG:boto:Token: None 2024-04-12T03:54:05.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.282 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c7b9cfc9d285bc-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.286 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.287 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.287 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.287 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.287 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:54:05.287 DEBUG:boto:Data: 2024-04-12T03:54:05.287 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.287 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.287 DEBUG:boto:Params: {} 2024-04-12T03:54:05.288 DEBUG:boto:Token: None 2024-04-12T03:54:05.288 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69 2024-04-12T03:54:05.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2NNYkEVwuSaMp2RjhSpBFb0UjEU= 2024-04-12T03:54:05.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2NNYkEVwuSaMp2RjhSpBFb0UjEU=', 'Content-Length': '0'} 2024-04-12T03:54:05.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2ca6e112500f79-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.290 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.290 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.290 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.290 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:54:05.290 DEBUG:boto:Data: 2024-04-12T03:54:05.290 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.291 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.291 DEBUG:boto:Params: {} 2024-04-12T03:54:05.291 DEBUG:boto:Token: None 2024-04-12T03:54:05.291 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69 2024-04-12T03:54:05.291 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2NNYkEVwuSaMp2RjhSpBFb0UjEU= 2024-04-12T03:54:05.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2NNYkEVwuSaMp2RjhSpBFb0UjEU=', 'Content-Length': '0'} 2024-04-12T03:54:05.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c28bf8c3de3d7d-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T03:54:05.293 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.293 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.293 DEBUG:boto:Method: GET 2024-04-12T03:54:05.293 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:54:05.293 DEBUG:boto:Data: 2024-04-12T03:54:05.293 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.294 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.294 DEBUG:boto:Params: {} 2024-04-12T03:54:05.294 DEBUG:boto:Token: None 2024-04-12T03:54:05.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69 2024-04-12T03:54:05.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xG+u9oysUlDxGbg3nlusl4PnZb8= 2024-04-12T03:54:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xG+u9oysUlDxGbg3nlusl4PnZb8=', 'Content-Length': '0'} 2024-04-12T03:54:05.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25e5afeb7da4365-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.296 DEBUG:boto:path=/dgbpxg-61/obj69 2024-04-12T03:54:05.296 DEBUG:boto:auth_path=/dgbpxg-61/obj69 2024-04-12T03:54:05.296 DEBUG:boto:Method: GET 2024-04-12T03:54:05.296 DEBUG:boto:Path: /dgbpxg-61/obj69 2024-04-12T03:54:05.296 DEBUG:boto:Data: 2024-04-12T03:54:05.296 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.297 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.297 DEBUG:boto:Params: {} 2024-04-12T03:54:05.297 DEBUG:boto:Token: None 2024-04-12T03:54:05.297 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj69 2024-04-12T03:54:05.297 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xG+u9oysUlDxGbg3nlusl4PnZb8= 2024-04-12T03:54:05.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xG+u9oysUlDxGbg3nlusl4PnZb8=', 'Content-Length': '0'} 2024-04-12T03:54:05.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87c4997cd998ce5-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.299 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.299 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.299 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.300 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.300 DEBUG:boto:Method: GET 2024-04-12T03:54:05.300 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.300 DEBUG:boto:Data: 2024-04-12T03:54:05.300 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.300 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.300 DEBUG:boto:Params: {} 2024-04-12T03:54:05.300 DEBUG:boto:Token: None 2024-04-12T03:54:05.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.301 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3c18ca5c0ef03ae-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.303 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.303 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.303 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.303 DEBUG:boto:path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.303 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.303 DEBUG:boto:Method: GET 2024-04-12T03:54:05.304 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj69 2024-04-12T03:54:05.304 DEBUG:boto:Data: 2024-04-12T03:54:05.304 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.304 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.304 DEBUG:boto:Params: {} 2024-04-12T03:54:05.304 DEBUG:boto:Token: None 2024-04-12T03:54:05.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.304 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d38315fb383cbf7-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.350 DEBUG:boto:b'dgbpxg-61obj691000falseobj692024-04-12T03:46:16.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:54:05.351 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.351 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.351 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.351 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.351 DEBUG:boto:Method: GET 2024-04-12T03:54:05.351 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.351 DEBUG:boto:Data: 2024-04-12T03:54:05.351 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.352 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.352 DEBUG:boto:Params: {} 2024-04-12T03:54:05.352 DEBUG:boto:Token: None 2024-04-12T03:54:05.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc= 2024-04-12T03:54:05.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc=', 'Content-Length': '0'} 2024-04-12T03:54:05.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201f13bc30158c96-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.354 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.355 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.355 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.355 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.355 DEBUG:boto:Method: GET 2024-04-12T03:54:05.355 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.355 DEBUG:boto:Data: 2024-04-12T03:54:05.355 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.355 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.355 DEBUG:boto:Params: {} 2024-04-12T03:54:05.355 DEBUG:boto:Token: None 2024-04-12T03:54:05.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.356 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc= 2024-04-12T03:54:05.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc=', 'Content-Length': '0'} 2024-04-12T03:54:05.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032bef4fbe2907c31-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.358 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.358 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.358 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.358 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.358 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.358 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.358 DEBUG:boto:Data: 2024-04-12T03:54:05.358 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.359 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.359 DEBUG:boto:Params: {} 2024-04-12T03:54:05.359 DEBUG:boto:Token: None 2024-04-12T03:54:05.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.359 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tsmQt3lQRZEuobst64IemO9Cahk= 2024-04-12T03:54:05.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tsmQt3lQRZEuobst64IemO9Cahk=', 'Content-Length': '0'} 2024-04-12T03:54:05.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079708e8ab37c19fd-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.361 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.361 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.361 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.361 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.361 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.361 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.361 DEBUG:boto:Data: 2024-04-12T03:54:05.361 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.361 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.362 DEBUG:boto:Params: {} 2024-04-12T03:54:05.362 DEBUG:boto:Token: None 2024-04-12T03:54:05.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.362 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tsmQt3lQRZEuobst64IemO9Cahk= 2024-04-12T03:54:05.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tsmQt3lQRZEuobst64IemO9Cahk=', 'Content-Length': '0'} 2024-04-12T03:54:05.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5237e7785ffd17-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:54:05.364 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.364 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.364 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.364 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.364 DEBUG:boto:Method: GET 2024-04-12T03:54:05.364 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.364 DEBUG:boto:Data: 2024-04-12T03:54:05.364 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.365 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.365 DEBUG:boto:Params: {} 2024-04-12T03:54:05.365 DEBUG:boto:Token: None 2024-04-12T03:54:05.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.365 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc= 2024-04-12T03:54:05.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc=', 'Content-Length': '0'} 2024-04-12T03:54:05.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2ed70b8795bd77-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.367 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.367 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.367 DEBUG:boto:path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.367 DEBUG:boto:auth_path=/dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.367 DEBUG:boto:Method: GET 2024-04-12T03:54:05.367 DEBUG:boto:Path: /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.367 DEBUG:boto:Data: 2024-04-12T03:54:05.367 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.368 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.368 DEBUG:boto:Params: {} 2024-04-12T03:54:05.368 DEBUG:boto:Token: None 2024-04-12T03:54:05.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74?versionId=null 2024-04-12T03:54:05.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc= 2024-04-12T03:54:05.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T/vKRamR6Vc+z5pOTUebvudqyyc=', 'Content-Length': '0'} 2024-04-12T03:54:05.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007675906666e1f2e1-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.370 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.370 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.370 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.370 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.370 DEBUG:boto:Method: GET 2024-04-12T03:54:05.371 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.371 DEBUG:boto:Data: 2024-04-12T03:54:05.371 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.371 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.371 DEBUG:boto:Params: {} 2024-04-12T03:54:05.371 DEBUG:boto:Token: None 2024-04-12T03:54:05.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.371 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85b436e1de625cc-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.373 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.374 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.374 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.374 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.374 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.374 DEBUG:boto:Method: GET 2024-04-12T03:54:05.374 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.374 DEBUG:boto:Data: 2024-04-12T03:54:05.375 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.375 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.375 DEBUG:boto:Params: {} 2024-04-12T03:54:05.375 DEBUG:boto:Token: None 2024-04-12T03:54:05.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.375 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083dad1a116884e54-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.379 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.379 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.380 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.380 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.380 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:54:05.380 DEBUG:boto:Data: 2024-04-12T03:54:05.380 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.380 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.380 DEBUG:boto:Params: {} 2024-04-12T03:54:05.380 DEBUG:boto:Token: None 2024-04-12T03:54:05.381 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74 2024-04-12T03:54:05.381 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V2KjWIzCtKJmk/8ESOuBJ2+W8OQ= 2024-04-12T03:54:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V2KjWIzCtKJmk/8ESOuBJ2+W8OQ=', 'Content-Length': '0'} 2024-04-12T03:54:05.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f1031de246ab95-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.383 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.383 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.383 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.383 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:54:05.383 DEBUG:boto:Data: 2024-04-12T03:54:05.383 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.383 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.383 DEBUG:boto:Params: {} 2024-04-12T03:54:05.383 DEBUG:boto:Token: None 2024-04-12T03:54:05.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74 2024-04-12T03:54:05.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:V2KjWIzCtKJmk/8ESOuBJ2+W8OQ= 2024-04-12T03:54:05.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V2KjWIzCtKJmk/8ESOuBJ2+W8OQ=', 'Content-Length': '0'} 2024-04-12T03:54:05.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe0c10f19c4da75-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T03:54:05.385 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.385 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.386 DEBUG:boto:Method: GET 2024-04-12T03:54:05.386 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:54:05.386 DEBUG:boto:Data: 2024-04-12T03:54:05.386 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.386 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.386 DEBUG:boto:Params: {} 2024-04-12T03:54:05.386 DEBUG:boto:Token: None 2024-04-12T03:54:05.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74 2024-04-12T03:54:05.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vG7qOHfjMSMHbsJ8BdEZgZZ/7Xk= 2024-04-12T03:54:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vG7qOHfjMSMHbsJ8BdEZgZZ/7Xk=', 'Content-Length': '0'} 2024-04-12T03:54:05.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048996d8eff6b2966-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.434 DEBUG:boto:path=/dgbpxg-61/obj74 2024-04-12T03:54:05.435 DEBUG:boto:auth_path=/dgbpxg-61/obj74 2024-04-12T03:54:05.435 DEBUG:boto:Method: GET 2024-04-12T03:54:05.435 DEBUG:boto:Path: /dgbpxg-61/obj74 2024-04-12T03:54:05.435 DEBUG:boto:Data: 2024-04-12T03:54:05.435 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.435 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.435 DEBUG:boto:Params: {} 2024-04-12T03:54:05.435 DEBUG:boto:Token: None 2024-04-12T03:54:05.435 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj74 2024-04-12T03:54:05.436 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vG7qOHfjMSMHbsJ8BdEZgZZ/7Xk= 2024-04-12T03:54:05.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vG7qOHfjMSMHbsJ8BdEZgZZ/7Xk=', 'Content-Length': '0'} 2024-04-12T03:54:05.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:51:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007163f117eb44b57d-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.438 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.438 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.438 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.438 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.438 DEBUG:boto:Method: GET 2024-04-12T03:54:05.438 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.438 DEBUG:boto:Data: 2024-04-12T03:54:05.438 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.439 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.439 DEBUG:boto:Params: {} 2024-04-12T03:54:05.439 DEBUG:boto:Token: None 2024-04-12T03:54:05.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.439 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ecc851b4f4484e-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.441 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.442 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.442 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.442 DEBUG:boto:path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.442 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.442 DEBUG:boto:Method: GET 2024-04-12T03:54:05.442 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj74 2024-04-12T03:54:05.442 DEBUG:boto:Data: 2024-04-12T03:54:05.442 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.442 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.443 DEBUG:boto:Params: {} 2024-04-12T03:54:05.443 DEBUG:boto:Token: None 2024-04-12T03:54:05.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.443 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9bb7a1bffd12e6-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.447 DEBUG:boto:b'dgbpxg-61obj741000falseobj742024-04-12T03:51:42.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:54:05.447 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.447 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.447 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.447 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.447 DEBUG:boto:Method: GET 2024-04-12T03:54:05.447 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.447 DEBUG:boto:Data: 2024-04-12T03:54:05.448 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.448 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.448 DEBUG:boto:Params: {} 2024-04-12T03:54:05.448 DEBUG:boto:Token: None 2024-04-12T03:54:05.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.448 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk= 2024-04-12T03:54:05.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk=', 'Content-Length': '0'} 2024-04-12T03:54:05.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c96e3779a059eab-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.450 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.450 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.451 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.451 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.451 DEBUG:boto:Method: GET 2024-04-12T03:54:05.451 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.451 DEBUG:boto:Data: 2024-04-12T03:54:05.451 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.451 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.451 DEBUG:boto:Params: {} 2024-04-12T03:54:05.451 DEBUG:boto:Token: None 2024-04-12T03:54:05.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk= 2024-04-12T03:54:05.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk=', 'Content-Length': '0'} 2024-04-12T03:54:05.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003873f8bda9cfde20-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.494 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.494 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.494 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.495 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.495 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.495 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.495 DEBUG:boto:Data: 2024-04-12T03:54:05.495 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.495 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.495 DEBUG:boto:Params: {} 2024-04-12T03:54:05.495 DEBUG:boto:Token: None 2024-04-12T03:54:05.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q0ES/g/Pg/d21JpxMXv6lGIU2Ko= 2024-04-12T03:54:05.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q0ES/g/Pg/d21JpxMXv6lGIU2Ko=', 'Content-Length': '0'} 2024-04-12T03:54:05.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf29f0b7b8508e76-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.498 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.498 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.498 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.498 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.498 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.498 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.498 DEBUG:boto:Data: 2024-04-12T03:54:05.498 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.498 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.499 DEBUG:boto:Params: {} 2024-04-12T03:54:05.499 DEBUG:boto:Token: None 2024-04-12T03:54:05.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.499 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q0ES/g/Pg/d21JpxMXv6lGIU2Ko= 2024-04-12T03:54:05.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q0ES/g/Pg/d21JpxMXv6lGIU2Ko=', 'Content-Length': '0'} 2024-04-12T03:54:05.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed43cc004700bd81-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:54:05.501 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.501 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.501 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.501 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.502 DEBUG:boto:Method: GET 2024-04-12T03:54:05.502 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.502 DEBUG:boto:Data: 2024-04-12T03:54:05.502 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.502 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.502 DEBUG:boto:Params: {} 2024-04-12T03:54:05.502 DEBUG:boto:Token: None 2024-04-12T03:54:05.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.502 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk= 2024-04-12T03:54:05.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk=', 'Content-Length': '0'} 2024-04-12T03:54:05.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e4d6c90e12af0c-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.504 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.505 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.505 DEBUG:boto:path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.505 DEBUG:boto:auth_path=/dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.505 DEBUG:boto:Method: GET 2024-04-12T03:54:05.505 DEBUG:boto:Path: /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.505 DEBUG:boto:Data: 2024-04-12T03:54:05.505 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.505 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.505 DEBUG:boto:Params: {} 2024-04-12T03:54:05.506 DEBUG:boto:Token: None 2024-04-12T03:54:05.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76?versionId=null 2024-04-12T03:54:05.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk= 2024-04-12T03:54:05.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QLhGZvplNzlI5nsPKt0hvpJMhYk=', 'Content-Length': '0'} 2024-04-12T03:54:05.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f802d511248618-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.508 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.508 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.508 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.508 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.508 DEBUG:boto:Method: GET 2024-04-12T03:54:05.508 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.508 DEBUG:boto:Data: 2024-04-12T03:54:05.508 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.509 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.509 DEBUG:boto:Params: {} 2024-04-12T03:54:05.509 DEBUG:boto:Token: None 2024-04-12T03:54:05.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.509 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002caeeedeeecdda97-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.511 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.512 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.512 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.512 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.512 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.512 DEBUG:boto:Method: GET 2024-04-12T03:54:05.512 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.512 DEBUG:boto:Data: 2024-04-12T03:54:05.512 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.512 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.513 DEBUG:boto:Params: {} 2024-04-12T03:54:05.513 DEBUG:boto:Token: None 2024-04-12T03:54:05.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.513 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046272a5c5200a125-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.517 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.517 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.517 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.517 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.517 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:54:05.517 DEBUG:boto:Data: 2024-04-12T03:54:05.517 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.518 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.518 DEBUG:boto:Params: {} 2024-04-12T03:54:05.518 DEBUG:boto:Token: None 2024-04-12T03:54:05.518 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76 2024-04-12T03:54:05.518 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gd22MoFopLCBQBd3+Wmcb24dUFU= 2024-04-12T03:54:05.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gd22MoFopLCBQBd3+Wmcb24dUFU=', 'Content-Length': '0'} 2024-04-12T03:54:05.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002443a01f167cce74-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.520 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.520 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.520 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.520 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:54:05.520 DEBUG:boto:Data: 2024-04-12T03:54:05.520 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.521 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.521 DEBUG:boto:Params: {} 2024-04-12T03:54:05.521 DEBUG:boto:Token: None 2024-04-12T03:54:05.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76 2024-04-12T03:54:05.521 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gd22MoFopLCBQBd3+Wmcb24dUFU= 2024-04-12T03:54:05.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gd22MoFopLCBQBd3+Wmcb24dUFU=', 'Content-Length': '0'} 2024-04-12T03:54:05.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caf72ed2286c5f61-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T03:54:05.523 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.523 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.523 DEBUG:boto:Method: GET 2024-04-12T03:54:05.523 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:54:05.523 DEBUG:boto:Data: 2024-04-12T03:54:05.523 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.523 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.524 DEBUG:boto:Params: {} 2024-04-12T03:54:05.524 DEBUG:boto:Token: None 2024-04-12T03:54:05.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76 2024-04-12T03:54:05.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:csDwESfi/Kv1Sgx/B97OFuSe6hM= 2024-04-12T03:54:05.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:csDwESfi/Kv1Sgx/B97OFuSe6hM=', 'Content-Length': '0'} 2024-04-12T03:54:05.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ae9428f047ec45d-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.526 DEBUG:boto:path=/dgbpxg-61/obj76 2024-04-12T03:54:05.526 DEBUG:boto:auth_path=/dgbpxg-61/obj76 2024-04-12T03:54:05.526 DEBUG:boto:Method: GET 2024-04-12T03:54:05.526 DEBUG:boto:Path: /dgbpxg-61/obj76 2024-04-12T03:54:05.526 DEBUG:boto:Data: 2024-04-12T03:54:05.526 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.527 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.527 DEBUG:boto:Params: {} 2024-04-12T03:54:05.527 DEBUG:boto:Token: None 2024-04-12T03:54:05.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj76 2024-04-12T03:54:05.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:csDwESfi/Kv1Sgx/B97OFuSe6hM= 2024-04-12T03:54:05.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:csDwESfi/Kv1Sgx/B97OFuSe6hM=', 'Content-Length': '0'} 2024-04-12T03:54:05.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b98e85a5943b4b-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.570 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.570 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.571 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.571 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.571 DEBUG:boto:Method: GET 2024-04-12T03:54:05.571 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.571 DEBUG:boto:Data: 2024-04-12T03:54:05.571 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.571 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.571 DEBUG:boto:Params: {} 2024-04-12T03:54:05.571 DEBUG:boto:Token: None 2024-04-12T03:54:05.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072fa4cf051bb0d33-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.574 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.574 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.574 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.574 DEBUG:boto:path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.574 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.574 DEBUG:boto:Method: GET 2024-04-12T03:54:05.574 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj76 2024-04-12T03:54:05.575 DEBUG:boto:Data: 2024-04-12T03:54:05.575 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.575 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.575 DEBUG:boto:Params: {} 2024-04-12T03:54:05.575 DEBUG:boto:Token: None 2024-04-12T03:54:05.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf9374e515aee6d-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.579 DEBUG:boto:b'dgbpxg-61obj761000falseobj762024-04-12T03:43:54.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:54:05.579 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.579 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.580 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.580 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.580 DEBUG:boto:Method: GET 2024-04-12T03:54:05.580 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.580 DEBUG:boto:Data: 2024-04-12T03:54:05.580 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.580 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.580 DEBUG:boto:Params: {} 2024-04-12T03:54:05.580 DEBUG:boto:Token: None 2024-04-12T03:54:05.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.581 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA= 2024-04-12T03:54:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA=', 'Content-Length': '0'} 2024-04-12T03:54:05.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019c2a7a60e0f890-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.583 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.583 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.583 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.583 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.583 DEBUG:boto:Method: GET 2024-04-12T03:54:05.583 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.583 DEBUG:boto:Data: 2024-04-12T03:54:05.583 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.583 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.583 DEBUG:boto:Params: {} 2024-04-12T03:54:05.584 DEBUG:boto:Token: None 2024-04-12T03:54:05.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA= 2024-04-12T03:54:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA=', 'Content-Length': '0'} 2024-04-12T03:54:05.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f651d8821576ea5f-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.586 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.586 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.586 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.586 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.586 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.587 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.587 DEBUG:boto:Data: 2024-04-12T03:54:05.587 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.587 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.587 DEBUG:boto:Params: {} 2024-04-12T03:54:05.587 DEBUG:boto:Token: None 2024-04-12T03:54:05.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DUoVVrffNgpu0nQ1fygdNSJPnPA= 2024-04-12T03:54:05.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DUoVVrffNgpu0nQ1fygdNSJPnPA=', 'Content-Length': '0'} 2024-04-12T03:54:05.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000253f2ea219239ad-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.589 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.589 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.589 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.590 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.590 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.590 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.590 DEBUG:boto:Data: 2024-04-12T03:54:05.590 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.590 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.590 DEBUG:boto:Params: {} 2024-04-12T03:54:05.590 DEBUG:boto:Token: None 2024-04-12T03:54:05.590 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.590 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DUoVVrffNgpu0nQ1fygdNSJPnPA= 2024-04-12T03:54:05.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DUoVVrffNgpu0nQ1fygdNSJPnPA=', 'Content-Length': '0'} 2024-04-12T03:54:05.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0095dc62db53595-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:54:05.592 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.593 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.593 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.593 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.593 DEBUG:boto:Method: GET 2024-04-12T03:54:05.593 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.593 DEBUG:boto:Data: 2024-04-12T03:54:05.593 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.593 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.593 DEBUG:boto:Params: {} 2024-04-12T03:54:05.593 DEBUG:boto:Token: None 2024-04-12T03:54:05.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA= 2024-04-12T03:54:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA=', 'Content-Length': '0'} 2024-04-12T03:54:05.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d41f107b381cf6-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.638 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.638 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.638 DEBUG:boto:path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.639 DEBUG:boto:auth_path=/dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.639 DEBUG:boto:Method: GET 2024-04-12T03:54:05.639 DEBUG:boto:Path: /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.639 DEBUG:boto:Data: 2024-04-12T03:54:05.639 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.639 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.639 DEBUG:boto:Params: {} 2024-04-12T03:54:05.639 DEBUG:boto:Token: None 2024-04-12T03:54:05.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85?versionId=null 2024-04-12T03:54:05.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA= 2024-04-12T03:54:05.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DX4aWXdjnqxSYAF8G0ZuhSfNuA=', 'Content-Length': '0'} 2024-04-12T03:54:05.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5368cee8be633a1-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.642 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.642 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.642 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.642 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.642 DEBUG:boto:Method: GET 2024-04-12T03:54:05.642 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.642 DEBUG:boto:Data: 2024-04-12T03:54:05.642 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.642 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.642 DEBUG:boto:Params: {} 2024-04-12T03:54:05.643 DEBUG:boto:Token: None 2024-04-12T03:54:05.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9eb7c647ebcd291-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.645 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.645 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.646 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.646 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.646 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.646 DEBUG:boto:Method: GET 2024-04-12T03:54:05.646 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.646 DEBUG:boto:Data: 2024-04-12T03:54:05.646 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.646 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.646 DEBUG:boto:Params: {} 2024-04-12T03:54:05.646 DEBUG:boto:Token: None 2024-04-12T03:54:05.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8f123d29b0c815c-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.651 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.651 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.651 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.651 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.651 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:54:05.651 DEBUG:boto:Data: 2024-04-12T03:54:05.651 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.651 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.652 DEBUG:boto:Params: {} 2024-04-12T03:54:05.652 DEBUG:boto:Token: None 2024-04-12T03:54:05.652 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85 2024-04-12T03:54:05.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IJMmzOkCRb2Jtzql5LpmPcGniUk= 2024-04-12T03:54:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IJMmzOkCRb2Jtzql5LpmPcGniUk=', 'Content-Length': '0'} 2024-04-12T03:54:05.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da13d3c32f21971-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.654 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.654 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.654 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.654 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:54:05.654 DEBUG:boto:Data: 2024-04-12T03:54:05.655 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.655 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.655 DEBUG:boto:Params: {} 2024-04-12T03:54:05.655 DEBUG:boto:Token: None 2024-04-12T03:54:05.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85 2024-04-12T03:54:05.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IJMmzOkCRb2Jtzql5LpmPcGniUk= 2024-04-12T03:54:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IJMmzOkCRb2Jtzql5LpmPcGniUk=', 'Content-Length': '0'} 2024-04-12T03:54:05.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054da446538191379-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T03:54:05.657 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.657 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.657 DEBUG:boto:Method: GET 2024-04-12T03:54:05.657 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:54:05.657 DEBUG:boto:Data: 2024-04-12T03:54:05.657 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.658 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.658 DEBUG:boto:Params: {} 2024-04-12T03:54:05.658 DEBUG:boto:Token: None 2024-04-12T03:54:05.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85 2024-04-12T03:54:05.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:781ehbajBh/QsukZbV6y4vQfI94= 2024-04-12T03:54:05.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:781ehbajBh/QsukZbV6y4vQfI94=', 'Content-Length': '0'} 2024-04-12T03:54:05.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ccc8a32fdd97186-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.660 DEBUG:boto:path=/dgbpxg-61/obj85 2024-04-12T03:54:05.660 DEBUG:boto:auth_path=/dgbpxg-61/obj85 2024-04-12T03:54:05.660 DEBUG:boto:Method: GET 2024-04-12T03:54:05.661 DEBUG:boto:Path: /dgbpxg-61/obj85 2024-04-12T03:54:05.661 DEBUG:boto:Data: 2024-04-12T03:54:05.661 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.661 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.661 DEBUG:boto:Params: {} 2024-04-12T03:54:05.661 DEBUG:boto:Token: None 2024-04-12T03:54:05.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj85 2024-04-12T03:54:05.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:781ehbajBh/QsukZbV6y4vQfI94= 2024-04-12T03:54:05.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:781ehbajBh/QsukZbV6y4vQfI94=', 'Content-Length': '0'} 2024-04-12T03:54:05.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:41:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5731dcf5b8b9a62-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.707 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.707 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.707 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.707 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.707 DEBUG:boto:Method: GET 2024-04-12T03:54:05.707 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.707 DEBUG:boto:Data: 2024-04-12T03:54:05.707 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.708 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.708 DEBUG:boto:Params: {} 2024-04-12T03:54:05.708 DEBUG:boto:Token: None 2024-04-12T03:54:05.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.708 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b15ae95849b28031-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.710 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.711 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.711 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.711 DEBUG:boto:path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.711 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.711 DEBUG:boto:Method: GET 2024-04-12T03:54:05.711 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj85 2024-04-12T03:54:05.711 DEBUG:boto:Data: 2024-04-12T03:54:05.711 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.711 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.712 DEBUG:boto:Params: {} 2024-04-12T03:54:05.712 DEBUG:boto:Token: None 2024-04-12T03:54:05.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cf67e2e61d4a113-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.715 DEBUG:boto:b'dgbpxg-61obj851000falseobj852024-04-12T03:41:32.649Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:54:05.716 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.716 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.716 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.716 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.716 DEBUG:boto:Method: GET 2024-04-12T03:54:05.716 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.717 DEBUG:boto:Data: 2024-04-12T03:54:05.717 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.717 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.717 DEBUG:boto:Params: {} 2024-04-12T03:54:05.717 DEBUG:boto:Token: None 2024-04-12T03:54:05.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.717 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228= 2024-04-12T03:54:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228=', 'Content-Length': '0'} 2024-04-12T03:54:05.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7fbd17c6b010394-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.719 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.720 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.720 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.720 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.720 DEBUG:boto:Method: GET 2024-04-12T03:54:05.720 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.720 DEBUG:boto:Data: 2024-04-12T03:54:05.720 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.720 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.720 DEBUG:boto:Params: {} 2024-04-12T03:54:05.720 DEBUG:boto:Token: None 2024-04-12T03:54:05.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228= 2024-04-12T03:54:05.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228=', 'Content-Length': '0'} 2024-04-12T03:54:05.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015868823908ddc7d-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.722 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.723 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.723 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.723 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.723 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.723 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.723 DEBUG:boto:Data: 2024-04-12T03:54:05.723 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.723 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.723 DEBUG:boto:Params: {} 2024-04-12T03:54:05.724 DEBUG:boto:Token: None 2024-04-12T03:54:05.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+MiwZnLiPGi6weVPFtydiflsn4= 2024-04-12T03:54:05.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+MiwZnLiPGi6weVPFtydiflsn4=', 'Content-Length': '0'} 2024-04-12T03:54:05.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808efe4555751e5a-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.726 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.726 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.726 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.726 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.726 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.726 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.726 DEBUG:boto:Data: 2024-04-12T03:54:05.726 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.726 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.727 DEBUG:boto:Params: {} 2024-04-12T03:54:05.727 DEBUG:boto:Token: None 2024-04-12T03:54:05.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.727 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1+MiwZnLiPGi6weVPFtydiflsn4= 2024-04-12T03:54:05.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1+MiwZnLiPGi6weVPFtydiflsn4=', 'Content-Length': '0'} 2024-04-12T03:54:05.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d66fa5031f614b-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:54:05.729 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.729 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.729 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.729 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.729 DEBUG:boto:Method: GET 2024-04-12T03:54:05.729 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.729 DEBUG:boto:Data: 2024-04-12T03:54:05.730 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.730 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.730 DEBUG:boto:Params: {} 2024-04-12T03:54:05.730 DEBUG:boto:Token: None 2024-04-12T03:54:05.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228= 2024-04-12T03:54:05.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228=', 'Content-Length': '0'} 2024-04-12T03:54:05.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed355ccce007e97-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.774 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.774 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.775 DEBUG:boto:path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.775 DEBUG:boto:auth_path=/dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.775 DEBUG:boto:Method: GET 2024-04-12T03:54:05.775 DEBUG:boto:Path: /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.775 DEBUG:boto:Data: 2024-04-12T03:54:05.775 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.775 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.775 DEBUG:boto:Params: {} 2024-04-12T03:54:05.775 DEBUG:boto:Token: None 2024-04-12T03:54:05.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87?versionId=null 2024-04-12T03:54:05.776 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228= 2024-04-12T03:54:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3FA8dv/D8uyO7NufEEc0JPKV228=', 'Content-Length': '0'} 2024-04-12T03:54:05.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce7cbef585fe466-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.778 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.778 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.778 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.778 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.778 DEBUG:boto:Method: GET 2024-04-12T03:54:05.778 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.778 DEBUG:boto:Data: 2024-04-12T03:54:05.778 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.778 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.778 DEBUG:boto:Params: {} 2024-04-12T03:54:05.779 DEBUG:boto:Token: None 2024-04-12T03:54:05.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.779 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559a8a285360079c-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.781 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.781 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.781 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.782 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.782 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.782 DEBUG:boto:Method: GET 2024-04-12T03:54:05.782 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.782 DEBUG:boto:Data: 2024-04-12T03:54:05.782 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.782 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.782 DEBUG:boto:Params: {} 2024-04-12T03:54:05.782 DEBUG:boto:Token: None 2024-04-12T03:54:05.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d48e3a6a3925e17-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.786 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.787 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.787 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.787 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.787 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:54:05.787 DEBUG:boto:Data: 2024-04-12T03:54:05.787 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.787 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.787 DEBUG:boto:Params: {} 2024-04-12T03:54:05.788 DEBUG:boto:Token: None 2024-04-12T03:54:05.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87 2024-04-12T03:54:05.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SFqa9Ktnt1b4Xs9oyMxkIqx/HnI= 2024-04-12T03:54:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SFqa9Ktnt1b4Xs9oyMxkIqx/HnI=', 'Content-Length': '0'} 2024-04-12T03:54:05.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c3cd290a959bb2-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.790 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.790 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.790 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.790 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:54:05.790 DEBUG:boto:Data: 2024-04-12T03:54:05.790 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.790 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.790 DEBUG:boto:Params: {} 2024-04-12T03:54:05.791 DEBUG:boto:Token: None 2024-04-12T03:54:05.791 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87 2024-04-12T03:54:05.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SFqa9Ktnt1b4Xs9oyMxkIqx/HnI= 2024-04-12T03:54:05.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SFqa9Ktnt1b4Xs9oyMxkIqx/HnI=', 'Content-Length': '0'} 2024-04-12T03:54:05.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253bc5b2d9722695-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T03:54:05.793 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.793 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.793 DEBUG:boto:Method: GET 2024-04-12T03:54:05.793 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:54:05.793 DEBUG:boto:Data: 2024-04-12T03:54:05.793 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.793 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.793 DEBUG:boto:Params: {} 2024-04-12T03:54:05.793 DEBUG:boto:Token: None 2024-04-12T03:54:05.794 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87 2024-04-12T03:54:05.794 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XwbviyYYvO+KoFNuhO/LnDMTYW4= 2024-04-12T03:54:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XwbviyYYvO+KoFNuhO/LnDMTYW4=', 'Content-Length': '0'} 2024-04-12T03:54:05.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a2ff73c9d8a8f9-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.796 DEBUG:boto:path=/dgbpxg-61/obj87 2024-04-12T03:54:05.796 DEBUG:boto:auth_path=/dgbpxg-61/obj87 2024-04-12T03:54:05.796 DEBUG:boto:Method: GET 2024-04-12T03:54:05.796 DEBUG:boto:Path: /dgbpxg-61/obj87 2024-04-12T03:54:05.796 DEBUG:boto:Data: 2024-04-12T03:54:05.796 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.797 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.797 DEBUG:boto:Params: {} 2024-04-12T03:54:05.797 DEBUG:boto:Token: None 2024-04-12T03:54:05.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj87 2024-04-12T03:54:05.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XwbviyYYvO+KoFNuhO/LnDMTYW4= 2024-04-12T03:54:05.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XwbviyYYvO+KoFNuhO/LnDMTYW4=', 'Content-Length': '0'} 2024-04-12T03:54:05.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514a28d3f255a342-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.799 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.799 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.799 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.799 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.799 DEBUG:boto:Method: GET 2024-04-12T03:54:05.799 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.799 DEBUG:boto:Data: 2024-04-12T03:54:05.799 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.800 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.800 DEBUG:boto:Params: {} 2024-04-12T03:54:05.800 DEBUG:boto:Token: None 2024-04-12T03:54:05.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000096113fee34b8ffb-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.802 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.803 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.803 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.803 DEBUG:boto:path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.803 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.803 DEBUG:boto:Method: GET 2024-04-12T03:54:05.803 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj87 2024-04-12T03:54:05.803 DEBUG:boto:Data: 2024-04-12T03:54:05.803 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.804 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.804 DEBUG:boto:Params: {} 2024-04-12T03:54:05.804 DEBUG:boto:Token: None 2024-04-12T03:54:05.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.804 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c1fe3dd92d90ea-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.808 DEBUG:boto:b'dgbpxg-61obj871000falseobj872024-04-12T03:47:19.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:54:05.808 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.808 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.808 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.808 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.808 DEBUG:boto:Method: GET 2024-04-12T03:54:05.808 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.808 DEBUG:boto:Data: 2024-04-12T03:54:05.809 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.809 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.809 DEBUG:boto:Params: {} 2024-04-12T03:54:05.809 DEBUG:boto:Token: None 2024-04-12T03:54:05.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.809 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY= 2024-04-12T03:54:05.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY=', 'Content-Length': '0'} 2024-04-12T03:54:05.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001240ddf8f856e959-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.811 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.811 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.811 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.812 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.812 DEBUG:boto:Method: GET 2024-04-12T03:54:05.812 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.812 DEBUG:boto:Data: 2024-04-12T03:54:05.812 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.812 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.812 DEBUG:boto:Params: {} 2024-04-12T03:54:05.812 DEBUG:boto:Token: None 2024-04-12T03:54:05.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.813 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY= 2024-04-12T03:54:05.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY=', 'Content-Length': '0'} 2024-04-12T03:54:05.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f003fc056b2b310-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.858 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.859 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.859 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.859 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.859 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.859 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.859 DEBUG:boto:Data: 2024-04-12T03:54:05.859 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.859 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.860 DEBUG:boto:Params: {} 2024-04-12T03:54:05.860 DEBUG:boto:Token: None 2024-04-12T03:54:05.860 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.860 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:URWhfu464s9c6gx6A8A4F+3/SQ0= 2024-04-12T03:54:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:URWhfu464s9c6gx6A8A4F+3/SQ0=', 'Content-Length': '0'} 2024-04-12T03:54:05.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008159b70f4c9de78e-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.862 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.862 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.862 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.862 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.862 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.862 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.863 DEBUG:boto:Data: 2024-04-12T03:54:05.863 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.863 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.863 DEBUG:boto:Params: {} 2024-04-12T03:54:05.863 DEBUG:boto:Token: None 2024-04-12T03:54:05.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:URWhfu464s9c6gx6A8A4F+3/SQ0= 2024-04-12T03:54:05.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:URWhfu464s9c6gx6A8A4F+3/SQ0=', 'Content-Length': '0'} 2024-04-12T03:54:05.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1cfd1e83e7f6a49-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:54:05.865 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.866 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.866 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.866 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.866 DEBUG:boto:Method: GET 2024-04-12T03:54:05.866 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.866 DEBUG:boto:Data: 2024-04-12T03:54:05.866 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.866 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.866 DEBUG:boto:Params: {} 2024-04-12T03:54:05.866 DEBUG:boto:Token: None 2024-04-12T03:54:05.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY= 2024-04-12T03:54:05.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY=', 'Content-Length': '0'} 2024-04-12T03:54:05.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8fb826b8c78d2b-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.869 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.869 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.869 DEBUG:boto:path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.869 DEBUG:boto:auth_path=/dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.869 DEBUG:boto:Method: GET 2024-04-12T03:54:05.869 DEBUG:boto:Path: /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.869 DEBUG:boto:Data: 2024-04-12T03:54:05.869 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.869 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.870 DEBUG:boto:Params: {} 2024-04-12T03:54:05.870 DEBUG:boto:Token: None 2024-04-12T03:54:05.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92?versionId=null 2024-04-12T03:54:05.870 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY= 2024-04-12T03:54:05.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bAgI+b1TN2nGogr5eta+1LhbJjY=', 'Content-Length': '0'} 2024-04-12T03:54:05.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038eee3e033aa7e17-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.872 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.872 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.872 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.872 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.872 DEBUG:boto:Method: GET 2024-04-12T03:54:05.873 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.873 DEBUG:boto:Data: 2024-04-12T03:54:05.873 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.873 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.873 DEBUG:boto:Params: {} 2024-04-12T03:54:05.873 DEBUG:boto:Token: None 2024-04-12T03:54:05.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000885036fef356ba13-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.876 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.876 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.876 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.876 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.876 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.876 DEBUG:boto:Method: GET 2024-04-12T03:54:05.877 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.877 DEBUG:boto:Data: 2024-04-12T03:54:05.877 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.877 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.877 DEBUG:boto:Params: {} 2024-04-12T03:54:05.877 DEBUG:boto:Token: None 2024-04-12T03:54:05.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.877 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9448a6a00a6fb74-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.881 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.881 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.882 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.882 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.882 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:54:05.882 DEBUG:boto:Data: 2024-04-12T03:54:05.882 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.882 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.882 DEBUG:boto:Params: {} 2024-04-12T03:54:05.882 DEBUG:boto:Token: None 2024-04-12T03:54:05.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92 2024-04-12T03:54:05.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1CfdgDbIqrL5+V1xXtxqJBqrdlo= 2024-04-12T03:54:05.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1CfdgDbIqrL5+V1xXtxqJBqrdlo=', 'Content-Length': '0'} 2024-04-12T03:54:05.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa95cea6780fabcb-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.884 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.885 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.885 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.885 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:54:05.885 DEBUG:boto:Data: 2024-04-12T03:54:05.885 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.885 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.885 DEBUG:boto:Params: {} 2024-04-12T03:54:05.885 DEBUG:boto:Token: None 2024-04-12T03:54:05.885 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92 2024-04-12T03:54:05.886 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1CfdgDbIqrL5+V1xXtxqJBqrdlo= 2024-04-12T03:54:05.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1CfdgDbIqrL5+V1xXtxqJBqrdlo=', 'Content-Length': '0'} 2024-04-12T03:54:05.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808f0d22dd14c2e1-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T03:54:05.888 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.888 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.888 DEBUG:boto:Method: GET 2024-04-12T03:54:05.888 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:54:05.888 DEBUG:boto:Data: 2024-04-12T03:54:05.888 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.888 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.888 DEBUG:boto:Params: {} 2024-04-12T03:54:05.888 DEBUG:boto:Token: None 2024-04-12T03:54:05.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92 2024-04-12T03:54:05.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yNCN43BtwJmNUfaJ5vTW5lNEf5o= 2024-04-12T03:54:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yNCN43BtwJmNUfaJ5vTW5lNEf5o=', 'Content-Length': '0'} 2024-04-12T03:54:05.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1c948074436e6d-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.934 DEBUG:boto:path=/dgbpxg-61/obj92 2024-04-12T03:54:05.935 DEBUG:boto:auth_path=/dgbpxg-61/obj92 2024-04-12T03:54:05.935 DEBUG:boto:Method: GET 2024-04-12T03:54:05.935 DEBUG:boto:Path: /dgbpxg-61/obj92 2024-04-12T03:54:05.935 DEBUG:boto:Data: 2024-04-12T03:54:05.935 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.935 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.935 DEBUG:boto:Params: {} 2024-04-12T03:54:05.935 DEBUG:boto:Token: None 2024-04-12T03:54:05.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj92 2024-04-12T03:54:05.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yNCN43BtwJmNUfaJ5vTW5lNEf5o= 2024-04-12T03:54:05.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yNCN43BtwJmNUfaJ5vTW5lNEf5o=', 'Content-Length': '0'} 2024-04-12T03:54:05.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f388f4b17ad8069-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.938 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.938 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.938 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.938 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.938 DEBUG:boto:Method: GET 2024-04-12T03:54:05.938 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.938 DEBUG:boto:Data: 2024-04-12T03:54:05.938 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.939 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.939 DEBUG:boto:Params: {} 2024-04-12T03:54:05.939 DEBUG:boto:Token: None 2024-04-12T03:54:05.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c729eef087677fcc-006618b05d-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.941 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.942 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:05.942 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:05.942 DEBUG:boto:path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.942 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.942 DEBUG:boto:Method: GET 2024-04-12T03:54:05.942 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj92 2024-04-12T03:54:05.942 DEBUG:boto:Data: 2024-04-12T03:54:05.942 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.942 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.943 DEBUG:boto:Params: {} 2024-04-12T03:54:05.943 DEBUG:boto:Token: None 2024-04-12T03:54:05.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/ 2024-04-12T03:54:05.943 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8= 2024-04-12T03:54:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rNeL8CaZ1bl5iYBB1MIRejg4tK8=', 'Content-Length': '0'} 2024-04-12T03:54:05.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000963097be47b4be2a-006618b05d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.946 DEBUG:boto:b'dgbpxg-61obj921000falseobj922024-04-12T03:46:16.731Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:05.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:54:05.947 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:05.947 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:05.947 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.947 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.947 DEBUG:boto:Method: GET 2024-04-12T03:54:05.947 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.948 DEBUG:boto:Data: 2024-04-12T03:54:05.948 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.948 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.948 DEBUG:boto:Params: {} 2024-04-12T03:54:05.948 DEBUG:boto:Token: None 2024-04-12T03:54:05.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ubx7TPaQQx55WWQskUCo3ciIJwE= 2024-04-12T03:54:05.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ubx7TPaQQx55WWQskUCo3ciIJwE=', 'Content-Length': '0'} 2024-04-12T03:54:05.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021dd972f2e6b669e-006618b05d-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.950 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:05.950 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:05.950 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.951 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.951 DEBUG:boto:Method: GET 2024-04-12T03:54:05.951 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.951 DEBUG:boto:Data: 2024-04-12T03:54:05.951 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.951 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.951 DEBUG:boto:Params: {} 2024-04-12T03:54:05.951 DEBUG:boto:Token: None 2024-04-12T03:54:05.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ubx7TPaQQx55WWQskUCo3ciIJwE= 2024-04-12T03:54:05.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ubx7TPaQQx55WWQskUCo3ciIJwE=', 'Content-Length': '0'} 2024-04-12T03:54:05.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f7ccd151727d65-006618b05d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.994 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:05.995 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:05.995 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.995 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.995 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.995 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.995 DEBUG:boto:Data: 2024-04-12T03:54:05.995 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:05.995 DEBUG:boto:Port: 8001 2024-04-12T03:54:05.995 DEBUG:boto:Params: {} 2024-04-12T03:54:05.995 DEBUG:boto:Token: None 2024-04-12T03:54:05.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKNVPMsBoiCqrWPT2PBilFWss4I= 2024-04-12T03:54:05.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKNVPMsBoiCqrWPT2PBilFWss4I=', 'Content-Length': '0'} 2024-04-12T03:54:05.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e877c7d3fde4ffde-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:05.998 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:05.998 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:05.998 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.998 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.998 DEBUG:boto:Method: HEAD 2024-04-12T03:54:05.998 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:05.999 DEBUG:boto:Data: 2024-04-12T03:54:05.999 DEBUG:boto:Headers: {} 2024-04-12T03:54:05.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:05.999 DEBUG:boto:Port: 8000 2024-04-12T03:54:05.999 DEBUG:boto:Params: {} 2024-04-12T03:54:05.999 DEBUG:boto:Token: None 2024-04-12T03:54:05.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:05 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QKNVPMsBoiCqrWPT2PBilFWss4I= 2024-04-12T03:54:06.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QKNVPMsBoiCqrWPT2PBilFWss4I=', 'Content-Length': '0'} 2024-04-12T03:54:06.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002404fbb0f63cfc68-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:54:06.002 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.002 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.002 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.002 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.002 DEBUG:boto:Method: GET 2024-04-12T03:54:06.002 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.002 DEBUG:boto:Data: 2024-04-12T03:54:06.002 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.002 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.003 DEBUG:boto:Params: {} 2024-04-12T03:54:06.003 DEBUG:boto:Token: None 2024-04-12T03:54:06.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NhYlJyyMW0gA0HbY+vCFAH93KIU= 2024-04-12T03:54:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NhYlJyyMW0gA0HbY+vCFAH93KIU=', 'Content-Length': '0'} 2024-04-12T03:54:06.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191dfdb09b7d01ee-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.005 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.005 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.005 DEBUG:boto:path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.005 DEBUG:boto:auth_path=/dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.005 DEBUG:boto:Method: GET 2024-04-12T03:54:06.005 DEBUG:boto:Path: /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.005 DEBUG:boto:Data: 2024-04-12T03:54:06.005 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.006 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.006 DEBUG:boto:Params: {} 2024-04-12T03:54:06.006 DEBUG:boto:Token: None 2024-04-12T03:54:06.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93?versionId=null 2024-04-12T03:54:06.006 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NhYlJyyMW0gA0HbY+vCFAH93KIU= 2024-04-12T03:54:06.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NhYlJyyMW0gA0HbY+vCFAH93KIU=', 'Content-Length': '0'} 2024-04-12T03:54:06.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007736929ee2cf817c-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.008 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.008 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.008 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.008 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.008 DEBUG:boto:Method: GET 2024-04-12T03:54:06.009 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.009 DEBUG:boto:Data: 2024-04-12T03:54:06.009 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.009 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.009 DEBUG:boto:Params: {} 2024-04-12T03:54:06.009 DEBUG:boto:Token: None 2024-04-12T03:54:06.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504cb399f615117a-006618b05e-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.011 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.012 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.012 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.012 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.012 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.012 DEBUG:boto:Method: GET 2024-04-12T03:54:06.012 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.012 DEBUG:boto:Data: 2024-04-12T03:54:06.012 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.013 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.013 DEBUG:boto:Params: {} 2024-04-12T03:54:06.013 DEBUG:boto:Token: None 2024-04-12T03:54:06.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f975dc1f6dab92-006618b05e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.017 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.017 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.017 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.017 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.017 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:54:06.017 DEBUG:boto:Data: 2024-04-12T03:54:06.017 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.018 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.018 DEBUG:boto:Params: {} 2024-04-12T03:54:06.018 DEBUG:boto:Token: None 2024-04-12T03:54:06.018 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93 2024-04-12T03:54:06.018 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SnjO5e8FJQisSl4d7qdqknQETeM= 2024-04-12T03:54:06.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SnjO5e8FJQisSl4d7qdqknQETeM=', 'Content-Length': '0'} 2024-04-12T03:54:06.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3a6f5d59e16928-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.020 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.020 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.020 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.020 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:54:06.020 DEBUG:boto:Data: 2024-04-12T03:54:06.020 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.020 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.021 DEBUG:boto:Params: {} 2024-04-12T03:54:06.021 DEBUG:boto:Token: None 2024-04-12T03:54:06.021 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93 2024-04-12T03:54:06.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SnjO5e8FJQisSl4d7qdqknQETeM= 2024-04-12T03:54:06.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SnjO5e8FJQisSl4d7qdqknQETeM=', 'Content-Length': '0'} 2024-04-12T03:54:06.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66f68359e8b3e77-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T03:54:06.023 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.023 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.023 DEBUG:boto:Method: GET 2024-04-12T03:54:06.023 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:54:06.023 DEBUG:boto:Data: 2024-04-12T03:54:06.023 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.024 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.024 DEBUG:boto:Params: {} 2024-04-12T03:54:06.024 DEBUG:boto:Token: None 2024-04-12T03:54:06.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93 2024-04-12T03:54:06.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hO7YDxgo6AKMXpifUJ2faZFklwI= 2024-04-12T03:54:06.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hO7YDxgo6AKMXpifUJ2faZFklwI=', 'Content-Length': '0'} 2024-04-12T03:54:06.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd42b3a05735dd37-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.026 DEBUG:boto:path=/dgbpxg-61/obj93 2024-04-12T03:54:06.026 DEBUG:boto:auth_path=/dgbpxg-61/obj93 2024-04-12T03:54:06.027 DEBUG:boto:Method: GET 2024-04-12T03:54:06.027 DEBUG:boto:Path: /dgbpxg-61/obj93 2024-04-12T03:54:06.027 DEBUG:boto:Data: 2024-04-12T03:54:06.027 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.027 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.027 DEBUG:boto:Params: {} 2024-04-12T03:54:06.027 DEBUG:boto:Token: None 2024-04-12T03:54:06.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj93 2024-04-12T03:54:06.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hO7YDxgo6AKMXpifUJ2faZFklwI= 2024-04-12T03:54:06.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hO7YDxgo6AKMXpifUJ2faZFklwI=', 'Content-Length': '0'} 2024-04-12T03:54:06.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008020c36fcd7cdbb2-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.070 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.071 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.071 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.071 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.071 DEBUG:boto:Method: GET 2024-04-12T03:54:06.071 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.071 DEBUG:boto:Data: 2024-04-12T03:54:06.071 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.071 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.072 DEBUG:boto:Params: {} 2024-04-12T03:54:06.072 DEBUG:boto:Token: None 2024-04-12T03:54:06.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce8a07125593715-006618b05e-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.074 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.075 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.075 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.075 DEBUG:boto:path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.075 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.075 DEBUG:boto:Method: GET 2024-04-12T03:54:06.075 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj93 2024-04-12T03:54:06.075 DEBUG:boto:Data: 2024-04-12T03:54:06.075 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.076 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.076 DEBUG:boto:Params: {} 2024-04-12T03:54:06.076 DEBUG:boto:Token: None 2024-04-12T03:54:06.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e845b74ad0fc02f-006618b05e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.079 DEBUG:boto:b'dgbpxg-61obj931000falseobj932024-04-12T03:50:18.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:54:06.080 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.080 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.080 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.080 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.080 DEBUG:boto:Method: GET 2024-04-12T03:54:06.080 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.081 DEBUG:boto:Data: 2024-04-12T03:54:06.081 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.081 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.081 DEBUG:boto:Params: {} 2024-04-12T03:54:06.081 DEBUG:boto:Token: None 2024-04-12T03:54:06.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.081 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k= 2024-04-12T03:54:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k=', 'Content-Length': '0'} 2024-04-12T03:54:06.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfaffd5bb788c687-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.126 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.127 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.127 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.127 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.127 DEBUG:boto:Method: GET 2024-04-12T03:54:06.127 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.127 DEBUG:boto:Data: 2024-04-12T03:54:06.127 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.127 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.127 DEBUG:boto:Params: {} 2024-04-12T03:54:06.128 DEBUG:boto:Token: None 2024-04-12T03:54:06.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k= 2024-04-12T03:54:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k=', 'Content-Length': '0'} 2024-04-12T03:54:06.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009137bac17eb0ead-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.130 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.130 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.130 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.130 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.130 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.130 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.130 DEBUG:boto:Data: 2024-04-12T03:54:06.131 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.131 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.131 DEBUG:boto:Params: {} 2024-04-12T03:54:06.131 DEBUG:boto:Token: None 2024-04-12T03:54:06.131 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EBGgu4HVcE5QUNnKb8hc/K/NUGU= 2024-04-12T03:54:06.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EBGgu4HVcE5QUNnKb8hc/K/NUGU=', 'Content-Length': '0'} 2024-04-12T03:54:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014184bd43dd3ff30-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.133 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.133 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.133 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.133 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.133 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.133 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.133 DEBUG:boto:Data: 2024-04-12T03:54:06.134 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.134 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.134 DEBUG:boto:Params: {} 2024-04-12T03:54:06.134 DEBUG:boto:Token: None 2024-04-12T03:54:06.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EBGgu4HVcE5QUNnKb8hc/K/NUGU= 2024-04-12T03:54:06.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EBGgu4HVcE5QUNnKb8hc/K/NUGU=', 'Content-Length': '0'} 2024-04-12T03:54:06.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017374a3800f675a6-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:54:06.136 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.137 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.137 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.137 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.137 DEBUG:boto:Method: GET 2024-04-12T03:54:06.137 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.137 DEBUG:boto:Data: 2024-04-12T03:54:06.137 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.137 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.137 DEBUG:boto:Params: {} 2024-04-12T03:54:06.138 DEBUG:boto:Token: None 2024-04-12T03:54:06.138 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k= 2024-04-12T03:54:06.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k=', 'Content-Length': '0'} 2024-04-12T03:54:06.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300f45b5b516bb8e-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.140 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.140 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.140 DEBUG:boto:path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.140 DEBUG:boto:auth_path=/dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.140 DEBUG:boto:Method: GET 2024-04-12T03:54:06.140 DEBUG:boto:Path: /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.140 DEBUG:boto:Data: 2024-04-12T03:54:06.140 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.140 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.140 DEBUG:boto:Params: {} 2024-04-12T03:54:06.141 DEBUG:boto:Token: None 2024-04-12T03:54:06.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95?versionId=null 2024-04-12T03:54:06.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k= 2024-04-12T03:54:06.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fp8FQ1jO3iNA1TZO3Z1ZEDOSI/k=', 'Content-Length': '0'} 2024-04-12T03:54:06.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006236d4c072b96c7b-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.143 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.143 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.143 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.143 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.143 DEBUG:boto:Method: GET 2024-04-12T03:54:06.143 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.143 DEBUG:boto:Data: 2024-04-12T03:54:06.143 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.143 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.143 DEBUG:boto:Params: {} 2024-04-12T03:54:06.144 DEBUG:boto:Token: None 2024-04-12T03:54:06.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed746fe138ec3fbd-006618b05e-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.146 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.146 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.146 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.147 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.147 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.147 DEBUG:boto:Method: GET 2024-04-12T03:54:06.147 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.147 DEBUG:boto:Data: 2024-04-12T03:54:06.147 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.147 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.147 DEBUG:boto:Params: {} 2024-04-12T03:54:06.147 DEBUG:boto:Token: None 2024-04-12T03:54:06.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b2a9cd7a8bf928-006618b05e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.151 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.152 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.152 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.152 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.152 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:54:06.152 DEBUG:boto:Data: 2024-04-12T03:54:06.152 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.152 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.152 DEBUG:boto:Params: {} 2024-04-12T03:54:06.152 DEBUG:boto:Token: None 2024-04-12T03:54:06.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95 2024-04-12T03:54:06.153 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d4XIKjIxtsR3U5/bD6d//OXfXoM= 2024-04-12T03:54:06.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d4XIKjIxtsR3U5/bD6d//OXfXoM=', 'Content-Length': '0'} 2024-04-12T03:54:06.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081bb10863989a958-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.154 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.155 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.155 DEBUG:boto:Method: HEAD 2024-04-12T03:54:06.155 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:54:06.155 DEBUG:boto:Data: 2024-04-12T03:54:06.155 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.155 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.155 DEBUG:boto:Params: {} 2024-04-12T03:54:06.155 DEBUG:boto:Token: None 2024-04-12T03:54:06.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95 2024-04-12T03:54:06.156 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:d4XIKjIxtsR3U5/bD6d//OXfXoM= 2024-04-12T03:54:06.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:d4XIKjIxtsR3U5/bD6d//OXfXoM=', 'Content-Length': '0'} 2024-04-12T03:54:06.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8ec6ec9b143b42-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T03:54:06.157 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.158 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.158 DEBUG:boto:Method: GET 2024-04-12T03:54:06.158 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:54:06.158 DEBUG:boto:Data: 2024-04-12T03:54:06.158 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.158 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.158 DEBUG:boto:Params: {} 2024-04-12T03:54:06.158 DEBUG:boto:Token: None 2024-04-12T03:54:06.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95 2024-04-12T03:54:06.159 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6wSIM2PsJjlcxXcaiZd8YMDt3g= 2024-04-12T03:54:06.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6wSIM2PsJjlcxXcaiZd8YMDt3g=', 'Content-Length': '0'} 2024-04-12T03:54:06.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-61:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046176c0f3f1237d3-006618b05e-20c9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.161 DEBUG:boto:path=/dgbpxg-61/obj95 2024-04-12T03:54:06.161 DEBUG:boto:auth_path=/dgbpxg-61/obj95 2024-04-12T03:54:06.161 DEBUG:boto:Method: GET 2024-04-12T03:54:06.161 DEBUG:boto:Path: /dgbpxg-61/obj95 2024-04-12T03:54:06.161 DEBUG:boto:Data: 2024-04-12T03:54:06.161 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.161 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.161 DEBUG:boto:Params: {} 2024-04-12T03:54:06.161 DEBUG:boto:Token: None 2024-04-12T03:54:06.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/obj95 2024-04-12T03:54:06.162 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C6wSIM2PsJjlcxXcaiZd8YMDt3g= 2024-04-12T03:54:06.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C6wSIM2PsJjlcxXcaiZd8YMDt3g=', 'Content-Length': '0'} 2024-04-12T03:54:06.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:43:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb07b6c782c9504-006618b05e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.206 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.207 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.207 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.207 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.207 DEBUG:boto:Method: GET 2024-04-12T03:54:06.207 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.207 DEBUG:boto:Data: 2024-04-12T03:54:06.207 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:54:06.207 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.207 DEBUG:boto:Params: {} 2024-04-12T03:54:06.207 DEBUG:boto:Token: None 2024-04-12T03:54:06.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea73bd887f756594-006618b05e-20c9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.210 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.211 DEBUG:boto:path=/dgbpxg-61/ 2024-04-12T03:54:06.211 DEBUG:boto:auth_path=/dgbpxg-61/ 2024-04-12T03:54:06.211 DEBUG:boto:path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.211 DEBUG:boto:auth_path=/dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.211 DEBUG:boto:Method: GET 2024-04-12T03:54:06.211 DEBUG:boto:Path: /dgbpxg-61/?prefix=obj95 2024-04-12T03:54:06.211 DEBUG:boto:Data: 2024-04-12T03:54:06.211 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:54:06.211 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.212 DEBUG:boto:Params: {} 2024-04-12T03:54:06.212 DEBUG:boto:Token: None 2024-04-12T03:54:06.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:54:06 GMT /dgbpxg-61/ 2024-04-12T03:54:06.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs= 2024-04-12T03:54:06.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:54:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pr6gn8WCag3MxGT4G/vzMaCMzLs=', 'Content-Length': '0'} 2024-04-12T03:54:06.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1dbdea7cc80c621-006618b05e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.216 DEBUG:boto:b'dgbpxg-61obj951000falseobj952024-04-12T03:43:54.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:54:06.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-61 zones={a2, a1} 2024-04-12T03:54:06.217 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2024-04-12T03:54:06.221 INFO:rgw_multi.tests:create role zone=a1 name=roles-dgbpxg-1 2024-04-12T03:54:06.221 DEBUG:boto:Method: POST 2024-04-12T03:54:06.221 DEBUG:boto:Path: / 2024-04-12T03:54:06.222 DEBUG:boto:Data: 2024-04-12T03:54:06.222 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:54:06.222 DEBUG:boto:Port: 8000 2024-04-12T03:54:06.222 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-12T03:54:06.222 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:54:06.222 DEBUG:boto:Token: None 2024-04-12T03:54:06.223 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20240412T035406Z host;x-amz-date 1132d4aa5676eeac5069f0516209befa984406533d1744e8d6c7d8505711d4a2 2024-04-12T03:54:06.223 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035406Z 20240412/front/smithi099/aws4_request 31bca2632e52adabbc79fb54158be2f034794a1c3f6ad7c9dcbbc609f9d3a867 2024-04-12T03:54:06.223 DEBUG:boto:Signature: 4ad1888b2aae65e01887f48111f4d5eeffca90bf1fbaa52da1661a715e267b77 2024-04-12T03:54:06.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035406Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=4ad1888b2aae65e01887f48111f4d5eeffca90bf1fbaa52da1661a715e267b77', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2024-04-12T03:54:06.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028d67fb8915d0204-006618b05e-1921-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.242 DEBUG:boto:b'1243114b-de3a-4e10-9200-03198c61ec85roles-dgbpxg-1/arn:aws:iam:::role/roles-dgbpxg-12024-04-12T03:54:06.246Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000028d67fb8915d0204-006618b05e-1921-a1' 2024-04-12T03:54:06.243 INFO:rgw_multi.tests:create role zone=a2 name=roles-dgbpxg-2 2024-04-12T03:54:06.243 DEBUG:boto:Method: POST 2024-04-12T03:54:06.243 DEBUG:boto:Path: / 2024-04-12T03:54:06.243 DEBUG:boto:Data: 2024-04-12T03:54:06.243 DEBUG:boto:Headers: {} 2024-04-12T03:54:06.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:54:06.243 DEBUG:boto:Port: 8001 2024-04-12T03:54:06.244 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-12T03:54:06.244 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:54:06.244 DEBUG:boto:Token: None 2024-04-12T03:54:06.244 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20240412T035406Z host;x-amz-date d43f8fbb9c55cdf62965b27ed313e786b20cf1dd8a322c7b0a296c79b63dbe00 2024-04-12T03:54:06.244 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035406Z 20240412/front/smithi099/aws4_request 7b442ac6fefb343ed56442bfc7892bfc8f9c929ddcaeededfc88845c13af6bd0 2024-04-12T03:54:06.244 DEBUG:boto:Signature: 60fa6fb97837d2fe51d321ff895a7d73ad5bcbd1156d289c3749072f215bd9aa 2024-04-12T03:54:06.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035406Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=60fa6fb97837d2fe51d321ff895a7d73ad5bcbd1156d289c3749072f215bd9aa', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2024-04-12T03:54:06.268 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f24b14cbf55c31a2-006618b05e-20c9-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 12 Apr 2024 03:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:54:06.268 DEBUG:boto:b'750300b2-ddff-4406-8081-a357153d71b5roles-dgbpxg-2/arn:aws:iam:::role/roles-dgbpxg-22024-04-12T03:54:06.274Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000f24b14cbf55c31a2-006618b05e-20c9-a2' 2024-04-12T03:54:06.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:06.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:06.269 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894046.270097_1417.1", 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:54:06.270097Z" 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:54:06.362 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:06.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:54:06.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:06.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:06.363 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:06.451+0000 7f3558d01780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:06.452+0000 7f3558d01780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:06.455 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:06.455 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:06Z" 2024-04-12T03:54:06.461 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:06.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:06Z" } 2024-04-12T03:54:06.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:06.462 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:06.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:06.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:11.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:11.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:11.464 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:11.562+0000 7fbbc0a4b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:11.563+0000 7fbbc0a4b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:11.567 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:11.567 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:11.567 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:11.568 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:11.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:11.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:11.572 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:11Z" 2024-04-12T03:54:11.573 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:11.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:11Z" } 2024-04-12T03:54:11.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:11.575 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:11.575 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:11.575 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:16.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:16.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:16.577 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:16.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:16.679+0000 7fc9967f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:16.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:16.680+0000 7fc9967f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:16.684 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:16.684 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:16.684 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:16Z" 2024-04-12T03:54:16.690 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:16.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:16Z" } 2024-04-12T03:54:16.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:16.691 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:16.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:16.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:21.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:21.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:21.692 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:21.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:21.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:21.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:21.792+0000 7f9d05e1f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:21.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:21.793+0000 7f9d05e1f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:21.797 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:21.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:21.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:21.800 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:21.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:21.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:21Z" 2024-04-12T03:54:21.803 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:21.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:21Z" } 2024-04-12T03:54:21.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:21.804 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:21.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:21.805 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:26.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:26.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:26.805 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:26.862 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:26.862 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:26.905+0000 7f3ec50c7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:26.906+0000 7f3ec50c7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:26.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:26.915 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:26.916 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:26.917 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:26.918 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:26.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:26.919 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:26Z" 2024-04-12T03:54:26.919 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:26.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:26Z" } 2024-04-12T03:54:26.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:26.920 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:26.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:26.920 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:31.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:31.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:31.922 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:31.979 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:31.979 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:32.021+0000 7f5b8db4a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:32.022+0000 7f5b8db4a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:32.026 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:32.026 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:32Z" 2024-04-12T03:54:32.032 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:32.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:32Z" } 2024-04-12T03:54:32.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:32.034 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:32.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:32.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:37.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:37.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:37.035 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:37.090 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:37.090 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:37.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:37.131+0000 7fc9780ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:37.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:37.132+0000 7fc9780ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:37.136 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.137 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:37.138 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:37.139 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:37.140 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:37.141 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:37.142 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:37.143 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:37.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:37.143 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:37Z" 2024-04-12T03:54:37.143 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:37.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:37Z" } 2024-04-12T03:54:37.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:37.144 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:37.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:37.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:42.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:42.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:42.146 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:42.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:42.238+0000 7fc98c9d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:42.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:42.239+0000 7fc98c9d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:42.244 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:42.245 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:42.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:42.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:42.248 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.249 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:42.250 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:42.250 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:42.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:42.250 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:42Z" 2024-04-12T03:54:42.250 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:42.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:42Z" } 2024-04-12T03:54:42.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:42.251 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:42.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:42.251 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:47.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:47.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:47.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:47.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:47.351+0000 7f83c4c93780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:47.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:47.352+0000 7f83c4c93780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:47.356 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:47Z" 2024-04-12T03:54:47.362 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:47.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:47Z" } 2024-04-12T03:54:47.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:47.363 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:47.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:47.364 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:52.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:52.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:52.365 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:52.463+0000 7fe37a627780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:52.464+0000 7fe37a627780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:52.468 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:52.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:52.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:52.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:52.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:52Z" 2024-04-12T03:54:52.474 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:52.475 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:52Z" } 2024-04-12T03:54:52.475 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:52.476 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:52.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:52.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:54:57.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:57.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:54:57.478 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:54:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:57.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:57.576+0000 7f82ea204780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:54:57.577+0000 7f82ea204780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:54:57.581 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:54:57.581 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:54:57.581 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:54:57.581 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:57.582 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:57.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:54:57.585 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:54:57.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:54:57Z" 2024-04-12T03:54:57.587 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:54:57.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:54:57Z" } 2024-04-12T03:54:57.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:54:57.588 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:54:57.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:54:57.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:02.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:02.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:02.591 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:02.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:02.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:02.688+0000 7f966f07a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:02.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:02.689+0000 7f966f07a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:02.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:02.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:02.702 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:02.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:02.702 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:02Z" 2024-04-12T03:55:02.702 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:02.703 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:02Z" } 2024-04-12T03:55:02.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:02.703 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:02.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:02.703 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:07.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:07.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:07.705 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:07.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:07.804+0000 7f87d80c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:07.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:07.805+0000 7f87d80c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:07.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:07.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:07.811 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:07.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:07.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:07.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:07.815 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:07Z" 2024-04-12T03:55:07.815 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:07.816 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:07Z" } 2024-04-12T03:55:07.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:07.816 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:07.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:07.817 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:12.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:12.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:12.818 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:12.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:12.911+0000 7f58a6a4c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:12.913+0000 7f58a6a4c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:12.915 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:12.915 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:12.916 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:12.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:12.918 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:12.919 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:12.920 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:12.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:12Z" 2024-04-12T03:55:12.922 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:12.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:12Z" } 2024-04-12T03:55:12.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:12.924 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:12.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:12.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:17.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:17.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:17.926 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:17.982 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:17.982 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:18.023+0000 7f3494f0f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:18.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:18.024+0000 7f3494f0f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:18.030 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:18.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:18.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:18.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:18.035 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:18.036 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:18Z" 2024-04-12T03:55:18.037 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:18.038 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:18Z" } 2024-04-12T03:55:18.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:18.038 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:18.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:18.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:23.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:23.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:23.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:23.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:23.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:23.139+0000 7fccc68e6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:23.140+0000 7fccc68e6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:23.144 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:23.144 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:23Z" 2024-04-12T03:55:23.151 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:23.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:23Z" } 2024-04-12T03:55:23.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:23.152 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:23.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:23.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:28.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:28.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:28.154 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:28.210 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:28.210 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:28.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:28.252+0000 7f507b7e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:28.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:28.253+0000 7f507b7e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:28.257 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:28Z" 2024-04-12T03:55:28.263 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:28.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:28Z" } 2024-04-12T03:55:28.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:28.265 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:28.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:28.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:33.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:33.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:33.266 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:33.321 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:33.321 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:33.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:33.365+0000 7faaf378d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:33.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:33.366+0000 7faaf378d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:33.369 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:33Z" 2024-04-12T03:55:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:33.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:33Z" } 2024-04-12T03:55:33.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:33.377 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:33.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:33.377 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:38.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:38.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:38.379 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:38.434 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:38.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:38.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:38.477+0000 7f3df97f6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:38.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:38.478+0000 7f3df97f6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712893902.652031_1378.1", 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.652031Z", 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:38Z" 2024-04-12T03:55:38.488 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:38.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893902.652031_1378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.652031Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:38Z" } 2024-04-12T03:55:38.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893902.652031_1378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.652031Z', 'realm_epoch': 2}}] 2024-04-12T03:55:38.490 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:38.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712893902.652031_1378.1'} 2024-04-12T03:55:38.490 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712893902.652031_1378.1 2024-04-12T03:55:43.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:43.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:43.492 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:43.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:43.548 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:43.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:43.591+0000 7fe2bba6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:43.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:43.592+0000 7fe2bba6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:43.596 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:43.596 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:43.596 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:43.596 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:43.597 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:43.598 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:43.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894046.246070_1414.1", 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:54:06.246070Z", 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:43.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:43Z" 2024-04-12T03:55:43.602 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:43.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894046.246070_1414.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:54:06.246070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:43Z" } 2024-04-12T03:55:43.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894046.246070_1414.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:54:06.246070Z', 'realm_epoch': 2}}] 2024-04-12T03:55:43.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:43.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.246070_1414.1'} 2024-04-12T03:55:43.604 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894046.270097_1417.1 target=1_1712894046.246070_1414.1 2024-04-12T03:55:48.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.605 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:48.662 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:48.662 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:48.707+0000 7fc25b977780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:48.708+0000 7fc25b977780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.712 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.713 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.714 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894046.270097_1417.1", 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:54:06.270097Z", 2024-04-12T03:55:48.716 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:48Z" 2024-04-12T03:55:48.717 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:48.718 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894046.270097_1417.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:54:06.270097Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:48Z" } 2024-04-12T03:55:48.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894046.270097_1417.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:54:06.270097Z', 'realm_epoch': 2}}] 2024-04-12T03:55:48.719 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:48.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894046.270097_1417.1'} 2024-04-12T03:55:48.719 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:55:48.719 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dgbpxg-1 zones={a2, a1} 2024-04-12T03:55:48.719 DEBUG:boto:Method: POST 2024-04-12T03:55:48.720 DEBUG:boto:Path: / 2024-04-12T03:55:48.720 DEBUG:boto:Data: 2024-04-12T03:55:48.720 DEBUG:boto:Headers: {} 2024-04-12T03:55:48.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:55:48.720 DEBUG:boto:Port: 8001 2024-04-12T03:55:48.720 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T03:55:48.721 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:55:48.721 DEBUG:boto:Token: None 2024-04-12T03:55:48.721 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20240412T035548Z host;x-amz-date 98e7c3fabda0df322a9b34ff13baebef00b9a7ff460b18703baa08b1b849a699 2024-04-12T03:55:48.722 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035548Z 20240412/front/smithi099/aws4_request 3d859d1c78fd3b8673d75c7e10a4722c7f8fe85dbae125b486c6d90ab035a3af 2024-04-12T03:55:48.722 DEBUG:boto:Signature: acdac77f9542dc8a10ff0deec0584962d1133973cb1a67306f3952a7b4ef7193 2024-04-12T03:55:48.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035548Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=acdac77f9542dc8a10ff0deec0584962d1133973cb1a67306f3952a7b4ef7193', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2024-04-12T03:55:48.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff3b44d11b5cf0ff-006618b0c4-20c9-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 03:55:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:48.735 DEBUG:boto:b'tx00000ff3b44d11b5cf0ff-006618b0c4-20c9-a21243114b-de3a-4e10-9200-03198c61ec85roles-dgbpxg-1/arn:aws:iam:::role/roles-dgbpxg-12024-04-12T03:54:06.246Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T03:55:48.736 DEBUG:boto:Method: POST 2024-04-12T03:55:48.736 DEBUG:boto:Path: / 2024-04-12T03:55:48.736 DEBUG:boto:Data: 2024-04-12T03:55:48.736 DEBUG:boto:Headers: {} 2024-04-12T03:55:48.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:55:48.736 DEBUG:boto:Port: 8000 2024-04-12T03:55:48.736 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T03:55:48.736 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:55:48.736 DEBUG:boto:Token: None 2024-04-12T03:55:48.737 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20240412T035548Z host;x-amz-date 98e7c3fabda0df322a9b34ff13baebef00b9a7ff460b18703baa08b1b849a699 2024-04-12T03:55:48.737 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035548Z 20240412/front/smithi099/aws4_request 7a1560e68cd0d034c5d47cf7c0a951df46091e81b013a716d5da7d2ac62d3ade 2024-04-12T03:55:48.737 DEBUG:boto:Signature: ca5e6bb2bbe1a69acaf9b55ad37020d24028935c1fd5bf885ddcaedf560db5ab 2024-04-12T03:55:48.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035548Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=ca5e6bb2bbe1a69acaf9b55ad37020d24028935c1fd5bf885ddcaedf560db5ab', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2024-04-12T03:55:48.748 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085a229cc1c45cd95-006618b0c4-1921-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 03:55:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:48.748 DEBUG:boto:b'tx0000085a229cc1c45cd95-006618b0c4-1921-a11243114b-de3a-4e10-9200-03198c61ec85roles-dgbpxg-1/arn:aws:iam:::role/roles-dgbpxg-12024-04-12T03:54:06.246Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T03:55:48.749 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dgbpxg-1 2024-04-12T03:55:48.749 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dgbpxg-1 zones={a2, a1} 2024-04-12T03:55:48.749 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dgbpxg-2 zones={a1, a2} 2024-04-12T03:55:48.749 DEBUG:boto:Method: POST 2024-04-12T03:55:48.749 DEBUG:boto:Path: / 2024-04-12T03:55:48.749 DEBUG:boto:Data: 2024-04-12T03:55:48.749 DEBUG:boto:Headers: {} 2024-04-12T03:55:48.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:55:48.750 DEBUG:boto:Port: 8000 2024-04-12T03:55:48.750 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T03:55:48.750 DEBUG:boto:Token: None 2024-04-12T03:55:48.750 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20240412T035548Z host;x-amz-date 537ceae7591bdd4378bafcc91a960e4d882aa04265ddb40120c714c3c7263586 2024-04-12T03:55:48.750 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035548Z 20240412/front/smithi099/aws4_request a947c1563650a7da5e3c3c130b30f9572076175a3e641adb9a0e1315c858405f 2024-04-12T03:55:48.750 DEBUG:boto:Signature: 069b7e42029dbc88b9d0bd3b29592f4ffd735ceda1f92e652633136786a65e5d 2024-04-12T03:55:48.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035548Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=069b7e42029dbc88b9d0bd3b29592f4ffd735ceda1f92e652633136786a65e5d', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2024-04-12T03:55:48.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087b014db53946f06-006618b0c4-1921-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 03:55:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:48.751 DEBUG:boto:b'tx0000087b014db53946f06-006618b0c4-1921-a1750300b2-ddff-4406-8081-a357153d71b5roles-dgbpxg-2/arn:aws:iam:::role/roles-dgbpxg-22024-04-12T03:54:06.267Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T03:55:48.752 DEBUG:boto:Method: POST 2024-04-12T03:55:48.752 DEBUG:boto:Path: / 2024-04-12T03:55:48.752 DEBUG:boto:Data: 2024-04-12T03:55:48.752 DEBUG:boto:Headers: {} 2024-04-12T03:55:48.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2024-04-12T03:55:48.753 DEBUG:boto:Port: 8001 2024-04-12T03:55:48.753 DEBUG:boto:Params: {'RoleName': 'roles-dgbpxg-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T03:55:48.753 DEBUG:boto:Token: None 2024-04-12T03:55:48.753 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20240412T035548Z host;x-amz-date 537ceae7591bdd4378bafcc91a960e4d882aa04265ddb40120c714c3c7263586 2024-04-12T03:55:48.753 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T035548Z 20240412/front/smithi099/aws4_request a8b883b8705c8fb889176f7e4f3bc218ae883ce2ba7e3d6dbd480ce64685ea46 2024-04-12T03:55:48.753 DEBUG:boto:Signature: 1918a3e0fcc043eb0883d472898d1b12c409618f9f75db2eafffea808969b713 2024-04-12T03:55:48.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T035548Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JB2ZC967UHS0RDZ3J3O2/20240412/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=1918a3e0fcc043eb0883d472898d1b12c409618f9f75db2eafffea808969b713', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2024-04-12T03:55:48.755 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd4c04401e290b81-006618b0c4-20c9-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 03:55:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:48.755 DEBUG:boto:b'tx00000bd4c04401e290b81-006618b0c4-20c9-a2750300b2-ddff-4406-8081-a357153d71b5roles-dgbpxg-2/arn:aws:iam:::role/roles-dgbpxg-22024-04-12T03:54:06.267Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T03:55:48.756 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dgbpxg-2 2024-04-12T03:55:48.756 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dgbpxg-2 zones={a1, a2} 2024-04-12T03:55:48.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-12T03:55:48.757 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:48.758 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:48.758 DEBUG:boto:Method: PUT 2024-04-12T03:55:48.758 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:48.758 DEBUG:boto:Data: 2024-04-12T03:55:48.758 DEBUG:boto:Headers: {} 2024-04-12T03:55:48.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:48.758 DEBUG:boto:Port: 8000 2024-04-12T03:55:48.758 DEBUG:boto:Params: {} 2024-04-12T03:55:48.758 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:55:48.759 DEBUG:boto:Token: None 2024-04-12T03:55:48.759 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:55:48 GMT /dgbpxg-62/ 2024-04-12T03:55:48.759 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+FC5rJMkIQ1jpNjuLxO77MNh3OA= 2024-04-12T03:55:48.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:48 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+FC5rJMkIQ1jpNjuLxO77MNh3OA=', 'Content-Length': '0'} 2024-04-12T03:55:48.783 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a051edfe447a770e-006618b0c4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:48.784 DEBUG:rgw_multi.tests:created bucket=dgbpxg-62 2024-04-12T03:55:48.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.784 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:48.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:48.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:55:48.877 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:55:48.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.877 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:48.877 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894148.793559_1440.1", 2024-04-12T03:55:48.879 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:55:48.793559Z" 2024-04-12T03:55:48.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.879 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:55:48.880 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894148.793559_1440.1'} 2024-04-12T03:55:48.880 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:55:48.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:48.880 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:48.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:48.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:48.970+0000 7f71267c5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:48.971+0000 7f71267c5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:55:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.976 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894148.793559_1440.1", 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:55:48.793559Z", 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:55:48.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:55:48Z" 2024-04-12T03:55:48.979 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:55:48.980 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894148.793559_1440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:48.793559Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:48Z" } 2024-04-12T03:55:48.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894148.793559_1440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:48.793559Z', 'realm_epoch': 2}}] 2024-04-12T03:55:48.981 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894148.793559_1440.1'} 2024-04-12T03:55:48.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894148.793559_1440.1'} 2024-04-12T03:55:48.981 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:55:48.981 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:55:48.981 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:55:48.982 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:55:48.993+0000 7f6c50504700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 104864) UID: 0 2024-04-12T03:55:48.982 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:55:48.993+0000 7f6c556d97c0 -1 shutting down 2024-04-12T03:55:54.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:55:55.082 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:55:55.083 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:55.083 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:55.083 DEBUG:boto:Method: HEAD 2024-04-12T03:55:55.083 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:55.083 DEBUG:boto:Data: 2024-04-12T03:55:55.083 DEBUG:boto:Headers: {} 2024-04-12T03:55:55.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.083 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.083 DEBUG:boto:Params: {} 2024-04-12T03:55:55.083 DEBUG:boto:Token: None 2024-04-12T03:55:55.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/ 2024-04-12T03:55:55.084 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo= 2024-04-12T03:55:55.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo=', 'Content-Length': '0'} 2024-04-12T03:55:55.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000167eb8c0d92c8383-006618b0cb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.088 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:55:55.088 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:55:55.088 DEBUG:boto:Method: PUT 2024-04-12T03:55:55.089 DEBUG:boto:Path: /dgbpxg-62/obj11 2024-04-12T03:55:55.089 DEBUG:boto:Data: 2024-04-12T03:55:55.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:55.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.089 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.089 DEBUG:boto:Params: {} 2024-04-12T03:55:55.089 DEBUG:boto:Token: None 2024-04-12T03:55:55.089 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/obj11 2024-04-12T03:55:55.089 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YmHRSdc6RBMTQuHLPh+0yYaRMvs= 2024-04-12T03:55:55.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YmHRSdc6RBMTQuHLPh+0yYaRMvs='} 2024-04-12T03:55:55.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e23b7555523e23c2-006618b0cb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.097 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:55.097 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:55.097 DEBUG:boto:Method: HEAD 2024-04-12T03:55:55.097 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:55.097 DEBUG:boto:Data: 2024-04-12T03:55:55.098 DEBUG:boto:Headers: {} 2024-04-12T03:55:55.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.098 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.098 DEBUG:boto:Params: {} 2024-04-12T03:55:55.098 DEBUG:boto:Token: None 2024-04-12T03:55:55.098 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/ 2024-04-12T03:55:55.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo= 2024-04-12T03:55:55.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo=', 'Content-Length': '0'} 2024-04-12T03:55:55.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5a551799a7d8121-006618b0cb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.101 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:55:55.101 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:55:55.101 DEBUG:boto:Method: PUT 2024-04-12T03:55:55.101 DEBUG:boto:Path: /dgbpxg-62/obj22 2024-04-12T03:55:55.101 DEBUG:boto:Data: 2024-04-12T03:55:55.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:55.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.101 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.102 DEBUG:boto:Params: {} 2024-04-12T03:55:55.102 DEBUG:boto:Token: None 2024-04-12T03:55:55.102 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/obj22 2024-04-12T03:55:55.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LshaGDNMWZLuD7clWNUbua/63fY= 2024-04-12T03:55:55.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LshaGDNMWZLuD7clWNUbua/63fY='} 2024-04-12T03:55:55.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd1396d3addbeb02-006618b0cb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.107 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:55.107 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:55.107 DEBUG:boto:Method: HEAD 2024-04-12T03:55:55.107 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:55.107 DEBUG:boto:Data: 2024-04-12T03:55:55.107 DEBUG:boto:Headers: {} 2024-04-12T03:55:55.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.107 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.107 DEBUG:boto:Params: {} 2024-04-12T03:55:55.107 DEBUG:boto:Token: None 2024-04-12T03:55:55.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/ 2024-04-12T03:55:55.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo= 2024-04-12T03:55:55.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo=', 'Content-Length': '0'} 2024-04-12T03:55:55.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff121ac3ff287f8-006618b0cb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.110 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:55:55.110 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:55:55.110 DEBUG:boto:Method: PUT 2024-04-12T03:55:55.110 DEBUG:boto:Path: /dgbpxg-62/obj33 2024-04-12T03:55:55.110 DEBUG:boto:Data: 2024-04-12T03:55:55.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:55.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.111 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.111 DEBUG:boto:Params: {} 2024-04-12T03:55:55.111 DEBUG:boto:Token: None 2024-04-12T03:55:55.111 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/obj33 2024-04-12T03:55:55.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vTVE7YeyLXZZvFStIL38VjsntV0= 2024-04-12T03:55:55.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vTVE7YeyLXZZvFStIL38VjsntV0='} 2024-04-12T03:55:55.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000516b95319672ceeb-006618b0cb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.158 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:55.159 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:55.159 DEBUG:boto:Method: HEAD 2024-04-12T03:55:55.159 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:55.159 DEBUG:boto:Data: 2024-04-12T03:55:55.159 DEBUG:boto:Headers: {} 2024-04-12T03:55:55.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.159 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.159 DEBUG:boto:Params: {} 2024-04-12T03:55:55.159 DEBUG:boto:Token: None 2024-04-12T03:55:55.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/ 2024-04-12T03:55:55.160 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo= 2024-04-12T03:55:55.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo=', 'Content-Length': '0'} 2024-04-12T03:55:55.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ca104f06c144719-006618b0cb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.162 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:55:55.162 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:55:55.163 DEBUG:boto:Method: PUT 2024-04-12T03:55:55.163 DEBUG:boto:Path: /dgbpxg-62/obj44 2024-04-12T03:55:55.163 DEBUG:boto:Data: 2024-04-12T03:55:55.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:55.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.163 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.163 DEBUG:boto:Params: {} 2024-04-12T03:55:55.163 DEBUG:boto:Token: None 2024-04-12T03:55:55.163 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/obj44 2024-04-12T03:55:55.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S3ddX/aGJcO62CBHFOiy5U3eKEE= 2024-04-12T03:55:55.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S3ddX/aGJcO62CBHFOiy5U3eKEE='} 2024-04-12T03:55:55.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd018c5eed49ff85-006618b0cb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.170 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:55:55.170 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:55:55.171 DEBUG:boto:Method: HEAD 2024-04-12T03:55:55.171 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:55:55.171 DEBUG:boto:Data: 2024-04-12T03:55:55.171 DEBUG:boto:Headers: {} 2024-04-12T03:55:55.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.171 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.171 DEBUG:boto:Params: {} 2024-04-12T03:55:55.171 DEBUG:boto:Token: None 2024-04-12T03:55:55.171 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/ 2024-04-12T03:55:55.171 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo= 2024-04-12T03:55:55.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:67lJZRlAskyrp1zmnKZsEnJrBxo=', 'Content-Length': '0'} 2024-04-12T03:55:55.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f23646cbc9f4082-006618b0cb-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.174 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:55:55.175 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:55:55.175 DEBUG:boto:Method: PUT 2024-04-12T03:55:55.175 DEBUG:boto:Path: /dgbpxg-62/obj55 2024-04-12T03:55:55.175 DEBUG:boto:Data: 2024-04-12T03:55:55.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:55.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:55:55.175 DEBUG:boto:Port: 8000 2024-04-12T03:55:55.175 DEBUG:boto:Params: {} 2024-04-12T03:55:55.175 DEBUG:boto:Token: None 2024-04-12T03:55:55.176 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:55 GMT /dgbpxg-62/obj55 2024-04-12T03:55:55.176 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZK4MNJjxwWdJS7FEtkLRoumsh4A= 2024-04-12T03:55:55.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:55 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZK4MNJjxwWdJS7FEtkLRoumsh4A='} 2024-04-12T03:55:55.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c47bd54929f35c21-006618b0cb-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:55.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:55:55.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:55:55.181 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:55:55.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:55.236 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:55.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:55.283+0000 7f0a91342780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T03:55:55.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:55.284+0000 7f0a91342780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T03:55:55.295 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:55:55.296 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:55:55.298 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:55:55.298 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:55:55.317 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:55:55.318 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:55:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:55:55.319 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:55:56.320 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:55:56.342 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:55:56.342 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:55:56.342 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:55:56.343 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T03:55:56.343 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-62 2024-04-12T03:55:56.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:55:56.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:55:56.344 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:55:56.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:56.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:55:56.485+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:00.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:56:01.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:01.484+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:06.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:06.485+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:11.486+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:16.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:16.487+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:21.488+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:26.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:26.488+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:31.489+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:36.490+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:41.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:41.490+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:46.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:46.490+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:51.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:51.490+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:56:56.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:56:56.492+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:00.346 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T03:57:00.357+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:57:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:01.492+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:06.493+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:11.494+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:16.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:16.494+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:21.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:21.495+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:26.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:26.496+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:29.136 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:57:29.147+0000 7fb523f8e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:57:31.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:31.496+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:36.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:36.496+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:41.498+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:46.499+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:51.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:51.499+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:57:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:57:56.500+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:01.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:01.501+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:06.502+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:11.502+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:16.502+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:21.503+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:26.504+0000 7fbdf17b9780 1 waiting to reach incremental sync.. 2024-04-12T03:58:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:31.506+0000 7fbdf17b9780 1 bucket sync caught up with source: 2024-04-12T03:58:31.494 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.159.6, 00000000002.71.6, 00000000002.49.6] 2024-04-12T03:58:31.494 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.159.6, 00000000002.71.6, 00000000002.49.6] 2024-04-12T03:58:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:31.506+0000 7fbdf17b9780 0 bucket checkpoint complete 2024-04-12T03:58:31.512 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-62 2024-04-12T03:58:31.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:31.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:31.513 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:58:31.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:31.574 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:31.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:31.623+0000 7fe73d23d780 1 bucket sync caught up with source: 2024-04-12T03:58:31.611 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1223.6, 00000000002.2843.6, 00000000002.534.6] 2024-04-12T03:58:31.611 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1223.6, 00000000002.2843.6, 00000000002.534.6] 2024-04-12T03:58:31.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:31.623+0000 7fe73d23d780 0 bucket checkpoint complete 2024-04-12T03:58:31.625 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-62 zones={a2, a1} 2024-04-12T03:58:31.625 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.626 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.626 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.626 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:58:31.626 DEBUG:boto:Data: 2024-04-12T03:58:31.626 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.626 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.626 DEBUG:boto:Params: {} 2024-04-12T03:58:31.627 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:58:31.627 DEBUG:boto:Token: None 2024-04-12T03:58:31.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.627 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zyn4oBzJ3uf/l06enp3+sh6EvmY= 2024-04-12T03:58:31.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zyn4oBzJ3uf/l06enp3+sh6EvmY=', 'Content-Length': '0'} 2024-04-12T03:58:31.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089a3829f751db954-006618b167-21a9-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.659 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.659 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.659 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.659 DEBUG:boto:Path: /dgbpxg-62/ 2024-04-12T03:58:31.659 DEBUG:boto:Data: 2024-04-12T03:58:31.659 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.659 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.659 DEBUG:boto:Params: {} 2024-04-12T03:58:31.660 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:58:31.660 DEBUG:boto:Token: None 2024-04-12T03:58:31.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zyn4oBzJ3uf/l06enp3+sh6EvmY= 2024-04-12T03:58:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zyn4oBzJ3uf/l06enp3+sh6EvmY=', 'Content-Length': '0'} 2024-04-12T03:58:31.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f24ff479753aa24e-006618b167-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:58:31.673 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.673 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.673 DEBUG:boto:path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.673 DEBUG:boto:auth_path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.673 DEBUG:boto:Method: GET 2024-04-12T03:58:31.673 DEBUG:boto:Path: /dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.673 DEBUG:boto:Data: 2024-04-12T03:58:31.673 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.673 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.673 DEBUG:boto:Params: {} 2024-04-12T03:58:31.674 DEBUG:boto:Token: None 2024-04-12T03:58:31.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/?versions 2024-04-12T03:58:31.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE= 2024-04-12T03:58:31.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE=', 'Content-Length': '0'} 2024-04-12T03:58:31.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c721fa7bf46184-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.676 DEBUG:boto:b'dgbpxg-62999falseobj11nulltrue2024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:58:31.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:58:31.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:58:31.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:58:31.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:58:31.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:58:31.678 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.678 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.678 DEBUG:boto:path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.678 DEBUG:boto:auth_path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.679 DEBUG:boto:Method: GET 2024-04-12T03:58:31.679 DEBUG:boto:Path: /dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.679 DEBUG:boto:Data: 2024-04-12T03:58:31.679 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.679 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.679 DEBUG:boto:Params: {} 2024-04-12T03:58:31.679 DEBUG:boto:Token: None 2024-04-12T03:58:31.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/?versions 2024-04-12T03:58:31.679 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE= 2024-04-12T03:58:31.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE=', 'Content-Length': '0'} 2024-04-12T03:58:31.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d884067ea5804c4a-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.682 DEBUG:boto:b'dgbpxg-62999falseobj11nulltrue2024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T03:58:31.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T03:58:31.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T03:58:31.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T03:58:31.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T03:58:31.683 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.683 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.683 DEBUG:boto:path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.683 DEBUG:boto:auth_path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.683 DEBUG:boto:Method: GET 2024-04-12T03:58:31.683 DEBUG:boto:Path: /dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.683 DEBUG:boto:Data: 2024-04-12T03:58:31.683 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.683 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.683 DEBUG:boto:Params: {} 2024-04-12T03:58:31.684 DEBUG:boto:Token: None 2024-04-12T03:58:31.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/?versions 2024-04-12T03:58:31.684 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE= 2024-04-12T03:58:31.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE=', 'Content-Length': '0'} 2024-04-12T03:58:31.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ac156e8ab91496-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.686 DEBUG:boto:b'dgbpxg-62999falseobj11nulltrue2024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.687 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.687 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.687 DEBUG:boto:path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.687 DEBUG:boto:auth_path=/dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.687 DEBUG:boto:Method: GET 2024-04-12T03:58:31.687 DEBUG:boto:Path: /dgbpxg-62/?versions&max-keys=999 2024-04-12T03:58:31.687 DEBUG:boto:Data: 2024-04-12T03:58:31.687 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.687 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.687 DEBUG:boto:Params: {} 2024-04-12T03:58:31.688 DEBUG:boto:Token: None 2024-04-12T03:58:31.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/?versions 2024-04-12T03:58:31.688 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE= 2024-04-12T03:58:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SGkovPNW/xxqmJ7yZUJ/DK1JxUE=', 'Content-Length': '0'} 2024-04-12T03:58:31.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15c983686aa95f5-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.690 DEBUG:boto:b'dgbpxg-62999falseobj11nulltrue2024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:58:31.691 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.691 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.691 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.691 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.691 DEBUG:boto:Method: GET 2024-04-12T03:58:31.691 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.691 DEBUG:boto:Data: 2024-04-12T03:58:31.691 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.692 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.692 DEBUG:boto:Params: {} 2024-04-12T03:58:31.692 DEBUG:boto:Token: None 2024-04-12T03:58:31.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y= 2024-04-12T03:58:31.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y=', 'Content-Length': '0'} 2024-04-12T03:58:31.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073397a06381057d0-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.694 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.694 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.695 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.695 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.695 DEBUG:boto:Method: GET 2024-04-12T03:58:31.695 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.695 DEBUG:boto:Data: 2024-04-12T03:58:31.695 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.695 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.695 DEBUG:boto:Params: {} 2024-04-12T03:58:31.695 DEBUG:boto:Token: None 2024-04-12T03:58:31.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.696 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y= 2024-04-12T03:58:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y=', 'Content-Length': '0'} 2024-04-12T03:58:31.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7966bdeeae619c-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.698 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.698 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.698 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.698 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.698 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.698 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.698 DEBUG:boto:Data: 2024-04-12T03:58:31.699 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.699 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.699 DEBUG:boto:Params: {} 2024-04-12T03:58:31.699 DEBUG:boto:Token: None 2024-04-12T03:58:31.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:igyAA6p1Fo2ID5ykjM1dh5IQP/U= 2024-04-12T03:58:31.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:igyAA6p1Fo2ID5ykjM1dh5IQP/U=', 'Content-Length': '0'} 2024-04-12T03:58:31.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ee320d0e78e204-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.701 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.701 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.701 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.701 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.701 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.702 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.702 DEBUG:boto:Data: 2024-04-12T03:58:31.702 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.702 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.702 DEBUG:boto:Params: {} 2024-04-12T03:58:31.702 DEBUG:boto:Token: None 2024-04-12T03:58:31.702 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:igyAA6p1Fo2ID5ykjM1dh5IQP/U= 2024-04-12T03:58:31.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:igyAA6p1Fo2ID5ykjM1dh5IQP/U=', 'Content-Length': '0'} 2024-04-12T03:58:31.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001277d58828e8a5e7-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:58:31.704 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.704 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.704 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.705 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.705 DEBUG:boto:Method: GET 2024-04-12T03:58:31.705 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.705 DEBUG:boto:Data: 2024-04-12T03:58:31.705 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.705 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.705 DEBUG:boto:Params: {} 2024-04-12T03:58:31.705 DEBUG:boto:Token: None 2024-04-12T03:58:31.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.705 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y= 2024-04-12T03:58:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y=', 'Content-Length': '0'} 2024-04-12T03:58:31.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e538944e36553e2-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.707 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.707 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.707 DEBUG:boto:path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.707 DEBUG:boto:auth_path=/dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.708 DEBUG:boto:Method: GET 2024-04-12T03:58:31.708 DEBUG:boto:Path: /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.708 DEBUG:boto:Data: 2024-04-12T03:58:31.708 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.708 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.708 DEBUG:boto:Params: {} 2024-04-12T03:58:31.708 DEBUG:boto:Token: None 2024-04-12T03:58:31.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11?versionId=null 2024-04-12T03:58:31.709 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y= 2024-04-12T03:58:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KYxu691T9VMIqHOm0jbV7FjBH5Y=', 'Content-Length': '0'} 2024-04-12T03:58:31.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118515362a53173a-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.711 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.711 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.711 DEBUG:boto:path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.711 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.711 DEBUG:boto:Method: GET 2024-04-12T03:58:31.711 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.711 DEBUG:boto:Data: 2024-04-12T03:58:31.711 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.711 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.711 DEBUG:boto:Params: {} 2024-04-12T03:58:31.712 DEBUG:boto:Token: None 2024-04-12T03:58:31.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1c25864556bbe13-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.714 DEBUG:boto:b'dgbpxg-62obj111000falseobj112024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.715 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.715 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.715 DEBUG:boto:path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.715 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.715 DEBUG:boto:Method: GET 2024-04-12T03:58:31.715 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.715 DEBUG:boto:Data: 2024-04-12T03:58:31.715 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.715 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.715 DEBUG:boto:Params: {} 2024-04-12T03:58:31.716 DEBUG:boto:Token: None 2024-04-12T03:58:31.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf5a08386278517-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.718 DEBUG:boto:b'dgbpxg-62obj111000falseobj112024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.718 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.719 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.719 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.719 DEBUG:boto:Path: /dgbpxg-62/obj11 2024-04-12T03:58:31.719 DEBUG:boto:Data: 2024-04-12T03:58:31.719 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.719 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.719 DEBUG:boto:Params: {} 2024-04-12T03:58:31.719 DEBUG:boto:Token: None 2024-04-12T03:58:31.719 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11 2024-04-12T03:58:31.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QWij2qQAOiSYwY1OFUM9oBshtxM= 2024-04-12T03:58:31.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QWij2qQAOiSYwY1OFUM9oBshtxM=', 'Content-Length': '0'} 2024-04-12T03:58:31.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89ce82aa82cee94-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.721 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.721 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.721 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.722 DEBUG:boto:Path: /dgbpxg-62/obj11 2024-04-12T03:58:31.722 DEBUG:boto:Data: 2024-04-12T03:58:31.722 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.722 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.722 DEBUG:boto:Params: {} 2024-04-12T03:58:31.722 DEBUG:boto:Token: None 2024-04-12T03:58:31.722 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11 2024-04-12T03:58:31.722 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QWij2qQAOiSYwY1OFUM9oBshtxM= 2024-04-12T03:58:31.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QWij2qQAOiSYwY1OFUM9oBshtxM=', 'Content-Length': '0'} 2024-04-12T03:58:31.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4428d8ec9e3bbd-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T03:58:31.725 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.725 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.725 DEBUG:boto:Method: GET 2024-04-12T03:58:31.725 DEBUG:boto:Path: /dgbpxg-62/obj11 2024-04-12T03:58:31.725 DEBUG:boto:Data: 2024-04-12T03:58:31.725 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.725 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.726 DEBUG:boto:Params: {} 2024-04-12T03:58:31.726 DEBUG:boto:Token: None 2024-04-12T03:58:31.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11 2024-04-12T03:58:31.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4c1N+aNGmHexONjQyQyAwrBxlCk= 2024-04-12T03:58:31.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4c1N+aNGmHexONjQyQyAwrBxlCk=', 'Content-Length': '0'} 2024-04-12T03:58:31.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a176025d8f553a55-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.728 DEBUG:boto:path=/dgbpxg-62/obj11 2024-04-12T03:58:31.728 DEBUG:boto:auth_path=/dgbpxg-62/obj11 2024-04-12T03:58:31.728 DEBUG:boto:Method: GET 2024-04-12T03:58:31.728 DEBUG:boto:Path: /dgbpxg-62/obj11 2024-04-12T03:58:31.728 DEBUG:boto:Data: 2024-04-12T03:58:31.728 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.728 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.729 DEBUG:boto:Params: {} 2024-04-12T03:58:31.729 DEBUG:boto:Token: None 2024-04-12T03:58:31.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj11 2024-04-12T03:58:31.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4c1N+aNGmHexONjQyQyAwrBxlCk= 2024-04-12T03:58:31.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4c1N+aNGmHexONjQyQyAwrBxlCk=', 'Content-Length': '0'} 2024-04-12T03:58:31.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f60c6dc77beafd9-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.731 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.731 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.731 DEBUG:boto:path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.731 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.732 DEBUG:boto:Method: GET 2024-04-12T03:58:31.732 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.732 DEBUG:boto:Data: 2024-04-12T03:58:31.732 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.732 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.732 DEBUG:boto:Params: {} 2024-04-12T03:58:31.732 DEBUG:boto:Token: None 2024-04-12T03:58:31.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044804d7ef4bfb092-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.735 DEBUG:boto:b'dgbpxg-62obj111000falseobj112024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.735 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.735 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.735 DEBUG:boto:path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.735 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.736 DEBUG:boto:Method: GET 2024-04-12T03:58:31.736 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj11 2024-04-12T03:58:31.736 DEBUG:boto:Data: 2024-04-12T03:58:31.736 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.736 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.736 DEBUG:boto:Params: {} 2024-04-12T03:58:31.736 DEBUG:boto:Token: None 2024-04-12T03:58:31.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d3018234c593cd-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.738 DEBUG:boto:b'dgbpxg-62obj111000falseobj112024-04-12T03:55:55.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:58:31.739 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.739 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.739 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.739 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.739 DEBUG:boto:Method: GET 2024-04-12T03:58:31.739 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.739 DEBUG:boto:Data: 2024-04-12T03:58:31.740 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.740 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.740 DEBUG:boto:Params: {} 2024-04-12T03:58:31.740 DEBUG:boto:Token: None 2024-04-12T03:58:31.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA= 2024-04-12T03:58:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA=', 'Content-Length': '0'} 2024-04-12T03:58:31.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d08deddee9d6c32-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.782 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.782 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.783 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.783 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.783 DEBUG:boto:Method: GET 2024-04-12T03:58:31.783 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.783 DEBUG:boto:Data: 2024-04-12T03:58:31.783 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.783 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.783 DEBUG:boto:Params: {} 2024-04-12T03:58:31.783 DEBUG:boto:Token: None 2024-04-12T03:58:31.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA= 2024-04-12T03:58:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA=', 'Content-Length': '0'} 2024-04-12T03:58:31.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea5e7cf49944feb6-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.785 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.785 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.786 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.786 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.786 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.786 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.786 DEBUG:boto:Data: 2024-04-12T03:58:31.786 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.786 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.786 DEBUG:boto:Params: {} 2024-04-12T03:58:31.786 DEBUG:boto:Token: None 2024-04-12T03:58:31.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.787 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5PnAuQNvzhkKAEcePFvHeZVaciQ= 2024-04-12T03:58:31.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5PnAuQNvzhkKAEcePFvHeZVaciQ=', 'Content-Length': '0'} 2024-04-12T03:58:31.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf2274688192fb0-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.789 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.789 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.789 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.789 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.789 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.789 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.789 DEBUG:boto:Data: 2024-04-12T03:58:31.789 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.789 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.789 DEBUG:boto:Params: {} 2024-04-12T03:58:31.790 DEBUG:boto:Token: None 2024-04-12T03:58:31.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5PnAuQNvzhkKAEcePFvHeZVaciQ= 2024-04-12T03:58:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5PnAuQNvzhkKAEcePFvHeZVaciQ=', 'Content-Length': '0'} 2024-04-12T03:58:31.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea7055df4075a52-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:58:31.792 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.792 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.792 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.792 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.792 DEBUG:boto:Method: GET 2024-04-12T03:58:31.792 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.792 DEBUG:boto:Data: 2024-04-12T03:58:31.792 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.792 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.793 DEBUG:boto:Params: {} 2024-04-12T03:58:31.793 DEBUG:boto:Token: None 2024-04-12T03:58:31.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA= 2024-04-12T03:58:31.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA=', 'Content-Length': '0'} 2024-04-12T03:58:31.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8aa3966dc97cbea-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.795 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.795 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.795 DEBUG:boto:path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.795 DEBUG:boto:auth_path=/dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.795 DEBUG:boto:Method: GET 2024-04-12T03:58:31.795 DEBUG:boto:Path: /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.795 DEBUG:boto:Data: 2024-04-12T03:58:31.795 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.796 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.796 DEBUG:boto:Params: {} 2024-04-12T03:58:31.796 DEBUG:boto:Token: None 2024-04-12T03:58:31.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22?versionId=null 2024-04-12T03:58:31.796 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA= 2024-04-12T03:58:31.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zJUifEWwfr2kKNw7p6sbPlDL6oA=', 'Content-Length': '0'} 2024-04-12T03:58:31.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1eb63938212c26-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.798 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.798 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.798 DEBUG:boto:path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.798 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.798 DEBUG:boto:Method: GET 2024-04-12T03:58:31.798 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.799 DEBUG:boto:Data: 2024-04-12T03:58:31.799 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.799 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.799 DEBUG:boto:Params: {} 2024-04-12T03:58:31.799 DEBUG:boto:Token: None 2024-04-12T03:58:31.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e2c651330bd1de-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.801 DEBUG:boto:b'dgbpxg-62obj221000falseobj222024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.802 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.802 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.802 DEBUG:boto:path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.802 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.802 DEBUG:boto:Method: GET 2024-04-12T03:58:31.802 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.802 DEBUG:boto:Data: 2024-04-12T03:58:31.802 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.803 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.803 DEBUG:boto:Params: {} 2024-04-12T03:58:31.803 DEBUG:boto:Token: None 2024-04-12T03:58:31.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2b08a295df16be-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.806 DEBUG:boto:b'dgbpxg-62obj221000falseobj222024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.806 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.806 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.806 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.806 DEBUG:boto:Path: /dgbpxg-62/obj22 2024-04-12T03:58:31.806 DEBUG:boto:Data: 2024-04-12T03:58:31.806 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.807 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.807 DEBUG:boto:Params: {} 2024-04-12T03:58:31.807 DEBUG:boto:Token: None 2024-04-12T03:58:31.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22 2024-04-12T03:58:31.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBGfvaTkPq5OE6EjUJmiaeffLxM= 2024-04-12T03:58:31.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBGfvaTkPq5OE6EjUJmiaeffLxM=', 'Content-Length': '0'} 2024-04-12T03:58:31.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a9c08a2c98edc4-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.809 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.809 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.809 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.809 DEBUG:boto:Path: /dgbpxg-62/obj22 2024-04-12T03:58:31.809 DEBUG:boto:Data: 2024-04-12T03:58:31.809 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.810 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.810 DEBUG:boto:Params: {} 2024-04-12T03:58:31.810 DEBUG:boto:Token: None 2024-04-12T03:58:31.810 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22 2024-04-12T03:58:31.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FBGfvaTkPq5OE6EjUJmiaeffLxM= 2024-04-12T03:58:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FBGfvaTkPq5OE6EjUJmiaeffLxM=', 'Content-Length': '0'} 2024-04-12T03:58:31.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000109dbaf2044232a5-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T03:58:31.812 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.812 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.813 DEBUG:boto:Method: GET 2024-04-12T03:58:31.813 DEBUG:boto:Path: /dgbpxg-62/obj22 2024-04-12T03:58:31.813 DEBUG:boto:Data: 2024-04-12T03:58:31.813 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.813 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.813 DEBUG:boto:Params: {} 2024-04-12T03:58:31.813 DEBUG:boto:Token: None 2024-04-12T03:58:31.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22 2024-04-12T03:58:31.814 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GRKrdC1ZXek34jjNBySIoybgexI= 2024-04-12T03:58:31.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GRKrdC1ZXek34jjNBySIoybgexI=', 'Content-Length': '0'} 2024-04-12T03:58:31.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573f42f0178b8976-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.815 DEBUG:boto:path=/dgbpxg-62/obj22 2024-04-12T03:58:31.816 DEBUG:boto:auth_path=/dgbpxg-62/obj22 2024-04-12T03:58:31.816 DEBUG:boto:Method: GET 2024-04-12T03:58:31.816 DEBUG:boto:Path: /dgbpxg-62/obj22 2024-04-12T03:58:31.816 DEBUG:boto:Data: 2024-04-12T03:58:31.816 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.816 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.816 DEBUG:boto:Params: {} 2024-04-12T03:58:31.816 DEBUG:boto:Token: None 2024-04-12T03:58:31.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj22 2024-04-12T03:58:31.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GRKrdC1ZXek34jjNBySIoybgexI= 2024-04-12T03:58:31.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GRKrdC1ZXek34jjNBySIoybgexI=', 'Content-Length': '0'} 2024-04-12T03:58:31.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a036e97a6cba46-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.818 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.818 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.818 DEBUG:boto:path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.819 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.819 DEBUG:boto:Method: GET 2024-04-12T03:58:31.819 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.819 DEBUG:boto:Data: 2024-04-12T03:58:31.819 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.819 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.819 DEBUG:boto:Params: {} 2024-04-12T03:58:31.819 DEBUG:boto:Token: None 2024-04-12T03:58:31.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000779e33b20830ca64-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.862 DEBUG:boto:b'dgbpxg-62obj221000falseobj222024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.863 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.863 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.863 DEBUG:boto:path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.863 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.863 DEBUG:boto:Method: GET 2024-04-12T03:58:31.863 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj22 2024-04-12T03:58:31.863 DEBUG:boto:Data: 2024-04-12T03:58:31.863 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.863 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.864 DEBUG:boto:Params: {} 2024-04-12T03:58:31.864 DEBUG:boto:Token: None 2024-04-12T03:58:31.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfe23ec37dbdc0f-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.867 DEBUG:boto:b'dgbpxg-62obj221000falseobj222024-04-12T03:55:55.115Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:58:31.867 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.867 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.867 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.867 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.867 DEBUG:boto:Method: GET 2024-04-12T03:58:31.868 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.868 DEBUG:boto:Data: 2024-04-12T03:58:31.868 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.868 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.868 DEBUG:boto:Params: {} 2024-04-12T03:58:31.868 DEBUG:boto:Token: None 2024-04-12T03:58:31.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs= 2024-04-12T03:58:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs=', 'Content-Length': '0'} 2024-04-12T03:58:31.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006946f2fbbcdaa36d-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.870 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.870 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.870 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.870 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.870 DEBUG:boto:Method: GET 2024-04-12T03:58:31.871 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.871 DEBUG:boto:Data: 2024-04-12T03:58:31.871 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.871 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.871 DEBUG:boto:Params: {} 2024-04-12T03:58:31.871 DEBUG:boto:Token: None 2024-04-12T03:58:31.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs= 2024-04-12T03:58:31.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs=', 'Content-Length': '0'} 2024-04-12T03:58:31.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ee1c96af76e763-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.914 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.914 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.915 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.915 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.915 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.915 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.915 DEBUG:boto:Data: 2024-04-12T03:58:31.915 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.915 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.915 DEBUG:boto:Params: {} 2024-04-12T03:58:31.915 DEBUG:boto:Token: None 2024-04-12T03:58:31.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HH6W/9z53SvtEpL5sDDa9GkOcvE= 2024-04-12T03:58:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HH6W/9z53SvtEpL5sDDa9GkOcvE=', 'Content-Length': '0'} 2024-04-12T03:58:31.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a2541f3ffa7d72-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.918 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.918 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.918 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.918 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.918 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.918 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.918 DEBUG:boto:Data: 2024-04-12T03:58:31.918 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.918 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.919 DEBUG:boto:Params: {} 2024-04-12T03:58:31.919 DEBUG:boto:Token: None 2024-04-12T03:58:31.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HH6W/9z53SvtEpL5sDDa9GkOcvE= 2024-04-12T03:58:31.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HH6W/9z53SvtEpL5sDDa9GkOcvE=', 'Content-Length': '0'} 2024-04-12T03:58:31.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690b46cfa2805a78-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:58:31.921 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.921 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.921 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.921 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.921 DEBUG:boto:Method: GET 2024-04-12T03:58:31.921 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.921 DEBUG:boto:Data: 2024-04-12T03:58:31.922 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.922 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.922 DEBUG:boto:Params: {} 2024-04-12T03:58:31.922 DEBUG:boto:Token: None 2024-04-12T03:58:31.922 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs= 2024-04-12T03:58:31.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs=', 'Content-Length': '0'} 2024-04-12T03:58:31.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b0c1b68238bb2d-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.924 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.924 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.924 DEBUG:boto:path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.924 DEBUG:boto:auth_path=/dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.924 DEBUG:boto:Method: GET 2024-04-12T03:58:31.924 DEBUG:boto:Path: /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.924 DEBUG:boto:Data: 2024-04-12T03:58:31.925 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.925 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.925 DEBUG:boto:Params: {} 2024-04-12T03:58:31.925 DEBUG:boto:Token: None 2024-04-12T03:58:31.925 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33?versionId=null 2024-04-12T03:58:31.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs= 2024-04-12T03:58:31.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XX8uDJItVQ+Rt+GYr0zhv7AG8Xs=', 'Content-Length': '0'} 2024-04-12T03:58:31.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029547a62ad0c75a9-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.927 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.927 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.927 DEBUG:boto:path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.927 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.928 DEBUG:boto:Method: GET 2024-04-12T03:58:31.928 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.928 DEBUG:boto:Data: 2024-04-12T03:58:31.928 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.928 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.928 DEBUG:boto:Params: {} 2024-04-12T03:58:31.928 DEBUG:boto:Token: None 2024-04-12T03:58:31.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.929 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008f4ab1fca4ae070-006618b167-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.931 DEBUG:boto:b'dgbpxg-62obj331000falseobj332024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.931 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.932 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.932 DEBUG:boto:path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.932 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.932 DEBUG:boto:Method: GET 2024-04-12T03:58:31.932 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.932 DEBUG:boto:Data: 2024-04-12T03:58:31.932 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.932 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.932 DEBUG:boto:Params: {} 2024-04-12T03:58:31.932 DEBUG:boto:Token: None 2024-04-12T03:58:31.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb60cc3a92a7b68-006618b167-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.935 DEBUG:boto:b'dgbpxg-62obj331000falseobj332024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.935 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.935 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.935 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.936 DEBUG:boto:Path: /dgbpxg-62/obj33 2024-04-12T03:58:31.936 DEBUG:boto:Data: 2024-04-12T03:58:31.936 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.936 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.936 DEBUG:boto:Params: {} 2024-04-12T03:58:31.936 DEBUG:boto:Token: None 2024-04-12T03:58:31.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33 2024-04-12T03:58:31.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q8TNE3sjJyz8WeYKoyIdSariQ6g= 2024-04-12T03:58:31.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q8TNE3sjJyz8WeYKoyIdSariQ6g=', 'Content-Length': '0'} 2024-04-12T03:58:31.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64002fcf00c22db-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.938 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.938 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.938 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.938 DEBUG:boto:Path: /dgbpxg-62/obj33 2024-04-12T03:58:31.938 DEBUG:boto:Data: 2024-04-12T03:58:31.938 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.939 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.939 DEBUG:boto:Params: {} 2024-04-12T03:58:31.939 DEBUG:boto:Token: None 2024-04-12T03:58:31.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33 2024-04-12T03:58:31.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q8TNE3sjJyz8WeYKoyIdSariQ6g= 2024-04-12T03:58:31.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q8TNE3sjJyz8WeYKoyIdSariQ6g=', 'Content-Length': '0'} 2024-04-12T03:58:31.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebf1a4e2db52df9-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T03:58:31.941 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.941 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.941 DEBUG:boto:Method: GET 2024-04-12T03:58:31.941 DEBUG:boto:Path: /dgbpxg-62/obj33 2024-04-12T03:58:31.941 DEBUG:boto:Data: 2024-04-12T03:58:31.941 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.941 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.942 DEBUG:boto:Params: {} 2024-04-12T03:58:31.942 DEBUG:boto:Token: None 2024-04-12T03:58:31.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33 2024-04-12T03:58:31.942 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNG7cKJHfrW9Dk6Gpz4IFqv0PsM= 2024-04-12T03:58:31.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNG7cKJHfrW9Dk6Gpz4IFqv0PsM=', 'Content-Length': '0'} 2024-04-12T03:58:31.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a714eab8003d307-006618b167-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.944 DEBUG:boto:path=/dgbpxg-62/obj33 2024-04-12T03:58:31.944 DEBUG:boto:auth_path=/dgbpxg-62/obj33 2024-04-12T03:58:31.944 DEBUG:boto:Method: GET 2024-04-12T03:58:31.944 DEBUG:boto:Path: /dgbpxg-62/obj33 2024-04-12T03:58:31.944 DEBUG:boto:Data: 2024-04-12T03:58:31.944 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.944 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.945 DEBUG:boto:Params: {} 2024-04-12T03:58:31.945 DEBUG:boto:Token: None 2024-04-12T03:58:31.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/obj33 2024-04-12T03:58:31.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DNG7cKJHfrW9Dk6Gpz4IFqv0PsM= 2024-04-12T03:58:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DNG7cKJHfrW9Dk6Gpz4IFqv0PsM=', 'Content-Length': '0'} 2024-04-12T03:58:31.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b978e09677f3f61-006618b167-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.990 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.991 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.991 DEBUG:boto:path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.991 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.991 DEBUG:boto:Method: GET 2024-04-12T03:58:31.991 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.991 DEBUG:boto:Data: 2024-04-12T03:58:31.991 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.991 DEBUG:boto:Port: 8001 2024-04-12T03:58:31.991 DEBUG:boto:Params: {} 2024-04-12T03:58:31.991 DEBUG:boto:Token: None 2024-04-12T03:58:31.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.992 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d833001617ccb69c-006618b168-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.994 DEBUG:boto:b'dgbpxg-62obj331000falseobj332024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.994 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:31.994 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:31.995 DEBUG:boto:path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.995 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.995 DEBUG:boto:Method: GET 2024-04-12T03:58:31.995 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj33 2024-04-12T03:58:31.995 DEBUG:boto:Data: 2024-04-12T03:58:31.995 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:31.995 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.995 DEBUG:boto:Params: {} 2024-04-12T03:58:31.995 DEBUG:boto:Token: None 2024-04-12T03:58:31.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:31 GMT /dgbpxg-62/ 2024-04-12T03:58:31.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4= 2024-04-12T03:58:31.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VAj0h+P4lFgg9r5eImDmb8IywI4=', 'Content-Length': '0'} 2024-04-12T03:58:31.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec3c38891a70867-006618b168-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.998 DEBUG:boto:b'dgbpxg-62obj331000falseobj332024-04-12T03:55:55.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:31.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:58:31.999 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:31.999 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:31.999 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:31.999 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:31.999 DEBUG:boto:Method: GET 2024-04-12T03:58:31.999 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:31.999 DEBUG:boto:Data: 2024-04-12T03:58:31.999 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:31.999 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.000 DEBUG:boto:Params: {} 2024-04-12T03:58:32.000 DEBUG:boto:Token: None 2024-04-12T03:58:32.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A= 2024-04-12T03:58:32.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A=', 'Content-Length': '0'} 2024-04-12T03:58:32.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b8f456968240d4-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.002 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.002 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.002 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.002 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.002 DEBUG:boto:Method: GET 2024-04-12T03:58:32.002 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.003 DEBUG:boto:Data: 2024-04-12T03:58:32.003 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.003 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.003 DEBUG:boto:Params: {} 2024-04-12T03:58:32.003 DEBUG:boto:Token: None 2024-04-12T03:58:32.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A= 2024-04-12T03:58:32.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A=', 'Content-Length': '0'} 2024-04-12T03:58:32.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641578f86b1d8e96-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.006 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.006 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.006 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.006 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.006 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.006 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.006 DEBUG:boto:Data: 2024-04-12T03:58:32.007 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.007 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.007 DEBUG:boto:Params: {} 2024-04-12T03:58:32.007 DEBUG:boto:Token: None 2024-04-12T03:58:32.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.007 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TGX2+aldvTI2anVUq+4VGEryhRQ= 2024-04-12T03:58:32.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TGX2+aldvTI2anVUq+4VGEryhRQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003898fdd5261e5d91-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.009 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.009 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.009 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.009 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.009 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.010 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.010 DEBUG:boto:Data: 2024-04-12T03:58:32.010 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.010 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.010 DEBUG:boto:Params: {} 2024-04-12T03:58:32.010 DEBUG:boto:Token: None 2024-04-12T03:58:32.011 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TGX2+aldvTI2anVUq+4VGEryhRQ= 2024-04-12T03:58:32.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TGX2+aldvTI2anVUq+4VGEryhRQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fbba22cd072005-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:58:32.013 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.013 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.013 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.013 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.013 DEBUG:boto:Method: GET 2024-04-12T03:58:32.013 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.013 DEBUG:boto:Data: 2024-04-12T03:58:32.013 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.014 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.014 DEBUG:boto:Params: {} 2024-04-12T03:58:32.014 DEBUG:boto:Token: None 2024-04-12T03:58:32.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A= 2024-04-12T03:58:32.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A=', 'Content-Length': '0'} 2024-04-12T03:58:32.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d0a4c0fe9fbf5b-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.016 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.016 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.016 DEBUG:boto:path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.016 DEBUG:boto:auth_path=/dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.017 DEBUG:boto:Method: GET 2024-04-12T03:58:32.017 DEBUG:boto:Path: /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.017 DEBUG:boto:Data: 2024-04-12T03:58:32.017 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.017 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.017 DEBUG:boto:Params: {} 2024-04-12T03:58:32.017 DEBUG:boto:Token: None 2024-04-12T03:58:32.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44?versionId=null 2024-04-12T03:58:32.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A= 2024-04-12T03:58:32.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vRWJIfW7M3/stcHyahWSjX2ut6A=', 'Content-Length': '0'} 2024-04-12T03:58:32.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028154d807cd1d50e-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.019 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.020 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.020 DEBUG:boto:path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.020 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.020 DEBUG:boto:Method: GET 2024-04-12T03:58:32.020 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.020 DEBUG:boto:Data: 2024-04-12T03:58:32.020 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.020 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.020 DEBUG:boto:Params: {} 2024-04-12T03:58:32.021 DEBUG:boto:Token: None 2024-04-12T03:58:32.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008048f62b6b11788e-006618b168-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.023 DEBUG:boto:b'dgbpxg-62obj441000falseobj442024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.023 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.024 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.024 DEBUG:boto:path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.024 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.024 DEBUG:boto:Method: GET 2024-04-12T03:58:32.024 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.024 DEBUG:boto:Data: 2024-04-12T03:58:32.024 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.024 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.024 DEBUG:boto:Params: {} 2024-04-12T03:58:32.025 DEBUG:boto:Token: None 2024-04-12T03:58:32.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed0995ae6c7afa0-006618b168-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.027 DEBUG:boto:b'dgbpxg-62obj441000falseobj442024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.028 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.028 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.028 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.028 DEBUG:boto:Path: /dgbpxg-62/obj44 2024-04-12T03:58:32.028 DEBUG:boto:Data: 2024-04-12T03:58:32.028 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.028 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.028 DEBUG:boto:Params: {} 2024-04-12T03:58:32.029 DEBUG:boto:Token: None 2024-04-12T03:58:32.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44 2024-04-12T03:58:32.029 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rck4B8o2N3V3NZ3ee+7oVbXTwUQ= 2024-04-12T03:58:32.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rck4B8o2N3V3NZ3ee+7oVbXTwUQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e18bf99dd67e15-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.031 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.031 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.031 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.031 DEBUG:boto:Path: /dgbpxg-62/obj44 2024-04-12T03:58:32.031 DEBUG:boto:Data: 2024-04-12T03:58:32.032 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.032 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.032 DEBUG:boto:Params: {} 2024-04-12T03:58:32.032 DEBUG:boto:Token: None 2024-04-12T03:58:32.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44 2024-04-12T03:58:32.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rck4B8o2N3V3NZ3ee+7oVbXTwUQ= 2024-04-12T03:58:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rck4B8o2N3V3NZ3ee+7oVbXTwUQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001cd9b97bc0dc3ca-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T03:58:32.034 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.034 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.034 DEBUG:boto:Method: GET 2024-04-12T03:58:32.034 DEBUG:boto:Path: /dgbpxg-62/obj44 2024-04-12T03:58:32.035 DEBUG:boto:Data: 2024-04-12T03:58:32.035 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.035 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.035 DEBUG:boto:Params: {} 2024-04-12T03:58:32.035 DEBUG:boto:Token: None 2024-04-12T03:58:32.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44 2024-04-12T03:58:32.035 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lzmrW+95USwbb93x3GstQ0w/C1Q= 2024-04-12T03:58:32.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lzmrW+95USwbb93x3GstQ0w/C1Q=', 'Content-Length': '0'} 2024-04-12T03:58:32.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f634902410f7202-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.078 DEBUG:boto:path=/dgbpxg-62/obj44 2024-04-12T03:58:32.079 DEBUG:boto:auth_path=/dgbpxg-62/obj44 2024-04-12T03:58:32.079 DEBUG:boto:Method: GET 2024-04-12T03:58:32.079 DEBUG:boto:Path: /dgbpxg-62/obj44 2024-04-12T03:58:32.079 DEBUG:boto:Data: 2024-04-12T03:58:32.079 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.079 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.079 DEBUG:boto:Params: {} 2024-04-12T03:58:32.079 DEBUG:boto:Token: None 2024-04-12T03:58:32.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj44 2024-04-12T03:58:32.080 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lzmrW+95USwbb93x3GstQ0w/C1Q= 2024-04-12T03:58:32.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lzmrW+95USwbb93x3GstQ0w/C1Q=', 'Content-Length': '0'} 2024-04-12T03:58:32.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf2cf193e2b6038-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.081 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.082 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.082 DEBUG:boto:path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.082 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.082 DEBUG:boto:Method: GET 2024-04-12T03:58:32.082 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.082 DEBUG:boto:Data: 2024-04-12T03:58:32.082 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.082 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.082 DEBUG:boto:Params: {} 2024-04-12T03:58:32.083 DEBUG:boto:Token: None 2024-04-12T03:58:32.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd92a61df768c206-006618b168-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.085 DEBUG:boto:b'dgbpxg-62obj441000falseobj442024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.086 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.086 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.086 DEBUG:boto:path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.086 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.086 DEBUG:boto:Method: GET 2024-04-12T03:58:32.086 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj44 2024-04-12T03:58:32.086 DEBUG:boto:Data: 2024-04-12T03:58:32.086 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.087 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.087 DEBUG:boto:Params: {} 2024-04-12T03:58:32.087 DEBUG:boto:Token: None 2024-04-12T03:58:32.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.087 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa93e7341e1485d7-006618b168-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.089 DEBUG:boto:b'dgbpxg-62obj441000falseobj442024-04-12T03:55:55.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:58:32.090 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.090 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.090 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.090 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.090 DEBUG:boto:Method: GET 2024-04-12T03:58:32.090 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.090 DEBUG:boto:Data: 2024-04-12T03:58:32.091 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.091 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.091 DEBUG:boto:Params: {} 2024-04-12T03:58:32.091 DEBUG:boto:Token: None 2024-04-12T03:58:32.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.091 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk= 2024-04-12T03:58:32.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk=', 'Content-Length': '0'} 2024-04-12T03:58:32.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000962d9250bb0e2400-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.093 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.093 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.094 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.094 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.094 DEBUG:boto:Method: GET 2024-04-12T03:58:32.094 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.094 DEBUG:boto:Data: 2024-04-12T03:58:32.094 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.094 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.094 DEBUG:boto:Params: {} 2024-04-12T03:58:32.094 DEBUG:boto:Token: None 2024-04-12T03:58:32.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.095 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk= 2024-04-12T03:58:32.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk=', 'Content-Length': '0'} 2024-04-12T03:58:32.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342cf4531605d1ce-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.097 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.097 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.097 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.097 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.097 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.097 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.097 DEBUG:boto:Data: 2024-04-12T03:58:32.097 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.098 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.098 DEBUG:boto:Params: {} 2024-04-12T03:58:32.098 DEBUG:boto:Token: None 2024-04-12T03:58:32.098 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0fx9tEBRDPU3/9pxJUHwiToucy8= 2024-04-12T03:58:32.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0fx9tEBRDPU3/9pxJUHwiToucy8=', 'Content-Length': '0'} 2024-04-12T03:58:32.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029bfc2a63209e41a-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.100 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.100 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.100 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.100 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.101 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.101 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.101 DEBUG:boto:Data: 2024-04-12T03:58:32.101 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.101 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.101 DEBUG:boto:Params: {} 2024-04-12T03:58:32.101 DEBUG:boto:Token: None 2024-04-12T03:58:32.101 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0fx9tEBRDPU3/9pxJUHwiToucy8= 2024-04-12T03:58:32.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0fx9tEBRDPU3/9pxJUHwiToucy8=', 'Content-Length': '0'} 2024-04-12T03:58:32.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009106aab8554c2322-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:58:32.104 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.104 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.104 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.104 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.104 DEBUG:boto:Method: GET 2024-04-12T03:58:32.104 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.104 DEBUG:boto:Data: 2024-04-12T03:58:32.104 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.105 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.105 DEBUG:boto:Params: {} 2024-04-12T03:58:32.105 DEBUG:boto:Token: None 2024-04-12T03:58:32.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk= 2024-04-12T03:58:32.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk=', 'Content-Length': '0'} 2024-04-12T03:58:32.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029cf11be0bf0091a-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.107 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.107 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.107 DEBUG:boto:path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.107 DEBUG:boto:auth_path=/dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.107 DEBUG:boto:Method: GET 2024-04-12T03:58:32.108 DEBUG:boto:Path: /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.108 DEBUG:boto:Data: 2024-04-12T03:58:32.108 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.108 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.108 DEBUG:boto:Params: {} 2024-04-12T03:58:32.108 DEBUG:boto:Token: None 2024-04-12T03:58:32.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55?versionId=null 2024-04-12T03:58:32.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk= 2024-04-12T03:58:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IAxSxxI4Z2LlvI85dGqhnJfjadk=', 'Content-Length': '0'} 2024-04-12T03:58:32.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000625f9e805dc0fe25-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.110 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.111 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.111 DEBUG:boto:path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.111 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.111 DEBUG:boto:Method: GET 2024-04-12T03:58:32.111 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.111 DEBUG:boto:Data: 2024-04-12T03:58:32.111 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.111 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.111 DEBUG:boto:Params: {} 2024-04-12T03:58:32.112 DEBUG:boto:Token: None 2024-04-12T03:58:32.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.112 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097aad2b8c137fb97-006618b168-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.114 DEBUG:boto:b'dgbpxg-62obj551000falseobj552024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.115 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.118 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.118 DEBUG:boto:path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.118 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.118 DEBUG:boto:Method: GET 2024-04-12T03:58:32.118 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.118 DEBUG:boto:Data: 2024-04-12T03:58:32.118 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.119 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.119 DEBUG:boto:Params: {} 2024-04-12T03:58:32.119 DEBUG:boto:Token: None 2024-04-12T03:58:32.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec62b9e5d9ae3317-006618b168-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.122 DEBUG:boto:b'dgbpxg-62obj551000falseobj552024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.122 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.122 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.122 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.122 DEBUG:boto:Path: /dgbpxg-62/obj55 2024-04-12T03:58:32.122 DEBUG:boto:Data: 2024-04-12T03:58:32.123 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.123 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.123 DEBUG:boto:Params: {} 2024-04-12T03:58:32.123 DEBUG:boto:Token: None 2024-04-12T03:58:32.123 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55 2024-04-12T03:58:32.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qH+kbOv//TFrUCeW9x2rH3Nino4= 2024-04-12T03:58:32.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qH+kbOv//TFrUCeW9x2rH3Nino4=', 'Content-Length': '0'} 2024-04-12T03:58:32.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b973eae16cb61519-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.125 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.125 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.125 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.125 DEBUG:boto:Path: /dgbpxg-62/obj55 2024-04-12T03:58:32.126 DEBUG:boto:Data: 2024-04-12T03:58:32.126 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.126 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.126 DEBUG:boto:Params: {} 2024-04-12T03:58:32.126 DEBUG:boto:Token: None 2024-04-12T03:58:32.126 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55 2024-04-12T03:58:32.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qH+kbOv//TFrUCeW9x2rH3Nino4= 2024-04-12T03:58:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qH+kbOv//TFrUCeW9x2rH3Nino4=', 'Content-Length': '0'} 2024-04-12T03:58:32.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351841fe42c908d3-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T03:58:32.128 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.128 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.128 DEBUG:boto:Method: GET 2024-04-12T03:58:32.128 DEBUG:boto:Path: /dgbpxg-62/obj55 2024-04-12T03:58:32.129 DEBUG:boto:Data: 2024-04-12T03:58:32.129 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.129 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.129 DEBUG:boto:Params: {} 2024-04-12T03:58:32.129 DEBUG:boto:Token: None 2024-04-12T03:58:32.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55 2024-04-12T03:58:32.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9FgSj68H37NT46uwQFuyPOgjur8= 2024-04-12T03:58:32.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9FgSj68H37NT46uwQFuyPOgjur8=', 'Content-Length': '0'} 2024-04-12T03:58:32.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-62:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1ab23deeed63ee4-006618b168-21a9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.131 DEBUG:boto:path=/dgbpxg-62/obj55 2024-04-12T03:58:32.132 DEBUG:boto:auth_path=/dgbpxg-62/obj55 2024-04-12T03:58:32.132 DEBUG:boto:Method: GET 2024-04-12T03:58:32.132 DEBUG:boto:Path: /dgbpxg-62/obj55 2024-04-12T03:58:32.132 DEBUG:boto:Data: 2024-04-12T03:58:32.132 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.132 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.132 DEBUG:boto:Params: {} 2024-04-12T03:58:32.132 DEBUG:boto:Token: None 2024-04-12T03:58:32.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/obj55 2024-04-12T03:58:32.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9FgSj68H37NT46uwQFuyPOgjur8= 2024-04-12T03:58:32.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9FgSj68H37NT46uwQFuyPOgjur8=', 'Content-Length': '0'} 2024-04-12T03:58:32.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d647345e644514-006618b168-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.135 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.135 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.135 DEBUG:boto:path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.135 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.135 DEBUG:boto:Method: GET 2024-04-12T03:58:32.135 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.135 DEBUG:boto:Data: 2024-04-12T03:58:32.136 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:32.136 DEBUG:boto:Port: 8001 2024-04-12T03:58:32.136 DEBUG:boto:Params: {} 2024-04-12T03:58:32.136 DEBUG:boto:Token: None 2024-04-12T03:58:32.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a420bad3b112f12d-006618b168-21a9-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.182 DEBUG:boto:b'dgbpxg-62obj551000falseobj552024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.183 DEBUG:boto:path=/dgbpxg-62/ 2024-04-12T03:58:32.183 DEBUG:boto:auth_path=/dgbpxg-62/ 2024-04-12T03:58:32.183 DEBUG:boto:path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.183 DEBUG:boto:auth_path=/dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.183 DEBUG:boto:Method: GET 2024-04-12T03:58:32.183 DEBUG:boto:Path: /dgbpxg-62/?prefix=obj55 2024-04-12T03:58:32.183 DEBUG:boto:Data: 2024-04-12T03:58:32.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.184 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.184 DEBUG:boto:Params: {} 2024-04-12T03:58:32.184 DEBUG:boto:Token: None 2024-04-12T03:58:32.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-62/ 2024-04-12T03:58:32.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ= 2024-04-12T03:58:32.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eF8WCnYI/ecy6QZydrdLIhi8KxQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0818a5a8f41b405-006618b168-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.187 DEBUG:boto:b'dgbpxg-62obj551000falseobj552024-04-12T03:55:55.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:32.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-62 zones={a2, a1} 2024-04-12T03:58:32.187 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:58:32.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.188 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.236 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.285 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000027953", 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:55:59.128164Z" 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000019599", 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:51:42.686320Z" 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032853", 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:55:55.180969Z" 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000029317", 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:55:59.129475Z" 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.287 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:58:32.288 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000027953', 1: '00000000000000000000:00000000000000019599', 2: '00000000000000000000:00000000000000032853', 3: '00000000000000000000:00000000000000029317'} 2024-04-12T03:58:32.288 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:58:32.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.289 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 7914148596058306417 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000027953", 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:55:59.128164Z" 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:58:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000019599", 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:51:42.686320Z" 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032853", 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 53, 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:55:55.180969Z" 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:58:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000029317", 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:55:59.129475Z" 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 62, 2024-04-12T03:58:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 62 2024-04-12T03:58:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:58:32Z" 2024-04-12T03:58:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:58:32.387 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7914148596058306417 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027953", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:59.128164Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019599", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:51:42.686320Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032853", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2024-04-12T03:55:55.180969Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000029317", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T03:55:59.129475Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2024-04-12T03:58:32Z" } 2024-04-12T03:58:32.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027953', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:59.128164Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019599', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:51:42.686320Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032853', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2024-04-12T03:55:55.180969Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000029317', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T03:55:59.129475Z'}}] 2024-04-12T03:58:32.388 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000027953', 1: '00000000000000000000:00000000000000019599', 2: '00000000000000000000:00000000000000032853', 3: '00000000000000000000:00000000000000029317'} 2024-04-12T03:58:32.388 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000027953', 1: '00000000000000000000:00000000000000019599', 2: '00000000000000000000:00000000000000032853', 3: '00000000000000000000:00000000000000029317'} 2024-04-12T03:58:32.388 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:58:32.388 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:58:32.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.388 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.484 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:58:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000011817", 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:57:29.576367Z" 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006009", 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:53:54.513070Z" 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015588", 2024-04-12T03:58:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:55:58.802711Z" 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015450", 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:56:17.417714Z" 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:58:32.487 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000011817', 1: '00000000000000000000:00000000000000006009', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T03:58:32.487 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:58:32.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.488 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.537 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.537 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:58:32.580 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000011817", 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.581 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:57:29.576367Z" 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006009", 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:53:54.513070Z" 2024-04-12T03:58:32.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015588", 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:55:58.802711Z" 2024-04-12T03:58:32.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015450", 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:56:17.417714Z" 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:58:32Z" 2024-04-12T03:58:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:58:32.586 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011817", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:29.576367Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006009", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:53:54.513070Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:58.802711Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015450", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:56:17.417714Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:58:32Z" } 2024-04-12T03:58:32.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011817', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:29.576367Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006009', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:53:54.513070Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:58.802711Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015450', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:56:17.417714Z'}}] 2024-04-12T03:58:32.586 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011817', 1: '00000000000000000000:00000000000000006009', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T03:58:32.587 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011817', 1: '00000000000000000000:00000000000000006009', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T03:58:32.587 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:58:32.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2024-04-12T03:58:32.588 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:32.589 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:32.589 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.589 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:32.589 DEBUG:boto:Data: 2024-04-12T03:58:32.589 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:32.589 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.589 DEBUG:boto:Params: {} 2024-04-12T03:58:32.589 DEBUG:boto:Token: None 2024-04-12T03:58:32.590 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:58:32 GMT /dgbpxg-63/ 2024-04-12T03:58:32.590 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gjGPjMXxoCr64QtbNVchHmZmhoQ= 2024-04-12T03:58:32.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gjGPjMXxoCr64QtbNVchHmZmhoQ=', 'Content-Length': '0'} 2024-04-12T03:58:32.605 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003feaf958378521f1-006618b168-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.605 DEBUG:rgw_multi.tests:created bucket=dgbpxg-63 2024-04-12T03:58:32.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.606 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.683 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T03:58:32.696 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T03:58:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:58:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T03:58:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894312.617262_1465.1", 2024-04-12T03:58:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:32.617262Z" 2024-04-12T03:58:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T03:58:32.699 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894312.617262_1465.1'} 2024-04-12T03:58:32.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:58:32.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.699 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:32.796+0000 7f4aa0143780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:32.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:32.796+0000 7f4aa0143780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T03:58:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T03:58:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T03:58:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T03:58:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:58:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:58:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T03:58:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894312.617262_1465.1", 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:32.617262Z", 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T03:58:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T03:58:32Z" 2024-04-12T03:58:32.803 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T03:58:32.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894312.617262_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:32.617262Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:58:32Z" } 2024-04-12T03:58:32.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894312.617262_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:32.617262Z', 'realm_epoch': 2}}] 2024-04-12T03:58:32.804 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894312.617262_1465.1'} 2024-04-12T03:58:32.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894312.617262_1465.1'} 2024-04-12T03:58:32.805 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:58:32.805 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:58:32.805 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:58:32.806 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:58:32.818+0000 7fb5307a7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 108410) UID: 0 2024-04-12T03:58:32.806 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T03:58:32.818+0000 7fb53597c7c0 -1 shutting down 2024-04-12T03:58:34.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:58:38.907 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:58:38.907 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:38.907 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:38.907 DEBUG:boto:Method: HEAD 2024-04-12T03:58:38.908 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:38.908 DEBUG:boto:Data: 2024-04-12T03:58:38.908 DEBUG:boto:Headers: {} 2024-04-12T03:58:38.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.908 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.908 DEBUG:boto:Params: {} 2024-04-12T03:58:38.908 DEBUG:boto:Token: None 2024-04-12T03:58:38.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/ 2024-04-12T03:58:38.909 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ= 2024-04-12T03:58:38.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ=', 'Content-Length': '0'} 2024-04-12T03:58:38.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6fc72b113ad89a4-006618b16e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.911 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T03:58:38.912 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T03:58:38.912 DEBUG:boto:Method: PUT 2024-04-12T03:58:38.912 DEBUG:boto:Path: /dgbpxg-63/obj11 2024-04-12T03:58:38.912 DEBUG:boto:Data: 2024-04-12T03:58:38.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:38.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.912 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.912 DEBUG:boto:Params: {} 2024-04-12T03:58:38.912 DEBUG:boto:Token: None 2024-04-12T03:58:38.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/obj11 2024-04-12T03:58:38.913 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z8MuvVOaxWa27KiYVKrX9Ua7XIA= 2024-04-12T03:58:38.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z8MuvVOaxWa27KiYVKrX9Ua7XIA='} 2024-04-12T03:58:38.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ed71e745ee718f-006618b16e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.920 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:38.921 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:38.921 DEBUG:boto:Method: HEAD 2024-04-12T03:58:38.921 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:38.921 DEBUG:boto:Data: 2024-04-12T03:58:38.921 DEBUG:boto:Headers: {} 2024-04-12T03:58:38.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.921 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.921 DEBUG:boto:Params: {} 2024-04-12T03:58:38.921 DEBUG:boto:Token: None 2024-04-12T03:58:38.921 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/ 2024-04-12T03:58:38.922 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ= 2024-04-12T03:58:38.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ=', 'Content-Length': '0'} 2024-04-12T03:58:38.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6bf6183823be746-006618b16e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.924 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T03:58:38.924 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T03:58:38.924 DEBUG:boto:Method: PUT 2024-04-12T03:58:38.924 DEBUG:boto:Path: /dgbpxg-63/obj22 2024-04-12T03:58:38.925 DEBUG:boto:Data: 2024-04-12T03:58:38.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:38.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.925 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.925 DEBUG:boto:Params: {} 2024-04-12T03:58:38.925 DEBUG:boto:Token: None 2024-04-12T03:58:38.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/obj22 2024-04-12T03:58:38.925 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gshnm746lSNQjAwL49lupZtLUjk= 2024-04-12T03:58:38.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gshnm746lSNQjAwL49lupZtLUjk='} 2024-04-12T03:58:38.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc230758cee67f7d-006618b16e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.971 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:38.971 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:38.971 DEBUG:boto:Method: HEAD 2024-04-12T03:58:38.971 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:38.971 DEBUG:boto:Data: 2024-04-12T03:58:38.971 DEBUG:boto:Headers: {} 2024-04-12T03:58:38.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.971 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.971 DEBUG:boto:Params: {} 2024-04-12T03:58:38.971 DEBUG:boto:Token: None 2024-04-12T03:58:38.972 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/ 2024-04-12T03:58:38.972 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ= 2024-04-12T03:58:38.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ=', 'Content-Length': '0'} 2024-04-12T03:58:38.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d920349a3fd842-006618b16e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.974 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T03:58:38.975 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T03:58:38.975 DEBUG:boto:Method: PUT 2024-04-12T03:58:38.975 DEBUG:boto:Path: /dgbpxg-63/obj33 2024-04-12T03:58:38.975 DEBUG:boto:Data: 2024-04-12T03:58:38.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:38.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.975 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.975 DEBUG:boto:Params: {} 2024-04-12T03:58:38.975 DEBUG:boto:Token: None 2024-04-12T03:58:38.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/obj33 2024-04-12T03:58:38.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByPe3vdxUH9mcDTiNmrhFgG6acs= 2024-04-12T03:58:38.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByPe3vdxUH9mcDTiNmrhFgG6acs='} 2024-04-12T03:58:38.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e82de64c9a25924-006618b16e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.982 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:38.982 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:38.983 DEBUG:boto:Method: HEAD 2024-04-12T03:58:38.983 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:38.983 DEBUG:boto:Data: 2024-04-12T03:58:38.983 DEBUG:boto:Headers: {} 2024-04-12T03:58:38.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.983 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.983 DEBUG:boto:Params: {} 2024-04-12T03:58:38.983 DEBUG:boto:Token: None 2024-04-12T03:58:38.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/ 2024-04-12T03:58:38.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ= 2024-04-12T03:58:38.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IXYWd5lAbfuZ7HQzmv8wVxTv3EQ=', 'Content-Length': '0'} 2024-04-12T03:58:38.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399716b6f64cf444-006618b16e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:38.987 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T03:58:38.987 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T03:58:38.987 DEBUG:boto:Method: PUT 2024-04-12T03:58:38.987 DEBUG:boto:Path: /dgbpxg-63/obj44 2024-04-12T03:58:38.987 DEBUG:boto:Data: 2024-04-12T03:58:38.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:38.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:38.987 DEBUG:boto:Port: 8000 2024-04-12T03:58:38.987 DEBUG:boto:Params: {} 2024-04-12T03:58:38.988 DEBUG:boto:Token: None 2024-04-12T03:58:38.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:38 GMT /dgbpxg-63/obj44 2024-04-12T03:58:38.988 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3HigELibykPj5iau9pPe94Nej9I= 2024-04-12T03:58:38.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3HigELibykPj5iau9pPe94Nej9I='} 2024-04-12T03:58:39.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021670edee466cd5f-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.031 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.031 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.031 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.031 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.031 DEBUG:boto:Data: 2024-04-12T03:58:39.031 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.031 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.032 DEBUG:boto:Params: {} 2024-04-12T03:58:39.032 DEBUG:boto:Token: None 2024-04-12T03:58:39.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.032 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c2b65a1598f70ea-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.035 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T03:58:39.035 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T03:58:39.035 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.035 DEBUG:boto:Path: /dgbpxg-63/obj55 2024-04-12T03:58:39.035 DEBUG:boto:Data: 2024-04-12T03:58:39.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.035 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.035 DEBUG:boto:Params: {} 2024-04-12T03:58:39.035 DEBUG:boto:Token: None 2024-04-12T03:58:39.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj55 2024-04-12T03:58:39.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:J3k/Mn0Gnl7ERBcU2rV8WoithLI= 2024-04-12T03:58:39.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:J3k/Mn0Gnl7ERBcU2rV8WoithLI='} 2024-04-12T03:58:39.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c32d5f6ba2f4f043-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:39.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:39.141 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:39.142 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:39.146 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2024-04-12T03:58:39.184 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.185 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.185 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.185 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.185 DEBUG:boto:Data: 2024-04-12T03:58:39.185 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.185 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.185 DEBUG:boto:Params: {} 2024-04-12T03:58:39.186 DEBUG:boto:Token: None 2024-04-12T03:58:39.186 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.186 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c31b54f16277ca5-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.189 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T03:58:39.189 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T03:58:39.189 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.189 DEBUG:boto:Path: /dgbpxg-63/obj17 2024-04-12T03:58:39.189 DEBUG:boto:Data: 2024-04-12T03:58:39.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.190 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.190 DEBUG:boto:Params: {} 2024-04-12T03:58:39.190 DEBUG:boto:Token: None 2024-04-12T03:58:39.190 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj17 2024-04-12T03:58:39.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s6kBD9Q1TgCF+o0dTAxq2CY0G20= 2024-04-12T03:58:39.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s6kBD9Q1TgCF+o0dTAxq2CY0G20='} 2024-04-12T03:58:39.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a8a7191042470f9-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.195 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.196 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.196 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.196 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.196 DEBUG:boto:Data: 2024-04-12T03:58:39.196 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.196 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.196 DEBUG:boto:Params: {} 2024-04-12T03:58:39.196 DEBUG:boto:Token: None 2024-04-12T03:58:39.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcac30d2a6c5eb4e-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.200 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T03:58:39.200 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T03:58:39.200 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.200 DEBUG:boto:Path: /dgbpxg-63/obj34 2024-04-12T03:58:39.200 DEBUG:boto:Data: 2024-04-12T03:58:39.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.200 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.200 DEBUG:boto:Params: {} 2024-04-12T03:58:39.201 DEBUG:boto:Token: None 2024-04-12T03:58:39.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj34 2024-04-12T03:58:39.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GhLTM5f6IerfYIsZkxVd+EgCnqk= 2024-04-12T03:58:39.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GhLTM5f6IerfYIsZkxVd+EgCnqk='} 2024-04-12T03:58:39.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000288b7cfd19484504-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.247 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.247 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.247 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.247 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.247 DEBUG:boto:Data: 2024-04-12T03:58:39.247 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.247 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.247 DEBUG:boto:Params: {} 2024-04-12T03:58:39.247 DEBUG:boto:Token: None 2024-04-12T03:58:39.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce6a7a31a7520b06-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.250 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T03:58:39.250 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T03:58:39.251 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.251 DEBUG:boto:Path: /dgbpxg-63/obj51 2024-04-12T03:58:39.251 DEBUG:boto:Data: 2024-04-12T03:58:39.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.251 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.251 DEBUG:boto:Params: {} 2024-04-12T03:58:39.251 DEBUG:boto:Token: None 2024-04-12T03:58:39.251 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj51 2024-04-12T03:58:39.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rrNWFthnnLvHtI2KG/50aYDSyoE= 2024-04-12T03:58:39.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rrNWFthnnLvHtI2KG/50aYDSyoE='} 2024-04-12T03:58:39.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000089f572a87aa3c47-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.256 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.256 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.256 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.257 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.257 DEBUG:boto:Data: 2024-04-12T03:58:39.257 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.257 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.257 DEBUG:boto:Params: {} 2024-04-12T03:58:39.257 DEBUG:boto:Token: None 2024-04-12T03:58:39.257 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c02c01e291b2c894-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.260 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T03:58:39.260 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T03:58:39.260 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.260 DEBUG:boto:Path: /dgbpxg-63/obj68 2024-04-12T03:58:39.260 DEBUG:boto:Data: 2024-04-12T03:58:39.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.261 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.261 DEBUG:boto:Params: {} 2024-04-12T03:58:39.261 DEBUG:boto:Token: None 2024-04-12T03:58:39.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj68 2024-04-12T03:58:39.261 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:M+o5695bjOHLPbPEnOAWVvT2p0I= 2024-04-12T03:58:39.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:M+o5695bjOHLPbPEnOAWVvT2p0I='} 2024-04-12T03:58:39.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006242bab80cd7ce38-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.307 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.307 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.307 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.307 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.307 DEBUG:boto:Data: 2024-04-12T03:58:39.307 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.308 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.308 DEBUG:boto:Params: {} 2024-04-12T03:58:39.308 DEBUG:boto:Token: None 2024-04-12T03:58:39.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.308 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c2f5d508271db63-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.311 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T03:58:39.311 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T03:58:39.311 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.312 DEBUG:boto:Path: /dgbpxg-63/obj85 2024-04-12T03:58:39.312 DEBUG:boto:Data: 2024-04-12T03:58:39.312 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.312 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.312 DEBUG:boto:Params: {} 2024-04-12T03:58:39.312 DEBUG:boto:Token: None 2024-04-12T03:58:39.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj85 2024-04-12T03:58:39.313 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zz3Reo5POVyz/IKLdUqe2N/6Q54= 2024-04-12T03:58:39.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zz3Reo5POVyz/IKLdUqe2N/6Q54='} 2024-04-12T03:58:39.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000354b964515c07f8-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.318 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:39.417 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:39.417 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:39.423 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2024-04-12T03:58:39.468 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.468 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.468 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.468 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.468 DEBUG:boto:Data: 2024-04-12T03:58:39.468 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.468 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.468 DEBUG:boto:Params: {} 2024-04-12T03:58:39.468 DEBUG:boto:Token: None 2024-04-12T03:58:39.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f75ddbec88d34e8-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.472 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T03:58:39.472 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T03:58:39.472 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.472 DEBUG:boto:Path: /dgbpxg-63/obj19 2024-04-12T03:58:39.472 DEBUG:boto:Data: 2024-04-12T03:58:39.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.473 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.473 DEBUG:boto:Params: {} 2024-04-12T03:58:39.473 DEBUG:boto:Token: None 2024-04-12T03:58:39.473 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj19 2024-04-12T03:58:39.473 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sd9tYdEzaUK5m12LcAmVoNs6JSk= 2024-04-12T03:58:39.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sd9tYdEzaUK5m12LcAmVoNs6JSk='} 2024-04-12T03:58:39.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077efcfa8e297451e-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.519 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.519 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.519 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.519 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.519 DEBUG:boto:Data: 2024-04-12T03:58:39.519 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.519 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.520 DEBUG:boto:Params: {} 2024-04-12T03:58:39.520 DEBUG:boto:Token: None 2024-04-12T03:58:39.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.520 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004da15115fd210895-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.523 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T03:58:39.523 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T03:58:39.523 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.523 DEBUG:boto:Path: /dgbpxg-63/obj38 2024-04-12T03:58:39.523 DEBUG:boto:Data: 2024-04-12T03:58:39.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.524 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.524 DEBUG:boto:Params: {} 2024-04-12T03:58:39.524 DEBUG:boto:Token: None 2024-04-12T03:58:39.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj38 2024-04-12T03:58:39.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gu9QYn4/Azh1YWWyW/WPh3M/GfQ= 2024-04-12T03:58:39.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gu9QYn4/Azh1YWWyW/WPh3M/GfQ='} 2024-04-12T03:58:39.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000de74d3be1625741-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.528 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.528 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.528 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.528 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.528 DEBUG:boto:Data: 2024-04-12T03:58:39.529 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.529 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.529 DEBUG:boto:Params: {} 2024-04-12T03:58:39.529 DEBUG:boto:Token: None 2024-04-12T03:58:39.529 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000111aafe299b8122a-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.532 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T03:58:39.532 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T03:58:39.532 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.532 DEBUG:boto:Path: /dgbpxg-63/obj57 2024-04-12T03:58:39.532 DEBUG:boto:Data: 2024-04-12T03:58:39.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.532 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.533 DEBUG:boto:Params: {} 2024-04-12T03:58:39.533 DEBUG:boto:Token: None 2024-04-12T03:58:39.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj57 2024-04-12T03:58:39.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PjtHSAOx3SU4psmMJVOkgxl/YCE= 2024-04-12T03:58:39.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PjtHSAOx3SU4psmMJVOkgxl/YCE='} 2024-04-12T03:58:39.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000243d4f48594b6b47-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.579 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.579 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.579 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.579 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.579 DEBUG:boto:Data: 2024-04-12T03:58:39.579 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.579 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.579 DEBUG:boto:Params: {} 2024-04-12T03:58:39.580 DEBUG:boto:Token: None 2024-04-12T03:58:39.580 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de72c28386b93ad2-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.583 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T03:58:39.583 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T03:58:39.583 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.583 DEBUG:boto:Path: /dgbpxg-63/obj76 2024-04-12T03:58:39.583 DEBUG:boto:Data: 2024-04-12T03:58:39.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.583 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.584 DEBUG:boto:Params: {} 2024-04-12T03:58:39.584 DEBUG:boto:Token: None 2024-04-12T03:58:39.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj76 2024-04-12T03:58:39.584 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lJvZ5WwfJb2BV6At4ZD6Qcluna0= 2024-04-12T03:58:39.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lJvZ5WwfJb2BV6At4ZD6Qcluna0='} 2024-04-12T03:58:39.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dec660fa68fbace8-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.589 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.589 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.589 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.589 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.589 DEBUG:boto:Data: 2024-04-12T03:58:39.590 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.590 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.590 DEBUG:boto:Params: {} 2024-04-12T03:58:39.590 DEBUG:boto:Token: None 2024-04-12T03:58:39.590 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.590 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc184300c950b602-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.593 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T03:58:39.593 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T03:58:39.593 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.593 DEBUG:boto:Path: /dgbpxg-63/obj95 2024-04-12T03:58:39.593 DEBUG:boto:Data: 2024-04-12T03:58:39.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.593 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.593 DEBUG:boto:Params: {} 2024-04-12T03:58:39.593 DEBUG:boto:Token: None 2024-04-12T03:58:39.593 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj95 2024-04-12T03:58:39.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xq8Begv8vNRwKBPar8O3arXxoFw= 2024-04-12T03:58:39.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xq8Begv8vNRwKBPar8O3arXxoFw='} 2024-04-12T03:58:39.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c84ce8bf41586227-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.643 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:39.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:58:39.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:39.739 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:39.739 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:39.746 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2024-04-12T03:58:39.792 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.792 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.793 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.793 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.793 DEBUG:boto:Data: 2024-04-12T03:58:39.793 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.793 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.793 DEBUG:boto:Params: {} 2024-04-12T03:58:39.793 DEBUG:boto:Token: None 2024-04-12T03:58:39.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.794 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096820b930d85f76a-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.797 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T03:58:39.797 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T03:58:39.797 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.797 DEBUG:boto:Path: /dgbpxg-63/obj23 2024-04-12T03:58:39.797 DEBUG:boto:Data: 2024-04-12T03:58:39.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.798 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.798 DEBUG:boto:Params: {} 2024-04-12T03:58:39.798 DEBUG:boto:Token: None 2024-04-12T03:58:39.798 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj23 2024-04-12T03:58:39.798 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VxUackudhhxjFdQhpM+e2JPfOug= 2024-04-12T03:58:39.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VxUackudhhxjFdQhpM+e2JPfOug='} 2024-04-12T03:58:39.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd0314abf67b70e8-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.803 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.804 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.804 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.804 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.804 DEBUG:boto:Data: 2024-04-12T03:58:39.804 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.804 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.804 DEBUG:boto:Params: {} 2024-04-12T03:58:39.804 DEBUG:boto:Token: None 2024-04-12T03:58:39.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.805 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f44f7af04ae667c1-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.807 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T03:58:39.807 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T03:58:39.808 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.808 DEBUG:boto:Path: /dgbpxg-63/obj46 2024-04-12T03:58:39.808 DEBUG:boto:Data: 2024-04-12T03:58:39.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.808 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.808 DEBUG:boto:Params: {} 2024-04-12T03:58:39.808 DEBUG:boto:Token: None 2024-04-12T03:58:39.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj46 2024-04-12T03:58:39.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iH3rOwtKeaZKT2cLFUxcd5GvG40= 2024-04-12T03:58:39.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iH3rOwtKeaZKT2cLFUxcd5GvG40='} 2024-04-12T03:58:39.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003149d8c189446249-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.813 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.813 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.813 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.813 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.813 DEBUG:boto:Data: 2024-04-12T03:58:39.813 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.814 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.814 DEBUG:boto:Params: {} 2024-04-12T03:58:39.814 DEBUG:boto:Token: None 2024-04-12T03:58:39.814 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.814 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af2409931153be54-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.817 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T03:58:39.817 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T03:58:39.817 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.817 DEBUG:boto:Path: /dgbpxg-63/obj69 2024-04-12T03:58:39.817 DEBUG:boto:Data: 2024-04-12T03:58:39.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.817 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.818 DEBUG:boto:Params: {} 2024-04-12T03:58:39.818 DEBUG:boto:Token: None 2024-04-12T03:58:39.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj69 2024-04-12T03:58:39.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XAN8Hj0xvRJ+Ko3Q0U5XqjGZgdk= 2024-04-12T03:58:39.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XAN8Hj0xvRJ+Ko3Q0U5XqjGZgdk='} 2024-04-12T03:58:39.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037e98827aa2e9036-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.863 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.863 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.863 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.863 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.863 DEBUG:boto:Data: 2024-04-12T03:58:39.863 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.863 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.863 DEBUG:boto:Params: {} 2024-04-12T03:58:39.864 DEBUG:boto:Token: None 2024-04-12T03:58:39.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093d47f9978bad20c-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.867 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T03:58:39.867 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T03:58:39.867 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.867 DEBUG:boto:Path: /dgbpxg-63/obj92 2024-04-12T03:58:39.867 DEBUG:boto:Data: 2024-04-12T03:58:39.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.867 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.867 DEBUG:boto:Params: {} 2024-04-12T03:58:39.868 DEBUG:boto:Token: None 2024-04-12T03:58:39.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj92 2024-04-12T03:58:39.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hUjN/yKFAnUhNeDp8nzdPpXFa+o= 2024-04-12T03:58:39.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hUjN/yKFAnUhNeDp8nzdPpXFa+o='} 2024-04-12T03:58:39.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a3e6fb7a938e6c3-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.873 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:39.873 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:39.874 DEBUG:boto:Method: HEAD 2024-04-12T03:58:39.874 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:39.874 DEBUG:boto:Data: 2024-04-12T03:58:39.874 DEBUG:boto:Headers: {} 2024-04-12T03:58:39.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.874 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.874 DEBUG:boto:Params: {} 2024-04-12T03:58:39.874 DEBUG:boto:Token: None 2024-04-12T03:58:39.874 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/ 2024-04-12T03:58:39.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA= 2024-04-12T03:58:39.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+aQXolERY/iNyPkLr5GZkJ3PkaA=', 'Content-Length': '0'} 2024-04-12T03:58:39.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026af9b1c0b22d512-006618b16f-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.877 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T03:58:39.877 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T03:58:39.877 DEBUG:boto:Method: PUT 2024-04-12T03:58:39.878 DEBUG:boto:Path: /dgbpxg-63/obj115 2024-04-12T03:58:39.878 DEBUG:boto:Data: 2024-04-12T03:58:39.878 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:39.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:39.878 DEBUG:boto:Port: 8000 2024-04-12T03:58:39.878 DEBUG:boto:Params: {} 2024-04-12T03:58:39.878 DEBUG:boto:Token: None 2024-04-12T03:58:39.878 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:39 GMT /dgbpxg-63/obj115 2024-04-12T03:58:39.878 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zjuJ+RdTs3hQwsqFdqj7xcLxCDA= 2024-04-12T03:58:39.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zjuJ+RdTs3hQwsqFdqj7xcLxCDA='} 2024-04-12T03:58:39.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091a8a39e667bc844-006618b16f-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:39.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:39.927 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:40.025 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:40.025 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:40.033 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2024-04-12T03:58:40.084 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.084 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.085 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.085 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.085 DEBUG:boto:Data: 2024-04-12T03:58:40.085 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.085 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.085 DEBUG:boto:Params: {} 2024-04-12T03:58:40.085 DEBUG:boto:Token: None 2024-04-12T03:58:40.085 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.085 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fda4cab6a5621435-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.089 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T03:58:40.089 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T03:58:40.089 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.089 DEBUG:boto:Path: /dgbpxg-63/obj29 2024-04-12T03:58:40.089 DEBUG:boto:Data: 2024-04-12T03:58:40.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.090 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.090 DEBUG:boto:Params: {} 2024-04-12T03:58:40.090 DEBUG:boto:Token: None 2024-04-12T03:58:40.090 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj29 2024-04-12T03:58:40.090 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q6I9qULx2H8Yli1usPgvXgCboIQ= 2024-04-12T03:58:40.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q6I9qULx2H8Yli1usPgvXgCboIQ='} 2024-04-12T03:58:40.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008a8ccd1619ab397-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.096 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.096 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.096 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.096 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.096 DEBUG:boto:Data: 2024-04-12T03:58:40.096 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.096 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.097 DEBUG:boto:Params: {} 2024-04-12T03:58:40.097 DEBUG:boto:Token: None 2024-04-12T03:58:40.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c620d1ee10f06b9-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.100 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T03:58:40.100 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T03:58:40.100 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.101 DEBUG:boto:Path: /dgbpxg-63/obj58 2024-04-12T03:58:40.101 DEBUG:boto:Data: 2024-04-12T03:58:40.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.101 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.101 DEBUG:boto:Params: {} 2024-04-12T03:58:40.101 DEBUG:boto:Token: None 2024-04-12T03:58:40.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj58 2024-04-12T03:58:40.101 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HnZ3DY6WoZluvYh86d3oq76QlR4= 2024-04-12T03:58:40.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HnZ3DY6WoZluvYh86d3oq76QlR4='} 2024-04-12T03:58:40.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003bbf5a8ab0a60dc3-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.147 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.147 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.147 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.147 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.147 DEBUG:boto:Data: 2024-04-12T03:58:40.147 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.147 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.147 DEBUG:boto:Params: {} 2024-04-12T03:58:40.147 DEBUG:boto:Token: None 2024-04-12T03:58:40.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c5629c7284936d3-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.151 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T03:58:40.151 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T03:58:40.151 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.151 DEBUG:boto:Path: /dgbpxg-63/obj87 2024-04-12T03:58:40.151 DEBUG:boto:Data: 2024-04-12T03:58:40.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.151 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.151 DEBUG:boto:Params: {} 2024-04-12T03:58:40.151 DEBUG:boto:Token: None 2024-04-12T03:58:40.152 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj87 2024-04-12T03:58:40.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gRdGdD8HljCPpTGBKzbJZNEcvQs= 2024-04-12T03:58:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gRdGdD8HljCPpTGBKzbJZNEcvQs='} 2024-04-12T03:58:40.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fef2b9b9e74d818-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.157 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.157 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.157 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.157 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.158 DEBUG:boto:Data: 2024-04-12T03:58:40.158 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.158 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.158 DEBUG:boto:Params: {} 2024-04-12T03:58:40.158 DEBUG:boto:Token: None 2024-04-12T03:58:40.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.158 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa8194e198a729c4-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.161 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T03:58:40.162 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T03:58:40.162 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.162 DEBUG:boto:Path: /dgbpxg-63/obj116 2024-04-12T03:58:40.162 DEBUG:boto:Data: 2024-04-12T03:58:40.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.162 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.162 DEBUG:boto:Params: {} 2024-04-12T03:58:40.162 DEBUG:boto:Token: None 2024-04-12T03:58:40.163 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj116 2024-04-12T03:58:40.163 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GFM0GVIXkzPG8dkjfLzHqQ6Eb/8= 2024-04-12T03:58:40.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GFM0GVIXkzPG8dkjfLzHqQ6Eb/8='} 2024-04-12T03:58:40.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000562a1320eff4c984-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.207 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.207 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.207 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.207 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.207 DEBUG:boto:Data: 2024-04-12T03:58:40.207 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.207 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.207 DEBUG:boto:Params: {} 2024-04-12T03:58:40.207 DEBUG:boto:Token: None 2024-04-12T03:58:40.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.208 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029b5273bc17177c6-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.211 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T03:58:40.211 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T03:58:40.211 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.211 DEBUG:boto:Path: /dgbpxg-63/obj145 2024-04-12T03:58:40.211 DEBUG:boto:Data: 2024-04-12T03:58:40.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.212 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.212 DEBUG:boto:Params: {} 2024-04-12T03:58:40.212 DEBUG:boto:Token: None 2024-04-12T03:58:40.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj145 2024-04-12T03:58:40.212 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZaGopVtYFog/+wVuUjiViplemrA= 2024-04-12T03:58:40.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZaGopVtYFog/+wVuUjiViplemrA='} 2024-04-12T03:58:40.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000994e4b78e14324c5-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.217 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:40.317 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:40.317 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:40.327 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2024-04-12T03:58:40.387 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.387 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.387 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.387 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.387 DEBUG:boto:Data: 2024-04-12T03:58:40.387 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.387 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.387 DEBUG:boto:Params: {} 2024-04-12T03:58:40.387 DEBUG:boto:Token: None 2024-04-12T03:58:40.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df0a3e2fb6cbff25-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.391 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T03:58:40.391 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T03:58:40.391 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.392 DEBUG:boto:Path: /dgbpxg-63/obj31 2024-04-12T03:58:40.392 DEBUG:boto:Data: 2024-04-12T03:58:40.392 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.392 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.392 DEBUG:boto:Params: {} 2024-04-12T03:58:40.392 DEBUG:boto:Token: None 2024-04-12T03:58:40.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj31 2024-04-12T03:58:40.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L0g3NjyQrN/pjFC9t/wsLv476aI= 2024-04-12T03:58:40.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L0g3NjyQrN/pjFC9t/wsLv476aI='} 2024-04-12T03:58:40.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089c753b7537730c3-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.435 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.435 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.435 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.435 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.435 DEBUG:boto:Data: 2024-04-12T03:58:40.435 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.435 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.435 DEBUG:boto:Params: {} 2024-04-12T03:58:40.435 DEBUG:boto:Token: None 2024-04-12T03:58:40.436 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.436 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ec25a50d89ac33-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.439 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T03:58:40.439 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T03:58:40.439 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.439 DEBUG:boto:Path: /dgbpxg-63/obj62 2024-04-12T03:58:40.439 DEBUG:boto:Data: 2024-04-12T03:58:40.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.440 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.440 DEBUG:boto:Params: {} 2024-04-12T03:58:40.440 DEBUG:boto:Token: None 2024-04-12T03:58:40.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj62 2024-04-12T03:58:40.440 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Xw5K3y+0jCbmhl5oxSVpeaLn7eE= 2024-04-12T03:58:40.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Xw5K3y+0jCbmhl5oxSVpeaLn7eE='} 2024-04-12T03:58:40.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000652c43d66583cde0-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.445 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.445 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.445 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.445 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.446 DEBUG:boto:Data: 2024-04-12T03:58:40.446 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.446 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.446 DEBUG:boto:Params: {} 2024-04-12T03:58:40.446 DEBUG:boto:Token: None 2024-04-12T03:58:40.446 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.446 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fad0f88b6fb0156-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.449 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T03:58:40.450 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T03:58:40.450 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.450 DEBUG:boto:Path: /dgbpxg-63/obj93 2024-04-12T03:58:40.450 DEBUG:boto:Data: 2024-04-12T03:58:40.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.450 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.450 DEBUG:boto:Params: {} 2024-04-12T03:58:40.450 DEBUG:boto:Token: None 2024-04-12T03:58:40.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj93 2024-04-12T03:58:40.451 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JoXePTlYoVldkEn7IG7Zw1kUACI= 2024-04-12T03:58:40.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JoXePTlYoVldkEn7IG7Zw1kUACI='} 2024-04-12T03:58:40.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008c7cf86c8f41b08-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.495 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.495 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.495 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.495 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.495 DEBUG:boto:Data: 2024-04-12T03:58:40.495 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.495 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.495 DEBUG:boto:Params: {} 2024-04-12T03:58:40.495 DEBUG:boto:Token: None 2024-04-12T03:58:40.496 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b87615843f27d033-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.499 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T03:58:40.499 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T03:58:40.499 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.499 DEBUG:boto:Path: /dgbpxg-63/obj124 2024-04-12T03:58:40.499 DEBUG:boto:Data: 2024-04-12T03:58:40.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.500 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.500 DEBUG:boto:Params: {} 2024-04-12T03:58:40.500 DEBUG:boto:Token: None 2024-04-12T03:58:40.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj124 2024-04-12T03:58:40.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1L3Hm41N3fZC40b1GTFjZWfPt/I= 2024-04-12T03:58:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1L3Hm41N3fZC40b1GTFjZWfPt/I='} 2024-04-12T03:58:40.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d7ea7c376774191-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.505 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.505 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.505 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.505 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.506 DEBUG:boto:Data: 2024-04-12T03:58:40.506 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.506 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.506 DEBUG:boto:Params: {} 2024-04-12T03:58:40.506 DEBUG:boto:Token: None 2024-04-12T03:58:40.506 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040930a59e8ce1333-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.509 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T03:58:40.509 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T03:58:40.510 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.510 DEBUG:boto:Path: /dgbpxg-63/obj155 2024-04-12T03:58:40.510 DEBUG:boto:Data: 2024-04-12T03:58:40.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.510 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.510 DEBUG:boto:Params: {} 2024-04-12T03:58:40.510 DEBUG:boto:Token: None 2024-04-12T03:58:40.510 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj155 2024-04-12T03:58:40.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/5tVbpDK6lx5k9qXfa+fQRRSjYw= 2024-04-12T03:58:40.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/5tVbpDK6lx5k9qXfa+fQRRSjYw='} 2024-04-12T03:58:40.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfd93eb0bfe3d634-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.559 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:40.663 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T03:58:40.663 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-63 2024-04-12T03:58:40.675 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2024-04-12T03:58:40.737 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.737 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.737 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.737 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.737 DEBUG:boto:Data: 2024-04-12T03:58:40.737 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.738 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.738 DEBUG:boto:Params: {} 2024-04-12T03:58:40.738 DEBUG:boto:Token: None 2024-04-12T03:58:40.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019c42deed636f02a-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.742 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T03:58:40.742 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T03:58:40.742 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.742 DEBUG:boto:Path: /dgbpxg-63/obj37 2024-04-12T03:58:40.742 DEBUG:boto:Data: 2024-04-12T03:58:40.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.742 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.743 DEBUG:boto:Params: {} 2024-04-12T03:58:40.743 DEBUG:boto:Token: None 2024-04-12T03:58:40.743 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj37 2024-04-12T03:58:40.743 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U5FgZUbjMPE728gj6m4OB/JtilY= 2024-04-12T03:58:40.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U5FgZUbjMPE728gj6m4OB/JtilY='} 2024-04-12T03:58:40.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f265edebba3a54fe-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.748 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.748 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.748 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.749 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.749 DEBUG:boto:Data: 2024-04-12T03:58:40.749 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.749 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.749 DEBUG:boto:Params: {} 2024-04-12T03:58:40.749 DEBUG:boto:Token: None 2024-04-12T03:58:40.749 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.749 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f524e5583e0a62ac-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.753 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T03:58:40.753 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T03:58:40.753 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.753 DEBUG:boto:Path: /dgbpxg-63/obj74 2024-04-12T03:58:40.753 DEBUG:boto:Data: 2024-04-12T03:58:40.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.753 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.753 DEBUG:boto:Params: {} 2024-04-12T03:58:40.754 DEBUG:boto:Token: None 2024-04-12T03:58:40.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj74 2024-04-12T03:58:40.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2++wznzGiur6bvyKDbweP50xN6Q= 2024-04-12T03:58:40.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2++wznzGiur6bvyKDbweP50xN6Q='} 2024-04-12T03:58:40.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a745718ee34be13-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.760 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.760 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.760 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.760 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.760 DEBUG:boto:Data: 2024-04-12T03:58:40.760 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.760 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.760 DEBUG:boto:Params: {} 2024-04-12T03:58:40.760 DEBUG:boto:Token: None 2024-04-12T03:58:40.761 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b1a1d7e92a69c4a-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.764 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T03:58:40.765 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T03:58:40.765 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.765 DEBUG:boto:Path: /dgbpxg-63/obj111 2024-04-12T03:58:40.765 DEBUG:boto:Data: 2024-04-12T03:58:40.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.765 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.765 DEBUG:boto:Params: {} 2024-04-12T03:58:40.765 DEBUG:boto:Token: None 2024-04-12T03:58:40.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj111 2024-04-12T03:58:40.766 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/f7iozfT2+xX95DPVM815o08yPU= 2024-04-12T03:58:40.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/f7iozfT2+xX95DPVM815o08yPU='} 2024-04-12T03:58:40.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073950ff53042ba9e-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.811 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.811 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.811 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.811 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.811 DEBUG:boto:Data: 2024-04-12T03:58:40.811 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.811 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.811 DEBUG:boto:Params: {} 2024-04-12T03:58:40.811 DEBUG:boto:Token: None 2024-04-12T03:58:40.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4251b296fa2cc0e-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.815 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T03:58:40.815 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T03:58:40.815 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.815 DEBUG:boto:Path: /dgbpxg-63/obj148 2024-04-12T03:58:40.815 DEBUG:boto:Data: 2024-04-12T03:58:40.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.816 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.816 DEBUG:boto:Params: {} 2024-04-12T03:58:40.816 DEBUG:boto:Token: None 2024-04-12T03:58:40.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj148 2024-04-12T03:58:40.816 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Oj0Opepx4kUUVz+CUn5/hBz5J+4= 2024-04-12T03:58:40.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Oj0Opepx4kUUVz+CUn5/hBz5J+4='} 2024-04-12T03:58:40.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dd10731951c1148-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.822 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T03:58:40.822 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T03:58:40.822 DEBUG:boto:Method: HEAD 2024-04-12T03:58:40.822 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T03:58:40.822 DEBUG:boto:Data: 2024-04-12T03:58:40.822 DEBUG:boto:Headers: {} 2024-04-12T03:58:40.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.822 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.822 DEBUG:boto:Params: {} 2024-04-12T03:58:40.823 DEBUG:boto:Token: None 2024-04-12T03:58:40.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/ 2024-04-12T03:58:40.823 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws= 2024-04-12T03:58:40.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x0YlWb6QKYEMY5IWJ19jbgVhCws=', 'Content-Length': '0'} 2024-04-12T03:58:40.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b7a61d51172b197-006618b170-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.826 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T03:58:40.826 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T03:58:40.826 DEBUG:boto:Method: PUT 2024-04-12T03:58:40.827 DEBUG:boto:Path: /dgbpxg-63/obj185 2024-04-12T03:58:40.827 DEBUG:boto:Data: 2024-04-12T03:58:40.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:40.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T03:58:40.827 DEBUG:boto:Port: 8000 2024-04-12T03:58:40.827 DEBUG:boto:Params: {} 2024-04-12T03:58:40.827 DEBUG:boto:Token: None 2024-04-12T03:58:40.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:40 GMT /dgbpxg-63/obj185 2024-04-12T03:58:40.827 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:m0mQi84RyTz075m1xZscLAucLS0= 2024-04-12T03:58:40.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m0mQi84RyTz075m1xZscLAucLS0='} 2024-04-12T03:58:40.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6295f5bf3c50058-006618b170-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:40.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:40.875 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:40.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:40.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:40.969+0000 7f0baf03c780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T03:58:40.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:40.970+0000 7f0baf03c780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T03:58:40.979 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:58:40.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:58:40.981 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:58:40.981 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:58:41.004 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:58:41.004 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:58:41.004 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:58:42.005 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T03:58:42.026 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:58:42.026 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:58:42.026 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:58:42.027 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T03:58:42.027 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-63 2024-04-12T03:58:42.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:42.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:42.028 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:58:42.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:42.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:42.171+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:58:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:47.172+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:58:52.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:52.172+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:58:57.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:58:57.172+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:02.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:02.173+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:07.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:07.174+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:12.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:12.175+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:17.175+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:22.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:22.176+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:27.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:27.176+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:32.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:32.177+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:37.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:37.177+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:42.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:42.177+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:47.179+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:52.179+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T03:59:57.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T03:59:57.180+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:02.180+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:07.181+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:12.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:12.181+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:17.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:17.182+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:22.182+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:27.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:27.182+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:32.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:32.184+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:37.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:37.185+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:42.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:42.186+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:47.186+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:52.187+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:00:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:00:57.187+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:02.187+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:07.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:07.188+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:12.188+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:17.190+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:22.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:22.191+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:27.192+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:32.192+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:37.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:37.193+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:42.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:42.193+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:47.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:47.193+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:52.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:52.194+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:01:57.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:01:57.195+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:02:02.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:02.196+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:02:03.575 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:02:03.588+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:02:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:07.197+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:02:12.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:12.197+0000 7fc1cfdf8780 1 waiting to reach incremental sync.. 2024-04-12T04:02:17.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:17.201+0000 7fc1cfdf8780 1 bucket sync caught up with source: 2024-04-12T04:02:17.188 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.123.6, , , , , , 00000000002.111.6, , , , , 00000000002.128.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.125.6] 2024-04-12T04:02:17.188 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.123.6, , , , , , 00000000002.111.6, , , , , 00000000002.128.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.125.6] 2024-04-12T04:02:17.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:17.201+0000 7fc1cfdf8780 0 bucket checkpoint complete 2024-04-12T04:02:17.207 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-63 2024-04-12T04:02:17.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:17.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:17.208 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:02:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:17.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:17.325+0000 7fd1e5236780 1 bucket sync caught up with source: 2024-04-12T04:02:17.312 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1616.6, 00000000002.2346.6, 00000000002.251.6, 00000000005.846.6, 00000000004.2870.6, 00000000001.3104.6, 00000000001.2343.6, 00000000004.694.6, 00000000007.1134.6, 00000000006.696.6] 2024-04-12T04:02:17.312 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1616.6, 00000000002.2346.6, 00000000002.251.6, 00000000005.846.6, 00000000004.2870.6, 00000000001.3104.6, 00000000001.2343.6, 00000000004.694.6, 00000000007.1134.6, 00000000006.696.6] 2024-04-12T04:02:17.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:17.325+0000 7fd1e5236780 0 bucket checkpoint complete 2024-04-12T04:02:17.325 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-63 zones={a2, a1} 2024-04-12T04:02:17.325 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.326 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.326 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.326 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T04:02:17.326 DEBUG:boto:Data: 2024-04-12T04:02:17.326 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.326 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.326 DEBUG:boto:Params: {} 2024-04-12T04:02:17.327 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:02:17.327 DEBUG:boto:Token: None 2024-04-12T04:02:17.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pM6MhWzIBkuo0q/haATDZUQ8EvM= 2024-04-12T04:02:17.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pM6MhWzIBkuo0q/haATDZUQ8EvM=', 'Content-Length': '0'} 2024-04-12T04:02:17.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061b7edafddcd2fb4-006618b249-2229-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.359 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.359 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.359 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.360 DEBUG:boto:Path: /dgbpxg-63/ 2024-04-12T04:02:17.360 DEBUG:boto:Data: 2024-04-12T04:02:17.360 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.360 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.360 DEBUG:boto:Params: {} 2024-04-12T04:02:17.360 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:02:17.360 DEBUG:boto:Token: None 2024-04-12T04:02:17.361 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.361 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pM6MhWzIBkuo0q/haATDZUQ8EvM= 2024-04-12T04:02:17.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pM6MhWzIBkuo0q/haATDZUQ8EvM=', 'Content-Length': '0'} 2024-04-12T04:02:17.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1287adcec955dca-006618b249-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:02:17.375 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.375 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.375 DEBUG:boto:path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.375 DEBUG:boto:auth_path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.375 DEBUG:boto:Method: GET 2024-04-12T04:02:17.375 DEBUG:boto:Path: /dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.375 DEBUG:boto:Data: 2024-04-12T04:02:17.376 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.376 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.376 DEBUG:boto:Params: {} 2024-04-12T04:02:17.376 DEBUG:boto:Token: None 2024-04-12T04:02:17.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/?versions 2024-04-12T04:02:17.376 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo= 2024-04-12T04:02:17.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo=', 'Content-Length': '0'} 2024-04-12T04:02:17.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048da4d5b44b13ea0-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.379 DEBUG:boto:b'dgbpxg-63999falseobj11nulltrue2024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:02:17.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:02:17.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:02:17.384 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:02:17.385 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.385 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.385 DEBUG:boto:path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.385 DEBUG:boto:auth_path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.385 DEBUG:boto:Method: GET 2024-04-12T04:02:17.385 DEBUG:boto:Path: /dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.385 DEBUG:boto:Data: 2024-04-12T04:02:17.385 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.385 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.385 DEBUG:boto:Params: {} 2024-04-12T04:02:17.386 DEBUG:boto:Token: None 2024-04-12T04:02:17.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/?versions 2024-04-12T04:02:17.386 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo= 2024-04-12T04:02:17.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo=', 'Content-Length': '0'} 2024-04-12T04:02:17.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047571429ae7e6044-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.390 DEBUG:boto:b'dgbpxg-63999falseobj11nulltrue2024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:02:17.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:02:17.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:02:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:02:17.395 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.395 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.395 DEBUG:boto:path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.395 DEBUG:boto:auth_path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.395 DEBUG:boto:Method: GET 2024-04-12T04:02:17.395 DEBUG:boto:Path: /dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.395 DEBUG:boto:Data: 2024-04-12T04:02:17.395 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.395 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.395 DEBUG:boto:Params: {} 2024-04-12T04:02:17.395 DEBUG:boto:Token: None 2024-04-12T04:02:17.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/?versions 2024-04-12T04:02:17.396 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo= 2024-04-12T04:02:17.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo=', 'Content-Length': '0'} 2024-04-12T04:02:17.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04fe36a187a60cd-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.398 DEBUG:boto:b'dgbpxg-63999falseobj11nulltrue2024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.400 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.400 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.400 DEBUG:boto:path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.400 DEBUG:boto:auth_path=/dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.401 DEBUG:boto:Method: GET 2024-04-12T04:02:17.401 DEBUG:boto:Path: /dgbpxg-63/?versions&max-keys=999 2024-04-12T04:02:17.401 DEBUG:boto:Data: 2024-04-12T04:02:17.401 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.401 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.401 DEBUG:boto:Params: {} 2024-04-12T04:02:17.401 DEBUG:boto:Token: None 2024-04-12T04:02:17.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/?versions 2024-04-12T04:02:17.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo= 2024-04-12T04:02:17.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LATgwzRNQNMWZkdDAJbX2RwhUUo=', 'Content-Length': '0'} 2024-04-12T04:02:17.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae365875aafc7259-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.405 DEBUG:boto:b'dgbpxg-63999falseobj11nulltrue2024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:02:17.408 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.408 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.408 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.409 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.409 DEBUG:boto:Method: GET 2024-04-12T04:02:17.409 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.409 DEBUG:boto:Data: 2024-04-12T04:02:17.409 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.409 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.409 DEBUG:boto:Params: {} 2024-04-12T04:02:17.409 DEBUG:boto:Token: None 2024-04-12T04:02:17.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw= 2024-04-12T04:02:17.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw=', 'Content-Length': '0'} 2024-04-12T04:02:17.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39d3b2ea0bf507d-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.454 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.455 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.455 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.455 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.456 DEBUG:boto:Method: GET 2024-04-12T04:02:17.456 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.456 DEBUG:boto:Data: 2024-04-12T04:02:17.456 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.456 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.456 DEBUG:boto:Params: {} 2024-04-12T04:02:17.456 DEBUG:boto:Token: None 2024-04-12T04:02:17.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.457 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw= 2024-04-12T04:02:17.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw=', 'Content-Length': '0'} 2024-04-12T04:02:17.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013ffd619b28659d-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.459 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.459 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.459 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.459 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.459 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.459 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.459 DEBUG:boto:Data: 2024-04-12T04:02:17.459 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.460 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.460 DEBUG:boto:Params: {} 2024-04-12T04:02:17.460 DEBUG:boto:Token: None 2024-04-12T04:02:17.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RriKnujI9FoN2TPPpmvhktIwYik= 2024-04-12T04:02:17.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RriKnujI9FoN2TPPpmvhktIwYik=', 'Content-Length': '0'} 2024-04-12T04:02:17.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7b659174a73ea9-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.462 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.462 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.462 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.462 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.462 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.462 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.462 DEBUG:boto:Data: 2024-04-12T04:02:17.462 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.463 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.463 DEBUG:boto:Params: {} 2024-04-12T04:02:17.463 DEBUG:boto:Token: None 2024-04-12T04:02:17.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RriKnujI9FoN2TPPpmvhktIwYik= 2024-04-12T04:02:17.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RriKnujI9FoN2TPPpmvhktIwYik=', 'Content-Length': '0'} 2024-04-12T04:02:17.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282b002f1d86c715-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:02:17.465 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.465 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.465 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.465 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.465 DEBUG:boto:Method: GET 2024-04-12T04:02:17.465 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.465 DEBUG:boto:Data: 2024-04-12T04:02:17.465 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.466 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.466 DEBUG:boto:Params: {} 2024-04-12T04:02:17.466 DEBUG:boto:Token: None 2024-04-12T04:02:17.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw= 2024-04-12T04:02:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw=', 'Content-Length': '0'} 2024-04-12T04:02:17.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005364eeb4c154fb75-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.468 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.468 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.468 DEBUG:boto:path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.468 DEBUG:boto:auth_path=/dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.468 DEBUG:boto:Method: GET 2024-04-12T04:02:17.468 DEBUG:boto:Path: /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.468 DEBUG:boto:Data: 2024-04-12T04:02:17.469 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.469 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.469 DEBUG:boto:Params: {} 2024-04-12T04:02:17.469 DEBUG:boto:Token: None 2024-04-12T04:02:17.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11?versionId=null 2024-04-12T04:02:17.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw= 2024-04-12T04:02:17.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rs30dXaC3TjUv+Ea9xO/BfpJtOw=', 'Content-Length': '0'} 2024-04-12T04:02:17.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa6961b24fb4ed1-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.474 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.474 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.474 DEBUG:boto:path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.474 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.474 DEBUG:boto:Method: GET 2024-04-12T04:02:17.474 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.474 DEBUG:boto:Data: 2024-04-12T04:02:17.474 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.475 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.475 DEBUG:boto:Params: {} 2024-04-12T04:02:17.475 DEBUG:boto:Token: None 2024-04-12T04:02:17.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8b26a452ffbb0e-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.478 DEBUG:boto:b'dgbpxg-63obj111000falseobj112024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.478 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.478 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.478 DEBUG:boto:path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.478 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.479 DEBUG:boto:Method: GET 2024-04-12T04:02:17.479 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.479 DEBUG:boto:Data: 2024-04-12T04:02:17.479 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.479 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.479 DEBUG:boto:Params: {} 2024-04-12T04:02:17.479 DEBUG:boto:Token: None 2024-04-12T04:02:17.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.479 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d05d456bf73a58-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.483 DEBUG:boto:b'dgbpxg-63obj111000falseobj112024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.484 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.484 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.484 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.484 DEBUG:boto:Path: /dgbpxg-63/obj11 2024-04-12T04:02:17.484 DEBUG:boto:Data: 2024-04-12T04:02:17.484 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.484 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.484 DEBUG:boto:Params: {} 2024-04-12T04:02:17.485 DEBUG:boto:Token: None 2024-04-12T04:02:17.485 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11 2024-04-12T04:02:17.485 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:up8Eh+zLdRPWkLuhUedAi6iZRXc= 2024-04-12T04:02:17.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:up8Eh+zLdRPWkLuhUedAi6iZRXc=', 'Content-Length': '0'} 2024-04-12T04:02:17.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086da7d9f497302d3-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.487 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.487 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.487 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.487 DEBUG:boto:Path: /dgbpxg-63/obj11 2024-04-12T04:02:17.487 DEBUG:boto:Data: 2024-04-12T04:02:17.487 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.487 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.487 DEBUG:boto:Params: {} 2024-04-12T04:02:17.487 DEBUG:boto:Token: None 2024-04-12T04:02:17.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11 2024-04-12T04:02:17.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:up8Eh+zLdRPWkLuhUedAi6iZRXc= 2024-04-12T04:02:17.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:up8Eh+zLdRPWkLuhUedAi6iZRXc=', 'Content-Length': '0'} 2024-04-12T04:02:17.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c98101f746b73a-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:02:17.489 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.490 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.490 DEBUG:boto:Method: GET 2024-04-12T04:02:17.490 DEBUG:boto:Path: /dgbpxg-63/obj11 2024-04-12T04:02:17.490 DEBUG:boto:Data: 2024-04-12T04:02:17.490 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.490 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.490 DEBUG:boto:Params: {} 2024-04-12T04:02:17.490 DEBUG:boto:Token: None 2024-04-12T04:02:17.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11 2024-04-12T04:02:17.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C5qf645XsqAQBI+yWhIkoBM2M6M= 2024-04-12T04:02:17.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C5qf645XsqAQBI+yWhIkoBM2M6M=', 'Content-Length': '0'} 2024-04-12T04:02:17.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003818dc6feffdc536-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.492 DEBUG:boto:path=/dgbpxg-63/obj11 2024-04-12T04:02:17.492 DEBUG:boto:auth_path=/dgbpxg-63/obj11 2024-04-12T04:02:17.493 DEBUG:boto:Method: GET 2024-04-12T04:02:17.493 DEBUG:boto:Path: /dgbpxg-63/obj11 2024-04-12T04:02:17.493 DEBUG:boto:Data: 2024-04-12T04:02:17.493 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.493 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.493 DEBUG:boto:Params: {} 2024-04-12T04:02:17.493 DEBUG:boto:Token: None 2024-04-12T04:02:17.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj11 2024-04-12T04:02:17.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C5qf645XsqAQBI+yWhIkoBM2M6M= 2024-04-12T04:02:17.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C5qf645XsqAQBI+yWhIkoBM2M6M=', 'Content-Length': '0'} 2024-04-12T04:02:17.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ddcf7fb34d879d-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.542 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.543 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.543 DEBUG:boto:path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.543 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.543 DEBUG:boto:Method: GET 2024-04-12T04:02:17.543 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.543 DEBUG:boto:Data: 2024-04-12T04:02:17.543 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.543 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.543 DEBUG:boto:Params: {} 2024-04-12T04:02:17.544 DEBUG:boto:Token: None 2024-04-12T04:02:17.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.544 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087897c1f08441487-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.546 DEBUG:boto:b'dgbpxg-63obj111000falseobj112024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.547 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.547 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.547 DEBUG:boto:path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.547 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.547 DEBUG:boto:Method: GET 2024-04-12T04:02:17.547 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj11 2024-04-12T04:02:17.548 DEBUG:boto:Data: 2024-04-12T04:02:17.548 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.548 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.548 DEBUG:boto:Params: {} 2024-04-12T04:02:17.548 DEBUG:boto:Token: None 2024-04-12T04:02:17.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee077a67a1d01eb5-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.552 DEBUG:boto:b'dgbpxg-63obj111000falseobj112024-04-12T03:58:38.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:02:17.553 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.553 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.553 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.553 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.553 DEBUG:boto:Method: GET 2024-04-12T04:02:17.553 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.553 DEBUG:boto:Data: 2024-04-12T04:02:17.554 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.554 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.554 DEBUG:boto:Params: {} 2024-04-12T04:02:17.554 DEBUG:boto:Token: None 2024-04-12T04:02:17.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.554 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8= 2024-04-12T04:02:17.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8=', 'Content-Length': '0'} 2024-04-12T04:02:17.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31437d6bc6e4017-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.598 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.598 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.599 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.599 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.599 DEBUG:boto:Method: GET 2024-04-12T04:02:17.599 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.599 DEBUG:boto:Data: 2024-04-12T04:02:17.599 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.599 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.599 DEBUG:boto:Params: {} 2024-04-12T04:02:17.599 DEBUG:boto:Token: None 2024-04-12T04:02:17.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8= 2024-04-12T04:02:17.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8=', 'Content-Length': '0'} 2024-04-12T04:02:17.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141c383240ea6c60-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.602 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.602 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.602 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.602 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.602 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.602 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.602 DEBUG:boto:Data: 2024-04-12T04:02:17.602 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.603 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.603 DEBUG:boto:Params: {} 2024-04-12T04:02:17.603 DEBUG:boto:Token: None 2024-04-12T04:02:17.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3YNE9SpxdIh6kTdLpMORF/PsVa8= 2024-04-12T04:02:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3YNE9SpxdIh6kTdLpMORF/PsVa8=', 'Content-Length': '0'} 2024-04-12T04:02:17.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a84f7acd52d8c45-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.605 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.605 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.605 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.605 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.606 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.606 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.606 DEBUG:boto:Data: 2024-04-12T04:02:17.606 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.606 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.606 DEBUG:boto:Params: {} 2024-04-12T04:02:17.606 DEBUG:boto:Token: None 2024-04-12T04:02:17.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3YNE9SpxdIh6kTdLpMORF/PsVa8= 2024-04-12T04:02:17.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3YNE9SpxdIh6kTdLpMORF/PsVa8=', 'Content-Length': '0'} 2024-04-12T04:02:17.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757501ecd3d0963b-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:02:17.609 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.610 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.610 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.610 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.610 DEBUG:boto:Method: GET 2024-04-12T04:02:17.610 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.610 DEBUG:boto:Data: 2024-04-12T04:02:17.610 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.610 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.611 DEBUG:boto:Params: {} 2024-04-12T04:02:17.611 DEBUG:boto:Token: None 2024-04-12T04:02:17.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8= 2024-04-12T04:02:17.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8=', 'Content-Length': '0'} 2024-04-12T04:02:17.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008585cc1a9020e3a9-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.613 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.613 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.613 DEBUG:boto:path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.613 DEBUG:boto:auth_path=/dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.613 DEBUG:boto:Method: GET 2024-04-12T04:02:17.613 DEBUG:boto:Path: /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.613 DEBUG:boto:Data: 2024-04-12T04:02:17.614 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.614 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.614 DEBUG:boto:Params: {} 2024-04-12T04:02:17.614 DEBUG:boto:Token: None 2024-04-12T04:02:17.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111?versionId=null 2024-04-12T04:02:17.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8= 2024-04-12T04:02:17.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WfEoF1j9aWHIdXJkTyVq10zNTL8=', 'Content-Length': '0'} 2024-04-12T04:02:17.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536ba0b03057df09-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.616 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.616 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.617 DEBUG:boto:path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.617 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.617 DEBUG:boto:Method: GET 2024-04-12T04:02:17.617 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.617 DEBUG:boto:Data: 2024-04-12T04:02:17.617 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.617 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.617 DEBUG:boto:Params: {} 2024-04-12T04:02:17.617 DEBUG:boto:Token: None 2024-04-12T04:02:17.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055941deb0821454e-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.620 DEBUG:boto:b'dgbpxg-63obj1111000falseobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.621 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.621 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.621 DEBUG:boto:path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.621 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.621 DEBUG:boto:Method: GET 2024-04-12T04:02:17.621 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.621 DEBUG:boto:Data: 2024-04-12T04:02:17.621 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.622 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.622 DEBUG:boto:Params: {} 2024-04-12T04:02:17.622 DEBUG:boto:Token: None 2024-04-12T04:02:17.622 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.622 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb90d3c93446bcc-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.626 DEBUG:boto:b'dgbpxg-63obj1111000falseobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.627 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.627 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.628 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.628 DEBUG:boto:Path: /dgbpxg-63/obj111 2024-04-12T04:02:17.628 DEBUG:boto:Data: 2024-04-12T04:02:17.628 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.628 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.629 DEBUG:boto:Params: {} 2024-04-12T04:02:17.629 DEBUG:boto:Token: None 2024-04-12T04:02:17.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111 2024-04-12T04:02:17.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2Tx0UhdSf8Gr5KUPzOD8EmOPqo= 2024-04-12T04:02:17.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2Tx0UhdSf8Gr5KUPzOD8EmOPqo=', 'Content-Length': '0'} 2024-04-12T04:02:17.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d436c1ad5cf600f-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.632 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.632 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.632 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.633 DEBUG:boto:Path: /dgbpxg-63/obj111 2024-04-12T04:02:17.633 DEBUG:boto:Data: 2024-04-12T04:02:17.633 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.633 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.634 DEBUG:boto:Params: {} 2024-04-12T04:02:17.634 DEBUG:boto:Token: None 2024-04-12T04:02:17.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111 2024-04-12T04:02:17.634 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:a2Tx0UhdSf8Gr5KUPzOD8EmOPqo= 2024-04-12T04:02:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:a2Tx0UhdSf8Gr5KUPzOD8EmOPqo=', 'Content-Length': '0'} 2024-04-12T04:02:17.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97bcf91d905362f-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:02:17.638 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.638 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.638 DEBUG:boto:Method: GET 2024-04-12T04:02:17.638 DEBUG:boto:Path: /dgbpxg-63/obj111 2024-04-12T04:02:17.638 DEBUG:boto:Data: 2024-04-12T04:02:17.638 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.639 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.639 DEBUG:boto:Params: {} 2024-04-12T04:02:17.639 DEBUG:boto:Token: None 2024-04-12T04:02:17.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111 2024-04-12T04:02:17.639 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r1+z8jp5t1OfLL5urs9CeIC0dCY= 2024-04-12T04:02:17.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r1+z8jp5t1OfLL5urs9CeIC0dCY=', 'Content-Length': '0'} 2024-04-12T04:02:17.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efff5ee40d63e237-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.641 DEBUG:boto:path=/dgbpxg-63/obj111 2024-04-12T04:02:17.641 DEBUG:boto:auth_path=/dgbpxg-63/obj111 2024-04-12T04:02:17.641 DEBUG:boto:Method: GET 2024-04-12T04:02:17.641 DEBUG:boto:Path: /dgbpxg-63/obj111 2024-04-12T04:02:17.641 DEBUG:boto:Data: 2024-04-12T04:02:17.641 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.642 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.642 DEBUG:boto:Params: {} 2024-04-12T04:02:17.642 DEBUG:boto:Token: None 2024-04-12T04:02:17.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj111 2024-04-12T04:02:17.642 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r1+z8jp5t1OfLL5urs9CeIC0dCY= 2024-04-12T04:02:17.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r1+z8jp5t1OfLL5urs9CeIC0dCY=', 'Content-Length': '0'} 2024-04-12T04:02:17.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd88cde7eaeca060-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.644 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.644 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.644 DEBUG:boto:path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.644 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.644 DEBUG:boto:Method: GET 2024-04-12T04:02:17.645 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.645 DEBUG:boto:Data: 2024-04-12T04:02:17.645 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.645 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.645 DEBUG:boto:Params: {} 2024-04-12T04:02:17.645 DEBUG:boto:Token: None 2024-04-12T04:02:17.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.645 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d7968892596f421-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.647 DEBUG:boto:b'dgbpxg-63obj1111000falseobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.648 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.648 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.648 DEBUG:boto:path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.648 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.648 DEBUG:boto:Method: GET 2024-04-12T04:02:17.648 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj111 2024-04-12T04:02:17.648 DEBUG:boto:Data: 2024-04-12T04:02:17.648 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.649 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.649 DEBUG:boto:Params: {} 2024-04-12T04:02:17.649 DEBUG:boto:Token: None 2024-04-12T04:02:17.649 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c33da514fe053f-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.653 DEBUG:boto:b'dgbpxg-63obj1111000falseobj1112024-04-12T03:58:40.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:02:17.654 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.654 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.654 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.654 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.654 DEBUG:boto:Method: GET 2024-04-12T04:02:17.654 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.654 DEBUG:boto:Data: 2024-04-12T04:02:17.654 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.655 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.655 DEBUG:boto:Params: {} 2024-04-12T04:02:17.655 DEBUG:boto:Token: None 2024-04-12T04:02:17.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw= 2024-04-12T04:02:17.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw=', 'Content-Length': '0'} 2024-04-12T04:02:17.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022223196dd42ff4f-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.657 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.657 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.657 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.657 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.657 DEBUG:boto:Method: GET 2024-04-12T04:02:17.657 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.658 DEBUG:boto:Data: 2024-04-12T04:02:17.658 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.658 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.658 DEBUG:boto:Params: {} 2024-04-12T04:02:17.658 DEBUG:boto:Token: None 2024-04-12T04:02:17.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.658 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw= 2024-04-12T04:02:17.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw=', 'Content-Length': '0'} 2024-04-12T04:02:17.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b419b99d3981c4e-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.661 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.661 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.661 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.661 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.661 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.661 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.661 DEBUG:boto:Data: 2024-04-12T04:02:17.661 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.662 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.662 DEBUG:boto:Params: {} 2024-04-12T04:02:17.662 DEBUG:boto:Token: None 2024-04-12T04:02:17.662 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.662 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c4R6Z2OXYjqEMMXQ1g+gdjNtLD4= 2024-04-12T04:02:17.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c4R6Z2OXYjqEMMXQ1g+gdjNtLD4=', 'Content-Length': '0'} 2024-04-12T04:02:17.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012767384f35ac808-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.664 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.664 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.664 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.664 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.664 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.664 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.664 DEBUG:boto:Data: 2024-04-12T04:02:17.664 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.665 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.665 DEBUG:boto:Params: {} 2024-04-12T04:02:17.665 DEBUG:boto:Token: None 2024-04-12T04:02:17.665 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.665 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c4R6Z2OXYjqEMMXQ1g+gdjNtLD4= 2024-04-12T04:02:17.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c4R6Z2OXYjqEMMXQ1g+gdjNtLD4=', 'Content-Length': '0'} 2024-04-12T04:02:17.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431e3d28600a7100-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:02:17.667 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.667 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.667 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.667 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.667 DEBUG:boto:Method: GET 2024-04-12T04:02:17.667 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.667 DEBUG:boto:Data: 2024-04-12T04:02:17.667 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.668 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.668 DEBUG:boto:Params: {} 2024-04-12T04:02:17.668 DEBUG:boto:Token: None 2024-04-12T04:02:17.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw= 2024-04-12T04:02:17.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw=', 'Content-Length': '0'} 2024-04-12T04:02:17.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cc34a82dfaee8e-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.670 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.670 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.670 DEBUG:boto:path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.670 DEBUG:boto:auth_path=/dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.670 DEBUG:boto:Method: GET 2024-04-12T04:02:17.674 DEBUG:boto:Path: /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.674 DEBUG:boto:Data: 2024-04-12T04:02:17.674 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.675 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.675 DEBUG:boto:Params: {} 2024-04-12T04:02:17.675 DEBUG:boto:Token: None 2024-04-12T04:02:17.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115?versionId=null 2024-04-12T04:02:17.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw= 2024-04-12T04:02:17.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nezD5smDzoVRDJ2bGN6grlXHnVw=', 'Content-Length': '0'} 2024-04-12T04:02:17.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02513c203d751a8-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.718 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.718 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.719 DEBUG:boto:path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.719 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.719 DEBUG:boto:Method: GET 2024-04-12T04:02:17.719 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.719 DEBUG:boto:Data: 2024-04-12T04:02:17.719 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.719 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.719 DEBUG:boto:Params: {} 2024-04-12T04:02:17.720 DEBUG:boto:Token: None 2024-04-12T04:02:17.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.720 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ea7f3d42e85500-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.722 DEBUG:boto:b'dgbpxg-63obj1151000falseobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.723 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.723 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.723 DEBUG:boto:path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.723 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.723 DEBUG:boto:Method: GET 2024-04-12T04:02:17.723 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.723 DEBUG:boto:Data: 2024-04-12T04:02:17.723 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.723 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.723 DEBUG:boto:Params: {} 2024-04-12T04:02:17.724 DEBUG:boto:Token: None 2024-04-12T04:02:17.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001037e8bc4c420e46-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.728 DEBUG:boto:b'dgbpxg-63obj1151000falseobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.729 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.729 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.729 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.729 DEBUG:boto:Path: /dgbpxg-63/obj115 2024-04-12T04:02:17.729 DEBUG:boto:Data: 2024-04-12T04:02:17.729 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.729 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.729 DEBUG:boto:Params: {} 2024-04-12T04:02:17.730 DEBUG:boto:Token: None 2024-04-12T04:02:17.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115 2024-04-12T04:02:17.730 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8aNXGBJGonniftcP7HQmJ9MIy4k= 2024-04-12T04:02:17.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8aNXGBJGonniftcP7HQmJ9MIy4k=', 'Content-Length': '0'} 2024-04-12T04:02:17.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8f183b0ec6e87a-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.732 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.732 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.732 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.732 DEBUG:boto:Path: /dgbpxg-63/obj115 2024-04-12T04:02:17.732 DEBUG:boto:Data: 2024-04-12T04:02:17.732 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.733 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.733 DEBUG:boto:Params: {} 2024-04-12T04:02:17.733 DEBUG:boto:Token: None 2024-04-12T04:02:17.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115 2024-04-12T04:02:17.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8aNXGBJGonniftcP7HQmJ9MIy4k= 2024-04-12T04:02:17.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8aNXGBJGonniftcP7HQmJ9MIy4k=', 'Content-Length': '0'} 2024-04-12T04:02:17.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4219faa866fd0dd-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:02:17.735 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.735 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.735 DEBUG:boto:Method: GET 2024-04-12T04:02:17.735 DEBUG:boto:Path: /dgbpxg-63/obj115 2024-04-12T04:02:17.735 DEBUG:boto:Data: 2024-04-12T04:02:17.736 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.736 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.736 DEBUG:boto:Params: {} 2024-04-12T04:02:17.736 DEBUG:boto:Token: None 2024-04-12T04:02:17.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115 2024-04-12T04:02:17.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rr6Qu8gzJjOeTgJLeCM+9yPGuC0= 2024-04-12T04:02:17.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rr6Qu8gzJjOeTgJLeCM+9yPGuC0=', 'Content-Length': '0'} 2024-04-12T04:02:17.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfa2ecb86476e89-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.738 DEBUG:boto:path=/dgbpxg-63/obj115 2024-04-12T04:02:17.738 DEBUG:boto:auth_path=/dgbpxg-63/obj115 2024-04-12T04:02:17.738 DEBUG:boto:Method: GET 2024-04-12T04:02:17.738 DEBUG:boto:Path: /dgbpxg-63/obj115 2024-04-12T04:02:17.738 DEBUG:boto:Data: 2024-04-12T04:02:17.738 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.739 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.739 DEBUG:boto:Params: {} 2024-04-12T04:02:17.739 DEBUG:boto:Token: None 2024-04-12T04:02:17.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj115 2024-04-12T04:02:17.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rr6Qu8gzJjOeTgJLeCM+9yPGuC0= 2024-04-12T04:02:17.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rr6Qu8gzJjOeTgJLeCM+9yPGuC0=', 'Content-Length': '0'} 2024-04-12T04:02:17.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009927c759a08516a8-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.786 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.786 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.787 DEBUG:boto:path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.787 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.787 DEBUG:boto:Method: GET 2024-04-12T04:02:17.787 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.787 DEBUG:boto:Data: 2024-04-12T04:02:17.787 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.787 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.787 DEBUG:boto:Params: {} 2024-04-12T04:02:17.788 DEBUG:boto:Token: None 2024-04-12T04:02:17.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a92b246ffa3a37e-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.790 DEBUG:boto:b'dgbpxg-63obj1151000falseobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.791 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.791 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.791 DEBUG:boto:path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.791 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.791 DEBUG:boto:Method: GET 2024-04-12T04:02:17.791 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj115 2024-04-12T04:02:17.792 DEBUG:boto:Data: 2024-04-12T04:02:17.792 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.792 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.792 DEBUG:boto:Params: {} 2024-04-12T04:02:17.792 DEBUG:boto:Token: None 2024-04-12T04:02:17.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.792 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486c95f68021cda3-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.796 DEBUG:boto:b'dgbpxg-63obj1151000falseobj1152024-04-12T03:58:39.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:02:17.797 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.797 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.797 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.797 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.797 DEBUG:boto:Method: GET 2024-04-12T04:02:17.797 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.797 DEBUG:boto:Data: 2024-04-12T04:02:17.797 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.798 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.798 DEBUG:boto:Params: {} 2024-04-12T04:02:17.798 DEBUG:boto:Token: None 2024-04-12T04:02:17.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.798 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA= 2024-04-12T04:02:17.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA=', 'Content-Length': '0'} 2024-04-12T04:02:17.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a897ce516a78122e-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.842 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.843 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.843 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.843 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.843 DEBUG:boto:Method: GET 2024-04-12T04:02:17.843 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.843 DEBUG:boto:Data: 2024-04-12T04:02:17.843 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.843 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.843 DEBUG:boto:Params: {} 2024-04-12T04:02:17.843 DEBUG:boto:Token: None 2024-04-12T04:02:17.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA= 2024-04-12T04:02:17.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA=', 'Content-Length': '0'} 2024-04-12T04:02:17.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91caccb2eb9e6fc-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.846 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.846 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.846 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.846 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.846 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.847 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.847 DEBUG:boto:Data: 2024-04-12T04:02:17.847 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.847 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.847 DEBUG:boto:Params: {} 2024-04-12T04:02:17.847 DEBUG:boto:Token: None 2024-04-12T04:02:17.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5dTvKLOaMtxXTHdzENGCaQYmgE= 2024-04-12T04:02:17.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5dTvKLOaMtxXTHdzENGCaQYmgE=', 'Content-Length': '0'} 2024-04-12T04:02:17.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b6cf0f9ad08ad8-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.849 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.850 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.850 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.850 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.850 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.850 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.850 DEBUG:boto:Data: 2024-04-12T04:02:17.850 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.850 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.851 DEBUG:boto:Params: {} 2024-04-12T04:02:17.851 DEBUG:boto:Token: None 2024-04-12T04:02:17.854 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y5dTvKLOaMtxXTHdzENGCaQYmgE= 2024-04-12T04:02:17.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y5dTvKLOaMtxXTHdzENGCaQYmgE=', 'Content-Length': '0'} 2024-04-12T04:02:17.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94907fa4e33f718-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:02:17.856 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.856 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.856 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.856 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.856 DEBUG:boto:Method: GET 2024-04-12T04:02:17.856 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.857 DEBUG:boto:Data: 2024-04-12T04:02:17.857 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.857 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.857 DEBUG:boto:Params: {} 2024-04-12T04:02:17.857 DEBUG:boto:Token: None 2024-04-12T04:02:17.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA= 2024-04-12T04:02:17.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA=', 'Content-Length': '0'} 2024-04-12T04:02:17.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092fcc396d52930f6-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.859 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.860 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.860 DEBUG:boto:path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.860 DEBUG:boto:auth_path=/dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.860 DEBUG:boto:Method: GET 2024-04-12T04:02:17.860 DEBUG:boto:Path: /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.860 DEBUG:boto:Data: 2024-04-12T04:02:17.860 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.860 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.860 DEBUG:boto:Params: {} 2024-04-12T04:02:17.860 DEBUG:boto:Token: None 2024-04-12T04:02:17.861 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116?versionId=null 2024-04-12T04:02:17.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA= 2024-04-12T04:02:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ByR9fWQZjB6/53ZvuA6iF9q0ZrA=', 'Content-Length': '0'} 2024-04-12T04:02:17.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de63c841585bc88-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.862 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.863 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.863 DEBUG:boto:path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.863 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.863 DEBUG:boto:Method: GET 2024-04-12T04:02:17.863 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.863 DEBUG:boto:Data: 2024-04-12T04:02:17.863 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.863 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.863 DEBUG:boto:Params: {} 2024-04-12T04:02:17.863 DEBUG:boto:Token: None 2024-04-12T04:02:17.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6fa4430f53f920a-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.866 DEBUG:boto:b'dgbpxg-63obj1161000falseobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.867 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.867 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.867 DEBUG:boto:path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.867 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.867 DEBUG:boto:Method: GET 2024-04-12T04:02:17.867 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.867 DEBUG:boto:Data: 2024-04-12T04:02:17.867 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.868 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.868 DEBUG:boto:Params: {} 2024-04-12T04:02:17.868 DEBUG:boto:Token: None 2024-04-12T04:02:17.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087fd7ac005181288-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.872 DEBUG:boto:b'dgbpxg-63obj1161000falseobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.872 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.872 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.872 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.872 DEBUG:boto:Path: /dgbpxg-63/obj116 2024-04-12T04:02:17.872 DEBUG:boto:Data: 2024-04-12T04:02:17.873 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.873 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.873 DEBUG:boto:Params: {} 2024-04-12T04:02:17.873 DEBUG:boto:Token: None 2024-04-12T04:02:17.873 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116 2024-04-12T04:02:17.873 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+8iFzP2q5udZK8WcTxufX3fwiic= 2024-04-12T04:02:17.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+8iFzP2q5udZK8WcTxufX3fwiic=', 'Content-Length': '0'} 2024-04-12T04:02:17.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42dc848a2bc582c-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.877 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.877 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.878 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.878 DEBUG:boto:Path: /dgbpxg-63/obj116 2024-04-12T04:02:17.878 DEBUG:boto:Data: 2024-04-12T04:02:17.878 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.878 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.878 DEBUG:boto:Params: {} 2024-04-12T04:02:17.878 DEBUG:boto:Token: None 2024-04-12T04:02:17.878 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116 2024-04-12T04:02:17.878 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+8iFzP2q5udZK8WcTxufX3fwiic= 2024-04-12T04:02:17.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+8iFzP2q5udZK8WcTxufX3fwiic=', 'Content-Length': '0'} 2024-04-12T04:02:17.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9467ee94166b96-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:02:17.881 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.881 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.881 DEBUG:boto:Method: GET 2024-04-12T04:02:17.881 DEBUG:boto:Path: /dgbpxg-63/obj116 2024-04-12T04:02:17.881 DEBUG:boto:Data: 2024-04-12T04:02:17.881 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.881 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.881 DEBUG:boto:Params: {} 2024-04-12T04:02:17.882 DEBUG:boto:Token: None 2024-04-12T04:02:17.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116 2024-04-12T04:02:17.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SSBJ/QPz3Z2XWLG7QAjq5D5H5LY= 2024-04-12T04:02:17.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SSBJ/QPz3Z2XWLG7QAjq5D5H5LY=', 'Content-Length': '0'} 2024-04-12T04:02:17.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9258572f851f6f-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.930 DEBUG:boto:path=/dgbpxg-63/obj116 2024-04-12T04:02:17.931 DEBUG:boto:auth_path=/dgbpxg-63/obj116 2024-04-12T04:02:17.931 DEBUG:boto:Method: GET 2024-04-12T04:02:17.931 DEBUG:boto:Path: /dgbpxg-63/obj116 2024-04-12T04:02:17.931 DEBUG:boto:Data: 2024-04-12T04:02:17.931 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.931 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.931 DEBUG:boto:Params: {} 2024-04-12T04:02:17.931 DEBUG:boto:Token: None 2024-04-12T04:02:17.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj116 2024-04-12T04:02:17.932 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SSBJ/QPz3Z2XWLG7QAjq5D5H5LY= 2024-04-12T04:02:17.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SSBJ/QPz3Z2XWLG7QAjq5D5H5LY=', 'Content-Length': '0'} 2024-04-12T04:02:17.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17501d90bc84d1f-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.934 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.934 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.934 DEBUG:boto:path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.934 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.934 DEBUG:boto:Method: GET 2024-04-12T04:02:17.934 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.935 DEBUG:boto:Data: 2024-04-12T04:02:17.935 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.935 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.935 DEBUG:boto:Params: {} 2024-04-12T04:02:17.935 DEBUG:boto:Token: None 2024-04-12T04:02:17.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.935 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc3b784697c08bea-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.937 DEBUG:boto:b'dgbpxg-63obj1161000falseobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.938 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.938 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.938 DEBUG:boto:path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.938 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.938 DEBUG:boto:Method: GET 2024-04-12T04:02:17.938 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj116 2024-04-12T04:02:17.938 DEBUG:boto:Data: 2024-04-12T04:02:17.939 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.939 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.939 DEBUG:boto:Params: {} 2024-04-12T04:02:17.939 DEBUG:boto:Token: None 2024-04-12T04:02:17.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b89676b85a41855c-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.943 DEBUG:boto:b'dgbpxg-63obj1161000falseobj1162024-04-12T03:58:40.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:02:17.943 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.944 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.944 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.944 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.944 DEBUG:boto:Method: GET 2024-04-12T04:02:17.944 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.944 DEBUG:boto:Data: 2024-04-12T04:02:17.944 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.944 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.944 DEBUG:boto:Params: {} 2024-04-12T04:02:17.944 DEBUG:boto:Token: None 2024-04-12T04:02:17.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw= 2024-04-12T04:02:17.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw=', 'Content-Length': '0'} 2024-04-12T04:02:17.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41eb95b49bf337d-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.947 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.947 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.947 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.947 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.947 DEBUG:boto:Method: GET 2024-04-12T04:02:17.947 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.947 DEBUG:boto:Data: 2024-04-12T04:02:17.948 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.948 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.948 DEBUG:boto:Params: {} 2024-04-12T04:02:17.948 DEBUG:boto:Token: None 2024-04-12T04:02:17.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw= 2024-04-12T04:02:17.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw=', 'Content-Length': '0'} 2024-04-12T04:02:17.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f98b70447ff5d4-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.950 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.950 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.950 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.951 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.951 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.951 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.951 DEBUG:boto:Data: 2024-04-12T04:02:17.951 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.951 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.951 DEBUG:boto:Params: {} 2024-04-12T04:02:17.951 DEBUG:boto:Token: None 2024-04-12T04:02:17.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mcHl3kM5pLL2+874WYESGW7XERs= 2024-04-12T04:02:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mcHl3kM5pLL2+874WYESGW7XERs=', 'Content-Length': '0'} 2024-04-12T04:02:17.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538f0241557f87bd-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.953 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.953 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.954 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.954 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.954 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.954 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.954 DEBUG:boto:Data: 2024-04-12T04:02:17.954 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.954 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.954 DEBUG:boto:Params: {} 2024-04-12T04:02:17.954 DEBUG:boto:Token: None 2024-04-12T04:02:17.954 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.955 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mcHl3kM5pLL2+874WYESGW7XERs= 2024-04-12T04:02:17.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mcHl3kM5pLL2+874WYESGW7XERs=', 'Content-Length': '0'} 2024-04-12T04:02:17.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55c0e85080e0f41-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:02:17.956 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.957 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.957 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.957 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.957 DEBUG:boto:Method: GET 2024-04-12T04:02:17.957 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.957 DEBUG:boto:Data: 2024-04-12T04:02:17.957 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.957 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.957 DEBUG:boto:Params: {} 2024-04-12T04:02:17.958 DEBUG:boto:Token: None 2024-04-12T04:02:17.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.958 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw= 2024-04-12T04:02:17.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw=', 'Content-Length': '0'} 2024-04-12T04:02:17.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079cf98fd714db6f7-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.960 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.960 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.960 DEBUG:boto:path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.960 DEBUG:boto:auth_path=/dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.960 DEBUG:boto:Method: GET 2024-04-12T04:02:17.960 DEBUG:boto:Path: /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.960 DEBUG:boto:Data: 2024-04-12T04:02:17.960 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.960 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.960 DEBUG:boto:Params: {} 2024-04-12T04:02:17.961 DEBUG:boto:Token: None 2024-04-12T04:02:17.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124?versionId=null 2024-04-12T04:02:17.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw= 2024-04-12T04:02:17.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5ZEdtaapxdk7QWz9aUB/fHdDqXw=', 'Content-Length': '0'} 2024-04-12T04:02:17.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f9fc527e647ece-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.963 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.963 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.963 DEBUG:boto:path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.963 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.963 DEBUG:boto:Method: GET 2024-04-12T04:02:17.963 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.963 DEBUG:boto:Data: 2024-04-12T04:02:17.963 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.963 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.964 DEBUG:boto:Params: {} 2024-04-12T04:02:17.964 DEBUG:boto:Token: None 2024-04-12T04:02:17.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca9ab565bb98c06-006618b249-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.966 DEBUG:boto:b'dgbpxg-63obj1241000falseobj1242024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.967 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:17.967 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:17.967 DEBUG:boto:path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.967 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.967 DEBUG:boto:Method: GET 2024-04-12T04:02:17.967 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj124 2024-04-12T04:02:17.967 DEBUG:boto:Data: 2024-04-12T04:02:17.967 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.968 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.968 DEBUG:boto:Params: {} 2024-04-12T04:02:17.968 DEBUG:boto:Token: None 2024-04-12T04:02:17.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/ 2024-04-12T04:02:17.968 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY= 2024-04-12T04:02:17.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6S04AT6gQIzAdU2EqEtbZAYC2dY=', 'Content-Length': '0'} 2024-04-12T04:02:17.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f01177c2555f15a-006618b249-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.972 DEBUG:boto:b'dgbpxg-63obj1241000falseobj1242024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:17.972 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.972 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.972 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.973 DEBUG:boto:Path: /dgbpxg-63/obj124 2024-04-12T04:02:17.973 DEBUG:boto:Data: 2024-04-12T04:02:17.973 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.973 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.973 DEBUG:boto:Params: {} 2024-04-12T04:02:17.973 DEBUG:boto:Token: None 2024-04-12T04:02:17.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124 2024-04-12T04:02:17.973 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GluKfeGuyS0+7KUi7chYBxk56Uc= 2024-04-12T04:02:17.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GluKfeGuyS0+7KUi7chYBxk56Uc=', 'Content-Length': '0'} 2024-04-12T04:02:17.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f790e41ab49f90e1-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.975 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.975 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.975 DEBUG:boto:Method: HEAD 2024-04-12T04:02:17.975 DEBUG:boto:Path: /dgbpxg-63/obj124 2024-04-12T04:02:17.976 DEBUG:boto:Data: 2024-04-12T04:02:17.976 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:17.976 DEBUG:boto:Port: 8000 2024-04-12T04:02:17.976 DEBUG:boto:Params: {} 2024-04-12T04:02:17.976 DEBUG:boto:Token: None 2024-04-12T04:02:17.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124 2024-04-12T04:02:17.976 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GluKfeGuyS0+7KUi7chYBxk56Uc= 2024-04-12T04:02:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GluKfeGuyS0+7KUi7chYBxk56Uc=', 'Content-Length': '0'} 2024-04-12T04:02:17.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4cf292de5f78a03-006618b249-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:17.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:02:17.978 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:17.978 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:17.978 DEBUG:boto:Method: GET 2024-04-12T04:02:17.979 DEBUG:boto:Path: /dgbpxg-63/obj124 2024-04-12T04:02:17.979 DEBUG:boto:Data: 2024-04-12T04:02:17.979 DEBUG:boto:Headers: {} 2024-04-12T04:02:17.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:17.979 DEBUG:boto:Port: 8001 2024-04-12T04:02:17.979 DEBUG:boto:Params: {} 2024-04-12T04:02:17.979 DEBUG:boto:Token: None 2024-04-12T04:02:17.979 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:17 GMT /dgbpxg-63/obj124 2024-04-12T04:02:17.979 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yD2jyQLTlXq/19pK3y71BWs7MPY= 2024-04-12T04:02:17.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yD2jyQLTlXq/19pK3y71BWs7MPY=', 'Content-Length': '0'} 2024-04-12T04:02:17.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d248c7720a7fac-006618b249-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.022 DEBUG:boto:path=/dgbpxg-63/obj124 2024-04-12T04:02:18.023 DEBUG:boto:auth_path=/dgbpxg-63/obj124 2024-04-12T04:02:18.023 DEBUG:boto:Method: GET 2024-04-12T04:02:18.023 DEBUG:boto:Path: /dgbpxg-63/obj124 2024-04-12T04:02:18.023 DEBUG:boto:Data: 2024-04-12T04:02:18.023 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.023 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.023 DEBUG:boto:Params: {} 2024-04-12T04:02:18.023 DEBUG:boto:Token: None 2024-04-12T04:02:18.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj124 2024-04-12T04:02:18.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qSm8BxAKxHcV+eB0T+CE3X2uGWM= 2024-04-12T04:02:18.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qSm8BxAKxHcV+eB0T+CE3X2uGWM=', 'Content-Length': '0'} 2024-04-12T04:02:18.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d11b42e9653990f-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.026 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.026 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.026 DEBUG:boto:path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.026 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.026 DEBUG:boto:Method: GET 2024-04-12T04:02:18.026 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.026 DEBUG:boto:Data: 2024-04-12T04:02:18.027 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.027 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.027 DEBUG:boto:Params: {} 2024-04-12T04:02:18.027 DEBUG:boto:Token: None 2024-04-12T04:02:18.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b017bbf10c2ecbf-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.030 DEBUG:boto:b'dgbpxg-63obj1241000falseobj1242024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.030 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.030 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.030 DEBUG:boto:path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.030 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.030 DEBUG:boto:Method: GET 2024-04-12T04:02:18.031 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj124 2024-04-12T04:02:18.031 DEBUG:boto:Data: 2024-04-12T04:02:18.031 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.031 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.031 DEBUG:boto:Params: {} 2024-04-12T04:02:18.031 DEBUG:boto:Token: None 2024-04-12T04:02:18.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1fa17a5aca32af-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.035 DEBUG:boto:b'dgbpxg-63obj1241000falseobj1242024-04-12T03:58:40.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:02:18.036 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.036 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.036 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.036 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.036 DEBUG:boto:Method: GET 2024-04-12T04:02:18.036 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.036 DEBUG:boto:Data: 2024-04-12T04:02:18.036 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.036 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.037 DEBUG:boto:Params: {} 2024-04-12T04:02:18.037 DEBUG:boto:Token: None 2024-04-12T04:02:18.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic= 2024-04-12T04:02:18.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic=', 'Content-Length': '0'} 2024-04-12T04:02:18.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a869f69aea1a6b-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.039 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.039 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.039 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.039 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.039 DEBUG:boto:Method: GET 2024-04-12T04:02:18.039 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.040 DEBUG:boto:Data: 2024-04-12T04:02:18.040 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.040 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.040 DEBUG:boto:Params: {} 2024-04-12T04:02:18.040 DEBUG:boto:Token: None 2024-04-12T04:02:18.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.040 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic= 2024-04-12T04:02:18.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic=', 'Content-Length': '0'} 2024-04-12T04:02:18.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb02b35d70948522-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.042 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.042 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.042 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.042 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.043 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.043 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.043 DEBUG:boto:Data: 2024-04-12T04:02:18.043 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.043 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.043 DEBUG:boto:Params: {} 2024-04-12T04:02:18.043 DEBUG:boto:Token: None 2024-04-12T04:02:18.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PM1hv5SAktN7437uOxDj8otkE4s= 2024-04-12T04:02:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PM1hv5SAktN7437uOxDj8otkE4s=', 'Content-Length': '0'} 2024-04-12T04:02:18.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfb787679593bff-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.045 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.045 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.045 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.045 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.045 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.046 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.046 DEBUG:boto:Data: 2024-04-12T04:02:18.046 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.046 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.046 DEBUG:boto:Params: {} 2024-04-12T04:02:18.046 DEBUG:boto:Token: None 2024-04-12T04:02:18.046 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.046 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PM1hv5SAktN7437uOxDj8otkE4s= 2024-04-12T04:02:18.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PM1hv5SAktN7437uOxDj8otkE4s=', 'Content-Length': '0'} 2024-04-12T04:02:18.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b9b8933f22d5e2-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:02:18.048 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.049 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.049 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.049 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.049 DEBUG:boto:Method: GET 2024-04-12T04:02:18.049 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.049 DEBUG:boto:Data: 2024-04-12T04:02:18.049 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.049 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.049 DEBUG:boto:Params: {} 2024-04-12T04:02:18.050 DEBUG:boto:Token: None 2024-04-12T04:02:18.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.050 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic= 2024-04-12T04:02:18.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic=', 'Content-Length': '0'} 2024-04-12T04:02:18.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4646867868eca8-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.052 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.052 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.052 DEBUG:boto:path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.052 DEBUG:boto:auth_path=/dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.052 DEBUG:boto:Method: GET 2024-04-12T04:02:18.052 DEBUG:boto:Path: /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.052 DEBUG:boto:Data: 2024-04-12T04:02:18.052 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.053 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.053 DEBUG:boto:Params: {} 2024-04-12T04:02:18.053 DEBUG:boto:Token: None 2024-04-12T04:02:18.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145?versionId=null 2024-04-12T04:02:18.053 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic= 2024-04-12T04:02:18.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YPMvSlRfSbD9e/kU4tNOCMrUAic=', 'Content-Length': '0'} 2024-04-12T04:02:18.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140c9ed3c0ecbb7f-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.055 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.055 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.055 DEBUG:boto:path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.055 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.055 DEBUG:boto:Method: GET 2024-04-12T04:02:18.055 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.055 DEBUG:boto:Data: 2024-04-12T04:02:18.056 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.056 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.056 DEBUG:boto:Params: {} 2024-04-12T04:02:18.056 DEBUG:boto:Token: None 2024-04-12T04:02:18.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002032b67a4209d196-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.058 DEBUG:boto:b'dgbpxg-63obj1451000falseobj1452024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.059 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.059 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.059 DEBUG:boto:path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.059 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.059 DEBUG:boto:Method: GET 2024-04-12T04:02:18.059 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.059 DEBUG:boto:Data: 2024-04-12T04:02:18.059 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.059 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.060 DEBUG:boto:Params: {} 2024-04-12T04:02:18.060 DEBUG:boto:Token: None 2024-04-12T04:02:18.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b6278fdae78137-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.063 DEBUG:boto:b'dgbpxg-63obj1451000falseobj1452024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.064 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.064 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.064 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.064 DEBUG:boto:Path: /dgbpxg-63/obj145 2024-04-12T04:02:18.064 DEBUG:boto:Data: 2024-04-12T04:02:18.064 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.064 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.064 DEBUG:boto:Params: {} 2024-04-12T04:02:18.065 DEBUG:boto:Token: None 2024-04-12T04:02:18.065 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145 2024-04-12T04:02:18.065 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:167a5M2cbD/XZg552OgUz/cRIZY= 2024-04-12T04:02:18.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:167a5M2cbD/XZg552OgUz/cRIZY=', 'Content-Length': '0'} 2024-04-12T04:02:18.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f86f6664d25266-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.067 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.067 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.067 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.067 DEBUG:boto:Path: /dgbpxg-63/obj145 2024-04-12T04:02:18.067 DEBUG:boto:Data: 2024-04-12T04:02:18.067 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.067 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.067 DEBUG:boto:Params: {} 2024-04-12T04:02:18.068 DEBUG:boto:Token: None 2024-04-12T04:02:18.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145 2024-04-12T04:02:18.068 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:167a5M2cbD/XZg552OgUz/cRIZY= 2024-04-12T04:02:18.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:167a5M2cbD/XZg552OgUz/cRIZY=', 'Content-Length': '0'} 2024-04-12T04:02:18.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb08e0737b6fbf1b-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:02:18.070 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.070 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.070 DEBUG:boto:Method: GET 2024-04-12T04:02:18.070 DEBUG:boto:Path: /dgbpxg-63/obj145 2024-04-12T04:02:18.070 DEBUG:boto:Data: 2024-04-12T04:02:18.070 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.070 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.071 DEBUG:boto:Params: {} 2024-04-12T04:02:18.071 DEBUG:boto:Token: None 2024-04-12T04:02:18.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145 2024-04-12T04:02:18.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XmsEUWzuvI/QCP8i3zLQaS1yqC4= 2024-04-12T04:02:18.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XmsEUWzuvI/QCP8i3zLQaS1yqC4=', 'Content-Length': '0'} 2024-04-12T04:02:18.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5f4e3ba2955f10-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.114 DEBUG:boto:path=/dgbpxg-63/obj145 2024-04-12T04:02:18.115 DEBUG:boto:auth_path=/dgbpxg-63/obj145 2024-04-12T04:02:18.115 DEBUG:boto:Method: GET 2024-04-12T04:02:18.115 DEBUG:boto:Path: /dgbpxg-63/obj145 2024-04-12T04:02:18.115 DEBUG:boto:Data: 2024-04-12T04:02:18.115 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.115 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.115 DEBUG:boto:Params: {} 2024-04-12T04:02:18.115 DEBUG:boto:Token: None 2024-04-12T04:02:18.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj145 2024-04-12T04:02:18.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XmsEUWzuvI/QCP8i3zLQaS1yqC4= 2024-04-12T04:02:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XmsEUWzuvI/QCP8i3zLQaS1yqC4=', 'Content-Length': '0'} 2024-04-12T04:02:18.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bddc73d8b127bb2-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.118 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.118 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.118 DEBUG:boto:path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.118 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.118 DEBUG:boto:Method: GET 2024-04-12T04:02:18.118 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.118 DEBUG:boto:Data: 2024-04-12T04:02:18.118 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.118 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.119 DEBUG:boto:Params: {} 2024-04-12T04:02:18.119 DEBUG:boto:Token: None 2024-04-12T04:02:18.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c15f85fbc87ff5-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.121 DEBUG:boto:b'dgbpxg-63obj1451000falseobj1452024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.122 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.122 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.122 DEBUG:boto:path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.122 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.122 DEBUG:boto:Method: GET 2024-04-12T04:02:18.122 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj145 2024-04-12T04:02:18.122 DEBUG:boto:Data: 2024-04-12T04:02:18.122 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.123 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.123 DEBUG:boto:Params: {} 2024-04-12T04:02:18.123 DEBUG:boto:Token: None 2024-04-12T04:02:18.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fef36d36434d883-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.127 DEBUG:boto:b'dgbpxg-63obj1451000falseobj1452024-04-12T03:58:40.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:02:18.127 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.127 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.128 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.128 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.128 DEBUG:boto:Method: GET 2024-04-12T04:02:18.128 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.128 DEBUG:boto:Data: 2024-04-12T04:02:18.128 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.128 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.128 DEBUG:boto:Params: {} 2024-04-12T04:02:18.128 DEBUG:boto:Token: None 2024-04-12T04:02:18.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4= 2024-04-12T04:02:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4=', 'Content-Length': '0'} 2024-04-12T04:02:18.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9a4eb744e57737-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.130 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.131 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.131 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.131 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.131 DEBUG:boto:Method: GET 2024-04-12T04:02:18.131 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.131 DEBUG:boto:Data: 2024-04-12T04:02:18.131 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.131 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.131 DEBUG:boto:Params: {} 2024-04-12T04:02:18.131 DEBUG:boto:Token: None 2024-04-12T04:02:18.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4= 2024-04-12T04:02:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4=', 'Content-Length': '0'} 2024-04-12T04:02:18.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a455258f1637f1-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.133 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.134 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.134 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.134 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.134 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.134 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.134 DEBUG:boto:Data: 2024-04-12T04:02:18.134 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.134 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.134 DEBUG:boto:Params: {} 2024-04-12T04:02:18.135 DEBUG:boto:Token: None 2024-04-12T04:02:18.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+JUoR5C0dM2hqTHikTtxrY9uEU= 2024-04-12T04:02:18.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+JUoR5C0dM2hqTHikTtxrY9uEU=', 'Content-Length': '0'} 2024-04-12T04:02:18.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427aee171f5fe26f-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.137 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.137 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.137 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.137 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.137 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.137 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.137 DEBUG:boto:Data: 2024-04-12T04:02:18.137 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.138 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.138 DEBUG:boto:Params: {} 2024-04-12T04:02:18.138 DEBUG:boto:Token: None 2024-04-12T04:02:18.138 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.138 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+JUoR5C0dM2hqTHikTtxrY9uEU= 2024-04-12T04:02:18.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+JUoR5C0dM2hqTHikTtxrY9uEU=', 'Content-Length': '0'} 2024-04-12T04:02:18.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4e7bf858226b34-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:02:18.140 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.140 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.140 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.140 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.140 DEBUG:boto:Method: GET 2024-04-12T04:02:18.140 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.140 DEBUG:boto:Data: 2024-04-12T04:02:18.140 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.140 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.141 DEBUG:boto:Params: {} 2024-04-12T04:02:18.141 DEBUG:boto:Token: None 2024-04-12T04:02:18.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.141 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4= 2024-04-12T04:02:18.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4=', 'Content-Length': '0'} 2024-04-12T04:02:18.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319d41ab50107443-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.143 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.143 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.143 DEBUG:boto:path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.143 DEBUG:boto:auth_path=/dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.143 DEBUG:boto:Method: GET 2024-04-12T04:02:18.143 DEBUG:boto:Path: /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.143 DEBUG:boto:Data: 2024-04-12T04:02:18.143 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.143 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.144 DEBUG:boto:Params: {} 2024-04-12T04:02:18.144 DEBUG:boto:Token: None 2024-04-12T04:02:18.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148?versionId=null 2024-04-12T04:02:18.144 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4= 2024-04-12T04:02:18.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q+CJ0OWt6LN9VgKK5Jvh0DEfJs4=', 'Content-Length': '0'} 2024-04-12T04:02:18.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c712d857302a6ad5-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.146 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.146 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.146 DEBUG:boto:path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.146 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.146 DEBUG:boto:Method: GET 2024-04-12T04:02:18.146 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.146 DEBUG:boto:Data: 2024-04-12T04:02:18.146 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.147 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.147 DEBUG:boto:Params: {} 2024-04-12T04:02:18.147 DEBUG:boto:Token: None 2024-04-12T04:02:18.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.147 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49caca7e1a6a703-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.190 DEBUG:boto:b'dgbpxg-63obj1481000falseobj1482024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.191 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.191 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.191 DEBUG:boto:path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.191 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.191 DEBUG:boto:Method: GET 2024-04-12T04:02:18.191 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.191 DEBUG:boto:Data: 2024-04-12T04:02:18.191 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.192 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.192 DEBUG:boto:Params: {} 2024-04-12T04:02:18.192 DEBUG:boto:Token: None 2024-04-12T04:02:18.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009742cdb7e89705e2-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.196 DEBUG:boto:b'dgbpxg-63obj1481000falseobj1482024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.196 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.196 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.196 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.196 DEBUG:boto:Path: /dgbpxg-63/obj148 2024-04-12T04:02:18.197 DEBUG:boto:Data: 2024-04-12T04:02:18.197 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.197 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.197 DEBUG:boto:Params: {} 2024-04-12T04:02:18.197 DEBUG:boto:Token: None 2024-04-12T04:02:18.197 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148 2024-04-12T04:02:18.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C1yGhEJnLxkMd8QBa1P+x2ft8Kc= 2024-04-12T04:02:18.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C1yGhEJnLxkMd8QBa1P+x2ft8Kc=', 'Content-Length': '0'} 2024-04-12T04:02:18.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ca5a12d8972116-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.199 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.199 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.200 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.200 DEBUG:boto:Path: /dgbpxg-63/obj148 2024-04-12T04:02:18.200 DEBUG:boto:Data: 2024-04-12T04:02:18.200 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.200 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.200 DEBUG:boto:Params: {} 2024-04-12T04:02:18.200 DEBUG:boto:Token: None 2024-04-12T04:02:18.200 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148 2024-04-12T04:02:18.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C1yGhEJnLxkMd8QBa1P+x2ft8Kc= 2024-04-12T04:02:18.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C1yGhEJnLxkMd8QBa1P+x2ft8Kc=', 'Content-Length': '0'} 2024-04-12T04:02:18.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7033c02eea8a0f6-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:02:18.202 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.203 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.203 DEBUG:boto:Method: GET 2024-04-12T04:02:18.203 DEBUG:boto:Path: /dgbpxg-63/obj148 2024-04-12T04:02:18.203 DEBUG:boto:Data: 2024-04-12T04:02:18.203 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.203 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.203 DEBUG:boto:Params: {} 2024-04-12T04:02:18.203 DEBUG:boto:Token: None 2024-04-12T04:02:18.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148 2024-04-12T04:02:18.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7R3SgIo8B81AgAPTgJNsp/0+CFs= 2024-04-12T04:02:18.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7R3SgIo8B81AgAPTgJNsp/0+CFs=', 'Content-Length': '0'} 2024-04-12T04:02:18.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e287d616fdd8344-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.205 DEBUG:boto:path=/dgbpxg-63/obj148 2024-04-12T04:02:18.205 DEBUG:boto:auth_path=/dgbpxg-63/obj148 2024-04-12T04:02:18.206 DEBUG:boto:Method: GET 2024-04-12T04:02:18.206 DEBUG:boto:Path: /dgbpxg-63/obj148 2024-04-12T04:02:18.206 DEBUG:boto:Data: 2024-04-12T04:02:18.206 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.206 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.206 DEBUG:boto:Params: {} 2024-04-12T04:02:18.206 DEBUG:boto:Token: None 2024-04-12T04:02:18.206 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj148 2024-04-12T04:02:18.207 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7R3SgIo8B81AgAPTgJNsp/0+CFs= 2024-04-12T04:02:18.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7R3SgIo8B81AgAPTgJNsp/0+CFs=', 'Content-Length': '0'} 2024-04-12T04:02:18.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8238a620a19841b-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.209 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.209 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.209 DEBUG:boto:path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.209 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.209 DEBUG:boto:Method: GET 2024-04-12T04:02:18.209 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.209 DEBUG:boto:Data: 2024-04-12T04:02:18.209 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.209 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.209 DEBUG:boto:Params: {} 2024-04-12T04:02:18.210 DEBUG:boto:Token: None 2024-04-12T04:02:18.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.210 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005043bcd1dd3563a1-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.212 DEBUG:boto:b'dgbpxg-63obj1481000falseobj1482024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.213 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.213 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.213 DEBUG:boto:path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.213 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.213 DEBUG:boto:Method: GET 2024-04-12T04:02:18.213 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj148 2024-04-12T04:02:18.213 DEBUG:boto:Data: 2024-04-12T04:02:18.213 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.214 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.214 DEBUG:boto:Params: {} 2024-04-12T04:02:18.214 DEBUG:boto:Token: None 2024-04-12T04:02:18.214 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.215 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb8cb4c5b74a474-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.218 DEBUG:boto:b'dgbpxg-63obj1481000falseobj1482024-04-12T03:58:40.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:02:18.219 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.219 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.219 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.219 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.219 DEBUG:boto:Method: GET 2024-04-12T04:02:18.219 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.219 DEBUG:boto:Data: 2024-04-12T04:02:18.219 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.219 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.219 DEBUG:boto:Params: {} 2024-04-12T04:02:18.220 DEBUG:boto:Token: None 2024-04-12T04:02:18.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw= 2024-04-12T04:02:18.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw=', 'Content-Length': '0'} 2024-04-12T04:02:18.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc05e3909d40fe2-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.222 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.222 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.222 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.222 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.222 DEBUG:boto:Method: GET 2024-04-12T04:02:18.222 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.222 DEBUG:boto:Data: 2024-04-12T04:02:18.222 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.223 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.223 DEBUG:boto:Params: {} 2024-04-12T04:02:18.223 DEBUG:boto:Token: None 2024-04-12T04:02:18.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.223 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw= 2024-04-12T04:02:18.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw=', 'Content-Length': '0'} 2024-04-12T04:02:18.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01d1373895f1fd8-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.225 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.225 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.225 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.225 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.225 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.225 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.225 DEBUG:boto:Data: 2024-04-12T04:02:18.226 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.226 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.226 DEBUG:boto:Params: {} 2024-04-12T04:02:18.226 DEBUG:boto:Token: None 2024-04-12T04:02:18.226 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.226 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pBWR4TBgmijBSV7rbxxTO8r3qrg= 2024-04-12T04:02:18.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pBWR4TBgmijBSV7rbxxTO8r3qrg=', 'Content-Length': '0'} 2024-04-12T04:02:18.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614253b40e02fa09-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.228 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.228 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.228 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.228 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.228 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.228 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.228 DEBUG:boto:Data: 2024-04-12T04:02:18.229 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.229 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.229 DEBUG:boto:Params: {} 2024-04-12T04:02:18.229 DEBUG:boto:Token: None 2024-04-12T04:02:18.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.229 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pBWR4TBgmijBSV7rbxxTO8r3qrg= 2024-04-12T04:02:18.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pBWR4TBgmijBSV7rbxxTO8r3qrg=', 'Content-Length': '0'} 2024-04-12T04:02:18.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a933c9826c7102-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:02:18.231 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.231 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.231 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.231 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.231 DEBUG:boto:Method: GET 2024-04-12T04:02:18.231 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.232 DEBUG:boto:Data: 2024-04-12T04:02:18.232 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.232 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.232 DEBUG:boto:Params: {} 2024-04-12T04:02:18.232 DEBUG:boto:Token: None 2024-04-12T04:02:18.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw= 2024-04-12T04:02:18.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw=', 'Content-Length': '0'} 2024-04-12T04:02:18.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006430f19981c911e2-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.234 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.234 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.234 DEBUG:boto:path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.234 DEBUG:boto:auth_path=/dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.235 DEBUG:boto:Method: GET 2024-04-12T04:02:18.235 DEBUG:boto:Path: /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.235 DEBUG:boto:Data: 2024-04-12T04:02:18.235 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.235 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.235 DEBUG:boto:Params: {} 2024-04-12T04:02:18.235 DEBUG:boto:Token: None 2024-04-12T04:02:18.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155?versionId=null 2024-04-12T04:02:18.235 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw= 2024-04-12T04:02:18.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:27gCyy56B8bZ3s0WpG899iCXgFw=', 'Content-Length': '0'} 2024-04-12T04:02:18.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e81d765e2c2287c-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.278 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.279 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.279 DEBUG:boto:path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.279 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.279 DEBUG:boto:Method: GET 2024-04-12T04:02:18.279 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.279 DEBUG:boto:Data: 2024-04-12T04:02:18.279 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.279 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.280 DEBUG:boto:Params: {} 2024-04-12T04:02:18.280 DEBUG:boto:Token: None 2024-04-12T04:02:18.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369b2a8cafa12f11-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.285 DEBUG:boto:b'dgbpxg-63obj1551000falseobj1552024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.286 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.286 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.286 DEBUG:boto:path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.286 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.286 DEBUG:boto:Method: GET 2024-04-12T04:02:18.286 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.286 DEBUG:boto:Data: 2024-04-12T04:02:18.286 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.287 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.287 DEBUG:boto:Params: {} 2024-04-12T04:02:18.287 DEBUG:boto:Token: None 2024-04-12T04:02:18.287 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.287 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085de1b165a126b36-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.291 DEBUG:boto:b'dgbpxg-63obj1551000falseobj1552024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.291 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.291 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.291 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.291 DEBUG:boto:Path: /dgbpxg-63/obj155 2024-04-12T04:02:18.291 DEBUG:boto:Data: 2024-04-12T04:02:18.291 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.292 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.292 DEBUG:boto:Params: {} 2024-04-12T04:02:18.292 DEBUG:boto:Token: None 2024-04-12T04:02:18.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155 2024-04-12T04:02:18.292 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZzbM+CTXBno7yJUEONZWaAJMjYk= 2024-04-12T04:02:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZzbM+CTXBno7yJUEONZWaAJMjYk=', 'Content-Length': '0'} 2024-04-12T04:02:18.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083fff692232d976-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.294 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.294 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.294 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.294 DEBUG:boto:Path: /dgbpxg-63/obj155 2024-04-12T04:02:18.294 DEBUG:boto:Data: 2024-04-12T04:02:18.294 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.294 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.294 DEBUG:boto:Params: {} 2024-04-12T04:02:18.295 DEBUG:boto:Token: None 2024-04-12T04:02:18.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155 2024-04-12T04:02:18.295 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZzbM+CTXBno7yJUEONZWaAJMjYk= 2024-04-12T04:02:18.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZzbM+CTXBno7yJUEONZWaAJMjYk=', 'Content-Length': '0'} 2024-04-12T04:02:18.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036829e9298e7890e-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:02:18.297 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.297 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.297 DEBUG:boto:Method: GET 2024-04-12T04:02:18.297 DEBUG:boto:Path: /dgbpxg-63/obj155 2024-04-12T04:02:18.297 DEBUG:boto:Data: 2024-04-12T04:02:18.297 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.297 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.298 DEBUG:boto:Params: {} 2024-04-12T04:02:18.298 DEBUG:boto:Token: None 2024-04-12T04:02:18.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155 2024-04-12T04:02:18.298 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p22vrDFb8I8Qo0IYfo0YPVItDEQ= 2024-04-12T04:02:18.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p22vrDFb8I8Qo0IYfo0YPVItDEQ=', 'Content-Length': '0'} 2024-04-12T04:02:18.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfbdf4bf612a33d8-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.343 DEBUG:boto:path=/dgbpxg-63/obj155 2024-04-12T04:02:18.343 DEBUG:boto:auth_path=/dgbpxg-63/obj155 2024-04-12T04:02:18.343 DEBUG:boto:Method: GET 2024-04-12T04:02:18.344 DEBUG:boto:Path: /dgbpxg-63/obj155 2024-04-12T04:02:18.344 DEBUG:boto:Data: 2024-04-12T04:02:18.344 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.344 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.344 DEBUG:boto:Params: {} 2024-04-12T04:02:18.344 DEBUG:boto:Token: None 2024-04-12T04:02:18.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj155 2024-04-12T04:02:18.344 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p22vrDFb8I8Qo0IYfo0YPVItDEQ= 2024-04-12T04:02:18.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p22vrDFb8I8Qo0IYfo0YPVItDEQ=', 'Content-Length': '0'} 2024-04-12T04:02:18.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b115b9bb51dd58-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.347 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.347 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.347 DEBUG:boto:path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.347 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.347 DEBUG:boto:Method: GET 2024-04-12T04:02:18.347 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.347 DEBUG:boto:Data: 2024-04-12T04:02:18.347 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.347 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.348 DEBUG:boto:Params: {} 2024-04-12T04:02:18.348 DEBUG:boto:Token: None 2024-04-12T04:02:18.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f9958d11afcdba-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.350 DEBUG:boto:b'dgbpxg-63obj1551000falseobj1552024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.351 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.351 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.351 DEBUG:boto:path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.351 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.351 DEBUG:boto:Method: GET 2024-04-12T04:02:18.351 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj155 2024-04-12T04:02:18.351 DEBUG:boto:Data: 2024-04-12T04:02:18.351 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.352 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.352 DEBUG:boto:Params: {} 2024-04-12T04:02:18.352 DEBUG:boto:Token: None 2024-04-12T04:02:18.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f618e2dde16ce5-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.398 DEBUG:boto:b'dgbpxg-63obj1551000falseobj1552024-04-12T03:58:40.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:02:18.399 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.399 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.399 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.399 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.399 DEBUG:boto:Method: GET 2024-04-12T04:02:18.399 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.400 DEBUG:boto:Data: 2024-04-12T04:02:18.400 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.400 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.400 DEBUG:boto:Params: {} 2024-04-12T04:02:18.400 DEBUG:boto:Token: None 2024-04-12T04:02:18.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc= 2024-04-12T04:02:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc=', 'Content-Length': '0'} 2024-04-12T04:02:18.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003709c7d46a163cad-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.403 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.403 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.403 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.403 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.403 DEBUG:boto:Method: GET 2024-04-12T04:02:18.403 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.403 DEBUG:boto:Data: 2024-04-12T04:02:18.403 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.403 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.403 DEBUG:boto:Params: {} 2024-04-12T04:02:18.404 DEBUG:boto:Token: None 2024-04-12T04:02:18.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc= 2024-04-12T04:02:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc=', 'Content-Length': '0'} 2024-04-12T04:02:18.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f5f00032e782d57-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.406 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.406 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.406 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.406 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.406 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.407 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.407 DEBUG:boto:Data: 2024-04-12T04:02:18.407 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.407 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.407 DEBUG:boto:Params: {} 2024-04-12T04:02:18.407 DEBUG:boto:Token: None 2024-04-12T04:02:18.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qHbaDHJAhtj07vepkh/EbFCIBs8= 2024-04-12T04:02:18.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qHbaDHJAhtj07vepkh/EbFCIBs8=', 'Content-Length': '0'} 2024-04-12T04:02:18.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd8d7792cf182cd-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.409 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.409 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.409 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.409 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.409 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.410 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.410 DEBUG:boto:Data: 2024-04-12T04:02:18.410 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.410 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.410 DEBUG:boto:Params: {} 2024-04-12T04:02:18.410 DEBUG:boto:Token: None 2024-04-12T04:02:18.410 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qHbaDHJAhtj07vepkh/EbFCIBs8= 2024-04-12T04:02:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qHbaDHJAhtj07vepkh/EbFCIBs8=', 'Content-Length': '0'} 2024-04-12T04:02:18.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219d6ab6451ca062-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:02:18.412 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.413 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.413 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.413 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.413 DEBUG:boto:Method: GET 2024-04-12T04:02:18.413 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.413 DEBUG:boto:Data: 2024-04-12T04:02:18.413 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.413 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.413 DEBUG:boto:Params: {} 2024-04-12T04:02:18.413 DEBUG:boto:Token: None 2024-04-12T04:02:18.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc= 2024-04-12T04:02:18.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc=', 'Content-Length': '0'} 2024-04-12T04:02:18.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000381396443b8f3a1e-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.415 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.415 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.416 DEBUG:boto:path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.416 DEBUG:boto:auth_path=/dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.416 DEBUG:boto:Method: GET 2024-04-12T04:02:18.416 DEBUG:boto:Path: /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.416 DEBUG:boto:Data: 2024-04-12T04:02:18.416 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.416 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.416 DEBUG:boto:Params: {} 2024-04-12T04:02:18.417 DEBUG:boto:Token: None 2024-04-12T04:02:18.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17?versionId=null 2024-04-12T04:02:18.417 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc= 2024-04-12T04:02:18.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1dpZ+OflUbimI9kNsUOHULjY8Nc=', 'Content-Length': '0'} 2024-04-12T04:02:18.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006649d1b2d7a6a642-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.419 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.419 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.419 DEBUG:boto:path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.419 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.419 DEBUG:boto:Method: GET 2024-04-12T04:02:18.419 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.419 DEBUG:boto:Data: 2024-04-12T04:02:18.419 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.420 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.420 DEBUG:boto:Params: {} 2024-04-12T04:02:18.420 DEBUG:boto:Token: None 2024-04-12T04:02:18.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.420 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000048347f37bc4ad2a-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.422 DEBUG:boto:b'dgbpxg-63obj171000falseobj172024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.423 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.423 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.423 DEBUG:boto:path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.423 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.423 DEBUG:boto:Method: GET 2024-04-12T04:02:18.423 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.423 DEBUG:boto:Data: 2024-04-12T04:02:18.423 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.424 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.424 DEBUG:boto:Params: {} 2024-04-12T04:02:18.424 DEBUG:boto:Token: None 2024-04-12T04:02:18.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed0e8260f3098f4-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.449 DEBUG:boto:b'dgbpxg-63obj171000falseobj172024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.449 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.449 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.449 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.449 DEBUG:boto:Path: /dgbpxg-63/obj17 2024-04-12T04:02:18.449 DEBUG:boto:Data: 2024-04-12T04:02:18.450 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.450 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.450 DEBUG:boto:Params: {} 2024-04-12T04:02:18.450 DEBUG:boto:Token: None 2024-04-12T04:02:18.450 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17 2024-04-12T04:02:18.450 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nlJJYdw1UQBDbrYAlJNPu5/gQGs= 2024-04-12T04:02:18.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nlJJYdw1UQBDbrYAlJNPu5/gQGs=', 'Content-Length': '0'} 2024-04-12T04:02:18.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003699decf057f1023-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.453 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.453 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.453 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.453 DEBUG:boto:Path: /dgbpxg-63/obj17 2024-04-12T04:02:18.453 DEBUG:boto:Data: 2024-04-12T04:02:18.453 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.453 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.453 DEBUG:boto:Params: {} 2024-04-12T04:02:18.453 DEBUG:boto:Token: None 2024-04-12T04:02:18.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17 2024-04-12T04:02:18.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nlJJYdw1UQBDbrYAlJNPu5/gQGs= 2024-04-12T04:02:18.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nlJJYdw1UQBDbrYAlJNPu5/gQGs=', 'Content-Length': '0'} 2024-04-12T04:02:18.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147f9bf5b9b24dc9-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:02:18.456 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.456 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.456 DEBUG:boto:Method: GET 2024-04-12T04:02:18.456 DEBUG:boto:Path: /dgbpxg-63/obj17 2024-04-12T04:02:18.456 DEBUG:boto:Data: 2024-04-12T04:02:18.456 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.456 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.457 DEBUG:boto:Params: {} 2024-04-12T04:02:18.457 DEBUG:boto:Token: None 2024-04-12T04:02:18.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17 2024-04-12T04:02:18.457 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJAR7kyNOeMtclX3OsY3jPrZxk8= 2024-04-12T04:02:18.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJAR7kyNOeMtclX3OsY3jPrZxk8=', 'Content-Length': '0'} 2024-04-12T04:02:18.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56cd11aa078cf09-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.506 DEBUG:boto:path=/dgbpxg-63/obj17 2024-04-12T04:02:18.507 DEBUG:boto:auth_path=/dgbpxg-63/obj17 2024-04-12T04:02:18.507 DEBUG:boto:Method: GET 2024-04-12T04:02:18.507 DEBUG:boto:Path: /dgbpxg-63/obj17 2024-04-12T04:02:18.507 DEBUG:boto:Data: 2024-04-12T04:02:18.507 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.507 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.507 DEBUG:boto:Params: {} 2024-04-12T04:02:18.508 DEBUG:boto:Token: None 2024-04-12T04:02:18.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj17 2024-04-12T04:02:18.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJAR7kyNOeMtclX3OsY3jPrZxk8= 2024-04-12T04:02:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJAR7kyNOeMtclX3OsY3jPrZxk8=', 'Content-Length': '0'} 2024-04-12T04:02:18.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d079e3dca3f477-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.510 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.510 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.510 DEBUG:boto:path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.510 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.510 DEBUG:boto:Method: GET 2024-04-12T04:02:18.511 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.511 DEBUG:boto:Data: 2024-04-12T04:02:18.511 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.511 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.511 DEBUG:boto:Params: {} 2024-04-12T04:02:18.511 DEBUG:boto:Token: None 2024-04-12T04:02:18.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002665cad7303951d4-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.514 DEBUG:boto:b'dgbpxg-63obj171000falseobj172024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.515 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.515 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.515 DEBUG:boto:path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.515 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.515 DEBUG:boto:Method: GET 2024-04-12T04:02:18.515 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj17 2024-04-12T04:02:18.515 DEBUG:boto:Data: 2024-04-12T04:02:18.515 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.516 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.516 DEBUG:boto:Params: {} 2024-04-12T04:02:18.516 DEBUG:boto:Token: None 2024-04-12T04:02:18.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032501ec18ae69f56-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.521 DEBUG:boto:b'dgbpxg-63obj171000falseobj172024-04-12T03:58:39.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:02:18.522 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.522 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.522 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.522 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.522 DEBUG:boto:Method: GET 2024-04-12T04:02:18.523 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.523 DEBUG:boto:Data: 2024-04-12T04:02:18.523 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.523 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.523 DEBUG:boto:Params: {} 2024-04-12T04:02:18.523 DEBUG:boto:Token: None 2024-04-12T04:02:18.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg= 2024-04-12T04:02:18.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg=', 'Content-Length': '0'} 2024-04-12T04:02:18.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5612ab9c728a28-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.525 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.526 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.526 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.526 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.526 DEBUG:boto:Method: GET 2024-04-12T04:02:18.526 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.526 DEBUG:boto:Data: 2024-04-12T04:02:18.526 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.526 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.526 DEBUG:boto:Params: {} 2024-04-12T04:02:18.527 DEBUG:boto:Token: None 2024-04-12T04:02:18.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg= 2024-04-12T04:02:18.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg=', 'Content-Length': '0'} 2024-04-12T04:02:18.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0840e5339a4c5f-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.529 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.529 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.529 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.529 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.529 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.529 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.529 DEBUG:boto:Data: 2024-04-12T04:02:18.529 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.530 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.530 DEBUG:boto:Params: {} 2024-04-12T04:02:18.530 DEBUG:boto:Token: None 2024-04-12T04:02:18.530 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:doyOWMWEBnick8SF2bHDfz+Bg+o= 2024-04-12T04:02:18.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:doyOWMWEBnick8SF2bHDfz+Bg+o=', 'Content-Length': '0'} 2024-04-12T04:02:18.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43c50ac96baf123-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.601 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.601 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.601 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.601 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.601 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.601 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.601 DEBUG:boto:Data: 2024-04-12T04:02:18.602 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.602 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.602 DEBUG:boto:Params: {} 2024-04-12T04:02:18.602 DEBUG:boto:Token: None 2024-04-12T04:02:18.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:doyOWMWEBnick8SF2bHDfz+Bg+o= 2024-04-12T04:02:18.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:doyOWMWEBnick8SF2bHDfz+Bg+o=', 'Content-Length': '0'} 2024-04-12T04:02:18.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af8f57bd196de2a-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:02:18.604 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.604 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.604 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.605 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.605 DEBUG:boto:Method: GET 2024-04-12T04:02:18.605 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.605 DEBUG:boto:Data: 2024-04-12T04:02:18.605 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.605 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.605 DEBUG:boto:Params: {} 2024-04-12T04:02:18.605 DEBUG:boto:Token: None 2024-04-12T04:02:18.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg= 2024-04-12T04:02:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg=', 'Content-Length': '0'} 2024-04-12T04:02:18.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2631dada90466f-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.608 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.608 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.608 DEBUG:boto:path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.608 DEBUG:boto:auth_path=/dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.608 DEBUG:boto:Method: GET 2024-04-12T04:02:18.608 DEBUG:boto:Path: /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.608 DEBUG:boto:Data: 2024-04-12T04:02:18.609 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.609 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.609 DEBUG:boto:Params: {} 2024-04-12T04:02:18.609 DEBUG:boto:Token: None 2024-04-12T04:02:18.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185?versionId=null 2024-04-12T04:02:18.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg= 2024-04-12T04:02:18.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:U/gs2/J2SQeLrnHw36e/U0VZPmg=', 'Content-Length': '0'} 2024-04-12T04:02:18.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd41c02ada5a5cbf-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.611 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.611 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.612 DEBUG:boto:path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.612 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.612 DEBUG:boto:Method: GET 2024-04-12T04:02:18.612 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.613 DEBUG:boto:Data: 2024-04-12T04:02:18.613 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.613 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.613 DEBUG:boto:Params: {} 2024-04-12T04:02:18.613 DEBUG:boto:Token: None 2024-04-12T04:02:18.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.614 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e2f9a8295d620ab-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.616 DEBUG:boto:b'dgbpxg-63obj1851000falseobj1852024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.616 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.616 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.617 DEBUG:boto:path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.617 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.617 DEBUG:boto:Method: GET 2024-04-12T04:02:18.617 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.617 DEBUG:boto:Data: 2024-04-12T04:02:18.617 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.617 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.617 DEBUG:boto:Params: {} 2024-04-12T04:02:18.617 DEBUG:boto:Token: None 2024-04-12T04:02:18.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.618 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000224931455d603a0-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.621 DEBUG:boto:b'dgbpxg-63obj1851000falseobj1852024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.622 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.622 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.622 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.622 DEBUG:boto:Path: /dgbpxg-63/obj185 2024-04-12T04:02:18.622 DEBUG:boto:Data: 2024-04-12T04:02:18.622 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.623 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.623 DEBUG:boto:Params: {} 2024-04-12T04:02:18.623 DEBUG:boto:Token: None 2024-04-12T04:02:18.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185 2024-04-12T04:02:18.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f6ZrEX22j+IDRsfWFGlVvjlMdhk= 2024-04-12T04:02:18.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f6ZrEX22j+IDRsfWFGlVvjlMdhk=', 'Content-Length': '0'} 2024-04-12T04:02:18.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72d700bf37bf4ac-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.625 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.625 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.625 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.625 DEBUG:boto:Path: /dgbpxg-63/obj185 2024-04-12T04:02:18.625 DEBUG:boto:Data: 2024-04-12T04:02:18.625 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.626 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.626 DEBUG:boto:Params: {} 2024-04-12T04:02:18.626 DEBUG:boto:Token: None 2024-04-12T04:02:18.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185 2024-04-12T04:02:18.626 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f6ZrEX22j+IDRsfWFGlVvjlMdhk= 2024-04-12T04:02:18.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f6ZrEX22j+IDRsfWFGlVvjlMdhk=', 'Content-Length': '0'} 2024-04-12T04:02:18.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe857f74e2d38e07-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:02:18.628 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.628 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.628 DEBUG:boto:Method: GET 2024-04-12T04:02:18.628 DEBUG:boto:Path: /dgbpxg-63/obj185 2024-04-12T04:02:18.628 DEBUG:boto:Data: 2024-04-12T04:02:18.629 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.629 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.629 DEBUG:boto:Params: {} 2024-04-12T04:02:18.629 DEBUG:boto:Token: None 2024-04-12T04:02:18.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185 2024-04-12T04:02:18.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JfXUxez2+RDZmWmfLMqxGeB+n6Y= 2024-04-12T04:02:18.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JfXUxez2+RDZmWmfLMqxGeB+n6Y=', 'Content-Length': '0'} 2024-04-12T04:02:18.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f1339cb32349f2-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.631 DEBUG:boto:path=/dgbpxg-63/obj185 2024-04-12T04:02:18.631 DEBUG:boto:auth_path=/dgbpxg-63/obj185 2024-04-12T04:02:18.631 DEBUG:boto:Method: GET 2024-04-12T04:02:18.631 DEBUG:boto:Path: /dgbpxg-63/obj185 2024-04-12T04:02:18.632 DEBUG:boto:Data: 2024-04-12T04:02:18.632 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.632 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.632 DEBUG:boto:Params: {} 2024-04-12T04:02:18.632 DEBUG:boto:Token: None 2024-04-12T04:02:18.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj185 2024-04-12T04:02:18.632 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JfXUxez2+RDZmWmfLMqxGeB+n6Y= 2024-04-12T04:02:18.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JfXUxez2+RDZmWmfLMqxGeB+n6Y=', 'Content-Length': '0'} 2024-04-12T04:02:18.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3956ac5fefaebd-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.634 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.634 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.634 DEBUG:boto:path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.634 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.635 DEBUG:boto:Method: GET 2024-04-12T04:02:18.635 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.635 DEBUG:boto:Data: 2024-04-12T04:02:18.635 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.635 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.635 DEBUG:boto:Params: {} 2024-04-12T04:02:18.635 DEBUG:boto:Token: None 2024-04-12T04:02:18.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.636 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c766004ed5c07f1-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.678 DEBUG:boto:b'dgbpxg-63obj1851000falseobj1852024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.679 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.679 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.679 DEBUG:boto:path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.679 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.679 DEBUG:boto:Method: GET 2024-04-12T04:02:18.679 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj185 2024-04-12T04:02:18.679 DEBUG:boto:Data: 2024-04-12T04:02:18.679 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.680 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.680 DEBUG:boto:Params: {} 2024-04-12T04:02:18.680 DEBUG:boto:Token: None 2024-04-12T04:02:18.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.680 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207fefb25ca40415-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.685 DEBUG:boto:b'dgbpxg-63obj1851000falseobj1852024-04-12T03:58:40.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:02:18.685 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.686 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.686 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.686 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.686 DEBUG:boto:Method: GET 2024-04-12T04:02:18.686 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.686 DEBUG:boto:Data: 2024-04-12T04:02:18.686 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.686 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.686 DEBUG:boto:Params: {} 2024-04-12T04:02:18.687 DEBUG:boto:Token: None 2024-04-12T04:02:18.687 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.687 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU= 2024-04-12T04:02:18.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU=', 'Content-Length': '0'} 2024-04-12T04:02:18.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055729a37580c58c9-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.689 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.689 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.689 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.689 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.689 DEBUG:boto:Method: GET 2024-04-12T04:02:18.689 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.689 DEBUG:boto:Data: 2024-04-12T04:02:18.689 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.690 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.690 DEBUG:boto:Params: {} 2024-04-12T04:02:18.690 DEBUG:boto:Token: None 2024-04-12T04:02:18.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.690 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU= 2024-04-12T04:02:18.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU=', 'Content-Length': '0'} 2024-04-12T04:02:18.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236a6236ad02eee0-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.734 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.735 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.735 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.735 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.735 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.735 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.735 DEBUG:boto:Data: 2024-04-12T04:02:18.735 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.735 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.735 DEBUG:boto:Params: {} 2024-04-12T04:02:18.736 DEBUG:boto:Token: None 2024-04-12T04:02:18.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZuRQbfb4lY5pssdeIQfPVUZryNg= 2024-04-12T04:02:18.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZuRQbfb4lY5pssdeIQfPVUZryNg=', 'Content-Length': '0'} 2024-04-12T04:02:18.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3fd006c7e5c6aac-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.738 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.738 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.738 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.738 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.738 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.738 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.739 DEBUG:boto:Data: 2024-04-12T04:02:18.739 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.739 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.739 DEBUG:boto:Params: {} 2024-04-12T04:02:18.739 DEBUG:boto:Token: None 2024-04-12T04:02:18.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.739 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZuRQbfb4lY5pssdeIQfPVUZryNg= 2024-04-12T04:02:18.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZuRQbfb4lY5pssdeIQfPVUZryNg=', 'Content-Length': '0'} 2024-04-12T04:02:18.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1af58d8172b071-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:02:18.742 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.742 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.742 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.742 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.742 DEBUG:boto:Method: GET 2024-04-12T04:02:18.742 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.742 DEBUG:boto:Data: 2024-04-12T04:02:18.742 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.742 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.743 DEBUG:boto:Params: {} 2024-04-12T04:02:18.743 DEBUG:boto:Token: None 2024-04-12T04:02:18.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.743 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU= 2024-04-12T04:02:18.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU=', 'Content-Length': '0'} 2024-04-12T04:02:18.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f963d8f82eeddad-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.745 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.745 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.745 DEBUG:boto:path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.746 DEBUG:boto:auth_path=/dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.746 DEBUG:boto:Method: GET 2024-04-12T04:02:18.746 DEBUG:boto:Path: /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.746 DEBUG:boto:Data: 2024-04-12T04:02:18.746 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.747 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.747 DEBUG:boto:Params: {} 2024-04-12T04:02:18.747 DEBUG:boto:Token: None 2024-04-12T04:02:18.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19?versionId=null 2024-04-12T04:02:18.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU= 2024-04-12T04:02:18.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iUWp5sUQC1Xg0C15CJV6wvHdSnU=', 'Content-Length': '0'} 2024-04-12T04:02:18.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dec25793d70181c3-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.749 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.749 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.749 DEBUG:boto:path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.750 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.750 DEBUG:boto:Method: GET 2024-04-12T04:02:18.750 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.750 DEBUG:boto:Data: 2024-04-12T04:02:18.750 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.750 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.750 DEBUG:boto:Params: {} 2024-04-12T04:02:18.750 DEBUG:boto:Token: None 2024-04-12T04:02:18.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.751 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782b2ee1557ab530-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.753 DEBUG:boto:b'dgbpxg-63obj191000falseobj192024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.754 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.754 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.754 DEBUG:boto:path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.754 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.754 DEBUG:boto:Method: GET 2024-04-12T04:02:18.754 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.754 DEBUG:boto:Data: 2024-04-12T04:02:18.754 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.755 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.755 DEBUG:boto:Params: {} 2024-04-12T04:02:18.755 DEBUG:boto:Token: None 2024-04-12T04:02:18.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.755 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013ad3452d1d7eed3-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.759 DEBUG:boto:b'dgbpxg-63obj191000falseobj192024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.759 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.759 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.760 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.760 DEBUG:boto:Path: /dgbpxg-63/obj19 2024-04-12T04:02:18.760 DEBUG:boto:Data: 2024-04-12T04:02:18.760 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.760 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.760 DEBUG:boto:Params: {} 2024-04-12T04:02:18.760 DEBUG:boto:Token: None 2024-04-12T04:02:18.760 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19 2024-04-12T04:02:18.760 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tdbzn3FTvIyLLGTNs/i2hWethBo= 2024-04-12T04:02:18.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tdbzn3FTvIyLLGTNs/i2hWethBo=', 'Content-Length': '0'} 2024-04-12T04:02:18.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ba8ce8880b2173-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.762 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.763 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.763 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.763 DEBUG:boto:Path: /dgbpxg-63/obj19 2024-04-12T04:02:18.763 DEBUG:boto:Data: 2024-04-12T04:02:18.763 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.763 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.763 DEBUG:boto:Params: {} 2024-04-12T04:02:18.763 DEBUG:boto:Token: None 2024-04-12T04:02:18.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19 2024-04-12T04:02:18.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Tdbzn3FTvIyLLGTNs/i2hWethBo= 2024-04-12T04:02:18.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Tdbzn3FTvIyLLGTNs/i2hWethBo=', 'Content-Length': '0'} 2024-04-12T04:02:18.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5dbbfe3e4b99993-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:02:18.766 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.766 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.766 DEBUG:boto:Method: GET 2024-04-12T04:02:18.766 DEBUG:boto:Path: /dgbpxg-63/obj19 2024-04-12T04:02:18.766 DEBUG:boto:Data: 2024-04-12T04:02:18.766 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.767 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.767 DEBUG:boto:Params: {} 2024-04-12T04:02:18.767 DEBUG:boto:Token: None 2024-04-12T04:02:18.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19 2024-04-12T04:02:18.767 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pnEsFm9NxnSjN0z5lcp1lKd7W0c= 2024-04-12T04:02:18.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pnEsFm9NxnSjN0z5lcp1lKd7W0c=', 'Content-Length': '0'} 2024-04-12T04:02:18.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf5cbf2fc6d0ad1-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.769 DEBUG:boto:path=/dgbpxg-63/obj19 2024-04-12T04:02:18.769 DEBUG:boto:auth_path=/dgbpxg-63/obj19 2024-04-12T04:02:18.769 DEBUG:boto:Method: GET 2024-04-12T04:02:18.769 DEBUG:boto:Path: /dgbpxg-63/obj19 2024-04-12T04:02:18.769 DEBUG:boto:Data: 2024-04-12T04:02:18.769 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.770 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.770 DEBUG:boto:Params: {} 2024-04-12T04:02:18.770 DEBUG:boto:Token: None 2024-04-12T04:02:18.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj19 2024-04-12T04:02:18.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pnEsFm9NxnSjN0z5lcp1lKd7W0c= 2024-04-12T04:02:18.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pnEsFm9NxnSjN0z5lcp1lKd7W0c=', 'Content-Length': '0'} 2024-04-12T04:02:18.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff4ff07db55a266-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.772 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.772 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.772 DEBUG:boto:path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.772 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.772 DEBUG:boto:Method: GET 2024-04-12T04:02:18.772 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.773 DEBUG:boto:Data: 2024-04-12T04:02:18.773 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.773 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.773 DEBUG:boto:Params: {} 2024-04-12T04:02:18.773 DEBUG:boto:Token: None 2024-04-12T04:02:18.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.773 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b8da0637037f57-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.775 DEBUG:boto:b'dgbpxg-63obj191000falseobj192024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.776 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.776 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.776 DEBUG:boto:path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.776 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.776 DEBUG:boto:Method: GET 2024-04-12T04:02:18.776 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj19 2024-04-12T04:02:18.776 DEBUG:boto:Data: 2024-04-12T04:02:18.776 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.777 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.777 DEBUG:boto:Params: {} 2024-04-12T04:02:18.777 DEBUG:boto:Token: None 2024-04-12T04:02:18.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.777 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d8dabb909ee183-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.781 DEBUG:boto:b'dgbpxg-63obj191000falseobj192024-04-12T03:58:39.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:02:18.782 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.782 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.782 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.782 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.782 DEBUG:boto:Method: GET 2024-04-12T04:02:18.782 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.782 DEBUG:boto:Data: 2024-04-12T04:02:18.782 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.783 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.783 DEBUG:boto:Params: {} 2024-04-12T04:02:18.783 DEBUG:boto:Token: None 2024-04-12T04:02:18.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.783 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A= 2024-04-12T04:02:18.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A=', 'Content-Length': '0'} 2024-04-12T04:02:18.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013d275b26d13439-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.785 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.785 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.785 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.785 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.785 DEBUG:boto:Method: GET 2024-04-12T04:02:18.786 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.786 DEBUG:boto:Data: 2024-04-12T04:02:18.786 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.786 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.786 DEBUG:boto:Params: {} 2024-04-12T04:02:18.786 DEBUG:boto:Token: None 2024-04-12T04:02:18.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.786 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A= 2024-04-12T04:02:18.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A=', 'Content-Length': '0'} 2024-04-12T04:02:18.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9a31e3b61be0fc-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.834 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.835 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.835 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.835 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.835 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.835 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.835 DEBUG:boto:Data: 2024-04-12T04:02:18.835 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.835 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.835 DEBUG:boto:Params: {} 2024-04-12T04:02:18.836 DEBUG:boto:Token: None 2024-04-12T04:02:18.836 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/lbIO4vnuBNTIJgSP2A3Fft48i4= 2024-04-12T04:02:18.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/lbIO4vnuBNTIJgSP2A3Fft48i4=', 'Content-Length': '0'} 2024-04-12T04:02:18.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d939d7bf644569b-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.838 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.838 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.838 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.838 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.838 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.838 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.838 DEBUG:boto:Data: 2024-04-12T04:02:18.838 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.839 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.839 DEBUG:boto:Params: {} 2024-04-12T04:02:18.839 DEBUG:boto:Token: None 2024-04-12T04:02:18.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/lbIO4vnuBNTIJgSP2A3Fft48i4= 2024-04-12T04:02:18.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/lbIO4vnuBNTIJgSP2A3Fft48i4=', 'Content-Length': '0'} 2024-04-12T04:02:18.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58d603f3321ccce-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:02:18.841 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.841 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.841 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.841 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.841 DEBUG:boto:Method: GET 2024-04-12T04:02:18.841 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.842 DEBUG:boto:Data: 2024-04-12T04:02:18.842 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.842 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.842 DEBUG:boto:Params: {} 2024-04-12T04:02:18.842 DEBUG:boto:Token: None 2024-04-12T04:02:18.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A= 2024-04-12T04:02:18.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A=', 'Content-Length': '0'} 2024-04-12T04:02:18.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009056abcc12f96c90-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.845 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.845 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.845 DEBUG:boto:path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.845 DEBUG:boto:auth_path=/dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.845 DEBUG:boto:Method: GET 2024-04-12T04:02:18.845 DEBUG:boto:Path: /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.846 DEBUG:boto:Data: 2024-04-12T04:02:18.846 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.846 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.846 DEBUG:boto:Params: {} 2024-04-12T04:02:18.846 DEBUG:boto:Token: None 2024-04-12T04:02:18.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22?versionId=null 2024-04-12T04:02:18.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A= 2024-04-12T04:02:18.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zH6m6BnlNrjos+6zQ8eXahRsj/A=', 'Content-Length': '0'} 2024-04-12T04:02:18.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3387d5316a5066-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.849 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.849 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.849 DEBUG:boto:path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.849 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.849 DEBUG:boto:Method: GET 2024-04-12T04:02:18.849 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.849 DEBUG:boto:Data: 2024-04-12T04:02:18.849 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.849 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.850 DEBUG:boto:Params: {} 2024-04-12T04:02:18.850 DEBUG:boto:Token: None 2024-04-12T04:02:18.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.850 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2dc0767ab67d540-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.852 DEBUG:boto:b'dgbpxg-63obj221000falseobj222024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.853 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.853 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.853 DEBUG:boto:path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.853 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.853 DEBUG:boto:Method: GET 2024-04-12T04:02:18.853 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.853 DEBUG:boto:Data: 2024-04-12T04:02:18.853 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.854 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.854 DEBUG:boto:Params: {} 2024-04-12T04:02:18.854 DEBUG:boto:Token: None 2024-04-12T04:02:18.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.854 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b2b947047a430c-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.858 DEBUG:boto:b'dgbpxg-63obj221000falseobj222024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.858 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.858 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.859 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.859 DEBUG:boto:Path: /dgbpxg-63/obj22 2024-04-12T04:02:18.859 DEBUG:boto:Data: 2024-04-12T04:02:18.859 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.859 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.859 DEBUG:boto:Params: {} 2024-04-12T04:02:18.859 DEBUG:boto:Token: None 2024-04-12T04:02:18.859 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22 2024-04-12T04:02:18.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yCyO8iQHBPLuUB2I7RxaRdKr2xs= 2024-04-12T04:02:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yCyO8iQHBPLuUB2I7RxaRdKr2xs=', 'Content-Length': '0'} 2024-04-12T04:02:18.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a794c15390168b5-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.861 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.861 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.861 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.861 DEBUG:boto:Path: /dgbpxg-63/obj22 2024-04-12T04:02:18.861 DEBUG:boto:Data: 2024-04-12T04:02:18.862 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.862 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.862 DEBUG:boto:Params: {} 2024-04-12T04:02:18.862 DEBUG:boto:Token: None 2024-04-12T04:02:18.862 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22 2024-04-12T04:02:18.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yCyO8iQHBPLuUB2I7RxaRdKr2xs= 2024-04-12T04:02:18.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yCyO8iQHBPLuUB2I7RxaRdKr2xs=', 'Content-Length': '0'} 2024-04-12T04:02:18.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c1301c4777d938-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:02:18.864 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.864 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.864 DEBUG:boto:Method: GET 2024-04-12T04:02:18.864 DEBUG:boto:Path: /dgbpxg-63/obj22 2024-04-12T04:02:18.865 DEBUG:boto:Data: 2024-04-12T04:02:18.865 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.865 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.865 DEBUG:boto:Params: {} 2024-04-12T04:02:18.865 DEBUG:boto:Token: None 2024-04-12T04:02:18.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22 2024-04-12T04:02:18.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wql/In7z44QxGltSw9LJUsLaZZE= 2024-04-12T04:02:18.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wql/In7z44QxGltSw9LJUsLaZZE=', 'Content-Length': '0'} 2024-04-12T04:02:18.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f98f94929c97047-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.867 DEBUG:boto:path=/dgbpxg-63/obj22 2024-04-12T04:02:18.867 DEBUG:boto:auth_path=/dgbpxg-63/obj22 2024-04-12T04:02:18.867 DEBUG:boto:Method: GET 2024-04-12T04:02:18.867 DEBUG:boto:Path: /dgbpxg-63/obj22 2024-04-12T04:02:18.867 DEBUG:boto:Data: 2024-04-12T04:02:18.867 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.868 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.868 DEBUG:boto:Params: {} 2024-04-12T04:02:18.868 DEBUG:boto:Token: None 2024-04-12T04:02:18.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj22 2024-04-12T04:02:18.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Wql/In7z44QxGltSw9LJUsLaZZE= 2024-04-12T04:02:18.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Wql/In7z44QxGltSw9LJUsLaZZE=', 'Content-Length': '0'} 2024-04-12T04:02:18.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8a031a3d7eeaf1-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.870 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.870 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.870 DEBUG:boto:path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.870 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.870 DEBUG:boto:Method: GET 2024-04-12T04:02:18.870 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.870 DEBUG:boto:Data: 2024-04-12T04:02:18.871 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.871 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.871 DEBUG:boto:Params: {} 2024-04-12T04:02:18.871 DEBUG:boto:Token: None 2024-04-12T04:02:18.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35c471df114625f-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.914 DEBUG:boto:b'dgbpxg-63obj221000falseobj222024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.915 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.915 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.915 DEBUG:boto:path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.915 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.915 DEBUG:boto:Method: GET 2024-04-12T04:02:18.915 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj22 2024-04-12T04:02:18.915 DEBUG:boto:Data: 2024-04-12T04:02:18.915 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.916 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.916 DEBUG:boto:Params: {} 2024-04-12T04:02:18.916 DEBUG:boto:Token: None 2024-04-12T04:02:18.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef3fca40ccf38155-006618b24a-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.920 DEBUG:boto:b'dgbpxg-63obj221000falseobj222024-04-12T03:58:38.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:02:18.920 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.920 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.920 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.920 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.921 DEBUG:boto:Method: GET 2024-04-12T04:02:18.921 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.921 DEBUG:boto:Data: 2024-04-12T04:02:18.921 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.921 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.921 DEBUG:boto:Params: {} 2024-04-12T04:02:18.921 DEBUG:boto:Token: None 2024-04-12T04:02:18.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo= 2024-04-12T04:02:18.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo=', 'Content-Length': '0'} 2024-04-12T04:02:18.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e569df797dff275-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.923 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.923 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.923 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.923 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.924 DEBUG:boto:Method: GET 2024-04-12T04:02:18.924 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.924 DEBUG:boto:Data: 2024-04-12T04:02:18.924 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.924 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.924 DEBUG:boto:Params: {} 2024-04-12T04:02:18.924 DEBUG:boto:Token: None 2024-04-12T04:02:18.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.924 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo= 2024-04-12T04:02:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo=', 'Content-Length': '0'} 2024-04-12T04:02:18.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077249f11adfdf3a8-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.926 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.927 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.927 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.927 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.927 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.927 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.927 DEBUG:boto:Data: 2024-04-12T04:02:18.927 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.927 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.927 DEBUG:boto:Params: {} 2024-04-12T04:02:18.927 DEBUG:boto:Token: None 2024-04-12T04:02:18.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dR9WqQxRRsO70nimXADOeC//7V4= 2024-04-12T04:02:18.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dR9WqQxRRsO70nimXADOeC//7V4=', 'Content-Length': '0'} 2024-04-12T04:02:18.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb52fbb57ce88e9-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.929 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.929 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.929 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.929 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.929 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.930 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.930 DEBUG:boto:Data: 2024-04-12T04:02:18.930 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.930 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.930 DEBUG:boto:Params: {} 2024-04-12T04:02:18.930 DEBUG:boto:Token: None 2024-04-12T04:02:18.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dR9WqQxRRsO70nimXADOeC//7V4= 2024-04-12T04:02:18.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dR9WqQxRRsO70nimXADOeC//7V4=', 'Content-Length': '0'} 2024-04-12T04:02:18.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008485afa03785f7b9-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:02:18.932 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.932 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.932 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.932 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.933 DEBUG:boto:Method: GET 2024-04-12T04:02:18.933 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.933 DEBUG:boto:Data: 2024-04-12T04:02:18.933 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.933 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.933 DEBUG:boto:Params: {} 2024-04-12T04:02:18.933 DEBUG:boto:Token: None 2024-04-12T04:02:18.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo= 2024-04-12T04:02:18.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo=', 'Content-Length': '0'} 2024-04-12T04:02:18.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bff049ff7168acc-006618b24a-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.935 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.935 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.935 DEBUG:boto:path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.935 DEBUG:boto:auth_path=/dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.935 DEBUG:boto:Method: GET 2024-04-12T04:02:18.935 DEBUG:boto:Path: /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.935 DEBUG:boto:Data: 2024-04-12T04:02:18.936 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.936 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.936 DEBUG:boto:Params: {} 2024-04-12T04:02:18.936 DEBUG:boto:Token: None 2024-04-12T04:02:18.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23?versionId=null 2024-04-12T04:02:18.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo= 2024-04-12T04:02:18.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MI4Jwr0diAJ5lbZJH82R2yzZCLo=', 'Content-Length': '0'} 2024-04-12T04:02:18.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35e53109dc94009-006618b24a-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.938 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.938 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.938 DEBUG:boto:path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.938 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.939 DEBUG:boto:Method: GET 2024-04-12T04:02:18.939 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.939 DEBUG:boto:Data: 2024-04-12T04:02:18.939 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.939 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.939 DEBUG:boto:Params: {} 2024-04-12T04:02:18.939 DEBUG:boto:Token: None 2024-04-12T04:02:18.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.939 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c259473fdf7530-006618b24a-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.982 DEBUG:boto:b'dgbpxg-63obj231000falseobj232024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.983 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:18.983 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:18.983 DEBUG:boto:path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.983 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.983 DEBUG:boto:Method: GET 2024-04-12T04:02:18.983 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj23 2024-04-12T04:02:18.983 DEBUG:boto:Data: 2024-04-12T04:02:18.983 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.984 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.984 DEBUG:boto:Params: {} 2024-04-12T04:02:18.984 DEBUG:boto:Token: None 2024-04-12T04:02:18.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/ 2024-04-12T04:02:18.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg= 2024-04-12T04:02:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kYkjHy3+RBzZDeFoKYUTISzRjGg=', 'Content-Length': '0'} 2024-04-12T04:02:18.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6a22e8048b82fe-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.988 DEBUG:boto:b'dgbpxg-63obj231000falseobj232024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:18.988 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.989 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.989 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.989 DEBUG:boto:Path: /dgbpxg-63/obj23 2024-04-12T04:02:18.989 DEBUG:boto:Data: 2024-04-12T04:02:18.989 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.989 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.989 DEBUG:boto:Params: {} 2024-04-12T04:02:18.989 DEBUG:boto:Token: None 2024-04-12T04:02:18.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23 2024-04-12T04:02:18.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/OEzd3Oa6fm+S+iouLxfz2mGIik= 2024-04-12T04:02:18.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/OEzd3Oa6fm+S+iouLxfz2mGIik=', 'Content-Length': '0'} 2024-04-12T04:02:18.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef880f585d3280ea-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.992 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.992 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.992 DEBUG:boto:Method: HEAD 2024-04-12T04:02:18.992 DEBUG:boto:Path: /dgbpxg-63/obj23 2024-04-12T04:02:18.992 DEBUG:boto:Data: 2024-04-12T04:02:18.992 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.992 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.992 DEBUG:boto:Params: {} 2024-04-12T04:02:18.993 DEBUG:boto:Token: None 2024-04-12T04:02:18.993 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23 2024-04-12T04:02:18.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/OEzd3Oa6fm+S+iouLxfz2mGIik= 2024-04-12T04:02:18.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/OEzd3Oa6fm+S+iouLxfz2mGIik=', 'Content-Length': '0'} 2024-04-12T04:02:18.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18eda7f8d1dbe62-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:02:18.995 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.995 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.995 DEBUG:boto:Method: GET 2024-04-12T04:02:18.995 DEBUG:boto:Path: /dgbpxg-63/obj23 2024-04-12T04:02:18.995 DEBUG:boto:Data: 2024-04-12T04:02:18.996 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:18.996 DEBUG:boto:Port: 8001 2024-04-12T04:02:18.996 DEBUG:boto:Params: {} 2024-04-12T04:02:18.996 DEBUG:boto:Token: None 2024-04-12T04:02:18.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23 2024-04-12T04:02:18.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1n4rctrJkC3c1L9JmozGFkwSEHM= 2024-04-12T04:02:18.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1n4rctrJkC3c1L9JmozGFkwSEHM=', 'Content-Length': '0'} 2024-04-12T04:02:18.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5a9b7765bfc6cf-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:18.998 DEBUG:boto:path=/dgbpxg-63/obj23 2024-04-12T04:02:18.998 DEBUG:boto:auth_path=/dgbpxg-63/obj23 2024-04-12T04:02:18.998 DEBUG:boto:Method: GET 2024-04-12T04:02:18.998 DEBUG:boto:Path: /dgbpxg-63/obj23 2024-04-12T04:02:18.998 DEBUG:boto:Data: 2024-04-12T04:02:18.998 DEBUG:boto:Headers: {} 2024-04-12T04:02:18.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:18.999 DEBUG:boto:Port: 8000 2024-04-12T04:02:18.999 DEBUG:boto:Params: {} 2024-04-12T04:02:18.999 DEBUG:boto:Token: None 2024-04-12T04:02:18.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:18 GMT /dgbpxg-63/obj23 2024-04-12T04:02:18.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1n4rctrJkC3c1L9JmozGFkwSEHM= 2024-04-12T04:02:18.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1n4rctrJkC3c1L9JmozGFkwSEHM=', 'Content-Length': '0'} 2024-04-12T04:02:19.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002308a4390c894ab5-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.042 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.043 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.043 DEBUG:boto:path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.043 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.043 DEBUG:boto:Method: GET 2024-04-12T04:02:19.043 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.043 DEBUG:boto:Data: 2024-04-12T04:02:19.043 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.043 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.043 DEBUG:boto:Params: {} 2024-04-12T04:02:19.043 DEBUG:boto:Token: None 2024-04-12T04:02:19.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017699bb7cf3684da-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.047 DEBUG:boto:b'dgbpxg-63obj231000falseobj232024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.047 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.048 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.048 DEBUG:boto:path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.048 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.048 DEBUG:boto:Method: GET 2024-04-12T04:02:19.048 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj23 2024-04-12T04:02:19.048 DEBUG:boto:Data: 2024-04-12T04:02:19.048 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.048 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.048 DEBUG:boto:Params: {} 2024-04-12T04:02:19.049 DEBUG:boto:Token: None 2024-04-12T04:02:19.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.049 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc17fde3e604f96-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.052 DEBUG:boto:b'dgbpxg-63obj231000falseobj232024-04-12T03:58:39.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:02:19.053 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.053 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.053 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.053 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.053 DEBUG:boto:Method: GET 2024-04-12T04:02:19.053 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.054 DEBUG:boto:Data: 2024-04-12T04:02:19.054 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.054 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.054 DEBUG:boto:Params: {} 2024-04-12T04:02:19.054 DEBUG:boto:Token: None 2024-04-12T04:02:19.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.054 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U= 2024-04-12T04:02:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U=', 'Content-Length': '0'} 2024-04-12T04:02:19.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001efdfa9eecaa3066-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.056 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.056 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.056 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.057 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.057 DEBUG:boto:Method: GET 2024-04-12T04:02:19.057 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.057 DEBUG:boto:Data: 2024-04-12T04:02:19.057 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.057 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.057 DEBUG:boto:Params: {} 2024-04-12T04:02:19.057 DEBUG:boto:Token: None 2024-04-12T04:02:19.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.057 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U= 2024-04-12T04:02:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U=', 'Content-Length': '0'} 2024-04-12T04:02:19.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5ad27595f4aee0-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.059 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.060 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.060 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.060 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.060 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.060 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.060 DEBUG:boto:Data: 2024-04-12T04:02:19.060 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.060 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.060 DEBUG:boto:Params: {} 2024-04-12T04:02:19.060 DEBUG:boto:Token: None 2024-04-12T04:02:19.061 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.061 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MyataISrpmkdFlGFCWPt5BkfMd0= 2024-04-12T04:02:19.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MyataISrpmkdFlGFCWPt5BkfMd0=', 'Content-Length': '0'} 2024-04-12T04:02:19.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ba13933fcfe423-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.062 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.062 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.063 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.063 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.063 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.063 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.063 DEBUG:boto:Data: 2024-04-12T04:02:19.063 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.063 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.063 DEBUG:boto:Params: {} 2024-04-12T04:02:19.063 DEBUG:boto:Token: None 2024-04-12T04:02:19.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MyataISrpmkdFlGFCWPt5BkfMd0= 2024-04-12T04:02:19.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MyataISrpmkdFlGFCWPt5BkfMd0=', 'Content-Length': '0'} 2024-04-12T04:02:19.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3cc0f0a04b2538-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:02:19.065 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.066 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.066 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.066 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.066 DEBUG:boto:Method: GET 2024-04-12T04:02:19.066 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.066 DEBUG:boto:Data: 2024-04-12T04:02:19.066 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.066 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.067 DEBUG:boto:Params: {} 2024-04-12T04:02:19.067 DEBUG:boto:Token: None 2024-04-12T04:02:19.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.067 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U= 2024-04-12T04:02:19.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U=', 'Content-Length': '0'} 2024-04-12T04:02:19.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b4f5a1de00db3ec-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.069 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.069 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.069 DEBUG:boto:path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.069 DEBUG:boto:auth_path=/dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.069 DEBUG:boto:Method: GET 2024-04-12T04:02:19.070 DEBUG:boto:Path: /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.070 DEBUG:boto:Data: 2024-04-12T04:02:19.070 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.070 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.070 DEBUG:boto:Params: {} 2024-04-12T04:02:19.070 DEBUG:boto:Token: None 2024-04-12T04:02:19.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29?versionId=null 2024-04-12T04:02:19.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U= 2024-04-12T04:02:19.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNgsLB4zJkX7yshlQrXKw9Lnv7U=', 'Content-Length': '0'} 2024-04-12T04:02:19.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2cbf6854e54110-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.118 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.119 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.119 DEBUG:boto:path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.119 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.119 DEBUG:boto:Method: GET 2024-04-12T04:02:19.119 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.119 DEBUG:boto:Data: 2024-04-12T04:02:19.119 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.119 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.119 DEBUG:boto:Params: {} 2024-04-12T04:02:19.120 DEBUG:boto:Token: None 2024-04-12T04:02:19.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec4f0f14c39ed94-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.122 DEBUG:boto:b'dgbpxg-63obj291000falseobj292024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.123 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.123 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.123 DEBUG:boto:path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.123 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.123 DEBUG:boto:Method: GET 2024-04-12T04:02:19.123 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.123 DEBUG:boto:Data: 2024-04-12T04:02:19.124 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.124 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.124 DEBUG:boto:Params: {} 2024-04-12T04:02:19.124 DEBUG:boto:Token: None 2024-04-12T04:02:19.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.124 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000625e2e53a0efcbc7-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.128 DEBUG:boto:b'dgbpxg-63obj291000falseobj292024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.128 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.128 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.128 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.129 DEBUG:boto:Path: /dgbpxg-63/obj29 2024-04-12T04:02:19.129 DEBUG:boto:Data: 2024-04-12T04:02:19.129 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.129 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.129 DEBUG:boto:Params: {} 2024-04-12T04:02:19.129 DEBUG:boto:Token: None 2024-04-12T04:02:19.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29 2024-04-12T04:02:19.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPvxhSVh4eF+yhNOj7u6VmKQDUs= 2024-04-12T04:02:19.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPvxhSVh4eF+yhNOj7u6VmKQDUs=', 'Content-Length': '0'} 2024-04-12T04:02:19.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ce3053d85604ef-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.131 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.131 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.131 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.131 DEBUG:boto:Path: /dgbpxg-63/obj29 2024-04-12T04:02:19.131 DEBUG:boto:Data: 2024-04-12T04:02:19.132 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.132 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.132 DEBUG:boto:Params: {} 2024-04-12T04:02:19.132 DEBUG:boto:Token: None 2024-04-12T04:02:19.132 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29 2024-04-12T04:02:19.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPvxhSVh4eF+yhNOj7u6VmKQDUs= 2024-04-12T04:02:19.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPvxhSVh4eF+yhNOj7u6VmKQDUs=', 'Content-Length': '0'} 2024-04-12T04:02:19.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090151bf3736a0e38-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:02:19.134 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.134 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.135 DEBUG:boto:Method: GET 2024-04-12T04:02:19.135 DEBUG:boto:Path: /dgbpxg-63/obj29 2024-04-12T04:02:19.135 DEBUG:boto:Data: 2024-04-12T04:02:19.135 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.135 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.135 DEBUG:boto:Params: {} 2024-04-12T04:02:19.135 DEBUG:boto:Token: None 2024-04-12T04:02:19.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29 2024-04-12T04:02:19.135 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZcwRe1ZotMzM+sDLplyB/n6Q5dg= 2024-04-12T04:02:19.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZcwRe1ZotMzM+sDLplyB/n6Q5dg=', 'Content-Length': '0'} 2024-04-12T04:02:19.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93a138202c898e2-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.182 DEBUG:boto:path=/dgbpxg-63/obj29 2024-04-12T04:02:19.183 DEBUG:boto:auth_path=/dgbpxg-63/obj29 2024-04-12T04:02:19.183 DEBUG:boto:Method: GET 2024-04-12T04:02:19.183 DEBUG:boto:Path: /dgbpxg-63/obj29 2024-04-12T04:02:19.183 DEBUG:boto:Data: 2024-04-12T04:02:19.183 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.183 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.183 DEBUG:boto:Params: {} 2024-04-12T04:02:19.183 DEBUG:boto:Token: None 2024-04-12T04:02:19.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj29 2024-04-12T04:02:19.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZcwRe1ZotMzM+sDLplyB/n6Q5dg= 2024-04-12T04:02:19.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZcwRe1ZotMzM+sDLplyB/n6Q5dg=', 'Content-Length': '0'} 2024-04-12T04:02:19.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2ca9283578fcbb-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.186 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.186 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.186 DEBUG:boto:path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.186 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.187 DEBUG:boto:Method: GET 2024-04-12T04:02:19.187 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.187 DEBUG:boto:Data: 2024-04-12T04:02:19.187 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.187 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.187 DEBUG:boto:Params: {} 2024-04-12T04:02:19.187 DEBUG:boto:Token: None 2024-04-12T04:02:19.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c08d55cffbd9878-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.190 DEBUG:boto:b'dgbpxg-63obj291000falseobj292024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.190 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.190 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.190 DEBUG:boto:path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.190 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.190 DEBUG:boto:Method: GET 2024-04-12T04:02:19.191 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj29 2024-04-12T04:02:19.191 DEBUG:boto:Data: 2024-04-12T04:02:19.191 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.191 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.191 DEBUG:boto:Params: {} 2024-04-12T04:02:19.191 DEBUG:boto:Token: None 2024-04-12T04:02:19.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.191 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10018681687fefd-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.195 DEBUG:boto:b'dgbpxg-63obj291000falseobj292024-04-12T03:58:40.105Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:02:19.195 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.195 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.196 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.196 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.196 DEBUG:boto:Method: GET 2024-04-12T04:02:19.196 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.196 DEBUG:boto:Data: 2024-04-12T04:02:19.196 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.196 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.196 DEBUG:boto:Params: {} 2024-04-12T04:02:19.196 DEBUG:boto:Token: None 2024-04-12T04:02:19.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo= 2024-04-12T04:02:19.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo=', 'Content-Length': '0'} 2024-04-12T04:02:19.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b319097a1518f029-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.198 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.198 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.199 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.199 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.199 DEBUG:boto:Method: GET 2024-04-12T04:02:19.199 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.199 DEBUG:boto:Data: 2024-04-12T04:02:19.199 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.199 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.199 DEBUG:boto:Params: {} 2024-04-12T04:02:19.199 DEBUG:boto:Token: None 2024-04-12T04:02:19.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo= 2024-04-12T04:02:19.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo=', 'Content-Length': '0'} 2024-04-12T04:02:19.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5cde0ace77dbd6-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.242 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.243 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.243 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.243 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.243 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.243 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.243 DEBUG:boto:Data: 2024-04-12T04:02:19.243 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.243 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.244 DEBUG:boto:Params: {} 2024-04-12T04:02:19.244 DEBUG:boto:Token: None 2024-04-12T04:02:19.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zVVBkD0oKyGBT95MwkfcBUG1rh8= 2024-04-12T04:02:19.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zVVBkD0oKyGBT95MwkfcBUG1rh8=', 'Content-Length': '0'} 2024-04-12T04:02:19.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085cd0f691c2b8423-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.246 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.246 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.246 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.246 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.246 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.246 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.246 DEBUG:boto:Data: 2024-04-12T04:02:19.247 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.247 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.247 DEBUG:boto:Params: {} 2024-04-12T04:02:19.247 DEBUG:boto:Token: None 2024-04-12T04:02:19.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zVVBkD0oKyGBT95MwkfcBUG1rh8= 2024-04-12T04:02:19.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zVVBkD0oKyGBT95MwkfcBUG1rh8=', 'Content-Length': '0'} 2024-04-12T04:02:19.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c8b879d06e83ca-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:02:19.249 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.249 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.249 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.250 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.250 DEBUG:boto:Method: GET 2024-04-12T04:02:19.250 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.250 DEBUG:boto:Data: 2024-04-12T04:02:19.250 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.250 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.250 DEBUG:boto:Params: {} 2024-04-12T04:02:19.250 DEBUG:boto:Token: None 2024-04-12T04:02:19.250 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo= 2024-04-12T04:02:19.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo=', 'Content-Length': '0'} 2024-04-12T04:02:19.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b56aadc007cb7c1-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.252 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.253 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.253 DEBUG:boto:path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.253 DEBUG:boto:auth_path=/dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.253 DEBUG:boto:Method: GET 2024-04-12T04:02:19.253 DEBUG:boto:Path: /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.253 DEBUG:boto:Data: 2024-04-12T04:02:19.253 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.253 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.253 DEBUG:boto:Params: {} 2024-04-12T04:02:19.253 DEBUG:boto:Token: None 2024-04-12T04:02:19.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31?versionId=null 2024-04-12T04:02:19.254 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo= 2024-04-12T04:02:19.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4tB86MMPpBHzQOoVzq/i2RZTcgo=', 'Content-Length': '0'} 2024-04-12T04:02:19.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ea2ebbdf9d2787-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.255 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.255 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.256 DEBUG:boto:path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.256 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.256 DEBUG:boto:Method: GET 2024-04-12T04:02:19.256 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.256 DEBUG:boto:Data: 2024-04-12T04:02:19.256 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.256 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.256 DEBUG:boto:Params: {} 2024-04-12T04:02:19.256 DEBUG:boto:Token: None 2024-04-12T04:02:19.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.257 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e611c8539ab525f-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.259 DEBUG:boto:b'dgbpxg-63obj311000falseobj312024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.259 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.260 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.260 DEBUG:boto:path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.260 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.260 DEBUG:boto:Method: GET 2024-04-12T04:02:19.260 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.260 DEBUG:boto:Data: 2024-04-12T04:02:19.260 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.260 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.260 DEBUG:boto:Params: {} 2024-04-12T04:02:19.261 DEBUG:boto:Token: None 2024-04-12T04:02:19.261 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.261 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f607c5211bd94b-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.264 DEBUG:boto:b'dgbpxg-63obj311000falseobj312024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.265 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.265 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.265 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.265 DEBUG:boto:Path: /dgbpxg-63/obj31 2024-04-12T04:02:19.265 DEBUG:boto:Data: 2024-04-12T04:02:19.265 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.265 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.265 DEBUG:boto:Params: {} 2024-04-12T04:02:19.266 DEBUG:boto:Token: None 2024-04-12T04:02:19.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31 2024-04-12T04:02:19.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WaApaR4sLvMVoQONJFfdNR5anZI= 2024-04-12T04:02:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WaApaR4sLvMVoQONJFfdNR5anZI=', 'Content-Length': '0'} 2024-04-12T04:02:19.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec7aec4742a76e4-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.268 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.268 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.268 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.268 DEBUG:boto:Path: /dgbpxg-63/obj31 2024-04-12T04:02:19.268 DEBUG:boto:Data: 2024-04-12T04:02:19.268 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.268 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.268 DEBUG:boto:Params: {} 2024-04-12T04:02:19.268 DEBUG:boto:Token: None 2024-04-12T04:02:19.269 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31 2024-04-12T04:02:19.269 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WaApaR4sLvMVoQONJFfdNR5anZI= 2024-04-12T04:02:19.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WaApaR4sLvMVoQONJFfdNR5anZI=', 'Content-Length': '0'} 2024-04-12T04:02:19.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2091c218009f994-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:02:19.271 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.271 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.271 DEBUG:boto:Method: GET 2024-04-12T04:02:19.271 DEBUG:boto:Path: /dgbpxg-63/obj31 2024-04-12T04:02:19.271 DEBUG:boto:Data: 2024-04-12T04:02:19.271 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.271 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.271 DEBUG:boto:Params: {} 2024-04-12T04:02:19.272 DEBUG:boto:Token: None 2024-04-12T04:02:19.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31 2024-04-12T04:02:19.272 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LNJKI/N3YCSCw9efBljWmctKoLo= 2024-04-12T04:02:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LNJKI/N3YCSCw9efBljWmctKoLo=', 'Content-Length': '0'} 2024-04-12T04:02:19.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a3a11d32c10f09-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.273 DEBUG:boto:path=/dgbpxg-63/obj31 2024-04-12T04:02:19.274 DEBUG:boto:auth_path=/dgbpxg-63/obj31 2024-04-12T04:02:19.274 DEBUG:boto:Method: GET 2024-04-12T04:02:19.274 DEBUG:boto:Path: /dgbpxg-63/obj31 2024-04-12T04:02:19.274 DEBUG:boto:Data: 2024-04-12T04:02:19.274 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.274 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.274 DEBUG:boto:Params: {} 2024-04-12T04:02:19.274 DEBUG:boto:Token: None 2024-04-12T04:02:19.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj31 2024-04-12T04:02:19.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LNJKI/N3YCSCw9efBljWmctKoLo= 2024-04-12T04:02:19.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LNJKI/N3YCSCw9efBljWmctKoLo=', 'Content-Length': '0'} 2024-04-12T04:02:19.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008530f7ec8f080406-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.322 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.323 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.323 DEBUG:boto:path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.323 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.323 DEBUG:boto:Method: GET 2024-04-12T04:02:19.323 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.323 DEBUG:boto:Data: 2024-04-12T04:02:19.323 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.324 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.324 DEBUG:boto:Params: {} 2024-04-12T04:02:19.324 DEBUG:boto:Token: None 2024-04-12T04:02:19.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1a94fc37edbf0c-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.327 DEBUG:boto:b'dgbpxg-63obj311000falseobj312024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.327 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.328 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.328 DEBUG:boto:path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.328 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.328 DEBUG:boto:Method: GET 2024-04-12T04:02:19.328 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj31 2024-04-12T04:02:19.328 DEBUG:boto:Data: 2024-04-12T04:02:19.328 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.328 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.328 DEBUG:boto:Params: {} 2024-04-12T04:02:19.329 DEBUG:boto:Token: None 2024-04-12T04:02:19.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.329 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac8c7fadc62a00c-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.333 DEBUG:boto:b'dgbpxg-63obj311000falseobj312024-04-12T03:58:40.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:02:19.333 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.333 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.334 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.334 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.334 DEBUG:boto:Method: GET 2024-04-12T04:02:19.334 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.334 DEBUG:boto:Data: 2024-04-12T04:02:19.334 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.334 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.334 DEBUG:boto:Params: {} 2024-04-12T04:02:19.334 DEBUG:boto:Token: None 2024-04-12T04:02:19.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms= 2024-04-12T04:02:19.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms=', 'Content-Length': '0'} 2024-04-12T04:02:19.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba74dd8e332ac323-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.378 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.379 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.379 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.379 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.379 DEBUG:boto:Method: GET 2024-04-12T04:02:19.379 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.379 DEBUG:boto:Data: 2024-04-12T04:02:19.380 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.380 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.380 DEBUG:boto:Params: {} 2024-04-12T04:02:19.380 DEBUG:boto:Token: None 2024-04-12T04:02:19.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.380 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms= 2024-04-12T04:02:19.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms=', 'Content-Length': '0'} 2024-04-12T04:02:19.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f65b5fedf88dd09-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.383 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.383 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.383 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.383 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.383 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.383 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.383 DEBUG:boto:Data: 2024-04-12T04:02:19.383 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.383 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.383 DEBUG:boto:Params: {} 2024-04-12T04:02:19.384 DEBUG:boto:Token: None 2024-04-12T04:02:19.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.384 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/IFka/OMzsHslb9jAP0UZdT0BBY= 2024-04-12T04:02:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/IFka/OMzsHslb9jAP0UZdT0BBY=', 'Content-Length': '0'} 2024-04-12T04:02:19.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdf3d3494db362e-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.386 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.386 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.386 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.386 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.386 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.386 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.386 DEBUG:boto:Data: 2024-04-12T04:02:19.387 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.387 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.387 DEBUG:boto:Params: {} 2024-04-12T04:02:19.387 DEBUG:boto:Token: None 2024-04-12T04:02:19.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.387 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/IFka/OMzsHslb9jAP0UZdT0BBY= 2024-04-12T04:02:19.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/IFka/OMzsHslb9jAP0UZdT0BBY=', 'Content-Length': '0'} 2024-04-12T04:02:19.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004adcbe86194ef81e-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:02:19.390 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.390 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.390 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.390 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.390 DEBUG:boto:Method: GET 2024-04-12T04:02:19.390 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.390 DEBUG:boto:Data: 2024-04-12T04:02:19.390 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.390 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.391 DEBUG:boto:Params: {} 2024-04-12T04:02:19.391 DEBUG:boto:Token: None 2024-04-12T04:02:19.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.391 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms= 2024-04-12T04:02:19.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms=', 'Content-Length': '0'} 2024-04-12T04:02:19.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9fcf6277520c11-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.393 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.393 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.393 DEBUG:boto:path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.393 DEBUG:boto:auth_path=/dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.393 DEBUG:boto:Method: GET 2024-04-12T04:02:19.393 DEBUG:boto:Path: /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.394 DEBUG:boto:Data: 2024-04-12T04:02:19.394 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.394 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.394 DEBUG:boto:Params: {} 2024-04-12T04:02:19.394 DEBUG:boto:Token: None 2024-04-12T04:02:19.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33?versionId=null 2024-04-12T04:02:19.394 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms= 2024-04-12T04:02:19.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qgQsqXic5kJGZOHUAS8XxiH9Wms=', 'Content-Length': '0'} 2024-04-12T04:02:19.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dceaae2af2f9cc85-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.396 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.397 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.397 DEBUG:boto:path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.397 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.397 DEBUG:boto:Method: GET 2024-04-12T04:02:19.397 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.397 DEBUG:boto:Data: 2024-04-12T04:02:19.397 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.397 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.398 DEBUG:boto:Params: {} 2024-04-12T04:02:19.398 DEBUG:boto:Token: None 2024-04-12T04:02:19.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725e38e3465a99d1-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.400 DEBUG:boto:b'dgbpxg-63obj331000falseobj332024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.401 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.401 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.401 DEBUG:boto:path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.401 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.401 DEBUG:boto:Method: GET 2024-04-12T04:02:19.401 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.401 DEBUG:boto:Data: 2024-04-12T04:02:19.401 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.402 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.402 DEBUG:boto:Params: {} 2024-04-12T04:02:19.402 DEBUG:boto:Token: None 2024-04-12T04:02:19.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.402 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b2053b1822cb557-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.406 DEBUG:boto:b'dgbpxg-63obj331000falseobj332024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.406 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.406 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.407 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.407 DEBUG:boto:Path: /dgbpxg-63/obj33 2024-04-12T04:02:19.407 DEBUG:boto:Data: 2024-04-12T04:02:19.407 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.407 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.407 DEBUG:boto:Params: {} 2024-04-12T04:02:19.407 DEBUG:boto:Token: None 2024-04-12T04:02:19.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33 2024-04-12T04:02:19.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dN+76/o08o8CVcalNWLCrNUDLpQ= 2024-04-12T04:02:19.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dN+76/o08o8CVcalNWLCrNUDLpQ=', 'Content-Length': '0'} 2024-04-12T04:02:19.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f36d79f16672e7-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.409 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.409 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.409 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.410 DEBUG:boto:Path: /dgbpxg-63/obj33 2024-04-12T04:02:19.410 DEBUG:boto:Data: 2024-04-12T04:02:19.410 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.410 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.410 DEBUG:boto:Params: {} 2024-04-12T04:02:19.410 DEBUG:boto:Token: None 2024-04-12T04:02:19.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33 2024-04-12T04:02:19.411 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dN+76/o08o8CVcalNWLCrNUDLpQ= 2024-04-12T04:02:19.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dN+76/o08o8CVcalNWLCrNUDLpQ=', 'Content-Length': '0'} 2024-04-12T04:02:19.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d031fec543d61ba2-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:02:19.412 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.412 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.413 DEBUG:boto:Method: GET 2024-04-12T04:02:19.413 DEBUG:boto:Path: /dgbpxg-63/obj33 2024-04-12T04:02:19.413 DEBUG:boto:Data: 2024-04-12T04:02:19.413 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.413 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.413 DEBUG:boto:Params: {} 2024-04-12T04:02:19.413 DEBUG:boto:Token: None 2024-04-12T04:02:19.413 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33 2024-04-12T04:02:19.414 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DxbNdUsfUHMaC1w8xRtX6n/4nw8= 2024-04-12T04:02:19.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DxbNdUsfUHMaC1w8xRtX6n/4nw8=', 'Content-Length': '0'} 2024-04-12T04:02:19.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc4c59917feb6f2-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.415 DEBUG:boto:path=/dgbpxg-63/obj33 2024-04-12T04:02:19.415 DEBUG:boto:auth_path=/dgbpxg-63/obj33 2024-04-12T04:02:19.415 DEBUG:boto:Method: GET 2024-04-12T04:02:19.415 DEBUG:boto:Path: /dgbpxg-63/obj33 2024-04-12T04:02:19.416 DEBUG:boto:Data: 2024-04-12T04:02:19.416 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.416 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.416 DEBUG:boto:Params: {} 2024-04-12T04:02:19.416 DEBUG:boto:Token: None 2024-04-12T04:02:19.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj33 2024-04-12T04:02:19.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DxbNdUsfUHMaC1w8xRtX6n/4nw8= 2024-04-12T04:02:19.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DxbNdUsfUHMaC1w8xRtX6n/4nw8=', 'Content-Length': '0'} 2024-04-12T04:02:19.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ae1e7a1bf09b99-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.418 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.418 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.418 DEBUG:boto:path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.418 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.419 DEBUG:boto:Method: GET 2024-04-12T04:02:19.419 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.419 DEBUG:boto:Data: 2024-04-12T04:02:19.419 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.419 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.419 DEBUG:boto:Params: {} 2024-04-12T04:02:19.419 DEBUG:boto:Token: None 2024-04-12T04:02:19.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.419 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afd88b8e2404c29b-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.422 DEBUG:boto:b'dgbpxg-63obj331000falseobj332024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.422 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.422 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.422 DEBUG:boto:path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.422 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.422 DEBUG:boto:Method: GET 2024-04-12T04:02:19.423 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj33 2024-04-12T04:02:19.423 DEBUG:boto:Data: 2024-04-12T04:02:19.423 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.423 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.423 DEBUG:boto:Params: {} 2024-04-12T04:02:19.423 DEBUG:boto:Token: None 2024-04-12T04:02:19.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.423 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010e77e7558a10174-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.427 DEBUG:boto:b'dgbpxg-63obj331000falseobj332024-04-12T03:58:38.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:02:19.427 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.428 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.428 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.428 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.428 DEBUG:boto:Method: GET 2024-04-12T04:02:19.428 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.428 DEBUG:boto:Data: 2024-04-12T04:02:19.428 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.428 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.428 DEBUG:boto:Params: {} 2024-04-12T04:02:19.429 DEBUG:boto:Token: None 2024-04-12T04:02:19.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.429 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew= 2024-04-12T04:02:19.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew=', 'Content-Length': '0'} 2024-04-12T04:02:19.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea13c7074893df42-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.431 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.432 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.432 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.432 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.432 DEBUG:boto:Method: GET 2024-04-12T04:02:19.432 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.432 DEBUG:boto:Data: 2024-04-12T04:02:19.432 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.432 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.433 DEBUG:boto:Params: {} 2024-04-12T04:02:19.433 DEBUG:boto:Token: None 2024-04-12T04:02:19.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.433 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew= 2024-04-12T04:02:19.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew=', 'Content-Length': '0'} 2024-04-12T04:02:19.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae83502523fad9e3-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.478 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.479 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.479 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.479 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.479 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.479 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.479 DEBUG:boto:Data: 2024-04-12T04:02:19.479 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.479 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.479 DEBUG:boto:Params: {} 2024-04-12T04:02:19.480 DEBUG:boto:Token: None 2024-04-12T04:02:19.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6nxCVtAhsCkw3kLpnXckEC2kTpo= 2024-04-12T04:02:19.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6nxCVtAhsCkw3kLpnXckEC2kTpo=', 'Content-Length': '0'} 2024-04-12T04:02:19.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10534443543a09a-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.482 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.482 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.482 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.482 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.482 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.482 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.483 DEBUG:boto:Data: 2024-04-12T04:02:19.483 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.483 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.483 DEBUG:boto:Params: {} 2024-04-12T04:02:19.483 DEBUG:boto:Token: None 2024-04-12T04:02:19.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.483 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6nxCVtAhsCkw3kLpnXckEC2kTpo= 2024-04-12T04:02:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6nxCVtAhsCkw3kLpnXckEC2kTpo=', 'Content-Length': '0'} 2024-04-12T04:02:19.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ccaad3b093ba33c-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:02:19.485 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.486 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.486 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.486 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.486 DEBUG:boto:Method: GET 2024-04-12T04:02:19.486 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.486 DEBUG:boto:Data: 2024-04-12T04:02:19.486 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.486 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.487 DEBUG:boto:Params: {} 2024-04-12T04:02:19.487 DEBUG:boto:Token: None 2024-04-12T04:02:19.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew= 2024-04-12T04:02:19.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew=', 'Content-Length': '0'} 2024-04-12T04:02:19.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003153cf79a565db83-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.489 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.489 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.489 DEBUG:boto:path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.489 DEBUG:boto:auth_path=/dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.489 DEBUG:boto:Method: GET 2024-04-12T04:02:19.490 DEBUG:boto:Path: /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.490 DEBUG:boto:Data: 2024-04-12T04:02:19.490 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.490 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.490 DEBUG:boto:Params: {} 2024-04-12T04:02:19.490 DEBUG:boto:Token: None 2024-04-12T04:02:19.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34?versionId=null 2024-04-12T04:02:19.491 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew= 2024-04-12T04:02:19.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F3c9wT5+31GSvPJPf4CLiSvKKew=', 'Content-Length': '0'} 2024-04-12T04:02:19.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000583343aeba757ea7-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.493 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.493 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.493 DEBUG:boto:path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.493 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.493 DEBUG:boto:Method: GET 2024-04-12T04:02:19.493 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.493 DEBUG:boto:Data: 2024-04-12T04:02:19.493 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.494 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.494 DEBUG:boto:Params: {} 2024-04-12T04:02:19.494 DEBUG:boto:Token: None 2024-04-12T04:02:19.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.494 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef64a7d034fbb9d0-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.496 DEBUG:boto:b'dgbpxg-63obj341000falseobj342024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.497 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.497 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.497 DEBUG:boto:path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.497 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.497 DEBUG:boto:Method: GET 2024-04-12T04:02:19.497 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.498 DEBUG:boto:Data: 2024-04-12T04:02:19.498 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.498 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.498 DEBUG:boto:Params: {} 2024-04-12T04:02:19.498 DEBUG:boto:Token: None 2024-04-12T04:02:19.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.498 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e53b6810297001e4-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.502 DEBUG:boto:b'dgbpxg-63obj341000falseobj342024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.503 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.503 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.503 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.503 DEBUG:boto:Path: /dgbpxg-63/obj34 2024-04-12T04:02:19.503 DEBUG:boto:Data: 2024-04-12T04:02:19.503 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.504 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.504 DEBUG:boto:Params: {} 2024-04-12T04:02:19.504 DEBUG:boto:Token: None 2024-04-12T04:02:19.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34 2024-04-12T04:02:19.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TOHKOwv6D47t8n05VrbbjuCDJF0= 2024-04-12T04:02:19.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TOHKOwv6D47t8n05VrbbjuCDJF0=', 'Content-Length': '0'} 2024-04-12T04:02:19.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327a8ebfce7a3a93-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.506 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.506 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.506 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.506 DEBUG:boto:Path: /dgbpxg-63/obj34 2024-04-12T04:02:19.506 DEBUG:boto:Data: 2024-04-12T04:02:19.507 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.507 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.507 DEBUG:boto:Params: {} 2024-04-12T04:02:19.507 DEBUG:boto:Token: None 2024-04-12T04:02:19.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34 2024-04-12T04:02:19.507 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TOHKOwv6D47t8n05VrbbjuCDJF0= 2024-04-12T04:02:19.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TOHKOwv6D47t8n05VrbbjuCDJF0=', 'Content-Length': '0'} 2024-04-12T04:02:19.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162fa849cce4dbcc-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:02:19.509 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.509 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.510 DEBUG:boto:Method: GET 2024-04-12T04:02:19.510 DEBUG:boto:Path: /dgbpxg-63/obj34 2024-04-12T04:02:19.510 DEBUG:boto:Data: 2024-04-12T04:02:19.510 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.510 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.510 DEBUG:boto:Params: {} 2024-04-12T04:02:19.510 DEBUG:boto:Token: None 2024-04-12T04:02:19.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34 2024-04-12T04:02:19.511 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+Ruc00ScT020afC0MtGkk2oaFM= 2024-04-12T04:02:19.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+Ruc00ScT020afC0MtGkk2oaFM=', 'Content-Length': '0'} 2024-04-12T04:02:19.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ddb50dfdbc9679e-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.514 DEBUG:boto:path=/dgbpxg-63/obj34 2024-04-12T04:02:19.514 DEBUG:boto:auth_path=/dgbpxg-63/obj34 2024-04-12T04:02:19.514 DEBUG:boto:Method: GET 2024-04-12T04:02:19.514 DEBUG:boto:Path: /dgbpxg-63/obj34 2024-04-12T04:02:19.514 DEBUG:boto:Data: 2024-04-12T04:02:19.514 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.514 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.515 DEBUG:boto:Params: {} 2024-04-12T04:02:19.515 DEBUG:boto:Token: None 2024-04-12T04:02:19.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj34 2024-04-12T04:02:19.515 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z+Ruc00ScT020afC0MtGkk2oaFM= 2024-04-12T04:02:19.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z+Ruc00ScT020afC0MtGkk2oaFM=', 'Content-Length': '0'} 2024-04-12T04:02:19.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012da90c6cb440383-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.517 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.517 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.517 DEBUG:boto:path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.518 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.518 DEBUG:boto:Method: GET 2024-04-12T04:02:19.518 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.518 DEBUG:boto:Data: 2024-04-12T04:02:19.518 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.518 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.518 DEBUG:boto:Params: {} 2024-04-12T04:02:19.519 DEBUG:boto:Token: None 2024-04-12T04:02:19.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe4c26a87a0cc8c-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.521 DEBUG:boto:b'dgbpxg-63obj341000falseobj342024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.522 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.522 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.522 DEBUG:boto:path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.522 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.522 DEBUG:boto:Method: GET 2024-04-12T04:02:19.523 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj34 2024-04-12T04:02:19.523 DEBUG:boto:Data: 2024-04-12T04:02:19.523 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.523 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.523 DEBUG:boto:Params: {} 2024-04-12T04:02:19.523 DEBUG:boto:Token: None 2024-04-12T04:02:19.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340746f6436b5f6b-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.527 DEBUG:boto:b'dgbpxg-63obj341000falseobj342024-04-12T03:58:39.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:02:19.528 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.528 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.528 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.528 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.528 DEBUG:boto:Method: GET 2024-04-12T04:02:19.528 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.529 DEBUG:boto:Data: 2024-04-12T04:02:19.529 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.529 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.529 DEBUG:boto:Params: {} 2024-04-12T04:02:19.529 DEBUG:boto:Token: None 2024-04-12T04:02:19.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE= 2024-04-12T04:02:19.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE=', 'Content-Length': '0'} 2024-04-12T04:02:19.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054a223776f0919e-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.531 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.532 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.532 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.532 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.532 DEBUG:boto:Method: GET 2024-04-12T04:02:19.532 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.532 DEBUG:boto:Data: 2024-04-12T04:02:19.532 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.533 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.533 DEBUG:boto:Params: {} 2024-04-12T04:02:19.533 DEBUG:boto:Token: None 2024-04-12T04:02:19.533 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.533 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE= 2024-04-12T04:02:19.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE=', 'Content-Length': '0'} 2024-04-12T04:02:19.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002adbe9436befa745-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.578 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.579 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.579 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.579 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.579 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.579 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.579 DEBUG:boto:Data: 2024-04-12T04:02:19.579 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.580 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.580 DEBUG:boto:Params: {} 2024-04-12T04:02:19.580 DEBUG:boto:Token: None 2024-04-12T04:02:19.580 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OrlU7Ek45QkwH9y2SpzyyGhitCY= 2024-04-12T04:02:19.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OrlU7Ek45QkwH9y2SpzyyGhitCY=', 'Content-Length': '0'} 2024-04-12T04:02:19.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f07a0af1f99d87-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.582 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.582 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.582 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.582 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.582 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.583 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.583 DEBUG:boto:Data: 2024-04-12T04:02:19.583 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.583 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.583 DEBUG:boto:Params: {} 2024-04-12T04:02:19.583 DEBUG:boto:Token: None 2024-04-12T04:02:19.583 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OrlU7Ek45QkwH9y2SpzyyGhitCY= 2024-04-12T04:02:19.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OrlU7Ek45QkwH9y2SpzyyGhitCY=', 'Content-Length': '0'} 2024-04-12T04:02:19.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034fd5d11f7ddd6da-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:02:19.585 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.586 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.586 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.586 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.586 DEBUG:boto:Method: GET 2024-04-12T04:02:19.586 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.586 DEBUG:boto:Data: 2024-04-12T04:02:19.586 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.586 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.587 DEBUG:boto:Params: {} 2024-04-12T04:02:19.587 DEBUG:boto:Token: None 2024-04-12T04:02:19.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE= 2024-04-12T04:02:19.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE=', 'Content-Length': '0'} 2024-04-12T04:02:19.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969ffb29e7c77e2f-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.589 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.589 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.589 DEBUG:boto:path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.589 DEBUG:boto:auth_path=/dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.589 DEBUG:boto:Method: GET 2024-04-12T04:02:19.590 DEBUG:boto:Path: /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.590 DEBUG:boto:Data: 2024-04-12T04:02:19.590 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.590 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.590 DEBUG:boto:Params: {} 2024-04-12T04:02:19.590 DEBUG:boto:Token: None 2024-04-12T04:02:19.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37?versionId=null 2024-04-12T04:02:19.590 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE= 2024-04-12T04:02:19.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8VOE43c0TGvFemWqYl6RBCyWpEE=', 'Content-Length': '0'} 2024-04-12T04:02:19.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89c2649e4be4da7-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.592 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.592 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.593 DEBUG:boto:path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.593 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.593 DEBUG:boto:Method: GET 2024-04-12T04:02:19.593 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.593 DEBUG:boto:Data: 2024-04-12T04:02:19.593 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.593 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.593 DEBUG:boto:Params: {} 2024-04-12T04:02:19.593 DEBUG:boto:Token: None 2024-04-12T04:02:19.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.594 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a924456e741993c-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.596 DEBUG:boto:b'dgbpxg-63obj371000falseobj372024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.596 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.596 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.596 DEBUG:boto:path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.597 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.597 DEBUG:boto:Method: GET 2024-04-12T04:02:19.597 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.597 DEBUG:boto:Data: 2024-04-12T04:02:19.597 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.597 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.597 DEBUG:boto:Params: {} 2024-04-12T04:02:19.597 DEBUG:boto:Token: None 2024-04-12T04:02:19.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a6fe085e43bd580-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.601 DEBUG:boto:b'dgbpxg-63obj371000falseobj372024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.601 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.601 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.601 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.601 DEBUG:boto:Path: /dgbpxg-63/obj37 2024-04-12T04:02:19.602 DEBUG:boto:Data: 2024-04-12T04:02:19.602 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.602 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.602 DEBUG:boto:Params: {} 2024-04-12T04:02:19.602 DEBUG:boto:Token: None 2024-04-12T04:02:19.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37 2024-04-12T04:02:19.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mEQhsXuJhOEW8K13vUqbKLvTLUw= 2024-04-12T04:02:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mEQhsXuJhOEW8K13vUqbKLvTLUw=', 'Content-Length': '0'} 2024-04-12T04:02:19.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007baa688772c154c2-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.604 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.604 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.604 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.604 DEBUG:boto:Path: /dgbpxg-63/obj37 2024-04-12T04:02:19.604 DEBUG:boto:Data: 2024-04-12T04:02:19.605 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.605 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.605 DEBUG:boto:Params: {} 2024-04-12T04:02:19.605 DEBUG:boto:Token: None 2024-04-12T04:02:19.605 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37 2024-04-12T04:02:19.605 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mEQhsXuJhOEW8K13vUqbKLvTLUw= 2024-04-12T04:02:19.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mEQhsXuJhOEW8K13vUqbKLvTLUw=', 'Content-Length': '0'} 2024-04-12T04:02:19.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4ae88d40a8b7af-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:02:19.607 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.607 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.607 DEBUG:boto:Method: GET 2024-04-12T04:02:19.607 DEBUG:boto:Path: /dgbpxg-63/obj37 2024-04-12T04:02:19.608 DEBUG:boto:Data: 2024-04-12T04:02:19.608 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.608 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.608 DEBUG:boto:Params: {} 2024-04-12T04:02:19.608 DEBUG:boto:Token: None 2024-04-12T04:02:19.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37 2024-04-12T04:02:19.608 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6xRAvvjYomHIuDOACCfyJzTgXZA= 2024-04-12T04:02:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6xRAvvjYomHIuDOACCfyJzTgXZA=', 'Content-Length': '0'} 2024-04-12T04:02:19.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a8735db3bc893e-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.610 DEBUG:boto:path=/dgbpxg-63/obj37 2024-04-12T04:02:19.610 DEBUG:boto:auth_path=/dgbpxg-63/obj37 2024-04-12T04:02:19.610 DEBUG:boto:Method: GET 2024-04-12T04:02:19.610 DEBUG:boto:Path: /dgbpxg-63/obj37 2024-04-12T04:02:19.611 DEBUG:boto:Data: 2024-04-12T04:02:19.611 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.611 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.611 DEBUG:boto:Params: {} 2024-04-12T04:02:19.611 DEBUG:boto:Token: None 2024-04-12T04:02:19.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj37 2024-04-12T04:02:19.611 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6xRAvvjYomHIuDOACCfyJzTgXZA= 2024-04-12T04:02:19.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6xRAvvjYomHIuDOACCfyJzTgXZA=', 'Content-Length': '0'} 2024-04-12T04:02:19.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f23305021e6a15d-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.654 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.655 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.655 DEBUG:boto:path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.655 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.655 DEBUG:boto:Method: GET 2024-04-12T04:02:19.655 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.655 DEBUG:boto:Data: 2024-04-12T04:02:19.655 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.655 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.655 DEBUG:boto:Params: {} 2024-04-12T04:02:19.656 DEBUG:boto:Token: None 2024-04-12T04:02:19.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b1fbed0d5dc585-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.659 DEBUG:boto:b'dgbpxg-63obj371000falseobj372024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.660 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.660 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.660 DEBUG:boto:path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.660 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.660 DEBUG:boto:Method: GET 2024-04-12T04:02:19.660 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj37 2024-04-12T04:02:19.660 DEBUG:boto:Data: 2024-04-12T04:02:19.660 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.661 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.661 DEBUG:boto:Params: {} 2024-04-12T04:02:19.661 DEBUG:boto:Token: None 2024-04-12T04:02:19.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.661 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000970a10c9cf829edc-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.664 DEBUG:boto:b'dgbpxg-63obj371000falseobj372024-04-12T03:58:40.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:02:19.665 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.665 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.665 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.665 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.665 DEBUG:boto:Method: GET 2024-04-12T04:02:19.665 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.665 DEBUG:boto:Data: 2024-04-12T04:02:19.665 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.666 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.666 DEBUG:boto:Params: {} 2024-04-12T04:02:19.666 DEBUG:boto:Token: None 2024-04-12T04:02:19.666 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.666 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo= 2024-04-12T04:02:19.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo=', 'Content-Length': '0'} 2024-04-12T04:02:19.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faebc3252d2094f-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.710 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.711 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.711 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.711 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.711 DEBUG:boto:Method: GET 2024-04-12T04:02:19.711 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.711 DEBUG:boto:Data: 2024-04-12T04:02:19.711 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.711 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.712 DEBUG:boto:Params: {} 2024-04-12T04:02:19.712 DEBUG:boto:Token: None 2024-04-12T04:02:19.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo= 2024-04-12T04:02:19.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo=', 'Content-Length': '0'} 2024-04-12T04:02:19.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b3df554547571e-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.714 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.714 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.715 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.715 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.715 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.715 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.715 DEBUG:boto:Data: 2024-04-12T04:02:19.715 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.715 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.715 DEBUG:boto:Params: {} 2024-04-12T04:02:19.715 DEBUG:boto:Token: None 2024-04-12T04:02:19.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HWmwpOhkFRC6ZCM1fLNEAOFQKFs= 2024-04-12T04:02:19.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HWmwpOhkFRC6ZCM1fLNEAOFQKFs=', 'Content-Length': '0'} 2024-04-12T04:02:19.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0e0dbd171850bb-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.718 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.718 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.718 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.718 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.718 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.719 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.719 DEBUG:boto:Data: 2024-04-12T04:02:19.719 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.719 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.719 DEBUG:boto:Params: {} 2024-04-12T04:02:19.719 DEBUG:boto:Token: None 2024-04-12T04:02:19.719 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HWmwpOhkFRC6ZCM1fLNEAOFQKFs= 2024-04-12T04:02:19.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HWmwpOhkFRC6ZCM1fLNEAOFQKFs=', 'Content-Length': '0'} 2024-04-12T04:02:19.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57df4ae2aa1716b-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:02:19.721 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.721 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.721 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.721 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.721 DEBUG:boto:Method: GET 2024-04-12T04:02:19.722 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.722 DEBUG:boto:Data: 2024-04-12T04:02:19.722 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.722 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.722 DEBUG:boto:Params: {} 2024-04-12T04:02:19.722 DEBUG:boto:Token: None 2024-04-12T04:02:19.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.722 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo= 2024-04-12T04:02:19.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo=', 'Content-Length': '0'} 2024-04-12T04:02:19.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8e1d3f6e0e509f-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.724 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.724 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.725 DEBUG:boto:path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.725 DEBUG:boto:auth_path=/dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.725 DEBUG:boto:Method: GET 2024-04-12T04:02:19.725 DEBUG:boto:Path: /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.725 DEBUG:boto:Data: 2024-04-12T04:02:19.725 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.725 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.725 DEBUG:boto:Params: {} 2024-04-12T04:02:19.725 DEBUG:boto:Token: None 2024-04-12T04:02:19.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38?versionId=null 2024-04-12T04:02:19.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo= 2024-04-12T04:02:19.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:he5nPs4GcA7spaplb7H4pUyr7jo=', 'Content-Length': '0'} 2024-04-12T04:02:19.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066267bbc3872a95d-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.727 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.728 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.728 DEBUG:boto:path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.728 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.728 DEBUG:boto:Method: GET 2024-04-12T04:02:19.728 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.728 DEBUG:boto:Data: 2024-04-12T04:02:19.728 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.728 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.729 DEBUG:boto:Params: {} 2024-04-12T04:02:19.729 DEBUG:boto:Token: None 2024-04-12T04:02:19.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518d5453e502b687-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.731 DEBUG:boto:b'dgbpxg-63obj381000falseobj382024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.732 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.732 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.732 DEBUG:boto:path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.732 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.732 DEBUG:boto:Method: GET 2024-04-12T04:02:19.732 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.732 DEBUG:boto:Data: 2024-04-12T04:02:19.732 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.732 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.733 DEBUG:boto:Params: {} 2024-04-12T04:02:19.733 DEBUG:boto:Token: None 2024-04-12T04:02:19.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.733 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dabbeb32e3b08698-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.736 DEBUG:boto:b'dgbpxg-63obj381000falseobj382024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.737 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.737 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.737 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.737 DEBUG:boto:Path: /dgbpxg-63/obj38 2024-04-12T04:02:19.737 DEBUG:boto:Data: 2024-04-12T04:02:19.737 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.737 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.737 DEBUG:boto:Params: {} 2024-04-12T04:02:19.737 DEBUG:boto:Token: None 2024-04-12T04:02:19.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38 2024-04-12T04:02:19.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aTvDaMGbDJORHIrsP0FiHolO0fE= 2024-04-12T04:02:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aTvDaMGbDJORHIrsP0FiHolO0fE=', 'Content-Length': '0'} 2024-04-12T04:02:19.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ee4fb913b2236d-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.739 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.740 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.740 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.740 DEBUG:boto:Path: /dgbpxg-63/obj38 2024-04-12T04:02:19.740 DEBUG:boto:Data: 2024-04-12T04:02:19.740 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.740 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.740 DEBUG:boto:Params: {} 2024-04-12T04:02:19.740 DEBUG:boto:Token: None 2024-04-12T04:02:19.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38 2024-04-12T04:02:19.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aTvDaMGbDJORHIrsP0FiHolO0fE= 2024-04-12T04:02:19.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aTvDaMGbDJORHIrsP0FiHolO0fE=', 'Content-Length': '0'} 2024-04-12T04:02:19.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64d02f31ac90204-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:02:19.742 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.743 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.743 DEBUG:boto:Method: GET 2024-04-12T04:02:19.743 DEBUG:boto:Path: /dgbpxg-63/obj38 2024-04-12T04:02:19.743 DEBUG:boto:Data: 2024-04-12T04:02:19.743 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.743 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.743 DEBUG:boto:Params: {} 2024-04-12T04:02:19.743 DEBUG:boto:Token: None 2024-04-12T04:02:19.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38 2024-04-12T04:02:19.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q6aXr2HqifqV40sCSNm167AfEN8= 2024-04-12T04:02:19.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q6aXr2HqifqV40sCSNm167AfEN8=', 'Content-Length': '0'} 2024-04-12T04:02:19.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32134a6058d9309-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.745 DEBUG:boto:path=/dgbpxg-63/obj38 2024-04-12T04:02:19.746 DEBUG:boto:auth_path=/dgbpxg-63/obj38 2024-04-12T04:02:19.746 DEBUG:boto:Method: GET 2024-04-12T04:02:19.746 DEBUG:boto:Path: /dgbpxg-63/obj38 2024-04-12T04:02:19.746 DEBUG:boto:Data: 2024-04-12T04:02:19.746 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.746 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.746 DEBUG:boto:Params: {} 2024-04-12T04:02:19.746 DEBUG:boto:Token: None 2024-04-12T04:02:19.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj38 2024-04-12T04:02:19.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q6aXr2HqifqV40sCSNm167AfEN8= 2024-04-12T04:02:19.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q6aXr2HqifqV40sCSNm167AfEN8=', 'Content-Length': '0'} 2024-04-12T04:02:19.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7652d6f69828dbf-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.749 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.749 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.749 DEBUG:boto:path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.749 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.749 DEBUG:boto:Method: GET 2024-04-12T04:02:19.749 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.749 DEBUG:boto:Data: 2024-04-12T04:02:19.749 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.750 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.750 DEBUG:boto:Params: {} 2024-04-12T04:02:19.750 DEBUG:boto:Token: None 2024-04-12T04:02:19.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.750 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000accf72441b72921a-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.752 DEBUG:boto:b'dgbpxg-63obj381000falseobj382024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.753 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.753 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.753 DEBUG:boto:path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.753 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.753 DEBUG:boto:Method: GET 2024-04-12T04:02:19.753 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj38 2024-04-12T04:02:19.753 DEBUG:boto:Data: 2024-04-12T04:02:19.753 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.754 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.754 DEBUG:boto:Params: {} 2024-04-12T04:02:19.754 DEBUG:boto:Token: None 2024-04-12T04:02:19.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.754 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052823fd08052512-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.759 DEBUG:boto:b'dgbpxg-63obj381000falseobj382024-04-12T03:58:39.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:02:19.760 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.760 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.760 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.760 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.760 DEBUG:boto:Method: GET 2024-04-12T04:02:19.760 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.760 DEBUG:boto:Data: 2024-04-12T04:02:19.761 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.761 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.761 DEBUG:boto:Params: {} 2024-04-12T04:02:19.761 DEBUG:boto:Token: None 2024-04-12T04:02:19.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.761 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc= 2024-04-12T04:02:19.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc=', 'Content-Length': '0'} 2024-04-12T04:02:19.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611f4b4996940b9c-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.763 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.763 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.763 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.763 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.763 DEBUG:boto:Method: GET 2024-04-12T04:02:19.764 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.764 DEBUG:boto:Data: 2024-04-12T04:02:19.764 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.764 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.764 DEBUG:boto:Params: {} 2024-04-12T04:02:19.764 DEBUG:boto:Token: None 2024-04-12T04:02:19.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.764 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc= 2024-04-12T04:02:19.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc=', 'Content-Length': '0'} 2024-04-12T04:02:19.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dad170726e7ec8-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.810 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.810 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.811 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.811 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.811 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.811 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.811 DEBUG:boto:Data: 2024-04-12T04:02:19.811 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.811 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.811 DEBUG:boto:Params: {} 2024-04-12T04:02:19.811 DEBUG:boto:Token: None 2024-04-12T04:02:19.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2Unk/mebfA4mCn4RB8b+yy1WaA= 2024-04-12T04:02:19.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2Unk/mebfA4mCn4RB8b+yy1WaA=', 'Content-Length': '0'} 2024-04-12T04:02:19.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a90435ce064021-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.814 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.814 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.814 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.814 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.814 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.814 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.814 DEBUG:boto:Data: 2024-04-12T04:02:19.814 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.815 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.815 DEBUG:boto:Params: {} 2024-04-12T04:02:19.815 DEBUG:boto:Token: None 2024-04-12T04:02:19.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.815 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:t2Unk/mebfA4mCn4RB8b+yy1WaA= 2024-04-12T04:02:19.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:t2Unk/mebfA4mCn4RB8b+yy1WaA=', 'Content-Length': '0'} 2024-04-12T04:02:19.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920ef0fedda68812-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:02:19.817 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.817 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.817 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.817 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.817 DEBUG:boto:Method: GET 2024-04-12T04:02:19.817 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.817 DEBUG:boto:Data: 2024-04-12T04:02:19.818 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.818 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.818 DEBUG:boto:Params: {} 2024-04-12T04:02:19.818 DEBUG:boto:Token: None 2024-04-12T04:02:19.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.818 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc= 2024-04-12T04:02:19.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc=', 'Content-Length': '0'} 2024-04-12T04:02:19.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f75435b395d63037-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.820 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.820 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.820 DEBUG:boto:path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.820 DEBUG:boto:auth_path=/dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.821 DEBUG:boto:Method: GET 2024-04-12T04:02:19.821 DEBUG:boto:Path: /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.821 DEBUG:boto:Data: 2024-04-12T04:02:19.821 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.821 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.821 DEBUG:boto:Params: {} 2024-04-12T04:02:19.821 DEBUG:boto:Token: None 2024-04-12T04:02:19.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44?versionId=null 2024-04-12T04:02:19.821 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc= 2024-04-12T04:02:19.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C3EiO/70FIiqlnL6M6bwLcb/bjc=', 'Content-Length': '0'} 2024-04-12T04:02:19.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a7710a193f26da-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.823 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.823 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.823 DEBUG:boto:path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.824 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.824 DEBUG:boto:Method: GET 2024-04-12T04:02:19.824 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.824 DEBUG:boto:Data: 2024-04-12T04:02:19.824 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.824 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.824 DEBUG:boto:Params: {} 2024-04-12T04:02:19.824 DEBUG:boto:Token: None 2024-04-12T04:02:19.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.825 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b24490d89dfb5f3-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.827 DEBUG:boto:b'dgbpxg-63obj441000falseobj442024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.827 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.827 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.827 DEBUG:boto:path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.828 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.828 DEBUG:boto:Method: GET 2024-04-12T04:02:19.828 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.828 DEBUG:boto:Data: 2024-04-12T04:02:19.828 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.828 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.828 DEBUG:boto:Params: {} 2024-04-12T04:02:19.828 DEBUG:boto:Token: None 2024-04-12T04:02:19.829 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.829 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8b6aedce08247b-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.832 DEBUG:boto:b'dgbpxg-63obj441000falseobj442024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.833 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.833 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.833 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.833 DEBUG:boto:Path: /dgbpxg-63/obj44 2024-04-12T04:02:19.833 DEBUG:boto:Data: 2024-04-12T04:02:19.833 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.833 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.833 DEBUG:boto:Params: {} 2024-04-12T04:02:19.834 DEBUG:boto:Token: None 2024-04-12T04:02:19.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44 2024-04-12T04:02:19.834 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nkuXOio9M6s8ChZJPzM+jUY65IA= 2024-04-12T04:02:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nkuXOio9M6s8ChZJPzM+jUY65IA=', 'Content-Length': '0'} 2024-04-12T04:02:19.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09bb0b1b86fb0be-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.836 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.836 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.836 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.836 DEBUG:boto:Path: /dgbpxg-63/obj44 2024-04-12T04:02:19.836 DEBUG:boto:Data: 2024-04-12T04:02:19.836 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.836 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.836 DEBUG:boto:Params: {} 2024-04-12T04:02:19.837 DEBUG:boto:Token: None 2024-04-12T04:02:19.837 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44 2024-04-12T04:02:19.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nkuXOio9M6s8ChZJPzM+jUY65IA= 2024-04-12T04:02:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nkuXOio9M6s8ChZJPzM+jUY65IA=', 'Content-Length': '0'} 2024-04-12T04:02:19.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1b50300fef8464-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:02:19.839 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.839 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.839 DEBUG:boto:Method: GET 2024-04-12T04:02:19.839 DEBUG:boto:Path: /dgbpxg-63/obj44 2024-04-12T04:02:19.839 DEBUG:boto:Data: 2024-04-12T04:02:19.839 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.839 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.840 DEBUG:boto:Params: {} 2024-04-12T04:02:19.840 DEBUG:boto:Token: None 2024-04-12T04:02:19.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44 2024-04-12T04:02:19.840 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TWWeWQHdieQAuANTeAAd3Jc+INQ= 2024-04-12T04:02:19.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TWWeWQHdieQAuANTeAAd3Jc+INQ=', 'Content-Length': '0'} 2024-04-12T04:02:19.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b781e0db019dd9b9-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.882 DEBUG:boto:path=/dgbpxg-63/obj44 2024-04-12T04:02:19.883 DEBUG:boto:auth_path=/dgbpxg-63/obj44 2024-04-12T04:02:19.883 DEBUG:boto:Method: GET 2024-04-12T04:02:19.883 DEBUG:boto:Path: /dgbpxg-63/obj44 2024-04-12T04:02:19.883 DEBUG:boto:Data: 2024-04-12T04:02:19.883 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.883 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.883 DEBUG:boto:Params: {} 2024-04-12T04:02:19.883 DEBUG:boto:Token: None 2024-04-12T04:02:19.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj44 2024-04-12T04:02:19.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TWWeWQHdieQAuANTeAAd3Jc+INQ= 2024-04-12T04:02:19.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TWWeWQHdieQAuANTeAAd3Jc+INQ=', 'Content-Length': '0'} 2024-04-12T04:02:19.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a70463ef86c7bd-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.885 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.886 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.886 DEBUG:boto:path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.886 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.886 DEBUG:boto:Method: GET 2024-04-12T04:02:19.886 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.886 DEBUG:boto:Data: 2024-04-12T04:02:19.886 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.886 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.886 DEBUG:boto:Params: {} 2024-04-12T04:02:19.887 DEBUG:boto:Token: None 2024-04-12T04:02:19.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.887 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9cfba58e26b2ab1-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.889 DEBUG:boto:b'dgbpxg-63obj441000falseobj442024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.889 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.890 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.890 DEBUG:boto:path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.890 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.890 DEBUG:boto:Method: GET 2024-04-12T04:02:19.890 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj44 2024-04-12T04:02:19.890 DEBUG:boto:Data: 2024-04-12T04:02:19.890 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.890 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.890 DEBUG:boto:Params: {} 2024-04-12T04:02:19.891 DEBUG:boto:Token: None 2024-04-12T04:02:19.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.891 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276d63f5dfc63133-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.895 DEBUG:boto:b'dgbpxg-63obj441000falseobj442024-04-12T03:58:39.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:02:19.895 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.895 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.896 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.896 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.896 DEBUG:boto:Method: GET 2024-04-12T04:02:19.896 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.896 DEBUG:boto:Data: 2024-04-12T04:02:19.896 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.896 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.896 DEBUG:boto:Params: {} 2024-04-12T04:02:19.896 DEBUG:boto:Token: None 2024-04-12T04:02:19.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.897 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws= 2024-04-12T04:02:19.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws=', 'Content-Length': '0'} 2024-04-12T04:02:19.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da5220a72288cc6-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.898 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.898 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.899 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.899 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.899 DEBUG:boto:Method: GET 2024-04-12T04:02:19.899 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.899 DEBUG:boto:Data: 2024-04-12T04:02:19.899 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.899 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.899 DEBUG:boto:Params: {} 2024-04-12T04:02:19.899 DEBUG:boto:Token: None 2024-04-12T04:02:19.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.900 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws= 2024-04-12T04:02:19.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws=', 'Content-Length': '0'} 2024-04-12T04:02:19.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000220dfe68013b684a-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.902 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.902 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.902 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.902 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.902 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.902 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.902 DEBUG:boto:Data: 2024-04-12T04:02:19.902 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.902 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.902 DEBUG:boto:Params: {} 2024-04-12T04:02:19.903 DEBUG:boto:Token: None 2024-04-12T04:02:19.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.903 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lbv2gtxKvOP3CmU8e3wYGggzU50= 2024-04-12T04:02:19.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lbv2gtxKvOP3CmU8e3wYGggzU50=', 'Content-Length': '0'} 2024-04-12T04:02:19.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a209a9b7c92f57-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.905 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.905 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.905 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.905 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.905 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.905 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.905 DEBUG:boto:Data: 2024-04-12T04:02:19.906 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.906 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.906 DEBUG:boto:Params: {} 2024-04-12T04:02:19.906 DEBUG:boto:Token: None 2024-04-12T04:02:19.906 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.906 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lbv2gtxKvOP3CmU8e3wYGggzU50= 2024-04-12T04:02:19.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lbv2gtxKvOP3CmU8e3wYGggzU50=', 'Content-Length': '0'} 2024-04-12T04:02:19.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb616760e228980-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:02:19.908 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.908 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.908 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.909 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.909 DEBUG:boto:Method: GET 2024-04-12T04:02:19.909 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.909 DEBUG:boto:Data: 2024-04-12T04:02:19.909 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.909 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.909 DEBUG:boto:Params: {} 2024-04-12T04:02:19.910 DEBUG:boto:Token: None 2024-04-12T04:02:19.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.910 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws= 2024-04-12T04:02:19.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws=', 'Content-Length': '0'} 2024-04-12T04:02:19.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17ec0488b0b614c-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.912 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.912 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.912 DEBUG:boto:path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.912 DEBUG:boto:auth_path=/dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.912 DEBUG:boto:Method: GET 2024-04-12T04:02:19.912 DEBUG:boto:Path: /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.912 DEBUG:boto:Data: 2024-04-12T04:02:19.912 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.912 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.912 DEBUG:boto:Params: {} 2024-04-12T04:02:19.913 DEBUG:boto:Token: None 2024-04-12T04:02:19.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46?versionId=null 2024-04-12T04:02:19.913 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws= 2024-04-12T04:02:19.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2zKAidMfDuyjRRla2pE0gA9Ykws=', 'Content-Length': '0'} 2024-04-12T04:02:19.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002567e130f843d4d4-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.915 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.915 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.915 DEBUG:boto:path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.915 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.915 DEBUG:boto:Method: GET 2024-04-12T04:02:19.916 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.916 DEBUG:boto:Data: 2024-04-12T04:02:19.916 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.916 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.916 DEBUG:boto:Params: {} 2024-04-12T04:02:19.916 DEBUG:boto:Token: None 2024-04-12T04:02:19.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.916 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac7736ac5b72d175-006618b24b-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.919 DEBUG:boto:b'dgbpxg-63obj461000falseobj462024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.921 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.921 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.922 DEBUG:boto:path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.922 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.922 DEBUG:boto:Method: GET 2024-04-12T04:02:19.922 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.922 DEBUG:boto:Data: 2024-04-12T04:02:19.922 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.922 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.922 DEBUG:boto:Params: {} 2024-04-12T04:02:19.922 DEBUG:boto:Token: None 2024-04-12T04:02:19.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005246ad147300e06a-006618b24b-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.926 DEBUG:boto:b'dgbpxg-63obj461000falseobj462024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.927 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.927 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.927 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.927 DEBUG:boto:Path: /dgbpxg-63/obj46 2024-04-12T04:02:19.927 DEBUG:boto:Data: 2024-04-12T04:02:19.927 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.927 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.928 DEBUG:boto:Params: {} 2024-04-12T04:02:19.928 DEBUG:boto:Token: None 2024-04-12T04:02:19.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46 2024-04-12T04:02:19.928 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qbf5DBpF0jyHduJJM+f6IxnWsUY= 2024-04-12T04:02:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qbf5DBpF0jyHduJJM+f6IxnWsUY=', 'Content-Length': '0'} 2024-04-12T04:02:19.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2a7d41c8610aa2e-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.930 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.930 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.930 DEBUG:boto:Method: HEAD 2024-04-12T04:02:19.930 DEBUG:boto:Path: /dgbpxg-63/obj46 2024-04-12T04:02:19.930 DEBUG:boto:Data: 2024-04-12T04:02:19.930 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.931 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.931 DEBUG:boto:Params: {} 2024-04-12T04:02:19.931 DEBUG:boto:Token: None 2024-04-12T04:02:19.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46 2024-04-12T04:02:19.931 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qbf5DBpF0jyHduJJM+f6IxnWsUY= 2024-04-12T04:02:19.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qbf5DBpF0jyHduJJM+f6IxnWsUY=', 'Content-Length': '0'} 2024-04-12T04:02:19.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006b504cf91c9db02-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:02:19.933 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.933 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.933 DEBUG:boto:Method: GET 2024-04-12T04:02:19.933 DEBUG:boto:Path: /dgbpxg-63/obj46 2024-04-12T04:02:19.933 DEBUG:boto:Data: 2024-04-12T04:02:19.933 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.933 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.934 DEBUG:boto:Params: {} 2024-04-12T04:02:19.934 DEBUG:boto:Token: None 2024-04-12T04:02:19.934 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46 2024-04-12T04:02:19.934 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6gR50mSSm9dI6e9S0zpYGFflgKk= 2024-04-12T04:02:19.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6gR50mSSm9dI6e9S0zpYGFflgKk=', 'Content-Length': '0'} 2024-04-12T04:02:19.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f814777caf1351bc-006618b24b-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.936 DEBUG:boto:path=/dgbpxg-63/obj46 2024-04-12T04:02:19.936 DEBUG:boto:auth_path=/dgbpxg-63/obj46 2024-04-12T04:02:19.936 DEBUG:boto:Method: GET 2024-04-12T04:02:19.936 DEBUG:boto:Path: /dgbpxg-63/obj46 2024-04-12T04:02:19.936 DEBUG:boto:Data: 2024-04-12T04:02:19.936 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.936 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.936 DEBUG:boto:Params: {} 2024-04-12T04:02:19.937 DEBUG:boto:Token: None 2024-04-12T04:02:19.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj46 2024-04-12T04:02:19.937 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6gR50mSSm9dI6e9S0zpYGFflgKk= 2024-04-12T04:02:19.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6gR50mSSm9dI6e9S0zpYGFflgKk=', 'Content-Length': '0'} 2024-04-12T04:02:19.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e53875be26a8b88-006618b24b-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.982 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.983 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.983 DEBUG:boto:path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.983 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.983 DEBUG:boto:Method: GET 2024-04-12T04:02:19.983 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.983 DEBUG:boto:Data: 2024-04-12T04:02:19.983 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.983 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.983 DEBUG:boto:Params: {} 2024-04-12T04:02:19.984 DEBUG:boto:Token: None 2024-04-12T04:02:19.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.984 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156a313887e27b72-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.987 DEBUG:boto:b'dgbpxg-63obj461000falseobj462024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.987 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:19.988 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:19.988 DEBUG:boto:path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.988 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.988 DEBUG:boto:Method: GET 2024-04-12T04:02:19.988 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj46 2024-04-12T04:02:19.988 DEBUG:boto:Data: 2024-04-12T04:02:19.988 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:19.988 DEBUG:boto:Port: 8000 2024-04-12T04:02:19.988 DEBUG:boto:Params: {} 2024-04-12T04:02:19.988 DEBUG:boto:Token: None 2024-04-12T04:02:19.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/ 2024-04-12T04:02:19.989 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI= 2024-04-12T04:02:19.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LcqIinsJIARBKGcTtdS51Q1CEEI=', 'Content-Length': '0'} 2024-04-12T04:02:19.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f361125c86787004-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:19.992 DEBUG:boto:b'dgbpxg-63obj461000falseobj462024-04-12T03:58:39.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:19.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:02:19.993 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:19.993 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:19.993 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:19.993 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:19.993 DEBUG:boto:Method: GET 2024-04-12T04:02:19.993 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:19.993 DEBUG:boto:Data: 2024-04-12T04:02:19.994 DEBUG:boto:Headers: {} 2024-04-12T04:02:19.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:19.994 DEBUG:boto:Port: 8001 2024-04-12T04:02:19.994 DEBUG:boto:Params: {} 2024-04-12T04:02:19.994 DEBUG:boto:Token: None 2024-04-12T04:02:19.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:19 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:19.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9qv4CQ6Nlsn3v2dpTkFpozUJQZU= 2024-04-12T04:02:19.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9qv4CQ6Nlsn3v2dpTkFpozUJQZU=', 'Content-Length': '0'} 2024-04-12T04:02:19.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac9ad3e31f4d5c2-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.042 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.043 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.043 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.043 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.043 DEBUG:boto:Method: GET 2024-04-12T04:02:20.043 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.043 DEBUG:boto:Data: 2024-04-12T04:02:20.043 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.043 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.043 DEBUG:boto:Params: {} 2024-04-12T04:02:20.044 DEBUG:boto:Token: None 2024-04-12T04:02:20.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.044 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc= 2024-04-12T04:02:20.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc=', 'Content-Length': '0'} 2024-04-12T04:02:20.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc12217e6464dc4a-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.046 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.046 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.046 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.047 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.047 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.047 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.047 DEBUG:boto:Data: 2024-04-12T04:02:20.047 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.047 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.047 DEBUG:boto:Params: {} 2024-04-12T04:02:20.047 DEBUG:boto:Token: None 2024-04-12T04:02:20.047 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQrRPCjV02j3t9xytp6pvG1SNiM= 2024-04-12T04:02:20.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQrRPCjV02j3t9xytp6pvG1SNiM=', 'Content-Length': '0'} 2024-04-12T04:02:20.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007141f2537c12830d-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.049 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.049 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.049 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.050 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.050 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.050 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.050 DEBUG:boto:Data: 2024-04-12T04:02:20.050 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.050 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.050 DEBUG:boto:Params: {} 2024-04-12T04:02:20.050 DEBUG:boto:Token: None 2024-04-12T04:02:20.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YQrRPCjV02j3t9xytp6pvG1SNiM= 2024-04-12T04:02:20.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YQrRPCjV02j3t9xytp6pvG1SNiM=', 'Content-Length': '0'} 2024-04-12T04:02:20.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e4b443f65aff23b-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:02:20.053 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.053 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.053 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.053 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.053 DEBUG:boto:Method: GET 2024-04-12T04:02:20.053 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.053 DEBUG:boto:Data: 2024-04-12T04:02:20.053 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.053 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.054 DEBUG:boto:Params: {} 2024-04-12T04:02:20.054 DEBUG:boto:Token: None 2024-04-12T04:02:20.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.054 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc= 2024-04-12T04:02:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc=', 'Content-Length': '0'} 2024-04-12T04:02:20.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff13d680e283dc11-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.056 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.056 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.056 DEBUG:boto:path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.056 DEBUG:boto:auth_path=/dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.056 DEBUG:boto:Method: GET 2024-04-12T04:02:20.056 DEBUG:boto:Path: /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.056 DEBUG:boto:Data: 2024-04-12T04:02:20.056 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.057 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.057 DEBUG:boto:Params: {} 2024-04-12T04:02:20.057 DEBUG:boto:Token: None 2024-04-12T04:02:20.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51?versionId=null 2024-04-12T04:02:20.057 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc= 2024-04-12T04:02:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9y/DQZ/wj7AYN3oqIR1x/6yJWLc=', 'Content-Length': '0'} 2024-04-12T04:02:20.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b815249a097aa1-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.059 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.059 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.059 DEBUG:boto:path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.059 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.059 DEBUG:boto:Method: GET 2024-04-12T04:02:20.059 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.059 DEBUG:boto:Data: 2024-04-12T04:02:20.060 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.060 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.060 DEBUG:boto:Params: {} 2024-04-12T04:02:20.060 DEBUG:boto:Token: None 2024-04-12T04:02:20.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.060 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6eaa2c2d14c6e3-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.063 DEBUG:boto:b'dgbpxg-63obj511000falseobj512024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.063 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.063 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.063 DEBUG:boto:path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.063 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.063 DEBUG:boto:Method: GET 2024-04-12T04:02:20.064 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.064 DEBUG:boto:Data: 2024-04-12T04:02:20.064 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.064 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.064 DEBUG:boto:Params: {} 2024-04-12T04:02:20.064 DEBUG:boto:Token: None 2024-04-12T04:02:20.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.064 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c49a9df0f38754d1-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.068 DEBUG:boto:b'dgbpxg-63obj511000falseobj512024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.068 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.070 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.070 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.071 DEBUG:boto:Path: /dgbpxg-63/obj51 2024-04-12T04:02:20.071 DEBUG:boto:Data: 2024-04-12T04:02:20.071 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.071 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.071 DEBUG:boto:Params: {} 2024-04-12T04:02:20.071 DEBUG:boto:Token: None 2024-04-12T04:02:20.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51 2024-04-12T04:02:20.071 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GD3I0k5x41c+ofrgrTwxuebUVtM= 2024-04-12T04:02:20.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GD3I0k5x41c+ofrgrTwxuebUVtM=', 'Content-Length': '0'} 2024-04-12T04:02:20.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077027a13ab314819-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.074 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.074 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.074 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.074 DEBUG:boto:Path: /dgbpxg-63/obj51 2024-04-12T04:02:20.074 DEBUG:boto:Data: 2024-04-12T04:02:20.074 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.075 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.075 DEBUG:boto:Params: {} 2024-04-12T04:02:20.075 DEBUG:boto:Token: None 2024-04-12T04:02:20.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51 2024-04-12T04:02:20.075 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GD3I0k5x41c+ofrgrTwxuebUVtM= 2024-04-12T04:02:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GD3I0k5x41c+ofrgrTwxuebUVtM=', 'Content-Length': '0'} 2024-04-12T04:02:20.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1b5797b1bd8e40-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:02:20.077 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.078 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.078 DEBUG:boto:Method: GET 2024-04-12T04:02:20.078 DEBUG:boto:Path: /dgbpxg-63/obj51 2024-04-12T04:02:20.078 DEBUG:boto:Data: 2024-04-12T04:02:20.078 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.078 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.078 DEBUG:boto:Params: {} 2024-04-12T04:02:20.078 DEBUG:boto:Token: None 2024-04-12T04:02:20.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51 2024-04-12T04:02:20.079 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P/qNzjcmSISiYsvtVQLv9bFhWLc= 2024-04-12T04:02:20.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P/qNzjcmSISiYsvtVQLv9bFhWLc=', 'Content-Length': '0'} 2024-04-12T04:02:20.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039581883a8036046-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.081 DEBUG:boto:path=/dgbpxg-63/obj51 2024-04-12T04:02:20.081 DEBUG:boto:auth_path=/dgbpxg-63/obj51 2024-04-12T04:02:20.081 DEBUG:boto:Method: GET 2024-04-12T04:02:20.081 DEBUG:boto:Path: /dgbpxg-63/obj51 2024-04-12T04:02:20.081 DEBUG:boto:Data: 2024-04-12T04:02:20.081 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.081 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.081 DEBUG:boto:Params: {} 2024-04-12T04:02:20.081 DEBUG:boto:Token: None 2024-04-12T04:02:20.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj51 2024-04-12T04:02:20.082 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P/qNzjcmSISiYsvtVQLv9bFhWLc= 2024-04-12T04:02:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P/qNzjcmSISiYsvtVQLv9bFhWLc=', 'Content-Length': '0'} 2024-04-12T04:02:20.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da07e77491bcf54e-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.126 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.126 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.127 DEBUG:boto:path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.127 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.127 DEBUG:boto:Method: GET 2024-04-12T04:02:20.127 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.127 DEBUG:boto:Data: 2024-04-12T04:02:20.127 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.127 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.127 DEBUG:boto:Params: {} 2024-04-12T04:02:20.127 DEBUG:boto:Token: None 2024-04-12T04:02:20.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.129 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007080fc318657f673-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.132 DEBUG:boto:b'dgbpxg-63obj511000falseobj512024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.132 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.132 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.132 DEBUG:boto:path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.132 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.132 DEBUG:boto:Method: GET 2024-04-12T04:02:20.133 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj51 2024-04-12T04:02:20.133 DEBUG:boto:Data: 2024-04-12T04:02:20.133 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.133 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.133 DEBUG:boto:Params: {} 2024-04-12T04:02:20.133 DEBUG:boto:Token: None 2024-04-12T04:02:20.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004771a6d14f387b4a-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.137 DEBUG:boto:b'dgbpxg-63obj511000falseobj512024-04-12T03:58:39.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:02:20.138 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.138 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.138 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.138 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.138 DEBUG:boto:Method: GET 2024-04-12T04:02:20.138 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.138 DEBUG:boto:Data: 2024-04-12T04:02:20.138 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.139 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.139 DEBUG:boto:Params: {} 2024-04-12T04:02:20.139 DEBUG:boto:Token: None 2024-04-12T04:02:20.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY= 2024-04-12T04:02:20.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY=', 'Content-Length': '0'} 2024-04-12T04:02:20.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc8a252d31328fb-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.182 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.182 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.183 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.183 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.183 DEBUG:boto:Method: GET 2024-04-12T04:02:20.183 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.183 DEBUG:boto:Data: 2024-04-12T04:02:20.183 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.183 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.183 DEBUG:boto:Params: {} 2024-04-12T04:02:20.183 DEBUG:boto:Token: None 2024-04-12T04:02:20.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY= 2024-04-12T04:02:20.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY=', 'Content-Length': '0'} 2024-04-12T04:02:20.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002778473675362509-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.186 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.186 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.186 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.186 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.186 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.186 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.187 DEBUG:boto:Data: 2024-04-12T04:02:20.187 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.187 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.187 DEBUG:boto:Params: {} 2024-04-12T04:02:20.187 DEBUG:boto:Token: None 2024-04-12T04:02:20.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IA8RERnMOn2Mnk3bXhEqulAIf1U= 2024-04-12T04:02:20.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IA8RERnMOn2Mnk3bXhEqulAIf1U=', 'Content-Length': '0'} 2024-04-12T04:02:20.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba2ac8e5b5589ef-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.189 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.189 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.189 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.189 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.189 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.190 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.190 DEBUG:boto:Data: 2024-04-12T04:02:20.190 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.190 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.190 DEBUG:boto:Params: {} 2024-04-12T04:02:20.190 DEBUG:boto:Token: None 2024-04-12T04:02:20.190 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.190 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IA8RERnMOn2Mnk3bXhEqulAIf1U= 2024-04-12T04:02:20.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IA8RERnMOn2Mnk3bXhEqulAIf1U=', 'Content-Length': '0'} 2024-04-12T04:02:20.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c790f8315eec2e5a-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:02:20.193 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.193 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.193 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.193 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.193 DEBUG:boto:Method: GET 2024-04-12T04:02:20.193 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.193 DEBUG:boto:Data: 2024-04-12T04:02:20.193 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.193 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.194 DEBUG:boto:Params: {} 2024-04-12T04:02:20.194 DEBUG:boto:Token: None 2024-04-12T04:02:20.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY= 2024-04-12T04:02:20.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY=', 'Content-Length': '0'} 2024-04-12T04:02:20.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745d95468671f97b-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.196 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.196 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.196 DEBUG:boto:path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.196 DEBUG:boto:auth_path=/dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.196 DEBUG:boto:Method: GET 2024-04-12T04:02:20.196 DEBUG:boto:Path: /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.196 DEBUG:boto:Data: 2024-04-12T04:02:20.196 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.197 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.197 DEBUG:boto:Params: {} 2024-04-12T04:02:20.197 DEBUG:boto:Token: None 2024-04-12T04:02:20.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55?versionId=null 2024-04-12T04:02:20.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY= 2024-04-12T04:02:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ae8MyXrQbtWzg5q6gXjrTDTcdrY=', 'Content-Length': '0'} 2024-04-12T04:02:20.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d30ce22499757505-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.199 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.200 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.200 DEBUG:boto:path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.200 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.200 DEBUG:boto:Method: GET 2024-04-12T04:02:20.200 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.200 DEBUG:boto:Data: 2024-04-12T04:02:20.200 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.200 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.201 DEBUG:boto:Params: {} 2024-04-12T04:02:20.201 DEBUG:boto:Token: None 2024-04-12T04:02:20.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.201 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb06ef781c8eaf5d-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.203 DEBUG:boto:b'dgbpxg-63obj551000falseobj552024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.204 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.204 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.204 DEBUG:boto:path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.204 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.205 DEBUG:boto:Method: GET 2024-04-12T04:02:20.205 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.205 DEBUG:boto:Data: 2024-04-12T04:02:20.205 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.205 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.205 DEBUG:boto:Params: {} 2024-04-12T04:02:20.205 DEBUG:boto:Token: None 2024-04-12T04:02:20.205 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.205 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e921b8a625401340-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.209 DEBUG:boto:b'dgbpxg-63obj551000falseobj552024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.209 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.210 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.210 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.210 DEBUG:boto:Path: /dgbpxg-63/obj55 2024-04-12T04:02:20.210 DEBUG:boto:Data: 2024-04-12T04:02:20.210 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.210 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.210 DEBUG:boto:Params: {} 2024-04-12T04:02:20.210 DEBUG:boto:Token: None 2024-04-12T04:02:20.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55 2024-04-12T04:02:20.211 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oxV1AXwlTKRfY35fCnESmVE72Lk= 2024-04-12T04:02:20.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oxV1AXwlTKRfY35fCnESmVE72Lk=', 'Content-Length': '0'} 2024-04-12T04:02:20.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a37c473fddfe2e-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.212 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.212 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.212 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.213 DEBUG:boto:Path: /dgbpxg-63/obj55 2024-04-12T04:02:20.213 DEBUG:boto:Data: 2024-04-12T04:02:20.213 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.213 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.213 DEBUG:boto:Params: {} 2024-04-12T04:02:20.213 DEBUG:boto:Token: None 2024-04-12T04:02:20.213 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55 2024-04-12T04:02:20.213 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oxV1AXwlTKRfY35fCnESmVE72Lk= 2024-04-12T04:02:20.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oxV1AXwlTKRfY35fCnESmVE72Lk=', 'Content-Length': '0'} 2024-04-12T04:02:20.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8e3ef0d9539cd3d-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:02:20.215 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.216 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.216 DEBUG:boto:Method: GET 2024-04-12T04:02:20.216 DEBUG:boto:Path: /dgbpxg-63/obj55 2024-04-12T04:02:20.216 DEBUG:boto:Data: 2024-04-12T04:02:20.216 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.216 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.216 DEBUG:boto:Params: {} 2024-04-12T04:02:20.216 DEBUG:boto:Token: None 2024-04-12T04:02:20.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55 2024-04-12T04:02:20.217 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4OlOijP3H/BQ2kkUc9HLiB3df8= 2024-04-12T04:02:20.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4OlOijP3H/BQ2kkUc9HLiB3df8=', 'Content-Length': '0'} 2024-04-12T04:02:20.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d947f2359fc114f9-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.218 DEBUG:boto:path=/dgbpxg-63/obj55 2024-04-12T04:02:20.219 DEBUG:boto:auth_path=/dgbpxg-63/obj55 2024-04-12T04:02:20.219 DEBUG:boto:Method: GET 2024-04-12T04:02:20.219 DEBUG:boto:Path: /dgbpxg-63/obj55 2024-04-12T04:02:20.219 DEBUG:boto:Data: 2024-04-12T04:02:20.219 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.219 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.219 DEBUG:boto:Params: {} 2024-04-12T04:02:20.219 DEBUG:boto:Token: None 2024-04-12T04:02:20.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj55 2024-04-12T04:02:20.220 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4OlOijP3H/BQ2kkUc9HLiB3df8= 2024-04-12T04:02:20.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4OlOijP3H/BQ2kkUc9HLiB3df8=', 'Content-Length': '0'} 2024-04-12T04:02:20.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ddab5a8ac2e6e89-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.262 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.262 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.263 DEBUG:boto:path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.263 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.263 DEBUG:boto:Method: GET 2024-04-12T04:02:20.263 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.263 DEBUG:boto:Data: 2024-04-12T04:02:20.263 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.263 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.263 DEBUG:boto:Params: {} 2024-04-12T04:02:20.263 DEBUG:boto:Token: None 2024-04-12T04:02:20.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.264 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000500d68b780778440-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.266 DEBUG:boto:b'dgbpxg-63obj551000falseobj552024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.266 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.266 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.267 DEBUG:boto:path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.267 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.267 DEBUG:boto:Method: GET 2024-04-12T04:02:20.267 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj55 2024-04-12T04:02:20.267 DEBUG:boto:Data: 2024-04-12T04:02:20.267 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.267 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.267 DEBUG:boto:Params: {} 2024-04-12T04:02:20.267 DEBUG:boto:Token: None 2024-04-12T04:02:20.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.267 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008edbe7fed34aeaca-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.271 DEBUG:boto:b'dgbpxg-63obj551000falseobj552024-04-12T03:58:39.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:02:20.272 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.272 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.272 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.272 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.272 DEBUG:boto:Method: GET 2024-04-12T04:02:20.272 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.272 DEBUG:boto:Data: 2024-04-12T04:02:20.272 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.272 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.272 DEBUG:boto:Params: {} 2024-04-12T04:02:20.273 DEBUG:boto:Token: None 2024-04-12T04:02:20.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk= 2024-04-12T04:02:20.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk=', 'Content-Length': '0'} 2024-04-12T04:02:20.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003206ece40139dc84-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.275 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.275 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.275 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.275 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.275 DEBUG:boto:Method: GET 2024-04-12T04:02:20.275 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.275 DEBUG:boto:Data: 2024-04-12T04:02:20.275 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.276 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.276 DEBUG:boto:Params: {} 2024-04-12T04:02:20.276 DEBUG:boto:Token: None 2024-04-12T04:02:20.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk= 2024-04-12T04:02:20.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk=', 'Content-Length': '0'} 2024-04-12T04:02:20.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001362c1f123cab832-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.278 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.278 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.278 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.279 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.279 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.279 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.279 DEBUG:boto:Data: 2024-04-12T04:02:20.279 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.279 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.279 DEBUG:boto:Params: {} 2024-04-12T04:02:20.279 DEBUG:boto:Token: None 2024-04-12T04:02:20.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BnnBWnOcPF4BoMU8KB2xGbKh+hM= 2024-04-12T04:02:20.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BnnBWnOcPF4BoMU8KB2xGbKh+hM=', 'Content-Length': '0'} 2024-04-12T04:02:20.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002886ff7a4bc1f421-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.281 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.282 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.282 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.282 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.282 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.282 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.282 DEBUG:boto:Data: 2024-04-12T04:02:20.282 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.282 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.282 DEBUG:boto:Params: {} 2024-04-12T04:02:20.283 DEBUG:boto:Token: None 2024-04-12T04:02:20.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:BnnBWnOcPF4BoMU8KB2xGbKh+hM= 2024-04-12T04:02:20.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:BnnBWnOcPF4BoMU8KB2xGbKh+hM=', 'Content-Length': '0'} 2024-04-12T04:02:20.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059701889b6f7183a-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:02:20.285 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.285 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.285 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.285 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.285 DEBUG:boto:Method: GET 2024-04-12T04:02:20.285 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.285 DEBUG:boto:Data: 2024-04-12T04:02:20.286 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.286 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.286 DEBUG:boto:Params: {} 2024-04-12T04:02:20.286 DEBUG:boto:Token: None 2024-04-12T04:02:20.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.286 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk= 2024-04-12T04:02:20.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk=', 'Content-Length': '0'} 2024-04-12T04:02:20.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3b2c91aeecbb9c-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.330 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.331 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.331 DEBUG:boto:path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.331 DEBUG:boto:auth_path=/dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.331 DEBUG:boto:Method: GET 2024-04-12T04:02:20.331 DEBUG:boto:Path: /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.331 DEBUG:boto:Data: 2024-04-12T04:02:20.331 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.331 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.331 DEBUG:boto:Params: {} 2024-04-12T04:02:20.332 DEBUG:boto:Token: None 2024-04-12T04:02:20.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57?versionId=null 2024-04-12T04:02:20.332 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk= 2024-04-12T04:02:20.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dso/aPA4EJFsFqYHqAjDg1ChBWk=', 'Content-Length': '0'} 2024-04-12T04:02:20.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd1de25e35a05ff-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.334 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.334 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.334 DEBUG:boto:path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.334 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.334 DEBUG:boto:Method: GET 2024-04-12T04:02:20.334 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.335 DEBUG:boto:Data: 2024-04-12T04:02:20.335 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.335 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.335 DEBUG:boto:Params: {} 2024-04-12T04:02:20.335 DEBUG:boto:Token: None 2024-04-12T04:02:20.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7eaa7137d77457-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.340 DEBUG:boto:b'dgbpxg-63obj571000falseobj572024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.340 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.340 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.340 DEBUG:boto:path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.340 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.341 DEBUG:boto:Method: GET 2024-04-12T04:02:20.341 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.341 DEBUG:boto:Data: 2024-04-12T04:02:20.341 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.341 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.341 DEBUG:boto:Params: {} 2024-04-12T04:02:20.341 DEBUG:boto:Token: None 2024-04-12T04:02:20.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f1eb49cd54d4003-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.345 DEBUG:boto:b'dgbpxg-63obj571000falseobj572024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.345 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.345 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.345 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.346 DEBUG:boto:Path: /dgbpxg-63/obj57 2024-04-12T04:02:20.346 DEBUG:boto:Data: 2024-04-12T04:02:20.346 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.346 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.346 DEBUG:boto:Params: {} 2024-04-12T04:02:20.346 DEBUG:boto:Token: None 2024-04-12T04:02:20.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57 2024-04-12T04:02:20.347 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VzapFinIOylGgjuE+hp0eUEtt90= 2024-04-12T04:02:20.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VzapFinIOylGgjuE+hp0eUEtt90=', 'Content-Length': '0'} 2024-04-12T04:02:20.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51f859f7c810c2c-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.348 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.348 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.348 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.348 DEBUG:boto:Path: /dgbpxg-63/obj57 2024-04-12T04:02:20.349 DEBUG:boto:Data: 2024-04-12T04:02:20.349 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.349 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.349 DEBUG:boto:Params: {} 2024-04-12T04:02:20.349 DEBUG:boto:Token: None 2024-04-12T04:02:20.349 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57 2024-04-12T04:02:20.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VzapFinIOylGgjuE+hp0eUEtt90= 2024-04-12T04:02:20.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VzapFinIOylGgjuE+hp0eUEtt90=', 'Content-Length': '0'} 2024-04-12T04:02:20.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80f7a7a8028d07c-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:02:20.351 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.351 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.351 DEBUG:boto:Method: GET 2024-04-12T04:02:20.351 DEBUG:boto:Path: /dgbpxg-63/obj57 2024-04-12T04:02:20.351 DEBUG:boto:Data: 2024-04-12T04:02:20.352 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.352 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.352 DEBUG:boto:Params: {} 2024-04-12T04:02:20.352 DEBUG:boto:Token: None 2024-04-12T04:02:20.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57 2024-04-12T04:02:20.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:laKOQs0JiRVdPPv+nVVNsou3hjw= 2024-04-12T04:02:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:laKOQs0JiRVdPPv+nVVNsou3hjw=', 'Content-Length': '0'} 2024-04-12T04:02:20.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c27a59ccd4de11-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.354 DEBUG:boto:path=/dgbpxg-63/obj57 2024-04-12T04:02:20.354 DEBUG:boto:auth_path=/dgbpxg-63/obj57 2024-04-12T04:02:20.354 DEBUG:boto:Method: GET 2024-04-12T04:02:20.354 DEBUG:boto:Path: /dgbpxg-63/obj57 2024-04-12T04:02:20.354 DEBUG:boto:Data: 2024-04-12T04:02:20.354 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.355 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.355 DEBUG:boto:Params: {} 2024-04-12T04:02:20.355 DEBUG:boto:Token: None 2024-04-12T04:02:20.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj57 2024-04-12T04:02:20.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:laKOQs0JiRVdPPv+nVVNsou3hjw= 2024-04-12T04:02:20.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:laKOQs0JiRVdPPv+nVVNsou3hjw=', 'Content-Length': '0'} 2024-04-12T04:02:20.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031edb3ed11c90004-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.398 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.399 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.399 DEBUG:boto:path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.399 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.399 DEBUG:boto:Method: GET 2024-04-12T04:02:20.399 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.399 DEBUG:boto:Data: 2024-04-12T04:02:20.399 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.399 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.399 DEBUG:boto:Params: {} 2024-04-12T04:02:20.400 DEBUG:boto:Token: None 2024-04-12T04:02:20.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.400 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3fb348592839a5-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.402 DEBUG:boto:b'dgbpxg-63obj571000falseobj572024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.403 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.403 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.403 DEBUG:boto:path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.403 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.403 DEBUG:boto:Method: GET 2024-04-12T04:02:20.403 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj57 2024-04-12T04:02:20.403 DEBUG:boto:Data: 2024-04-12T04:02:20.403 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.404 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.404 DEBUG:boto:Params: {} 2024-04-12T04:02:20.404 DEBUG:boto:Token: None 2024-04-12T04:02:20.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466b6be253432427-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.408 DEBUG:boto:b'dgbpxg-63obj571000falseobj572024-04-12T03:58:39.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:02:20.409 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.409 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.409 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.409 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.409 DEBUG:boto:Method: GET 2024-04-12T04:02:20.409 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.409 DEBUG:boto:Data: 2024-04-12T04:02:20.409 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.410 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.410 DEBUG:boto:Params: {} 2024-04-12T04:02:20.410 DEBUG:boto:Token: None 2024-04-12T04:02:20.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.410 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI= 2024-04-12T04:02:20.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI=', 'Content-Length': '0'} 2024-04-12T04:02:20.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0cfe3b49da085f0-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.454 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.455 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.455 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.455 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.455 DEBUG:boto:Method: GET 2024-04-12T04:02:20.455 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.455 DEBUG:boto:Data: 2024-04-12T04:02:20.455 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.455 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.455 DEBUG:boto:Params: {} 2024-04-12T04:02:20.456 DEBUG:boto:Token: None 2024-04-12T04:02:20.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI= 2024-04-12T04:02:20.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI=', 'Content-Length': '0'} 2024-04-12T04:02:20.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042966a23b7cc23a8-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.458 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.458 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.458 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.459 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.459 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.459 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.459 DEBUG:boto:Data: 2024-04-12T04:02:20.459 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.459 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.459 DEBUG:boto:Params: {} 2024-04-12T04:02:20.459 DEBUG:boto:Token: None 2024-04-12T04:02:20.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.459 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IIhTqzPuvdIKknNXcHDFMobO5iA= 2024-04-12T04:02:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IIhTqzPuvdIKknNXcHDFMobO5iA=', 'Content-Length': '0'} 2024-04-12T04:02:20.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f6b3d387555044d-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.461 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.461 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.461 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.462 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.462 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.462 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.462 DEBUG:boto:Data: 2024-04-12T04:02:20.462 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.462 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.462 DEBUG:boto:Params: {} 2024-04-12T04:02:20.462 DEBUG:boto:Token: None 2024-04-12T04:02:20.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IIhTqzPuvdIKknNXcHDFMobO5iA= 2024-04-12T04:02:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IIhTqzPuvdIKknNXcHDFMobO5iA=', 'Content-Length': '0'} 2024-04-12T04:02:20.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e22bcd65cb3962-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:02:20.465 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.465 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.465 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.465 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.465 DEBUG:boto:Method: GET 2024-04-12T04:02:20.465 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.465 DEBUG:boto:Data: 2024-04-12T04:02:20.466 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.466 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.466 DEBUG:boto:Params: {} 2024-04-12T04:02:20.466 DEBUG:boto:Token: None 2024-04-12T04:02:20.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.466 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI= 2024-04-12T04:02:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI=', 'Content-Length': '0'} 2024-04-12T04:02:20.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e14cd4957105440-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.468 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.468 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.468 DEBUG:boto:path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.468 DEBUG:boto:auth_path=/dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.468 DEBUG:boto:Method: GET 2024-04-12T04:02:20.468 DEBUG:boto:Path: /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.469 DEBUG:boto:Data: 2024-04-12T04:02:20.469 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.469 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.469 DEBUG:boto:Params: {} 2024-04-12T04:02:20.469 DEBUG:boto:Token: None 2024-04-12T04:02:20.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58?versionId=null 2024-04-12T04:02:20.469 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI= 2024-04-12T04:02:20.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/YotjNNM+WhKRTT8LY3ap+zZrsI=', 'Content-Length': '0'} 2024-04-12T04:02:20.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb233db9c0f0e47-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.471 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.471 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.471 DEBUG:boto:path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.471 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.471 DEBUG:boto:Method: GET 2024-04-12T04:02:20.472 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.472 DEBUG:boto:Data: 2024-04-12T04:02:20.472 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.472 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.472 DEBUG:boto:Params: {} 2024-04-12T04:02:20.472 DEBUG:boto:Token: None 2024-04-12T04:02:20.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002106fc781f3707dc-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.475 DEBUG:boto:b'dgbpxg-63obj581000falseobj582024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.475 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.475 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.475 DEBUG:boto:path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.475 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.475 DEBUG:boto:Method: GET 2024-04-12T04:02:20.476 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.476 DEBUG:boto:Data: 2024-04-12T04:02:20.476 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.476 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.476 DEBUG:boto:Params: {} 2024-04-12T04:02:20.476 DEBUG:boto:Token: None 2024-04-12T04:02:20.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e77988261d69ef-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.480 DEBUG:boto:b'dgbpxg-63obj581000falseobj582024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.480 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.480 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.480 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.481 DEBUG:boto:Path: /dgbpxg-63/obj58 2024-04-12T04:02:20.481 DEBUG:boto:Data: 2024-04-12T04:02:20.481 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.481 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.481 DEBUG:boto:Params: {} 2024-04-12T04:02:20.481 DEBUG:boto:Token: None 2024-04-12T04:02:20.481 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58 2024-04-12T04:02:20.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Id65kagZg7teDdCh9S/EgpnsQdg= 2024-04-12T04:02:20.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Id65kagZg7teDdCh9S/EgpnsQdg=', 'Content-Length': '0'} 2024-04-12T04:02:20.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe1ff4cecf8687a3-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.483 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.483 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.483 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.483 DEBUG:boto:Path: /dgbpxg-63/obj58 2024-04-12T04:02:20.484 DEBUG:boto:Data: 2024-04-12T04:02:20.484 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.484 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.484 DEBUG:boto:Params: {} 2024-04-12T04:02:20.484 DEBUG:boto:Token: None 2024-04-12T04:02:20.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58 2024-04-12T04:02:20.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Id65kagZg7teDdCh9S/EgpnsQdg= 2024-04-12T04:02:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Id65kagZg7teDdCh9S/EgpnsQdg=', 'Content-Length': '0'} 2024-04-12T04:02:20.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f85d749f9dc718e-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:02:20.486 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.486 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.486 DEBUG:boto:Method: GET 2024-04-12T04:02:20.486 DEBUG:boto:Path: /dgbpxg-63/obj58 2024-04-12T04:02:20.487 DEBUG:boto:Data: 2024-04-12T04:02:20.487 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.487 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.487 DEBUG:boto:Params: {} 2024-04-12T04:02:20.487 DEBUG:boto:Token: None 2024-04-12T04:02:20.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58 2024-04-12T04:02:20.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GLvYUrg9Wz6K3kXTRcMuQoSQjqM= 2024-04-12T04:02:20.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GLvYUrg9Wz6K3kXTRcMuQoSQjqM=', 'Content-Length': '0'} 2024-04-12T04:02:20.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf60dde5e392bbc-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.530 DEBUG:boto:path=/dgbpxg-63/obj58 2024-04-12T04:02:20.531 DEBUG:boto:auth_path=/dgbpxg-63/obj58 2024-04-12T04:02:20.531 DEBUG:boto:Method: GET 2024-04-12T04:02:20.531 DEBUG:boto:Path: /dgbpxg-63/obj58 2024-04-12T04:02:20.531 DEBUG:boto:Data: 2024-04-12T04:02:20.531 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.531 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.531 DEBUG:boto:Params: {} 2024-04-12T04:02:20.531 DEBUG:boto:Token: None 2024-04-12T04:02:20.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj58 2024-04-12T04:02:20.532 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GLvYUrg9Wz6K3kXTRcMuQoSQjqM= 2024-04-12T04:02:20.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GLvYUrg9Wz6K3kXTRcMuQoSQjqM=', 'Content-Length': '0'} 2024-04-12T04:02:20.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cc8c912d118e38-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.534 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.534 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.534 DEBUG:boto:path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.534 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.534 DEBUG:boto:Method: GET 2024-04-12T04:02:20.534 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.534 DEBUG:boto:Data: 2024-04-12T04:02:20.534 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.534 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.535 DEBUG:boto:Params: {} 2024-04-12T04:02:20.535 DEBUG:boto:Token: None 2024-04-12T04:02:20.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c3dd95e504e90e-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.537 DEBUG:boto:b'dgbpxg-63obj581000falseobj582024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.537 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.537 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.538 DEBUG:boto:path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.538 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.538 DEBUG:boto:Method: GET 2024-04-12T04:02:20.538 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj58 2024-04-12T04:02:20.538 DEBUG:boto:Data: 2024-04-12T04:02:20.538 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.538 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.538 DEBUG:boto:Params: {} 2024-04-12T04:02:20.539 DEBUG:boto:Token: None 2024-04-12T04:02:20.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156b909fbc1a05df-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.546 DEBUG:boto:b'dgbpxg-63obj581000falseobj582024-04-12T03:58:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:02:20.546 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.547 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.547 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.547 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.547 DEBUG:boto:Method: GET 2024-04-12T04:02:20.547 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.547 DEBUG:boto:Data: 2024-04-12T04:02:20.547 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.547 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.547 DEBUG:boto:Params: {} 2024-04-12T04:02:20.548 DEBUG:boto:Token: None 2024-04-12T04:02:20.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.548 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4= 2024-04-12T04:02:20.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4=', 'Content-Length': '0'} 2024-04-12T04:02:20.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f70f7e24589153-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.549 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.550 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.550 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.550 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.550 DEBUG:boto:Method: GET 2024-04-12T04:02:20.550 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.550 DEBUG:boto:Data: 2024-04-12T04:02:20.550 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.550 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.550 DEBUG:boto:Params: {} 2024-04-12T04:02:20.550 DEBUG:boto:Token: None 2024-04-12T04:02:20.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.551 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4= 2024-04-12T04:02:20.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4=', 'Content-Length': '0'} 2024-04-12T04:02:20.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ddbd83ae13374cd-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.553 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.553 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.553 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.553 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.553 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.553 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.553 DEBUG:boto:Data: 2024-04-12T04:02:20.553 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.553 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.554 DEBUG:boto:Params: {} 2024-04-12T04:02:20.554 DEBUG:boto:Token: None 2024-04-12T04:02:20.554 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.554 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jGcqgiL1prxxnIFDkpkf4uuRtvI= 2024-04-12T04:02:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jGcqgiL1prxxnIFDkpkf4uuRtvI=', 'Content-Length': '0'} 2024-04-12T04:02:20.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066fabe7613f321fb-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.556 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.556 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.556 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.556 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.556 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.556 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.556 DEBUG:boto:Data: 2024-04-12T04:02:20.556 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.556 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.556 DEBUG:boto:Params: {} 2024-04-12T04:02:20.557 DEBUG:boto:Token: None 2024-04-12T04:02:20.557 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.557 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jGcqgiL1prxxnIFDkpkf4uuRtvI= 2024-04-12T04:02:20.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jGcqgiL1prxxnIFDkpkf4uuRtvI=', 'Content-Length': '0'} 2024-04-12T04:02:20.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ab5db2347c2772-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:02:20.559 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.559 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.559 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.559 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.559 DEBUG:boto:Method: GET 2024-04-12T04:02:20.559 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.559 DEBUG:boto:Data: 2024-04-12T04:02:20.560 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.560 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.560 DEBUG:boto:Params: {} 2024-04-12T04:02:20.560 DEBUG:boto:Token: None 2024-04-12T04:02:20.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4= 2024-04-12T04:02:20.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4=', 'Content-Length': '0'} 2024-04-12T04:02:20.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100e81431056f87c-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.562 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.562 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.562 DEBUG:boto:path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.563 DEBUG:boto:auth_path=/dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.563 DEBUG:boto:Method: GET 2024-04-12T04:02:20.563 DEBUG:boto:Path: /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.563 DEBUG:boto:Data: 2024-04-12T04:02:20.563 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.563 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.563 DEBUG:boto:Params: {} 2024-04-12T04:02:20.563 DEBUG:boto:Token: None 2024-04-12T04:02:20.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62?versionId=null 2024-04-12T04:02:20.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4= 2024-04-12T04:02:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Kf3rk/lEeMz0RnjbhvyBFdSCf4=', 'Content-Length': '0'} 2024-04-12T04:02:20.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eb4346876e8f070-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.566 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.566 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.566 DEBUG:boto:path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.566 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.566 DEBUG:boto:Method: GET 2024-04-12T04:02:20.566 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.566 DEBUG:boto:Data: 2024-04-12T04:02:20.566 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.567 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.567 DEBUG:boto:Params: {} 2024-04-12T04:02:20.567 DEBUG:boto:Token: None 2024-04-12T04:02:20.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.567 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff4b29c1423302c-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.569 DEBUG:boto:b'dgbpxg-63obj621000falseobj622024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.570 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.570 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.570 DEBUG:boto:path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.570 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.570 DEBUG:boto:Method: GET 2024-04-12T04:02:20.570 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.570 DEBUG:boto:Data: 2024-04-12T04:02:20.570 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.571 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.571 DEBUG:boto:Params: {} 2024-04-12T04:02:20.571 DEBUG:boto:Token: None 2024-04-12T04:02:20.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61a91109bed1e6e-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.575 DEBUG:boto:b'dgbpxg-63obj621000falseobj622024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.575 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.575 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.575 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.576 DEBUG:boto:Path: /dgbpxg-63/obj62 2024-04-12T04:02:20.576 DEBUG:boto:Data: 2024-04-12T04:02:20.576 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.576 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.576 DEBUG:boto:Params: {} 2024-04-12T04:02:20.576 DEBUG:boto:Token: None 2024-04-12T04:02:20.576 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62 2024-04-12T04:02:20.576 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:muu3/H1KIB5SQb00Yi/uq9a73ow= 2024-04-12T04:02:20.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:muu3/H1KIB5SQb00Yi/uq9a73ow=', 'Content-Length': '0'} 2024-04-12T04:02:20.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059765a836c88d6df-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.578 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.578 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.579 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.579 DEBUG:boto:Path: /dgbpxg-63/obj62 2024-04-12T04:02:20.579 DEBUG:boto:Data: 2024-04-12T04:02:20.579 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.579 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.579 DEBUG:boto:Params: {} 2024-04-12T04:02:20.579 DEBUG:boto:Token: None 2024-04-12T04:02:20.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62 2024-04-12T04:02:20.580 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:muu3/H1KIB5SQb00Yi/uq9a73ow= 2024-04-12T04:02:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:muu3/H1KIB5SQb00Yi/uq9a73ow=', 'Content-Length': '0'} 2024-04-12T04:02:20.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4018ef0b49c1f07-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:02:20.581 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.581 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.582 DEBUG:boto:Method: GET 2024-04-12T04:02:20.582 DEBUG:boto:Path: /dgbpxg-63/obj62 2024-04-12T04:02:20.582 DEBUG:boto:Data: 2024-04-12T04:02:20.582 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.582 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.582 DEBUG:boto:Params: {} 2024-04-12T04:02:20.582 DEBUG:boto:Token: None 2024-04-12T04:02:20.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62 2024-04-12T04:02:20.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ARwfZb7cZjaBrGfOFi2m89Jfhfo= 2024-04-12T04:02:20.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ARwfZb7cZjaBrGfOFi2m89Jfhfo=', 'Content-Length': '0'} 2024-04-12T04:02:20.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf44d950251e2637-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.584 DEBUG:boto:path=/dgbpxg-63/obj62 2024-04-12T04:02:20.585 DEBUG:boto:auth_path=/dgbpxg-63/obj62 2024-04-12T04:02:20.585 DEBUG:boto:Method: GET 2024-04-12T04:02:20.585 DEBUG:boto:Path: /dgbpxg-63/obj62 2024-04-12T04:02:20.585 DEBUG:boto:Data: 2024-04-12T04:02:20.585 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.585 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.585 DEBUG:boto:Params: {} 2024-04-12T04:02:20.585 DEBUG:boto:Token: None 2024-04-12T04:02:20.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj62 2024-04-12T04:02:20.586 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ARwfZb7cZjaBrGfOFi2m89Jfhfo= 2024-04-12T04:02:20.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ARwfZb7cZjaBrGfOFi2m89Jfhfo=', 'Content-Length': '0'} 2024-04-12T04:02:20.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c1b7c31f08143ff-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.588 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.588 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.588 DEBUG:boto:path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.588 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.588 DEBUG:boto:Method: GET 2024-04-12T04:02:20.588 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.588 DEBUG:boto:Data: 2024-04-12T04:02:20.588 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.588 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.588 DEBUG:boto:Params: {} 2024-04-12T04:02:20.589 DEBUG:boto:Token: None 2024-04-12T04:02:20.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.589 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd039ba9fbc77df-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.591 DEBUG:boto:b'dgbpxg-63obj621000falseobj622024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.591 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.591 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.591 DEBUG:boto:path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.591 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.592 DEBUG:boto:Method: GET 2024-04-12T04:02:20.592 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj62 2024-04-12T04:02:20.592 DEBUG:boto:Data: 2024-04-12T04:02:20.592 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.592 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.592 DEBUG:boto:Params: {} 2024-04-12T04:02:20.592 DEBUG:boto:Token: None 2024-04-12T04:02:20.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.592 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bcea9822a7bd9ee-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.596 DEBUG:boto:b'dgbpxg-63obj621000falseobj622024-04-12T03:58:40.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:02:20.597 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.597 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.597 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.597 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.597 DEBUG:boto:Method: GET 2024-04-12T04:02:20.597 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.597 DEBUG:boto:Data: 2024-04-12T04:02:20.597 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.598 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.598 DEBUG:boto:Params: {} 2024-04-12T04:02:20.598 DEBUG:boto:Token: None 2024-04-12T04:02:20.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.598 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI= 2024-04-12T04:02:20.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI=', 'Content-Length': '0'} 2024-04-12T04:02:20.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000152931793239d0fa-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.600 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.600 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.600 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.600 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.600 DEBUG:boto:Method: GET 2024-04-12T04:02:20.601 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.601 DEBUG:boto:Data: 2024-04-12T04:02:20.601 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.601 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.601 DEBUG:boto:Params: {} 2024-04-12T04:02:20.601 DEBUG:boto:Token: None 2024-04-12T04:02:20.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.601 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI= 2024-04-12T04:02:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI=', 'Content-Length': '0'} 2024-04-12T04:02:20.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d2e05fe1c606da7-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.647 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.647 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.648 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.648 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.648 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.648 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.648 DEBUG:boto:Data: 2024-04-12T04:02:20.648 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.648 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.648 DEBUG:boto:Params: {} 2024-04-12T04:02:20.648 DEBUG:boto:Token: None 2024-04-12T04:02:20.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.649 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3HqFHl6Tr5LXU55+Bmm3y9nzD6Y= 2024-04-12T04:02:20.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3HqFHl6Tr5LXU55+Bmm3y9nzD6Y=', 'Content-Length': '0'} 2024-04-12T04:02:20.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002fac7ccb0bbab51-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.651 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.651 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.651 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.651 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.651 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.651 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.651 DEBUG:boto:Data: 2024-04-12T04:02:20.651 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.651 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.652 DEBUG:boto:Params: {} 2024-04-12T04:02:20.652 DEBUG:boto:Token: None 2024-04-12T04:02:20.652 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.652 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3HqFHl6Tr5LXU55+Bmm3y9nzD6Y= 2024-04-12T04:02:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3HqFHl6Tr5LXU55+Bmm3y9nzD6Y=', 'Content-Length': '0'} 2024-04-12T04:02:20.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000317963ef6b3b27c6-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:02:20.654 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.654 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.654 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.654 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.654 DEBUG:boto:Method: GET 2024-04-12T04:02:20.654 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.654 DEBUG:boto:Data: 2024-04-12T04:02:20.654 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.655 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.655 DEBUG:boto:Params: {} 2024-04-12T04:02:20.655 DEBUG:boto:Token: None 2024-04-12T04:02:20.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.655 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI= 2024-04-12T04:02:20.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI=', 'Content-Length': '0'} 2024-04-12T04:02:20.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb715e0ed1e9fcdb-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.698 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.698 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.698 DEBUG:boto:path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.699 DEBUG:boto:auth_path=/dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.699 DEBUG:boto:Method: GET 2024-04-12T04:02:20.699 DEBUG:boto:Path: /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.699 DEBUG:boto:Data: 2024-04-12T04:02:20.699 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.699 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.699 DEBUG:boto:Params: {} 2024-04-12T04:02:20.699 DEBUG:boto:Token: None 2024-04-12T04:02:20.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68?versionId=null 2024-04-12T04:02:20.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI= 2024-04-12T04:02:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:l7eJURC/s/SNVnHzwiGQx1t1pBI=', 'Content-Length': '0'} 2024-04-12T04:02:20.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4464da29610652b-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.701 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.701 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.701 DEBUG:boto:path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.702 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.702 DEBUG:boto:Method: GET 2024-04-12T04:02:20.702 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.702 DEBUG:boto:Data: 2024-04-12T04:02:20.702 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.702 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.702 DEBUG:boto:Params: {} 2024-04-12T04:02:20.702 DEBUG:boto:Token: None 2024-04-12T04:02:20.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.703 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d5d7def14b4366-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.705 DEBUG:boto:b'dgbpxg-63obj681000falseobj682024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.705 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.705 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.705 DEBUG:boto:path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.705 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.706 DEBUG:boto:Method: GET 2024-04-12T04:02:20.706 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.706 DEBUG:boto:Data: 2024-04-12T04:02:20.706 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.706 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.706 DEBUG:boto:Params: {} 2024-04-12T04:02:20.706 DEBUG:boto:Token: None 2024-04-12T04:02:20.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.706 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095070837ad3f6c98-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.710 DEBUG:boto:b'dgbpxg-63obj681000falseobj682024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.711 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.711 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.711 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.711 DEBUG:boto:Path: /dgbpxg-63/obj68 2024-04-12T04:02:20.711 DEBUG:boto:Data: 2024-04-12T04:02:20.711 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.711 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.711 DEBUG:boto:Params: {} 2024-04-12T04:02:20.711 DEBUG:boto:Token: None 2024-04-12T04:02:20.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68 2024-04-12T04:02:20.712 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9TdDtG8vxJfFdV+sejNd8X8yX70= 2024-04-12T04:02:20.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9TdDtG8vxJfFdV+sejNd8X8yX70=', 'Content-Length': '0'} 2024-04-12T04:02:20.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb496ef317042eb8-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.714 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.714 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.714 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.714 DEBUG:boto:Path: /dgbpxg-63/obj68 2024-04-12T04:02:20.714 DEBUG:boto:Data: 2024-04-12T04:02:20.714 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.714 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.714 DEBUG:boto:Params: {} 2024-04-12T04:02:20.714 DEBUG:boto:Token: None 2024-04-12T04:02:20.715 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68 2024-04-12T04:02:20.715 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9TdDtG8vxJfFdV+sejNd8X8yX70= 2024-04-12T04:02:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9TdDtG8vxJfFdV+sejNd8X8yX70=', 'Content-Length': '0'} 2024-04-12T04:02:20.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb5a6d24ce00fd98-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:02:20.717 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.717 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.717 DEBUG:boto:Method: GET 2024-04-12T04:02:20.717 DEBUG:boto:Path: /dgbpxg-63/obj68 2024-04-12T04:02:20.717 DEBUG:boto:Data: 2024-04-12T04:02:20.717 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.717 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.717 DEBUG:boto:Params: {} 2024-04-12T04:02:20.717 DEBUG:boto:Token: None 2024-04-12T04:02:20.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68 2024-04-12T04:02:20.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hp2o6s3zNvYRnn5jinl/GMGfEAc= 2024-04-12T04:02:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hp2o6s3zNvYRnn5jinl/GMGfEAc=', 'Content-Length': '0'} 2024-04-12T04:02:20.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c607f04350bf1d-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.719 DEBUG:boto:path=/dgbpxg-63/obj68 2024-04-12T04:02:20.719 DEBUG:boto:auth_path=/dgbpxg-63/obj68 2024-04-12T04:02:20.720 DEBUG:boto:Method: GET 2024-04-12T04:02:20.720 DEBUG:boto:Path: /dgbpxg-63/obj68 2024-04-12T04:02:20.720 DEBUG:boto:Data: 2024-04-12T04:02:20.720 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.720 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.720 DEBUG:boto:Params: {} 2024-04-12T04:02:20.720 DEBUG:boto:Token: None 2024-04-12T04:02:20.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj68 2024-04-12T04:02:20.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hp2o6s3zNvYRnn5jinl/GMGfEAc= 2024-04-12T04:02:20.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hp2o6s3zNvYRnn5jinl/GMGfEAc=', 'Content-Length': '0'} 2024-04-12T04:02:20.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026709ec53dcd5eab-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.722 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.722 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.723 DEBUG:boto:path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.723 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.723 DEBUG:boto:Method: GET 2024-04-12T04:02:20.723 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.723 DEBUG:boto:Data: 2024-04-12T04:02:20.723 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.723 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.723 DEBUG:boto:Params: {} 2024-04-12T04:02:20.723 DEBUG:boto:Token: None 2024-04-12T04:02:20.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000838a6445d7525bc-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.726 DEBUG:boto:b'dgbpxg-63obj681000falseobj682024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.726 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.727 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.727 DEBUG:boto:path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.727 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.727 DEBUG:boto:Method: GET 2024-04-12T04:02:20.727 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj68 2024-04-12T04:02:20.727 DEBUG:boto:Data: 2024-04-12T04:02:20.727 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.727 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.727 DEBUG:boto:Params: {} 2024-04-12T04:02:20.727 DEBUG:boto:Token: None 2024-04-12T04:02:20.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.728 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f13ac9fad073ec8-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.732 DEBUG:boto:b'dgbpxg-63obj681000falseobj682024-04-12T03:58:39.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:02:20.733 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.733 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.733 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.733 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.733 DEBUG:boto:Method: GET 2024-04-12T04:02:20.733 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.733 DEBUG:boto:Data: 2024-04-12T04:02:20.733 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.733 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.733 DEBUG:boto:Params: {} 2024-04-12T04:02:20.734 DEBUG:boto:Token: None 2024-04-12T04:02:20.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.734 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4= 2024-04-12T04:02:20.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4=', 'Content-Length': '0'} 2024-04-12T04:02:20.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424cc311f203a228-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.736 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.736 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.736 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.736 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.736 DEBUG:boto:Method: GET 2024-04-12T04:02:20.736 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.737 DEBUG:boto:Data: 2024-04-12T04:02:20.737 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.737 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.737 DEBUG:boto:Params: {} 2024-04-12T04:02:20.737 DEBUG:boto:Token: None 2024-04-12T04:02:20.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.737 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4= 2024-04-12T04:02:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4=', 'Content-Length': '0'} 2024-04-12T04:02:20.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dbf42dbf8814e59-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.782 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.783 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.783 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.783 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.783 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.783 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.783 DEBUG:boto:Data: 2024-04-12T04:02:20.783 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.783 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.783 DEBUG:boto:Params: {} 2024-04-12T04:02:20.784 DEBUG:boto:Token: None 2024-04-12T04:02:20.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/h2NhKDf358XnvOqc8kFs0MZ4Gk= 2024-04-12T04:02:20.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/h2NhKDf358XnvOqc8kFs0MZ4Gk=', 'Content-Length': '0'} 2024-04-12T04:02:20.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc60b71271ad5f6-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.786 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.786 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.786 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.786 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.786 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.786 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.786 DEBUG:boto:Data: 2024-04-12T04:02:20.787 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.787 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.787 DEBUG:boto:Params: {} 2024-04-12T04:02:20.787 DEBUG:boto:Token: None 2024-04-12T04:02:20.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.787 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/h2NhKDf358XnvOqc8kFs0MZ4Gk= 2024-04-12T04:02:20.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/h2NhKDf358XnvOqc8kFs0MZ4Gk=', 'Content-Length': '0'} 2024-04-12T04:02:20.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b030cd03e7177a9-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:02:20.789 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.789 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.789 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.789 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.789 DEBUG:boto:Method: GET 2024-04-12T04:02:20.789 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.789 DEBUG:boto:Data: 2024-04-12T04:02:20.790 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.790 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.790 DEBUG:boto:Params: {} 2024-04-12T04:02:20.790 DEBUG:boto:Token: None 2024-04-12T04:02:20.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.790 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4= 2024-04-12T04:02:20.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4=', 'Content-Length': '0'} 2024-04-12T04:02:20.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c3647988e5de93-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.792 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.792 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.792 DEBUG:boto:path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.792 DEBUG:boto:auth_path=/dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.792 DEBUG:boto:Method: GET 2024-04-12T04:02:20.792 DEBUG:boto:Path: /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.793 DEBUG:boto:Data: 2024-04-12T04:02:20.793 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.793 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.793 DEBUG:boto:Params: {} 2024-04-12T04:02:20.793 DEBUG:boto:Token: None 2024-04-12T04:02:20.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69?versionId=null 2024-04-12T04:02:20.793 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4= 2024-04-12T04:02:20.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fibVN9XR3ZxQ8G8fxxSUXMLtLt4=', 'Content-Length': '0'} 2024-04-12T04:02:20.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26e5f90eed14511-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.795 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.795 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.796 DEBUG:boto:path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.796 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.796 DEBUG:boto:Method: GET 2024-04-12T04:02:20.796 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.796 DEBUG:boto:Data: 2024-04-12T04:02:20.796 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.796 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.796 DEBUG:boto:Params: {} 2024-04-12T04:02:20.796 DEBUG:boto:Token: None 2024-04-12T04:02:20.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.797 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ddf327bd95a6c22-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.799 DEBUG:boto:b'dgbpxg-63obj691000falseobj692024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.799 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.799 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.799 DEBUG:boto:path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.799 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.800 DEBUG:boto:Method: GET 2024-04-12T04:02:20.800 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.800 DEBUG:boto:Data: 2024-04-12T04:02:20.800 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.800 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.800 DEBUG:boto:Params: {} 2024-04-12T04:02:20.800 DEBUG:boto:Token: None 2024-04-12T04:02:20.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047438c62aa5e4baa-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.804 DEBUG:boto:b'dgbpxg-63obj691000falseobj692024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.804 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.805 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.805 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.805 DEBUG:boto:Path: /dgbpxg-63/obj69 2024-04-12T04:02:20.805 DEBUG:boto:Data: 2024-04-12T04:02:20.805 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.805 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.805 DEBUG:boto:Params: {} 2024-04-12T04:02:20.805 DEBUG:boto:Token: None 2024-04-12T04:02:20.805 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69 2024-04-12T04:02:20.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dIu4rCHY6hb2pY91uA0z4vTgjEE= 2024-04-12T04:02:20.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dIu4rCHY6hb2pY91uA0z4vTgjEE=', 'Content-Length': '0'} 2024-04-12T04:02:20.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe530b2c02c7194-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.807 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.807 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.807 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.807 DEBUG:boto:Path: /dgbpxg-63/obj69 2024-04-12T04:02:20.808 DEBUG:boto:Data: 2024-04-12T04:02:20.808 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.808 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.808 DEBUG:boto:Params: {} 2024-04-12T04:02:20.808 DEBUG:boto:Token: None 2024-04-12T04:02:20.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69 2024-04-12T04:02:20.808 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dIu4rCHY6hb2pY91uA0z4vTgjEE= 2024-04-12T04:02:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dIu4rCHY6hb2pY91uA0z4vTgjEE=', 'Content-Length': '0'} 2024-04-12T04:02:20.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000742976e8cbd759bf-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:02:20.810 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.810 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.810 DEBUG:boto:Method: GET 2024-04-12T04:02:20.810 DEBUG:boto:Path: /dgbpxg-63/obj69 2024-04-12T04:02:20.810 DEBUG:boto:Data: 2024-04-12T04:02:20.811 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.811 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.811 DEBUG:boto:Params: {} 2024-04-12T04:02:20.811 DEBUG:boto:Token: None 2024-04-12T04:02:20.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69 2024-04-12T04:02:20.811 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pj120+soZwdGvywFd18OZSZu/7E= 2024-04-12T04:02:20.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pj120+soZwdGvywFd18OZSZu/7E=', 'Content-Length': '0'} 2024-04-12T04:02:20.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d108df1df6e4a9a-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.813 DEBUG:boto:path=/dgbpxg-63/obj69 2024-04-12T04:02:20.813 DEBUG:boto:auth_path=/dgbpxg-63/obj69 2024-04-12T04:02:20.813 DEBUG:boto:Method: GET 2024-04-12T04:02:20.813 DEBUG:boto:Path: /dgbpxg-63/obj69 2024-04-12T04:02:20.813 DEBUG:boto:Data: 2024-04-12T04:02:20.813 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.813 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.814 DEBUG:boto:Params: {} 2024-04-12T04:02:20.814 DEBUG:boto:Token: None 2024-04-12T04:02:20.814 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj69 2024-04-12T04:02:20.814 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Pj120+soZwdGvywFd18OZSZu/7E= 2024-04-12T04:02:20.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Pj120+soZwdGvywFd18OZSZu/7E=', 'Content-Length': '0'} 2024-04-12T04:02:20.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000759242bcd46fc279-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.862 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.863 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.863 DEBUG:boto:path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.863 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.863 DEBUG:boto:Method: GET 2024-04-12T04:02:20.863 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.863 DEBUG:boto:Data: 2024-04-12T04:02:20.863 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.863 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.864 DEBUG:boto:Params: {} 2024-04-12T04:02:20.864 DEBUG:boto:Token: None 2024-04-12T04:02:20.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.864 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b58cd6e38e18a2-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.867 DEBUG:boto:b'dgbpxg-63obj691000falseobj692024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.867 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.867 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.867 DEBUG:boto:path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.867 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.867 DEBUG:boto:Method: GET 2024-04-12T04:02:20.867 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj69 2024-04-12T04:02:20.868 DEBUG:boto:Data: 2024-04-12T04:02:20.868 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.868 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.868 DEBUG:boto:Params: {} 2024-04-12T04:02:20.868 DEBUG:boto:Token: None 2024-04-12T04:02:20.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1671a6e4a5796b9-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.872 DEBUG:boto:b'dgbpxg-63obj691000falseobj692024-04-12T03:58:39.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:02:20.873 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.873 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.873 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.873 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.873 DEBUG:boto:Method: GET 2024-04-12T04:02:20.873 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.873 DEBUG:boto:Data: 2024-04-12T04:02:20.873 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.873 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.873 DEBUG:boto:Params: {} 2024-04-12T04:02:20.874 DEBUG:boto:Token: None 2024-04-12T04:02:20.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.874 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY= 2024-04-12T04:02:20.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY=', 'Content-Length': '0'} 2024-04-12T04:02:20.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cddfbc4980096d68-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.918 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.918 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.918 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.919 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.919 DEBUG:boto:Method: GET 2024-04-12T04:02:20.919 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.919 DEBUG:boto:Data: 2024-04-12T04:02:20.919 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.919 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.919 DEBUG:boto:Params: {} 2024-04-12T04:02:20.919 DEBUG:boto:Token: None 2024-04-12T04:02:20.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.920 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY= 2024-04-12T04:02:20.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY=', 'Content-Length': '0'} 2024-04-12T04:02:20.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c37f14b48a5c0eb-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.922 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.922 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.922 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.922 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.922 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.922 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.923 DEBUG:boto:Data: 2024-04-12T04:02:20.923 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.923 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.923 DEBUG:boto:Params: {} 2024-04-12T04:02:20.923 DEBUG:boto:Token: None 2024-04-12T04:02:20.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x/3KvGfca+wnhdp5L5Va9077dX4= 2024-04-12T04:02:20.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x/3KvGfca+wnhdp5L5Va9077dX4=', 'Content-Length': '0'} 2024-04-12T04:02:20.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2215c72702d682-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.925 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.925 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.925 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.925 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.926 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.926 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.926 DEBUG:boto:Data: 2024-04-12T04:02:20.926 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.926 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.926 DEBUG:boto:Params: {} 2024-04-12T04:02:20.926 DEBUG:boto:Token: None 2024-04-12T04:02:20.926 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.927 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:x/3KvGfca+wnhdp5L5Va9077dX4= 2024-04-12T04:02:20.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:x/3KvGfca+wnhdp5L5Va9077dX4=', 'Content-Length': '0'} 2024-04-12T04:02:20.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f26f7f6322794d4-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:02:20.928 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.929 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.929 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.929 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.929 DEBUG:boto:Method: GET 2024-04-12T04:02:20.929 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.929 DEBUG:boto:Data: 2024-04-12T04:02:20.929 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.929 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.929 DEBUG:boto:Params: {} 2024-04-12T04:02:20.929 DEBUG:boto:Token: None 2024-04-12T04:02:20.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.930 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY= 2024-04-12T04:02:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY=', 'Content-Length': '0'} 2024-04-12T04:02:20.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c8a3300d820906-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.932 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.932 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.932 DEBUG:boto:path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.932 DEBUG:boto:auth_path=/dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.932 DEBUG:boto:Method: GET 2024-04-12T04:02:20.932 DEBUG:boto:Path: /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.932 DEBUG:boto:Data: 2024-04-12T04:02:20.932 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.933 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.933 DEBUG:boto:Params: {} 2024-04-12T04:02:20.933 DEBUG:boto:Token: None 2024-04-12T04:02:20.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74?versionId=null 2024-04-12T04:02:20.933 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY= 2024-04-12T04:02:20.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6ugQ60/HQG1BEGhUPE3BDSSPlEY=', 'Content-Length': '0'} 2024-04-12T04:02:20.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008beab7cc0de8e007-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.935 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.935 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.935 DEBUG:boto:path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.935 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.935 DEBUG:boto:Method: GET 2024-04-12T04:02:20.935 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.935 DEBUG:boto:Data: 2024-04-12T04:02:20.936 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.936 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.936 DEBUG:boto:Params: {} 2024-04-12T04:02:20.936 DEBUG:boto:Token: None 2024-04-12T04:02:20.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.936 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068925cf31fe0eabf-006618b24c-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.938 DEBUG:boto:b'dgbpxg-63obj741000falseobj742024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.939 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.939 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.939 DEBUG:boto:path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.939 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.939 DEBUG:boto:Method: GET 2024-04-12T04:02:20.939 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.939 DEBUG:boto:Data: 2024-04-12T04:02:20.939 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.940 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.940 DEBUG:boto:Params: {} 2024-04-12T04:02:20.940 DEBUG:boto:Token: None 2024-04-12T04:02:20.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:20.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4d00b0cba03799-006618b24c-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.944 DEBUG:boto:b'dgbpxg-63obj741000falseobj742024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:20.944 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.944 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.944 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.944 DEBUG:boto:Path: /dgbpxg-63/obj74 2024-04-12T04:02:20.944 DEBUG:boto:Data: 2024-04-12T04:02:20.944 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.945 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.945 DEBUG:boto:Params: {} 2024-04-12T04:02:20.945 DEBUG:boto:Token: None 2024-04-12T04:02:20.945 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74 2024-04-12T04:02:20.945 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9cVpnKgPqFVjaMrWtPYmxXR6bl4= 2024-04-12T04:02:20.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9cVpnKgPqFVjaMrWtPYmxXR6bl4=', 'Content-Length': '0'} 2024-04-12T04:02:20.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f0200695cdf733-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.947 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.947 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.947 DEBUG:boto:Method: HEAD 2024-04-12T04:02:20.947 DEBUG:boto:Path: /dgbpxg-63/obj74 2024-04-12T04:02:20.947 DEBUG:boto:Data: 2024-04-12T04:02:20.947 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.947 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.948 DEBUG:boto:Params: {} 2024-04-12T04:02:20.948 DEBUG:boto:Token: None 2024-04-12T04:02:20.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74 2024-04-12T04:02:20.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9cVpnKgPqFVjaMrWtPYmxXR6bl4= 2024-04-12T04:02:20.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9cVpnKgPqFVjaMrWtPYmxXR6bl4=', 'Content-Length': '0'} 2024-04-12T04:02:20.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595ca751be68b5f6-006618b24c-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:02:20.950 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.950 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.950 DEBUG:boto:Method: GET 2024-04-12T04:02:20.950 DEBUG:boto:Path: /dgbpxg-63/obj74 2024-04-12T04:02:20.950 DEBUG:boto:Data: 2024-04-12T04:02:20.950 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.951 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.951 DEBUG:boto:Params: {} 2024-04-12T04:02:20.951 DEBUG:boto:Token: None 2024-04-12T04:02:20.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74 2024-04-12T04:02:20.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZW5FBD4IGbRLGn4pAcnd5ZOcwk= 2024-04-12T04:02:20.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZW5FBD4IGbRLGn4pAcnd5ZOcwk=', 'Content-Length': '0'} 2024-04-12T04:02:20.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c1af4010adc4dc-006618b24c-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.994 DEBUG:boto:path=/dgbpxg-63/obj74 2024-04-12T04:02:20.994 DEBUG:boto:auth_path=/dgbpxg-63/obj74 2024-04-12T04:02:20.994 DEBUG:boto:Method: GET 2024-04-12T04:02:20.995 DEBUG:boto:Path: /dgbpxg-63/obj74 2024-04-12T04:02:20.995 DEBUG:boto:Data: 2024-04-12T04:02:20.995 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:20.995 DEBUG:boto:Port: 8000 2024-04-12T04:02:20.995 DEBUG:boto:Params: {} 2024-04-12T04:02:20.995 DEBUG:boto:Token: None 2024-04-12T04:02:20.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/obj74 2024-04-12T04:02:20.995 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:IZW5FBD4IGbRLGn4pAcnd5ZOcwk= 2024-04-12T04:02:20.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IZW5FBD4IGbRLGn4pAcnd5ZOcwk=', 'Content-Length': '0'} 2024-04-12T04:02:20.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387f86f716ee25a5-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:20.997 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:20.997 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:20.998 DEBUG:boto:path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.998 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.998 DEBUG:boto:Method: GET 2024-04-12T04:02:20.998 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj74 2024-04-12T04:02:20.998 DEBUG:boto:Data: 2024-04-12T04:02:20.998 DEBUG:boto:Headers: {} 2024-04-12T04:02:20.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:20.998 DEBUG:boto:Port: 8001 2024-04-12T04:02:20.998 DEBUG:boto:Params: {} 2024-04-12T04:02:20.998 DEBUG:boto:Token: None 2024-04-12T04:02:20.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:20 GMT /dgbpxg-63/ 2024-04-12T04:02:20.999 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64= 2024-04-12T04:02:20.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yrFvEBOk98JUaN8rgoWlbl65M64=', 'Content-Length': '0'} 2024-04-12T04:02:21.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bfaa41f3871a915-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.001 DEBUG:boto:b'dgbpxg-63obj741000falseobj742024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.002 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.002 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.002 DEBUG:boto:path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:21.002 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj74 2024-04-12T04:02:21.002 DEBUG:boto:Method: GET 2024-04-12T04:02:21.002 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj74 2024-04-12T04:02:21.002 DEBUG:boto:Data: 2024-04-12T04:02:21.002 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.003 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.003 DEBUG:boto:Params: {} 2024-04-12T04:02:21.003 DEBUG:boto:Token: None 2024-04-12T04:02:21.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.003 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b796fc6d589a4e-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.006 DEBUG:boto:b'dgbpxg-63obj741000falseobj742024-04-12T03:58:40.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:02:21.007 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.007 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.007 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.007 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.007 DEBUG:boto:Method: GET 2024-04-12T04:02:21.007 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.007 DEBUG:boto:Data: 2024-04-12T04:02:21.007 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.008 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.008 DEBUG:boto:Params: {} 2024-04-12T04:02:21.008 DEBUG:boto:Token: None 2024-04-12T04:02:21.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg= 2024-04-12T04:02:21.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg=', 'Content-Length': '0'} 2024-04-12T04:02:21.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c71802f6d7d979-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.010 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.010 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.010 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.010 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.010 DEBUG:boto:Method: GET 2024-04-12T04:02:21.010 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.010 DEBUG:boto:Data: 2024-04-12T04:02:21.010 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.011 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.011 DEBUG:boto:Params: {} 2024-04-12T04:02:21.011 DEBUG:boto:Token: None 2024-04-12T04:02:21.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.011 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg= 2024-04-12T04:02:21.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg=', 'Content-Length': '0'} 2024-04-12T04:02:21.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fbacad521e1cba-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.013 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.013 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.013 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.013 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.013 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.013 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.013 DEBUG:boto:Data: 2024-04-12T04:02:21.014 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.014 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.014 DEBUG:boto:Params: {} 2024-04-12T04:02:21.014 DEBUG:boto:Token: None 2024-04-12T04:02:21.014 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.014 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O3WE5FIx8WGId8QHFYcYePL9X3Q= 2024-04-12T04:02:21.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O3WE5FIx8WGId8QHFYcYePL9X3Q=', 'Content-Length': '0'} 2024-04-12T04:02:21.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062abbdc8e7c422e8-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.016 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.016 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.016 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.016 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.017 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.017 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.017 DEBUG:boto:Data: 2024-04-12T04:02:21.017 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.017 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.017 DEBUG:boto:Params: {} 2024-04-12T04:02:21.017 DEBUG:boto:Token: None 2024-04-12T04:02:21.017 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.017 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O3WE5FIx8WGId8QHFYcYePL9X3Q= 2024-04-12T04:02:21.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O3WE5FIx8WGId8QHFYcYePL9X3Q=', 'Content-Length': '0'} 2024-04-12T04:02:21.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df42338169819d6d-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:02:21.019 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.019 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.020 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.020 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.020 DEBUG:boto:Method: GET 2024-04-12T04:02:21.020 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.020 DEBUG:boto:Data: 2024-04-12T04:02:21.020 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.020 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.020 DEBUG:boto:Params: {} 2024-04-12T04:02:21.020 DEBUG:boto:Token: None 2024-04-12T04:02:21.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.021 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg= 2024-04-12T04:02:21.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg=', 'Content-Length': '0'} 2024-04-12T04:02:21.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000962817d0ebc93afa-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.023 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.023 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.023 DEBUG:boto:path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.023 DEBUG:boto:auth_path=/dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.023 DEBUG:boto:Method: GET 2024-04-12T04:02:21.023 DEBUG:boto:Path: /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.023 DEBUG:boto:Data: 2024-04-12T04:02:21.023 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.024 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.024 DEBUG:boto:Params: {} 2024-04-12T04:02:21.024 DEBUG:boto:Token: None 2024-04-12T04:02:21.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76?versionId=null 2024-04-12T04:02:21.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg= 2024-04-12T04:02:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YbbBPrWpPCjOTxrmR2muUqilPFg=', 'Content-Length': '0'} 2024-04-12T04:02:21.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1c6b0074fa9c57-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.026 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.026 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.026 DEBUG:boto:path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.026 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.027 DEBUG:boto:Method: GET 2024-04-12T04:02:21.027 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.027 DEBUG:boto:Data: 2024-04-12T04:02:21.027 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.027 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.027 DEBUG:boto:Params: {} 2024-04-12T04:02:21.027 DEBUG:boto:Token: None 2024-04-12T04:02:21.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007010d5ee02585bfd-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.029 DEBUG:boto:b'dgbpxg-63obj761000falseobj762024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.030 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.030 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.030 DEBUG:boto:path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.030 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.030 DEBUG:boto:Method: GET 2024-04-12T04:02:21.030 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.030 DEBUG:boto:Data: 2024-04-12T04:02:21.031 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.031 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.031 DEBUG:boto:Params: {} 2024-04-12T04:02:21.031 DEBUG:boto:Token: None 2024-04-12T04:02:21.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d66a62606b1b3dd1-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.035 DEBUG:boto:b'dgbpxg-63obj761000falseobj762024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.035 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.035 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.036 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.036 DEBUG:boto:Path: /dgbpxg-63/obj76 2024-04-12T04:02:21.036 DEBUG:boto:Data: 2024-04-12T04:02:21.036 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.036 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.036 DEBUG:boto:Params: {} 2024-04-12T04:02:21.036 DEBUG:boto:Token: None 2024-04-12T04:02:21.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76 2024-04-12T04:02:21.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lJyvsyMlSRc2jhC4wPKQT8otjks= 2024-04-12T04:02:21.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lJyvsyMlSRc2jhC4wPKQT8otjks=', 'Content-Length': '0'} 2024-04-12T04:02:21.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb084c51cf77d955-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.038 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.038 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.038 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.038 DEBUG:boto:Path: /dgbpxg-63/obj76 2024-04-12T04:02:21.038 DEBUG:boto:Data: 2024-04-12T04:02:21.038 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.039 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.039 DEBUG:boto:Params: {} 2024-04-12T04:02:21.039 DEBUG:boto:Token: None 2024-04-12T04:02:21.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76 2024-04-12T04:02:21.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lJyvsyMlSRc2jhC4wPKQT8otjks= 2024-04-12T04:02:21.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lJyvsyMlSRc2jhC4wPKQT8otjks=', 'Content-Length': '0'} 2024-04-12T04:02:21.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fef8e5e111a47b2-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:02:21.041 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.041 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.041 DEBUG:boto:Method: GET 2024-04-12T04:02:21.041 DEBUG:boto:Path: /dgbpxg-63/obj76 2024-04-12T04:02:21.041 DEBUG:boto:Data: 2024-04-12T04:02:21.041 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.042 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.042 DEBUG:boto:Params: {} 2024-04-12T04:02:21.042 DEBUG:boto:Token: None 2024-04-12T04:02:21.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76 2024-04-12T04:02:21.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yz5qE3xKotQoWg0ZIPq/Glv8b9Q= 2024-04-12T04:02:21.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yz5qE3xKotQoWg0ZIPq/Glv8b9Q=', 'Content-Length': '0'} 2024-04-12T04:02:21.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d939de6769a3e0-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.044 DEBUG:boto:path=/dgbpxg-63/obj76 2024-04-12T04:02:21.044 DEBUG:boto:auth_path=/dgbpxg-63/obj76 2024-04-12T04:02:21.044 DEBUG:boto:Method: GET 2024-04-12T04:02:21.044 DEBUG:boto:Path: /dgbpxg-63/obj76 2024-04-12T04:02:21.044 DEBUG:boto:Data: 2024-04-12T04:02:21.044 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.044 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.044 DEBUG:boto:Params: {} 2024-04-12T04:02:21.045 DEBUG:boto:Token: None 2024-04-12T04:02:21.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj76 2024-04-12T04:02:21.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yz5qE3xKotQoWg0ZIPq/Glv8b9Q= 2024-04-12T04:02:21.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yz5qE3xKotQoWg0ZIPq/Glv8b9Q=', 'Content-Length': '0'} 2024-04-12T04:02:21.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da83733fedfc8120-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.090 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.091 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.091 DEBUG:boto:path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.091 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.091 DEBUG:boto:Method: GET 2024-04-12T04:02:21.091 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.091 DEBUG:boto:Data: 2024-04-12T04:02:21.091 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.091 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.091 DEBUG:boto:Params: {} 2024-04-12T04:02:21.092 DEBUG:boto:Token: None 2024-04-12T04:02:21.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043da968cf83af7f4-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.094 DEBUG:boto:b'dgbpxg-63obj761000falseobj762024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.095 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.095 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.095 DEBUG:boto:path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.095 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.095 DEBUG:boto:Method: GET 2024-04-12T04:02:21.095 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj76 2024-04-12T04:02:21.095 DEBUG:boto:Data: 2024-04-12T04:02:21.095 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.095 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.095 DEBUG:boto:Params: {} 2024-04-12T04:02:21.096 DEBUG:boto:Token: None 2024-04-12T04:02:21.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db825747265a42d1-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.100 DEBUG:boto:b'dgbpxg-63obj761000falseobj762024-04-12T03:58:39.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:02:21.100 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.100 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.100 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.101 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.101 DEBUG:boto:Method: GET 2024-04-12T04:02:21.101 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.101 DEBUG:boto:Data: 2024-04-12T04:02:21.101 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.101 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.101 DEBUG:boto:Params: {} 2024-04-12T04:02:21.101 DEBUG:boto:Token: None 2024-04-12T04:02:21.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo= 2024-04-12T04:02:21.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo=', 'Content-Length': '0'} 2024-04-12T04:02:21.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d3d4c9571e9556a-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.104 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.104 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.104 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.104 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.104 DEBUG:boto:Method: GET 2024-04-12T04:02:21.104 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.104 DEBUG:boto:Data: 2024-04-12T04:02:21.104 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.104 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.104 DEBUG:boto:Params: {} 2024-04-12T04:02:21.105 DEBUG:boto:Token: None 2024-04-12T04:02:21.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo= 2024-04-12T04:02:21.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo=', 'Content-Length': '0'} 2024-04-12T04:02:21.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044515dbae27cd414-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.107 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.107 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.107 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.107 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.107 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.107 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.107 DEBUG:boto:Data: 2024-04-12T04:02:21.107 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.107 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.108 DEBUG:boto:Params: {} 2024-04-12T04:02:21.108 DEBUG:boto:Token: None 2024-04-12T04:02:21.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q2mBheJ2QeFNm35WfVQ8QYpOVJA= 2024-04-12T04:02:21.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2mBheJ2QeFNm35WfVQ8QYpOVJA=', 'Content-Length': '0'} 2024-04-12T04:02:21.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003fcd9e08a952222-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.110 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.110 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.110 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.110 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.110 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.110 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.110 DEBUG:boto:Data: 2024-04-12T04:02:21.110 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.110 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.111 DEBUG:boto:Params: {} 2024-04-12T04:02:21.111 DEBUG:boto:Token: None 2024-04-12T04:02:21.111 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.111 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:q2mBheJ2QeFNm35WfVQ8QYpOVJA= 2024-04-12T04:02:21.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2mBheJ2QeFNm35WfVQ8QYpOVJA=', 'Content-Length': '0'} 2024-04-12T04:02:21.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086991ae9410957f4-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:02:21.113 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.113 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.113 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.113 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.113 DEBUG:boto:Method: GET 2024-04-12T04:02:21.114 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.114 DEBUG:boto:Data: 2024-04-12T04:02:21.114 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.114 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.114 DEBUG:boto:Params: {} 2024-04-12T04:02:21.114 DEBUG:boto:Token: None 2024-04-12T04:02:21.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo= 2024-04-12T04:02:21.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo=', 'Content-Length': '0'} 2024-04-12T04:02:21.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d67f88f1476782-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.116 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.116 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.116 DEBUG:boto:path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.116 DEBUG:boto:auth_path=/dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.116 DEBUG:boto:Method: GET 2024-04-12T04:02:21.117 DEBUG:boto:Path: /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.117 DEBUG:boto:Data: 2024-04-12T04:02:21.117 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.117 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.117 DEBUG:boto:Params: {} 2024-04-12T04:02:21.117 DEBUG:boto:Token: None 2024-04-12T04:02:21.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85?versionId=null 2024-04-12T04:02:21.117 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo= 2024-04-12T04:02:21.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XpKhJJdO7WPe2EWGXjMUom4flBo=', 'Content-Length': '0'} 2024-04-12T04:02:21.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f1bd66f3d04749-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.163 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.163 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.163 DEBUG:boto:path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.163 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.163 DEBUG:boto:Method: GET 2024-04-12T04:02:21.163 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.164 DEBUG:boto:Data: 2024-04-12T04:02:21.164 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.164 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.164 DEBUG:boto:Params: {} 2024-04-12T04:02:21.164 DEBUG:boto:Token: None 2024-04-12T04:02:21.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.165 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ea05e6caac5af5-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.167 DEBUG:boto:b'dgbpxg-63obj851000falseobj852024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.168 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.168 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.168 DEBUG:boto:path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.168 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.168 DEBUG:boto:Method: GET 2024-04-12T04:02:21.168 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.168 DEBUG:boto:Data: 2024-04-12T04:02:21.168 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.169 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.169 DEBUG:boto:Params: {} 2024-04-12T04:02:21.169 DEBUG:boto:Token: None 2024-04-12T04:02:21.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.169 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d817b7089987b019-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.173 DEBUG:boto:b'dgbpxg-63obj851000falseobj852024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.173 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.173 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.173 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.174 DEBUG:boto:Path: /dgbpxg-63/obj85 2024-04-12T04:02:21.174 DEBUG:boto:Data: 2024-04-12T04:02:21.174 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.174 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.174 DEBUG:boto:Params: {} 2024-04-12T04:02:21.174 DEBUG:boto:Token: None 2024-04-12T04:02:21.174 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85 2024-04-12T04:02:21.174 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6Z2RUyVAZpLXGoEGPu8bjp3bVxs= 2024-04-12T04:02:21.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6Z2RUyVAZpLXGoEGPu8bjp3bVxs=', 'Content-Length': '0'} 2024-04-12T04:02:21.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f042e7ba2feef258-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.176 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.177 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.177 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.177 DEBUG:boto:Path: /dgbpxg-63/obj85 2024-04-12T04:02:21.177 DEBUG:boto:Data: 2024-04-12T04:02:21.177 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.177 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.177 DEBUG:boto:Params: {} 2024-04-12T04:02:21.178 DEBUG:boto:Token: None 2024-04-12T04:02:21.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85 2024-04-12T04:02:21.178 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6Z2RUyVAZpLXGoEGPu8bjp3bVxs= 2024-04-12T04:02:21.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6Z2RUyVAZpLXGoEGPu8bjp3bVxs=', 'Content-Length': '0'} 2024-04-12T04:02:21.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65e65d1e5665ae5-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:02:21.180 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.180 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.180 DEBUG:boto:Method: GET 2024-04-12T04:02:21.180 DEBUG:boto:Path: /dgbpxg-63/obj85 2024-04-12T04:02:21.180 DEBUG:boto:Data: 2024-04-12T04:02:21.181 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.181 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.181 DEBUG:boto:Params: {} 2024-04-12T04:02:21.181 DEBUG:boto:Token: None 2024-04-12T04:02:21.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85 2024-04-12T04:02:21.181 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C2KEnCAlDgIrxWl41Y3x+wbmsNY= 2024-04-12T04:02:21.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C2KEnCAlDgIrxWl41Y3x+wbmsNY=', 'Content-Length': '0'} 2024-04-12T04:02:21.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043deb8161695c492-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.230 DEBUG:boto:path=/dgbpxg-63/obj85 2024-04-12T04:02:21.231 DEBUG:boto:auth_path=/dgbpxg-63/obj85 2024-04-12T04:02:21.231 DEBUG:boto:Method: GET 2024-04-12T04:02:21.231 DEBUG:boto:Path: /dgbpxg-63/obj85 2024-04-12T04:02:21.231 DEBUG:boto:Data: 2024-04-12T04:02:21.231 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.231 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.231 DEBUG:boto:Params: {} 2024-04-12T04:02:21.232 DEBUG:boto:Token: None 2024-04-12T04:02:21.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj85 2024-04-12T04:02:21.232 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:C2KEnCAlDgIrxWl41Y3x+wbmsNY= 2024-04-12T04:02:21.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C2KEnCAlDgIrxWl41Y3x+wbmsNY=', 'Content-Length': '0'} 2024-04-12T04:02:21.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f714a4b4ffd09b-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.234 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.235 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.235 DEBUG:boto:path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.235 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.235 DEBUG:boto:Method: GET 2024-04-12T04:02:21.235 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.235 DEBUG:boto:Data: 2024-04-12T04:02:21.235 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.235 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.236 DEBUG:boto:Params: {} 2024-04-12T04:02:21.236 DEBUG:boto:Token: None 2024-04-12T04:02:21.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.236 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055584bd6da564f49-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.239 DEBUG:boto:b'dgbpxg-63obj851000falseobj852024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.239 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.240 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.240 DEBUG:boto:path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.240 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.240 DEBUG:boto:Method: GET 2024-04-12T04:02:21.240 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj85 2024-04-12T04:02:21.240 DEBUG:boto:Data: 2024-04-12T04:02:21.240 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.240 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.240 DEBUG:boto:Params: {} 2024-04-12T04:02:21.241 DEBUG:boto:Token: None 2024-04-12T04:02:21.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001402e56544399604-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.245 DEBUG:boto:b'dgbpxg-63obj851000falseobj852024-04-12T03:58:39.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:02:21.245 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.246 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.246 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.246 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.246 DEBUG:boto:Method: GET 2024-04-12T04:02:21.246 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.246 DEBUG:boto:Data: 2024-04-12T04:02:21.246 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.246 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.247 DEBUG:boto:Params: {} 2024-04-12T04:02:21.247 DEBUG:boto:Token: None 2024-04-12T04:02:21.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.247 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8= 2024-04-12T04:02:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8=', 'Content-Length': '0'} 2024-04-12T04:02:21.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bbea55873b862c7-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.249 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.249 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.250 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.250 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.250 DEBUG:boto:Method: GET 2024-04-12T04:02:21.250 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.250 DEBUG:boto:Data: 2024-04-12T04:02:21.250 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.250 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.250 DEBUG:boto:Params: {} 2024-04-12T04:02:21.251 DEBUG:boto:Token: None 2024-04-12T04:02:21.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8= 2024-04-12T04:02:21.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8=', 'Content-Length': '0'} 2024-04-12T04:02:21.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004448be35f65dea3f-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.294 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.295 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.295 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.295 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.295 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.295 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.295 DEBUG:boto:Data: 2024-04-12T04:02:21.295 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.295 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.295 DEBUG:boto:Params: {} 2024-04-12T04:02:21.296 DEBUG:boto:Token: None 2024-04-12T04:02:21.296 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.296 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:poe0T/wLouTcP9pjYaYXkmmPqEs= 2024-04-12T04:02:21.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:poe0T/wLouTcP9pjYaYXkmmPqEs=', 'Content-Length': '0'} 2024-04-12T04:02:21.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0c7eb216730907-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.298 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.298 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.298 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.298 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.298 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.298 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.299 DEBUG:boto:Data: 2024-04-12T04:02:21.299 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.299 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.299 DEBUG:boto:Params: {} 2024-04-12T04:02:21.299 DEBUG:boto:Token: None 2024-04-12T04:02:21.299 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.300 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:poe0T/wLouTcP9pjYaYXkmmPqEs= 2024-04-12T04:02:21.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:poe0T/wLouTcP9pjYaYXkmmPqEs=', 'Content-Length': '0'} 2024-04-12T04:02:21.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241fd43ebacffdea-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:02:21.301 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.302 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.302 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.302 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.302 DEBUG:boto:Method: GET 2024-04-12T04:02:21.302 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.302 DEBUG:boto:Data: 2024-04-12T04:02:21.302 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.302 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.303 DEBUG:boto:Params: {} 2024-04-12T04:02:21.303 DEBUG:boto:Token: None 2024-04-12T04:02:21.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.303 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8= 2024-04-12T04:02:21.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8=', 'Content-Length': '0'} 2024-04-12T04:02:21.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8215ff64de74ff-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.305 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.305 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.305 DEBUG:boto:path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.305 DEBUG:boto:auth_path=/dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.306 DEBUG:boto:Method: GET 2024-04-12T04:02:21.306 DEBUG:boto:Path: /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.306 DEBUG:boto:Data: 2024-04-12T04:02:21.306 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.306 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.306 DEBUG:boto:Params: {} 2024-04-12T04:02:21.306 DEBUG:boto:Token: None 2024-04-12T04:02:21.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87?versionId=null 2024-04-12T04:02:21.307 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8= 2024-04-12T04:02:21.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tbvRhpmeQSKHTq6eHu74igCUqP8=', 'Content-Length': '0'} 2024-04-12T04:02:21.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d344eb6d577811-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.309 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.309 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.309 DEBUG:boto:path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.310 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.310 DEBUG:boto:Method: GET 2024-04-12T04:02:21.310 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.310 DEBUG:boto:Data: 2024-04-12T04:02:21.310 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.311 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.311 DEBUG:boto:Params: {} 2024-04-12T04:02:21.311 DEBUG:boto:Token: None 2024-04-12T04:02:21.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.312 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3b350fc1469e4f-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.314 DEBUG:boto:b'dgbpxg-63obj871000falseobj872024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.315 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.315 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.315 DEBUG:boto:path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.315 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.315 DEBUG:boto:Method: GET 2024-04-12T04:02:21.315 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.315 DEBUG:boto:Data: 2024-04-12T04:02:21.315 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.315 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.316 DEBUG:boto:Params: {} 2024-04-12T04:02:21.316 DEBUG:boto:Token: None 2024-04-12T04:02:21.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.316 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4f847395c716fa-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.320 DEBUG:boto:b'dgbpxg-63obj871000falseobj872024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.320 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.321 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.321 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.321 DEBUG:boto:Path: /dgbpxg-63/obj87 2024-04-12T04:02:21.321 DEBUG:boto:Data: 2024-04-12T04:02:21.322 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.322 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.322 DEBUG:boto:Params: {} 2024-04-12T04:02:21.322 DEBUG:boto:Token: None 2024-04-12T04:02:21.322 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87 2024-04-12T04:02:21.322 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ugJ18WZHxcSLfU51W2m2UvSESXs= 2024-04-12T04:02:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ugJ18WZHxcSLfU51W2m2UvSESXs=', 'Content-Length': '0'} 2024-04-12T04:02:21.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050abceadbc6ddffe-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.324 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.324 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.325 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.325 DEBUG:boto:Path: /dgbpxg-63/obj87 2024-04-12T04:02:21.325 DEBUG:boto:Data: 2024-04-12T04:02:21.325 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.325 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.325 DEBUG:boto:Params: {} 2024-04-12T04:02:21.325 DEBUG:boto:Token: None 2024-04-12T04:02:21.325 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87 2024-04-12T04:02:21.326 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ugJ18WZHxcSLfU51W2m2UvSESXs= 2024-04-12T04:02:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ugJ18WZHxcSLfU51W2m2UvSESXs=', 'Content-Length': '0'} 2024-04-12T04:02:21.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd1ff82407777a9-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:02:21.328 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.328 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.328 DEBUG:boto:Method: GET 2024-04-12T04:02:21.328 DEBUG:boto:Path: /dgbpxg-63/obj87 2024-04-12T04:02:21.329 DEBUG:boto:Data: 2024-04-12T04:02:21.329 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.329 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.329 DEBUG:boto:Params: {} 2024-04-12T04:02:21.329 DEBUG:boto:Token: None 2024-04-12T04:02:21.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87 2024-04-12T04:02:21.330 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QAsFiupZOyULJArAIes9CcsJmJg= 2024-04-12T04:02:21.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QAsFiupZOyULJArAIes9CcsJmJg=', 'Content-Length': '0'} 2024-04-12T04:02:21.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daddd7f2c9a5363b-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.332 DEBUG:boto:path=/dgbpxg-63/obj87 2024-04-12T04:02:21.332 DEBUG:boto:auth_path=/dgbpxg-63/obj87 2024-04-12T04:02:21.332 DEBUG:boto:Method: GET 2024-04-12T04:02:21.332 DEBUG:boto:Path: /dgbpxg-63/obj87 2024-04-12T04:02:21.333 DEBUG:boto:Data: 2024-04-12T04:02:21.333 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.333 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.333 DEBUG:boto:Params: {} 2024-04-12T04:02:21.333 DEBUG:boto:Token: None 2024-04-12T04:02:21.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj87 2024-04-12T04:02:21.333 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QAsFiupZOyULJArAIes9CcsJmJg= 2024-04-12T04:02:21.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QAsFiupZOyULJArAIes9CcsJmJg=', 'Content-Length': '0'} 2024-04-12T04:02:21.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b4e5f225f709cd-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.335 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.335 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.336 DEBUG:boto:path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.336 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.336 DEBUG:boto:Method: GET 2024-04-12T04:02:21.336 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.336 DEBUG:boto:Data: 2024-04-12T04:02:21.336 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.336 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.336 DEBUG:boto:Params: {} 2024-04-12T04:02:21.337 DEBUG:boto:Token: None 2024-04-12T04:02:21.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.337 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2f3494a390f3cc-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.339 DEBUG:boto:b'dgbpxg-63obj871000falseobj872024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.340 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.340 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.340 DEBUG:boto:path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.340 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.340 DEBUG:boto:Method: GET 2024-04-12T04:02:21.340 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj87 2024-04-12T04:02:21.340 DEBUG:boto:Data: 2024-04-12T04:02:21.340 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.341 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.341 DEBUG:boto:Params: {} 2024-04-12T04:02:21.341 DEBUG:boto:Token: None 2024-04-12T04:02:21.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.341 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008098edfb6dfe8b00-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.386 DEBUG:boto:b'dgbpxg-63obj871000falseobj872024-04-12T03:58:40.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:02:21.387 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.387 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.387 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.387 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.387 DEBUG:boto:Method: GET 2024-04-12T04:02:21.387 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.388 DEBUG:boto:Data: 2024-04-12T04:02:21.388 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.388 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.388 DEBUG:boto:Params: {} 2024-04-12T04:02:21.388 DEBUG:boto:Token: None 2024-04-12T04:02:21.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.388 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao= 2024-04-12T04:02:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao=', 'Content-Length': '0'} 2024-04-12T04:02:21.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ca560c2487c5ff-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.390 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.390 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.390 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.391 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.391 DEBUG:boto:Method: GET 2024-04-12T04:02:21.391 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.391 DEBUG:boto:Data: 2024-04-12T04:02:21.391 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.391 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.391 DEBUG:boto:Params: {} 2024-04-12T04:02:21.391 DEBUG:boto:Token: None 2024-04-12T04:02:21.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.392 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao= 2024-04-12T04:02:21.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao=', 'Content-Length': '0'} 2024-04-12T04:02:21.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e9e89527a32ad9-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.393 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.394 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.394 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.394 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.394 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.394 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.394 DEBUG:boto:Data: 2024-04-12T04:02:21.394 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.394 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.394 DEBUG:boto:Params: {} 2024-04-12T04:02:21.395 DEBUG:boto:Token: None 2024-04-12T04:02:21.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.395 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xsJiekykKLnEEitnTBYdr1E/frY= 2024-04-12T04:02:21.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xsJiekykKLnEEitnTBYdr1E/frY=', 'Content-Length': '0'} 2024-04-12T04:02:21.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9ae67cc8b95898-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.396 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.397 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.397 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.397 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.397 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.397 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.397 DEBUG:boto:Data: 2024-04-12T04:02:21.397 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.397 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.397 DEBUG:boto:Params: {} 2024-04-12T04:02:21.397 DEBUG:boto:Token: None 2024-04-12T04:02:21.398 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.398 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xsJiekykKLnEEitnTBYdr1E/frY= 2024-04-12T04:02:21.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xsJiekykKLnEEitnTBYdr1E/frY=', 'Content-Length': '0'} 2024-04-12T04:02:21.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357d6592a13a0e02-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:02:21.400 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.400 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.400 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.400 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.400 DEBUG:boto:Method: GET 2024-04-12T04:02:21.400 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.400 DEBUG:boto:Data: 2024-04-12T04:02:21.400 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.400 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.400 DEBUG:boto:Params: {} 2024-04-12T04:02:21.401 DEBUG:boto:Token: None 2024-04-12T04:02:21.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.401 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao= 2024-04-12T04:02:21.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao=', 'Content-Length': '0'} 2024-04-12T04:02:21.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b292025c558d59f-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.403 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.403 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.403 DEBUG:boto:path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.403 DEBUG:boto:auth_path=/dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.403 DEBUG:boto:Method: GET 2024-04-12T04:02:21.403 DEBUG:boto:Path: /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.403 DEBUG:boto:Data: 2024-04-12T04:02:21.403 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.403 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.403 DEBUG:boto:Params: {} 2024-04-12T04:02:21.404 DEBUG:boto:Token: None 2024-04-12T04:02:21.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92?versionId=null 2024-04-12T04:02:21.404 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao= 2024-04-12T04:02:21.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gl8N0PdeFkKPp+O8mSyRK5jXfao=', 'Content-Length': '0'} 2024-04-12T04:02:21.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bb8bce2cc43d37-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.405 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.406 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.406 DEBUG:boto:path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.406 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.406 DEBUG:boto:Method: GET 2024-04-12T04:02:21.406 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.406 DEBUG:boto:Data: 2024-04-12T04:02:21.406 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.406 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.406 DEBUG:boto:Params: {} 2024-04-12T04:02:21.407 DEBUG:boto:Token: None 2024-04-12T04:02:21.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.407 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008433af7429bb66ad-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.450 DEBUG:boto:b'dgbpxg-63obj921000falseobj922024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.451 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.451 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.451 DEBUG:boto:path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.451 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.451 DEBUG:boto:Method: GET 2024-04-12T04:02:21.451 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.451 DEBUG:boto:Data: 2024-04-12T04:02:21.451 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.452 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.452 DEBUG:boto:Params: {} 2024-04-12T04:02:21.452 DEBUG:boto:Token: None 2024-04-12T04:02:21.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e983706bc289433b-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.456 DEBUG:boto:b'dgbpxg-63obj921000falseobj922024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.457 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.457 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.457 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.457 DEBUG:boto:Path: /dgbpxg-63/obj92 2024-04-12T04:02:21.457 DEBUG:boto:Data: 2024-04-12T04:02:21.457 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.457 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.457 DEBUG:boto:Params: {} 2024-04-12T04:02:21.457 DEBUG:boto:Token: None 2024-04-12T04:02:21.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92 2024-04-12T04:02:21.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mlrYk8HRrpwlBhvct1RVxju7oKg= 2024-04-12T04:02:21.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mlrYk8HRrpwlBhvct1RVxju7oKg=', 'Content-Length': '0'} 2024-04-12T04:02:21.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce970554ff79f1d-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.459 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.460 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.460 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.460 DEBUG:boto:Path: /dgbpxg-63/obj92 2024-04-12T04:02:21.460 DEBUG:boto:Data: 2024-04-12T04:02:21.460 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.460 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.460 DEBUG:boto:Params: {} 2024-04-12T04:02:21.460 DEBUG:boto:Token: None 2024-04-12T04:02:21.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92 2024-04-12T04:02:21.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mlrYk8HRrpwlBhvct1RVxju7oKg= 2024-04-12T04:02:21.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mlrYk8HRrpwlBhvct1RVxju7oKg=', 'Content-Length': '0'} 2024-04-12T04:02:21.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b627d57f0d1c6d-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:02:21.463 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.463 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.463 DEBUG:boto:Method: GET 2024-04-12T04:02:21.463 DEBUG:boto:Path: /dgbpxg-63/obj92 2024-04-12T04:02:21.463 DEBUG:boto:Data: 2024-04-12T04:02:21.463 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.463 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.463 DEBUG:boto:Params: {} 2024-04-12T04:02:21.463 DEBUG:boto:Token: None 2024-04-12T04:02:21.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92 2024-04-12T04:02:21.464 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xZ8Rxn9g16t6i+NB2xQoY78ugTI= 2024-04-12T04:02:21.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xZ8Rxn9g16t6i+NB2xQoY78ugTI=', 'Content-Length': '0'} 2024-04-12T04:02:21.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f37267a1ea3ebb5-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.466 DEBUG:boto:path=/dgbpxg-63/obj92 2024-04-12T04:02:21.466 DEBUG:boto:auth_path=/dgbpxg-63/obj92 2024-04-12T04:02:21.466 DEBUG:boto:Method: GET 2024-04-12T04:02:21.466 DEBUG:boto:Path: /dgbpxg-63/obj92 2024-04-12T04:02:21.466 DEBUG:boto:Data: 2024-04-12T04:02:21.466 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.466 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.466 DEBUG:boto:Params: {} 2024-04-12T04:02:21.467 DEBUG:boto:Token: None 2024-04-12T04:02:21.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj92 2024-04-12T04:02:21.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xZ8Rxn9g16t6i+NB2xQoY78ugTI= 2024-04-12T04:02:21.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xZ8Rxn9g16t6i+NB2xQoY78ugTI=', 'Content-Length': '0'} 2024-04-12T04:02:21.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003485b5591a73afe5-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.515 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.515 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.515 DEBUG:boto:path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.515 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.515 DEBUG:boto:Method: GET 2024-04-12T04:02:21.515 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.515 DEBUG:boto:Data: 2024-04-12T04:02:21.515 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.516 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.516 DEBUG:boto:Params: {} 2024-04-12T04:02:21.516 DEBUG:boto:Token: None 2024-04-12T04:02:21.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005750711f6728873-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.519 DEBUG:boto:b'dgbpxg-63obj921000falseobj922024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.519 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.519 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.520 DEBUG:boto:path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.520 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.520 DEBUG:boto:Method: GET 2024-04-12T04:02:21.520 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj92 2024-04-12T04:02:21.520 DEBUG:boto:Data: 2024-04-12T04:02:21.520 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.520 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.520 DEBUG:boto:Params: {} 2024-04-12T04:02:21.521 DEBUG:boto:Token: None 2024-04-12T04:02:21.521 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.521 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de680e17094d8dea-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.525 DEBUG:boto:b'dgbpxg-63obj921000falseobj922024-04-12T03:58:39.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:02:21.526 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.526 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.526 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.526 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.526 DEBUG:boto:Method: GET 2024-04-12T04:02:21.526 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.526 DEBUG:boto:Data: 2024-04-12T04:02:21.526 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.526 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.527 DEBUG:boto:Params: {} 2024-04-12T04:02:21.527 DEBUG:boto:Token: None 2024-04-12T04:02:21.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.527 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM= 2024-04-12T04:02:21.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM=', 'Content-Length': '0'} 2024-04-12T04:02:21.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ff7a8af9925891-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.529 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.529 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.529 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.529 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.530 DEBUG:boto:Method: GET 2024-04-12T04:02:21.530 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.530 DEBUG:boto:Data: 2024-04-12T04:02:21.530 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.530 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.530 DEBUG:boto:Params: {} 2024-04-12T04:02:21.530 DEBUG:boto:Token: None 2024-04-12T04:02:21.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM= 2024-04-12T04:02:21.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM=', 'Content-Length': '0'} 2024-04-12T04:02:21.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ca06560ba431ef-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.532 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.533 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.533 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.533 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.533 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.533 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.533 DEBUG:boto:Data: 2024-04-12T04:02:21.533 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.533 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.533 DEBUG:boto:Params: {} 2024-04-12T04:02:21.533 DEBUG:boto:Token: None 2024-04-12T04:02:21.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.534 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LDGPHeoaUIkXZY2Oyts/0raLkK4= 2024-04-12T04:02:21.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LDGPHeoaUIkXZY2Oyts/0raLkK4=', 'Content-Length': '0'} 2024-04-12T04:02:21.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fae036a740d40e6-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.538 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.538 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.538 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.538 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.538 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.538 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.538 DEBUG:boto:Data: 2024-04-12T04:02:21.538 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.539 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.539 DEBUG:boto:Params: {} 2024-04-12T04:02:21.539 DEBUG:boto:Token: None 2024-04-12T04:02:21.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.539 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LDGPHeoaUIkXZY2Oyts/0raLkK4= 2024-04-12T04:02:21.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LDGPHeoaUIkXZY2Oyts/0raLkK4=', 'Content-Length': '0'} 2024-04-12T04:02:21.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a755f8c55d2f150-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:02:21.541 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.541 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.541 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.541 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.541 DEBUG:boto:Method: GET 2024-04-12T04:02:21.542 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.542 DEBUG:boto:Data: 2024-04-12T04:02:21.542 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.542 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.542 DEBUG:boto:Params: {} 2024-04-12T04:02:21.542 DEBUG:boto:Token: None 2024-04-12T04:02:21.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.542 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM= 2024-04-12T04:02:21.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM=', 'Content-Length': '0'} 2024-04-12T04:02:21.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8fe3d0af2cded8-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.544 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.544 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.544 DEBUG:boto:path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.545 DEBUG:boto:auth_path=/dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.545 DEBUG:boto:Method: GET 2024-04-12T04:02:21.545 DEBUG:boto:Path: /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.545 DEBUG:boto:Data: 2024-04-12T04:02:21.545 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.545 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.545 DEBUG:boto:Params: {} 2024-04-12T04:02:21.545 DEBUG:boto:Token: None 2024-04-12T04:02:21.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93?versionId=null 2024-04-12T04:02:21.546 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM= 2024-04-12T04:02:21.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i30BZkg1GYrQU4YlOWOdCveWYEM=', 'Content-Length': '0'} 2024-04-12T04:02:21.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7eb52b8755f197-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.547 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.547 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.547 DEBUG:boto:path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.548 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.548 DEBUG:boto:Method: GET 2024-04-12T04:02:21.548 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.548 DEBUG:boto:Data: 2024-04-12T04:02:21.548 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.548 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.548 DEBUG:boto:Params: {} 2024-04-12T04:02:21.548 DEBUG:boto:Token: None 2024-04-12T04:02:21.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.549 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004564f395b7d5377d-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.551 DEBUG:boto:b'dgbpxg-63obj931000falseobj932024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.551 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.551 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.551 DEBUG:boto:path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.551 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.552 DEBUG:boto:Method: GET 2024-04-12T04:02:21.552 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.552 DEBUG:boto:Data: 2024-04-12T04:02:21.552 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.552 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.552 DEBUG:boto:Params: {} 2024-04-12T04:02:21.552 DEBUG:boto:Token: None 2024-04-12T04:02:21.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.553 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8cf46935d0ba213-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.598 DEBUG:boto:b'dgbpxg-63obj931000falseobj932024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.599 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.599 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.599 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.599 DEBUG:boto:Path: /dgbpxg-63/obj93 2024-04-12T04:02:21.599 DEBUG:boto:Data: 2024-04-12T04:02:21.599 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.599 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.600 DEBUG:boto:Params: {} 2024-04-12T04:02:21.600 DEBUG:boto:Token: None 2024-04-12T04:02:21.600 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93 2024-04-12T04:02:21.600 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Mu8O6wYuNr83Eu0rrhUEK9Oe/I= 2024-04-12T04:02:21.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Mu8O6wYuNr83Eu0rrhUEK9Oe/I=', 'Content-Length': '0'} 2024-04-12T04:02:21.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003549de94b4415007-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.602 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.602 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.603 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.603 DEBUG:boto:Path: /dgbpxg-63/obj93 2024-04-12T04:02:21.603 DEBUG:boto:Data: 2024-04-12T04:02:21.603 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.603 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.603 DEBUG:boto:Params: {} 2024-04-12T04:02:21.603 DEBUG:boto:Token: None 2024-04-12T04:02:21.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93 2024-04-12T04:02:21.604 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Mu8O6wYuNr83Eu0rrhUEK9Oe/I= 2024-04-12T04:02:21.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Mu8O6wYuNr83Eu0rrhUEK9Oe/I=', 'Content-Length': '0'} 2024-04-12T04:02:21.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e13386c8790924-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:02:21.606 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.606 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.606 DEBUG:boto:Method: GET 2024-04-12T04:02:21.606 DEBUG:boto:Path: /dgbpxg-63/obj93 2024-04-12T04:02:21.606 DEBUG:boto:Data: 2024-04-12T04:02:21.606 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.606 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.606 DEBUG:boto:Params: {} 2024-04-12T04:02:21.607 DEBUG:boto:Token: None 2024-04-12T04:02:21.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93 2024-04-12T04:02:21.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZKpo2Ekz336FAqlzUeK0pfa138k= 2024-04-12T04:02:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZKpo2Ekz336FAqlzUeK0pfa138k=', 'Content-Length': '0'} 2024-04-12T04:02:21.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b266ca7f2b5849e4-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.609 DEBUG:boto:path=/dgbpxg-63/obj93 2024-04-12T04:02:21.609 DEBUG:boto:auth_path=/dgbpxg-63/obj93 2024-04-12T04:02:21.609 DEBUG:boto:Method: GET 2024-04-12T04:02:21.609 DEBUG:boto:Path: /dgbpxg-63/obj93 2024-04-12T04:02:21.609 DEBUG:boto:Data: 2024-04-12T04:02:21.609 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.609 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.609 DEBUG:boto:Params: {} 2024-04-12T04:02:21.609 DEBUG:boto:Token: None 2024-04-12T04:02:21.610 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj93 2024-04-12T04:02:21.610 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZKpo2Ekz336FAqlzUeK0pfa138k= 2024-04-12T04:02:21.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZKpo2Ekz336FAqlzUeK0pfa138k=', 'Content-Length': '0'} 2024-04-12T04:02:21.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004619f7938d3359a3-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.612 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.612 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.612 DEBUG:boto:path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.612 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.612 DEBUG:boto:Method: GET 2024-04-12T04:02:21.612 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.612 DEBUG:boto:Data: 2024-04-12T04:02:21.612 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.613 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.613 DEBUG:boto:Params: {} 2024-04-12T04:02:21.613 DEBUG:boto:Token: None 2024-04-12T04:02:21.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007663b987c0c5fafe-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.616 DEBUG:boto:b'dgbpxg-63obj931000falseobj932024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.616 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.616 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.616 DEBUG:boto:path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.616 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.616 DEBUG:boto:Method: GET 2024-04-12T04:02:21.617 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj93 2024-04-12T04:02:21.617 DEBUG:boto:Data: 2024-04-12T04:02:21.617 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.617 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.617 DEBUG:boto:Params: {} 2024-04-12T04:02:21.617 DEBUG:boto:Token: None 2024-04-12T04:02:21.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.617 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085eb72be31c8b9d4-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.621 DEBUG:boto:b'dgbpxg-63obj931000falseobj932024-04-12T03:58:40.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:02:21.621 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.621 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.622 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.622 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.622 DEBUG:boto:Method: GET 2024-04-12T04:02:21.622 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.622 DEBUG:boto:Data: 2024-04-12T04:02:21.622 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.622 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.622 DEBUG:boto:Params: {} 2024-04-12T04:02:21.622 DEBUG:boto:Token: None 2024-04-12T04:02:21.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.623 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY= 2024-04-12T04:02:21.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY=', 'Content-Length': '0'} 2024-04-12T04:02:21.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dfd829bc51da372-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.666 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.667 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.667 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.667 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.667 DEBUG:boto:Method: GET 2024-04-12T04:02:21.667 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.667 DEBUG:boto:Data: 2024-04-12T04:02:21.667 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.667 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.668 DEBUG:boto:Params: {} 2024-04-12T04:02:21.668 DEBUG:boto:Token: None 2024-04-12T04:02:21.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY= 2024-04-12T04:02:21.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY=', 'Content-Length': '0'} 2024-04-12T04:02:21.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b2e39010293200-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.670 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.670 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.670 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.671 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.671 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.671 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.671 DEBUG:boto:Data: 2024-04-12T04:02:21.671 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.671 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.671 DEBUG:boto:Params: {} 2024-04-12T04:02:21.671 DEBUG:boto:Token: None 2024-04-12T04:02:21.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.672 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OIf5q6S100YT+TMlsqFElRucP+k= 2024-04-12T04:02:21.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OIf5q6S100YT+TMlsqFElRucP+k=', 'Content-Length': '0'} 2024-04-12T04:02:21.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c62ddd62c30463-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.674 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.674 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.674 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.674 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.674 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.674 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.674 DEBUG:boto:Data: 2024-04-12T04:02:21.674 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.675 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.675 DEBUG:boto:Params: {} 2024-04-12T04:02:21.675 DEBUG:boto:Token: None 2024-04-12T04:02:21.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.675 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OIf5q6S100YT+TMlsqFElRucP+k= 2024-04-12T04:02:21.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OIf5q6S100YT+TMlsqFElRucP+k=', 'Content-Length': '0'} 2024-04-12T04:02:21.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb331e81c56624c9-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:02:21.677 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.677 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.677 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.677 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.677 DEBUG:boto:Method: GET 2024-04-12T04:02:21.677 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.677 DEBUG:boto:Data: 2024-04-12T04:02:21.678 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.678 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.678 DEBUG:boto:Params: {} 2024-04-12T04:02:21.678 DEBUG:boto:Token: None 2024-04-12T04:02:21.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.678 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY= 2024-04-12T04:02:21.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY=', 'Content-Length': '0'} 2024-04-12T04:02:21.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2a776f4b65f7d0-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.680 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.680 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.681 DEBUG:boto:path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.681 DEBUG:boto:auth_path=/dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.681 DEBUG:boto:Method: GET 2024-04-12T04:02:21.681 DEBUG:boto:Path: /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.681 DEBUG:boto:Data: 2024-04-12T04:02:21.681 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.681 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.681 DEBUG:boto:Params: {} 2024-04-12T04:02:21.681 DEBUG:boto:Token: None 2024-04-12T04:02:21.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95?versionId=null 2024-04-12T04:02:21.682 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY= 2024-04-12T04:02:21.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6mhDZszvJ4pc1qai8Jl7UDPQUnY=', 'Content-Length': '0'} 2024-04-12T04:02:21.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002152f2a314243198-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.684 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.684 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.684 DEBUG:boto:path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.684 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.684 DEBUG:boto:Method: GET 2024-04-12T04:02:21.684 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.684 DEBUG:boto:Data: 2024-04-12T04:02:21.684 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.684 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.684 DEBUG:boto:Params: {} 2024-04-12T04:02:21.685 DEBUG:boto:Token: None 2024-04-12T04:02:21.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.685 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b42f00842499a5-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.687 DEBUG:boto:b'dgbpxg-63obj951000falseobj952024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.688 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.688 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.688 DEBUG:boto:path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.688 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.688 DEBUG:boto:Method: GET 2024-04-12T04:02:21.688 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.688 DEBUG:boto:Data: 2024-04-12T04:02:21.688 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.688 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.688 DEBUG:boto:Params: {} 2024-04-12T04:02:21.689 DEBUG:boto:Token: None 2024-04-12T04:02:21.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007312db9369d1b16a-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.693 DEBUG:boto:b'dgbpxg-63obj951000falseobj952024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.693 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.693 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.694 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.694 DEBUG:boto:Path: /dgbpxg-63/obj95 2024-04-12T04:02:21.694 DEBUG:boto:Data: 2024-04-12T04:02:21.694 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.694 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.694 DEBUG:boto:Params: {} 2024-04-12T04:02:21.694 DEBUG:boto:Token: None 2024-04-12T04:02:21.694 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95 2024-04-12T04:02:21.694 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H3PJTJbUzRyC3eRcrvkBTRFGNdg= 2024-04-12T04:02:21.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H3PJTJbUzRyC3eRcrvkBTRFGNdg=', 'Content-Length': '0'} 2024-04-12T04:02:21.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721cf2764c72770c-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.696 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.696 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.697 DEBUG:boto:Method: HEAD 2024-04-12T04:02:21.697 DEBUG:boto:Path: /dgbpxg-63/obj95 2024-04-12T04:02:21.697 DEBUG:boto:Data: 2024-04-12T04:02:21.697 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.697 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.697 DEBUG:boto:Params: {} 2024-04-12T04:02:21.697 DEBUG:boto:Token: None 2024-04-12T04:02:21.697 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95 2024-04-12T04:02:21.698 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:H3PJTJbUzRyC3eRcrvkBTRFGNdg= 2024-04-12T04:02:21.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H3PJTJbUzRyC3eRcrvkBTRFGNdg=', 'Content-Length': '0'} 2024-04-12T04:02:21.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a2c75bc512bcde-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:02:21.699 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.700 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.700 DEBUG:boto:Method: GET 2024-04-12T04:02:21.700 DEBUG:boto:Path: /dgbpxg-63/obj95 2024-04-12T04:02:21.700 DEBUG:boto:Data: 2024-04-12T04:02:21.700 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.700 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.700 DEBUG:boto:Params: {} 2024-04-12T04:02:21.700 DEBUG:boto:Token: None 2024-04-12T04:02:21.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95 2024-04-12T04:02:21.700 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q/zG4nLDRN+EwXeaHn77Vu/RSNo= 2024-04-12T04:02:21.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/zG4nLDRN+EwXeaHn77Vu/RSNo=', 'Content-Length': '0'} 2024-04-12T04:02:21.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-63:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffaaf4b862bf904-006618b24d-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.702 DEBUG:boto:path=/dgbpxg-63/obj95 2024-04-12T04:02:21.702 DEBUG:boto:auth_path=/dgbpxg-63/obj95 2024-04-12T04:02:21.703 DEBUG:boto:Method: GET 2024-04-12T04:02:21.703 DEBUG:boto:Path: /dgbpxg-63/obj95 2024-04-12T04:02:21.703 DEBUG:boto:Data: 2024-04-12T04:02:21.703 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.703 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.703 DEBUG:boto:Params: {} 2024-04-12T04:02:21.703 DEBUG:boto:Token: None 2024-04-12T04:02:21.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/obj95 2024-04-12T04:02:21.704 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Q/zG4nLDRN+EwXeaHn77Vu/RSNo= 2024-04-12T04:02:21.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/zG4nLDRN+EwXeaHn77Vu/RSNo=', 'Content-Length': '0'} 2024-04-12T04:02:21.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d302094689592ace-006618b24d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.705 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.705 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.706 DEBUG:boto:path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.706 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.706 DEBUG:boto:Method: GET 2024-04-12T04:02:21.706 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.706 DEBUG:boto:Data: 2024-04-12T04:02:21.706 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:21.706 DEBUG:boto:Port: 8001 2024-04-12T04:02:21.706 DEBUG:boto:Params: {} 2024-04-12T04:02:21.706 DEBUG:boto:Token: None 2024-04-12T04:02:21.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.707 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a282d696eda3bee2-006618b24d-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.710 DEBUG:boto:b'dgbpxg-63obj951000falseobj952024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.710 DEBUG:boto:path=/dgbpxg-63/ 2024-04-12T04:02:21.710 DEBUG:boto:auth_path=/dgbpxg-63/ 2024-04-12T04:02:21.710 DEBUG:boto:path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.710 DEBUG:boto:auth_path=/dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.710 DEBUG:boto:Method: GET 2024-04-12T04:02:21.710 DEBUG:boto:Path: /dgbpxg-63/?prefix=obj95 2024-04-12T04:02:21.711 DEBUG:boto:Data: 2024-04-12T04:02:21.711 DEBUG:boto:Headers: {} 2024-04-12T04:02:21.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:21.711 DEBUG:boto:Port: 8000 2024-04-12T04:02:21.711 DEBUG:boto:Params: {} 2024-04-12T04:02:21.711 DEBUG:boto:Token: None 2024-04-12T04:02:21.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:21 GMT /dgbpxg-63/ 2024-04-12T04:02:21.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU= 2024-04-12T04:02:21.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2DPM7ot05PO00K5Okc84m27GugU=', 'Content-Length': '0'} 2024-04-12T04:02:21.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001627aaec33eec63d-006618b24d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:21.715 DEBUG:boto:b'dgbpxg-63obj951000falseobj952024-04-12T03:58:39.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:21.715 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-63 zones={a2, a1} 2024-04-12T04:02:21.715 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T04:02:21.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.716 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:21.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:21.764 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:21.811 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034325", 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:55.131881Z" 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000023988", 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:55.133471Z" 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000039502", 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:40.845124Z" 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034975", 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:55.134822Z" 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.813 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:02:21.815 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000034325', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039502', 3: '00000000000000000000:00000000000000034975'} 2024-04-12T04:02:21.815 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:02:21.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.815 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:21.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:21.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:21.909 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:02:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:02:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:02:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:02:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10092454543754946800 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034325", 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:55.131881Z" 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000023988", 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:55.133471Z" 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000039502", 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 54, 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:40.845124Z" 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034975", 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:55.134822Z" 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:02:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 63, 2024-04-12T04:02:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 63 2024-04-12T04:02:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:02:21Z" 2024-04-12T04:02:21.915 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:02:21.916 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10092454543754946800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034325", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:55.131881Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023988", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:55.133471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000039502", "next_step_marker": "", "total_entries": 54, "pos": 0, "timestamp": "2024-04-12T03:58:40.845124Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034975", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T03:58:55.134822Z" } } ] }, "full_sync": { "total": 63, "complete": 63 }, "current_time": "2024-04-12T04:02:21Z" } 2024-04-12T04:02:21.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034325', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:55.131881Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023988', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:55.133471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000039502', 'next_step_marker': '', 'total_entries': 54, 'pos': 0, 'timestamp': '2024-04-12T03:58:40.845124Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034975', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T03:58:55.134822Z'}}] 2024-04-12T04:02:21.916 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034325', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039502', 3: '00000000000000000000:00000000000000034975'} 2024-04-12T04:02:21.916 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034325', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039502', 3: '00000000000000000000:00000000000000034975'} 2024-04-12T04:02:21.916 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:02:21.916 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T04:02:21.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:21.917 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:21.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:21.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:22.012 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:02:22.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012524", 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:00:31.101627Z" 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006432", 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:00:31.103654Z" 2024-04-12T04:02:22.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016152", 2024-04-12T04:02:22.014 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:00:31.105362Z" 2024-04-12T04:02:22.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016437", 2024-04-12T04:02:22.015 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:00:53.108620Z" 2024-04-12T04:02:22.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.015 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:02:22.016 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000012524', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016152', 3: '00000000000000000000:00000000000000016437'} 2024-04-12T04:02:22.016 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:02:22.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.016 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:22.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:22.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:22.111 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:02:22.111 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:02:22.111 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:02:22.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012524", 2024-04-12T04:02:22.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:00:31.101627Z" 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:22.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006432", 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:00:31.103654Z" 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016152", 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:00:31.105362Z" 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016437", 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:00:53.108620Z" 2024-04-12T04:02:22.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:02:22Z" 2024-04-12T04:02:22.117 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:02:22.119 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012524", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:00:31.101627Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006432", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:00:31.103654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:00:31.105362Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016437", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:00:53.108620Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:02:22Z" } 2024-04-12T04:02:22.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012524', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:00:31.101627Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006432', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:00:31.103654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:00:31.105362Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016437', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:00:53.108620Z'}}] 2024-04-12T04:02:22.119 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012524', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016152', 3: '00000000000000000000:00000000000000016437'} 2024-04-12T04:02:22.119 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012524', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016152', 3: '00000000000000000000:00000000000000016437'} 2024-04-12T04:02:22.119 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:02:22.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2024-04-12T04:02:22.121 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:22.122 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:22.122 DEBUG:boto:Method: PUT 2024-04-12T04:02:22.122 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:22.122 DEBUG:boto:Data: 2024-04-12T04:02:22.122 DEBUG:boto:Headers: {} 2024-04-12T04:02:22.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:22.122 DEBUG:boto:Port: 8000 2024-04-12T04:02:22.122 DEBUG:boto:Params: {} 2024-04-12T04:02:22.123 DEBUG:boto:Token: None 2024-04-12T04:02:22.123 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:02:22 GMT /dgbpxg-64/ 2024-04-12T04:02:22.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CjEEQWjcTUlL8ZDorPYicFb0Ats= 2024-04-12T04:02:22.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:22 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CjEEQWjcTUlL8ZDorPYicFb0Ats=', 'Content-Length': '0'} 2024-04-12T04:02:22.138 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac7d64e806e4a12e-006618b24e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:22.138 DEBUG:rgw_multi.tests:created bucket=dgbpxg-64 2024-04-12T04:02:22.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.139 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:22.186 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:22.187 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:22.216 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894542.151240_1633.1", 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:02:22.151240Z" 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.230 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:02:22.231 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894542.151240_1633.1'} 2024-04-12T04:02:22.232 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:02:22.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:02:22.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:02:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:22.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:22.326+0000 7fac21ecb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:02:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:22.327+0000 7fac21ecb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:02:22.326 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:02:22.326 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:02:22.326 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:02:22.326 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:02:22.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:02:22.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:02:22.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:02:22.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894542.151240_1633.1", 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:02:22.151240Z", 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:02:22.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:02:22Z" 2024-04-12T04:02:22.332 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:02:22.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894542.151240_1633.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:02:22.151240Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:02:22Z" } 2024-04-12T04:02:22.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894542.151240_1633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:02:22.151240Z', 'realm_epoch': 2}}] 2024-04-12T04:02:22.333 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894542.151240_1633.1'} 2024-04-12T04:02:22.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894542.151240_1633.1'} 2024-04-12T04:02:22.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:02:22.334 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:22.334 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:22.334 DEBUG:boto:Method: HEAD 2024-04-12T04:02:22.334 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:22.334 DEBUG:boto:Data: 2024-04-12T04:02:22.334 DEBUG:boto:Headers: {} 2024-04-12T04:02:22.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:22.334 DEBUG:boto:Port: 8000 2024-04-12T04:02:22.334 DEBUG:boto:Params: {} 2024-04-12T04:02:22.335 DEBUG:boto:Token: None 2024-04-12T04:02:22.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:22 GMT /dgbpxg-64/ 2024-04-12T04:02:22.335 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0RZp80Y9WJ3FDO+rOuKm5Gs1B1E= 2024-04-12T04:02:22.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:22 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0RZp80Y9WJ3FDO+rOuKm5Gs1B1E=', 'Content-Length': '0'} 2024-04-12T04:02:22.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e99ee299d5fbdc6-006618b24e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:22.337 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:22.338 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:22.338 DEBUG:boto:Method: PUT 2024-04-12T04:02:22.338 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:02:22.338 DEBUG:boto:Data: 2024-04-12T04:02:22.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:22.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:22.338 DEBUG:boto:Port: 8000 2024-04-12T04:02:22.338 DEBUG:boto:Params: {} 2024-04-12T04:02:22.338 DEBUG:boto:Token: None 2024-04-12T04:02:22.339 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:22 GMT /dgbpxg-64/dummy 2024-04-12T04:02:22.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CI8cZTfqsttOHPXaaWJ3x8Cc6Y8= 2024-04-12T04:02:22.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:22 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CI8cZTfqsttOHPXaaWJ3x8Cc6Y8='} 2024-04-12T04:02:22.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8bb26294aa3613c-006618b24e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:22.345 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-64 2024-04-12T04:02:22.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:22.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:22.345 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:02:22.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:22.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:22.450+0000 7fcb76b29780 1 waiting to reach incremental sync.. 2024-04-12T04:02:27.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:27.452+0000 7fcb76b29780 1 bucket sync caught up with source: 2024-04-12T04:02:27.438 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:27.438 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:27.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:27.452+0000 7fcb76b29780 0 bucket checkpoint complete 2024-04-12T04:02:27.456 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-64 2024-04-12T04:02:27.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:27.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:27.456 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:02:27.518 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:27.519 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:27.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:27.569+0000 7eff5ff68780 1 bucket sync caught up with source: 2024-04-12T04:02:27.555 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.2477.6, , , , , ] 2024-04-12T04:02:27.555 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.2477.6, , , , , ] 2024-04-12T04:02:27.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:27.569+0000 7eff5ff68780 0 bucket checkpoint complete 2024-04-12T04:02:27.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-64 zones={a2, a1} 2024-04-12T04:02:27.569 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.569 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.570 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.570 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:27.570 DEBUG:boto:Data: 2024-04-12T04:02:27.570 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.570 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.570 DEBUG:boto:Params: {} 2024-04-12T04:02:27.570 DEBUG:boto:Token: None 2024-04-12T04:02:27.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98k3qB6rCNwZE+KR/4clyppt/rs= 2024-04-12T04:02:27.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98k3qB6rCNwZE+KR/4clyppt/rs=', 'Content-Length': '0'} 2024-04-12T04:02:27.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e8b4cd4bc95a5f6-006618b253-2229-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.573 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.573 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.573 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.574 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:27.574 DEBUG:boto:Data: 2024-04-12T04:02:27.574 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.574 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.574 DEBUG:boto:Params: {} 2024-04-12T04:02:27.574 DEBUG:boto:Token: None 2024-04-12T04:02:27.574 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.574 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:98k3qB6rCNwZE+KR/4clyppt/rs= 2024-04-12T04:02:27.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:98k3qB6rCNwZE+KR/4clyppt/rs=', 'Content-Length': '0'} 2024-04-12T04:02:27.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea8297321d220d22-006618b253-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:02:27.577 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.577 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.577 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.577 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.577 DEBUG:boto:Method: GET 2024-04-12T04:02:27.577 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.578 DEBUG:boto:Data: 2024-04-12T04:02:27.578 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.578 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.578 DEBUG:boto:Params: {} 2024-04-12T04:02:27.578 DEBUG:boto:Token: None 2024-04-12T04:02:27.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/?versions 2024-04-12T04:02:27.578 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU= 2024-04-12T04:02:27.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU=', 'Content-Length': '0'} 2024-04-12T04:02:27.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5569fa8dec0bb1c-006618b253-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.581 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.581 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:02:27.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:02:27.581 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.582 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.582 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.582 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.582 DEBUG:boto:Method: GET 2024-04-12T04:02:27.582 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.582 DEBUG:boto:Data: 2024-04-12T04:02:27.582 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.582 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.582 DEBUG:boto:Params: {} 2024-04-12T04:02:27.583 DEBUG:boto:Token: None 2024-04-12T04:02:27.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/?versions 2024-04-12T04:02:27.583 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU= 2024-04-12T04:02:27.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU=', 'Content-Length': '0'} 2024-04-12T04:02:27.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8238b760abb107-006618b253-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.585 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.585 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:02:27.585 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.586 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.586 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.586 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.586 DEBUG:boto:Method: GET 2024-04-12T04:02:27.586 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.586 DEBUG:boto:Data: 2024-04-12T04:02:27.586 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.586 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.586 DEBUG:boto:Params: {} 2024-04-12T04:02:27.586 DEBUG:boto:Token: None 2024-04-12T04:02:27.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/?versions 2024-04-12T04:02:27.587 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU= 2024-04-12T04:02:27.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU=', 'Content-Length': '0'} 2024-04-12T04:02:27.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019916404ca35d452-006618b253-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.589 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.589 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.589 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.590 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.590 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.590 DEBUG:boto:Method: GET 2024-04-12T04:02:27.590 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:02:27.590 DEBUG:boto:Data: 2024-04-12T04:02:27.590 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.590 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.590 DEBUG:boto:Params: {} 2024-04-12T04:02:27.590 DEBUG:boto:Token: None 2024-04-12T04:02:27.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/?versions 2024-04-12T04:02:27.591 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU= 2024-04-12T04:02:27.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/mbdYnq14V+WD54mIGOdv71dboU=', 'Content-Length': '0'} 2024-04-12T04:02:27.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002938587bbf9173c6-006618b253-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.593 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:02:27.594 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.594 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.594 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.594 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.594 DEBUG:boto:Method: GET 2024-04-12T04:02:27.594 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.594 DEBUG:boto:Data: 2024-04-12T04:02:27.594 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.594 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.595 DEBUG:boto:Params: {} 2024-04-12T04:02:27.595 DEBUG:boto:Token: None 2024-04-12T04:02:27.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.595 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y= 2024-04-12T04:02:27.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y=', 'Content-Length': '0'} 2024-04-12T04:02:27.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb3070b7e5b18a62-006618b253-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.597 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.597 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.597 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.598 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.598 DEBUG:boto:Method: GET 2024-04-12T04:02:27.598 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.598 DEBUG:boto:Data: 2024-04-12T04:02:27.598 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.598 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.598 DEBUG:boto:Params: {} 2024-04-12T04:02:27.598 DEBUG:boto:Token: None 2024-04-12T04:02:27.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y= 2024-04-12T04:02:27.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y=', 'Content-Length': '0'} 2024-04-12T04:02:27.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be09b7f1e7fa003-006618b253-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.601 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.601 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.601 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.601 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.601 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.602 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.602 DEBUG:boto:Data: 2024-04-12T04:02:27.602 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.602 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.602 DEBUG:boto:Params: {} 2024-04-12T04:02:27.602 DEBUG:boto:Token: None 2024-04-12T04:02:27.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.602 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FpOXrKBWyM8nWkHOjxwgF3hPdyQ= 2024-04-12T04:02:27.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FpOXrKBWyM8nWkHOjxwgF3hPdyQ=', 'Content-Length': '0'} 2024-04-12T04:02:27.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff5692e11c3f7d8-006618b253-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.604 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.604 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.604 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.605 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.605 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.605 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.605 DEBUG:boto:Data: 2024-04-12T04:02:27.605 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.605 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.605 DEBUG:boto:Params: {} 2024-04-12T04:02:27.605 DEBUG:boto:Token: None 2024-04-12T04:02:27.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.606 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FpOXrKBWyM8nWkHOjxwgF3hPdyQ= 2024-04-12T04:02:27.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FpOXrKBWyM8nWkHOjxwgF3hPdyQ=', 'Content-Length': '0'} 2024-04-12T04:02:27.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b53152649d6e0598-006618b253-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:02:27.607 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.608 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.608 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.608 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.608 DEBUG:boto:Method: GET 2024-04-12T04:02:27.608 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.608 DEBUG:boto:Data: 2024-04-12T04:02:27.608 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.608 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.608 DEBUG:boto:Params: {} 2024-04-12T04:02:27.608 DEBUG:boto:Token: None 2024-04-12T04:02:27.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y= 2024-04-12T04:02:27.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y=', 'Content-Length': '0'} 2024-04-12T04:02:27.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc56e286070b6ed2-006618b253-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.610 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.611 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.611 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.611 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.611 DEBUG:boto:Method: GET 2024-04-12T04:02:27.611 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.611 DEBUG:boto:Data: 2024-04-12T04:02:27.611 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.611 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.611 DEBUG:boto:Params: {} 2024-04-12T04:02:27.612 DEBUG:boto:Token: None 2024-04-12T04:02:27.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:02:27.612 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y= 2024-04-12T04:02:27.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3MeBRkCATScypPmltOjAaIYxu9Y=', 'Content-Length': '0'} 2024-04-12T04:02:27.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84cd9f1d9f92ba-006618b253-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.614 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.614 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.614 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.614 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.614 DEBUG:boto:Method: GET 2024-04-12T04:02:27.615 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.615 DEBUG:boto:Data: 2024-04-12T04:02:27.615 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.615 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.615 DEBUG:boto:Params: {} 2024-04-12T04:02:27.615 DEBUG:boto:Token: None 2024-04-12T04:02:27.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.615 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo= 2024-04-12T04:02:27.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo=', 'Content-Length': '0'} 2024-04-12T04:02:27.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d89f68b230c0d5e-006618b253-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.658 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.659 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.659 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.659 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.659 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.659 DEBUG:boto:Method: GET 2024-04-12T04:02:27.659 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.659 DEBUG:boto:Data: 2024-04-12T04:02:27.660 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.660 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.660 DEBUG:boto:Params: {} 2024-04-12T04:02:27.660 DEBUG:boto:Token: None 2024-04-12T04:02:27.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.660 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo= 2024-04-12T04:02:27.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo=', 'Content-Length': '0'} 2024-04-12T04:02:27.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001738a86e349194e9-006618b253-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.663 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.663 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.663 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.663 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.663 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:02:27.663 DEBUG:boto:Data: 2024-04-12T04:02:27.664 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.664 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.664 DEBUG:boto:Params: {} 2024-04-12T04:02:27.664 DEBUG:boto:Token: None 2024-04-12T04:02:27.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy 2024-04-12T04:02:27.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O1RrXQscbB4qbVyO8RwR82zJhWo= 2024-04-12T04:02:27.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O1RrXQscbB4qbVyO8RwR82zJhWo=', 'Content-Length': '0'} 2024-04-12T04:02:27.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6daf27a4583173-006618b253-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.667 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.667 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.667 DEBUG:boto:Method: HEAD 2024-04-12T04:02:27.667 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:02:27.667 DEBUG:boto:Data: 2024-04-12T04:02:27.667 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.667 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.668 DEBUG:boto:Params: {} 2024-04-12T04:02:27.668 DEBUG:boto:Token: None 2024-04-12T04:02:27.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy 2024-04-12T04:02:27.668 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O1RrXQscbB4qbVyO8RwR82zJhWo= 2024-04-12T04:02:27.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O1RrXQscbB4qbVyO8RwR82zJhWo=', 'Content-Length': '0'} 2024-04-12T04:02:27.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc594c0e23be3614-006618b253-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:02:27.670 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.670 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.670 DEBUG:boto:Method: GET 2024-04-12T04:02:27.670 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:02:27.670 DEBUG:boto:Data: 2024-04-12T04:02:27.670 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.670 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.671 DEBUG:boto:Params: {} 2024-04-12T04:02:27.671 DEBUG:boto:Token: None 2024-04-12T04:02:27.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy 2024-04-12T04:02:27.671 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k1Jd/hZSUlW+1YpePodomVm50W0= 2024-04-12T04:02:27.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k1Jd/hZSUlW+1YpePodomVm50W0=', 'Content-Length': '0'} 2024-04-12T04:02:27.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ce1f6e45ae20ba-006618b253-2229-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.673 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:02:27.673 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:02:27.673 DEBUG:boto:Method: GET 2024-04-12T04:02:27.673 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:02:27.673 DEBUG:boto:Data: 2024-04-12T04:02:27.673 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.673 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.673 DEBUG:boto:Params: {} 2024-04-12T04:02:27.673 DEBUG:boto:Token: None 2024-04-12T04:02:27.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/dummy 2024-04-12T04:02:27.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:k1Jd/hZSUlW+1YpePodomVm50W0= 2024-04-12T04:02:27.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:k1Jd/hZSUlW+1YpePodomVm50W0=', 'Content-Length': '0'} 2024-04-12T04:02:27.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009faea72588073252-006618b253-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.722 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.722 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.722 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.722 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.723 DEBUG:boto:Method: GET 2024-04-12T04:02:27.723 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.723 DEBUG:boto:Data: 2024-04-12T04:02:27.723 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:27.723 DEBUG:boto:Port: 8001 2024-04-12T04:02:27.723 DEBUG:boto:Params: {} 2024-04-12T04:02:27.723 DEBUG:boto:Token: None 2024-04-12T04:02:27.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.724 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo= 2024-04-12T04:02:27.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo=', 'Content-Length': '0'} 2024-04-12T04:02:27.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6136f8f73b86c1d-006618b253-2229-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.726 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.727 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:27.727 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:27.727 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.727 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.727 DEBUG:boto:Method: GET 2024-04-12T04:02:27.727 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:02:27.727 DEBUG:boto:Data: 2024-04-12T04:02:27.727 DEBUG:boto:Headers: {} 2024-04-12T04:02:27.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:27.727 DEBUG:boto:Port: 8000 2024-04-12T04:02:27.728 DEBUG:boto:Params: {} 2024-04-12T04:02:27.728 DEBUG:boto:Token: None 2024-04-12T04:02:27.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:02:27 GMT /dgbpxg-64/ 2024-04-12T04:02:27.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo= 2024-04-12T04:02:27.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B4le9PiSBIvcy+29zAQdjUzOrAo=', 'Content-Length': '0'} 2024-04-12T04:02:27.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c2edc42a04252ed-006618b253-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:02:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:27.731 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:02:27.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-64 zones={a2, a1} 2024-04-12T04:02:27.732 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:02:27.732 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:02:27.733 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:02:27.747+0000 7faa2d101700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 110688) UID: 0 2024-04-12T04:02:27.733 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:02:27.747+0000 7faa322d67c0 -1 shutting down 2024-04-12T04:02:33.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:02:33.833 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:02:33.833 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:33.833 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:33.833 DEBUG:boto:Method: HEAD 2024-04-12T04:02:33.833 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:33.833 DEBUG:boto:Data: 2024-04-12T04:02:33.834 DEBUG:boto:Headers: {} 2024-04-12T04:02:33.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.834 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.834 DEBUG:boto:Params: {} 2024-04-12T04:02:33.834 DEBUG:boto:Token: None 2024-04-12T04:02:33.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/ 2024-04-12T04:02:33.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8= 2024-04-12T04:02:33.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8=', 'Content-Length': '0'} 2024-04-12T04:02:33.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b39ddea6b49496b4-006618b259-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.838 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:02:33.838 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:02:33.838 DEBUG:boto:Method: PUT 2024-04-12T04:02:33.838 DEBUG:boto:Path: /dgbpxg-64/obj11 2024-04-12T04:02:33.838 DEBUG:boto:Data: 2024-04-12T04:02:33.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:33.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.839 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.839 DEBUG:boto:Params: {} 2024-04-12T04:02:33.839 DEBUG:boto:Token: None 2024-04-12T04:02:33.839 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/obj11 2024-04-12T04:02:33.839 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qQqpfM+YKcExCUU1HNsOvS+GIOs= 2024-04-12T04:02:33.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qQqpfM+YKcExCUU1HNsOvS+GIOs='} 2024-04-12T04:02:33.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5ce580724b4d54a-006618b259-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.846 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:33.846 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:33.846 DEBUG:boto:Method: HEAD 2024-04-12T04:02:33.846 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:33.846 DEBUG:boto:Data: 2024-04-12T04:02:33.846 DEBUG:boto:Headers: {} 2024-04-12T04:02:33.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.846 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.847 DEBUG:boto:Params: {} 2024-04-12T04:02:33.847 DEBUG:boto:Token: None 2024-04-12T04:02:33.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/ 2024-04-12T04:02:33.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8= 2024-04-12T04:02:33.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8=', 'Content-Length': '0'} 2024-04-12T04:02:33.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020184b1e3576b4e3-006618b259-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.850 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:02:33.850 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:02:33.850 DEBUG:boto:Method: PUT 2024-04-12T04:02:33.850 DEBUG:boto:Path: /dgbpxg-64/obj22 2024-04-12T04:02:33.850 DEBUG:boto:Data: 2024-04-12T04:02:33.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:33.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.851 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.851 DEBUG:boto:Params: {} 2024-04-12T04:02:33.851 DEBUG:boto:Token: None 2024-04-12T04:02:33.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/obj22 2024-04-12T04:02:33.851 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kuMm+XV04dJUR9neE69P75bjsgo= 2024-04-12T04:02:33.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kuMm+XV04dJUR9neE69P75bjsgo='} 2024-04-12T04:02:33.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042ac1317d383d646-006618b259-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.856 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:33.856 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:33.856 DEBUG:boto:Method: HEAD 2024-04-12T04:02:33.856 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:33.856 DEBUG:boto:Data: 2024-04-12T04:02:33.856 DEBUG:boto:Headers: {} 2024-04-12T04:02:33.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.857 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.857 DEBUG:boto:Params: {} 2024-04-12T04:02:33.857 DEBUG:boto:Token: None 2024-04-12T04:02:33.857 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/ 2024-04-12T04:02:33.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8= 2024-04-12T04:02:33.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8=', 'Content-Length': '0'} 2024-04-12T04:02:33.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd17cce856bf03db-006618b259-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.860 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:02:33.860 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:02:33.860 DEBUG:boto:Method: PUT 2024-04-12T04:02:33.860 DEBUG:boto:Path: /dgbpxg-64/obj33 2024-04-12T04:02:33.860 DEBUG:boto:Data: 2024-04-12T04:02:33.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:33.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.861 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.861 DEBUG:boto:Params: {} 2024-04-12T04:02:33.861 DEBUG:boto:Token: None 2024-04-12T04:02:33.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/obj33 2024-04-12T04:02:33.861 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HTCBdug3kUkwLYic1NIvETyN454= 2024-04-12T04:02:33.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HTCBdug3kUkwLYic1NIvETyN454='} 2024-04-12T04:02:33.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c67eeb9c53444b2f-006618b259-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.907 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:33.907 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:33.907 DEBUG:boto:Method: HEAD 2024-04-12T04:02:33.907 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:33.907 DEBUG:boto:Data: 2024-04-12T04:02:33.907 DEBUG:boto:Headers: {} 2024-04-12T04:02:33.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.908 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.908 DEBUG:boto:Params: {} 2024-04-12T04:02:33.908 DEBUG:boto:Token: None 2024-04-12T04:02:33.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/ 2024-04-12T04:02:33.908 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8= 2024-04-12T04:02:33.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8=', 'Content-Length': '0'} 2024-04-12T04:02:33.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe57cef5744d4a7d-006618b259-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.911 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:02:33.911 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:02:33.911 DEBUG:boto:Method: PUT 2024-04-12T04:02:33.911 DEBUG:boto:Path: /dgbpxg-64/obj44 2024-04-12T04:02:33.911 DEBUG:boto:Data: 2024-04-12T04:02:33.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:33.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.912 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.912 DEBUG:boto:Params: {} 2024-04-12T04:02:33.912 DEBUG:boto:Token: None 2024-04-12T04:02:33.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/obj44 2024-04-12T04:02:33.912 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VszgtlJ738zMSGXsXIg9ycZbP3w= 2024-04-12T04:02:33.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VszgtlJ738zMSGXsXIg9ycZbP3w='} 2024-04-12T04:02:33.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a92a0f9b6c4e05c-006618b259-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.918 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:02:33.918 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:02:33.918 DEBUG:boto:Method: HEAD 2024-04-12T04:02:33.919 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:02:33.919 DEBUG:boto:Data: 2024-04-12T04:02:33.919 DEBUG:boto:Headers: {} 2024-04-12T04:02:33.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.919 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.919 DEBUG:boto:Params: {} 2024-04-12T04:02:33.919 DEBUG:boto:Token: None 2024-04-12T04:02:33.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/ 2024-04-12T04:02:33.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8= 2024-04-12T04:02:33.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jrVu5LTwX8q2/6Kz4jQXi2nHAP8=', 'Content-Length': '0'} 2024-04-12T04:02:33.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df6974898b21c38b-006618b259-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.922 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:02:33.922 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:02:33.922 DEBUG:boto:Method: PUT 2024-04-12T04:02:33.922 DEBUG:boto:Path: /dgbpxg-64/obj55 2024-04-12T04:02:33.922 DEBUG:boto:Data: 2024-04-12T04:02:33.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:02:33.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:02:33.923 DEBUG:boto:Port: 8000 2024-04-12T04:02:33.923 DEBUG:boto:Params: {} 2024-04-12T04:02:33.923 DEBUG:boto:Token: None 2024-04-12T04:02:33.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:02:33 GMT /dgbpxg-64/obj55 2024-04-12T04:02:33.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:MzAWjIQEi9VqZ+e8LHBkzA1nGYU= 2024-04-12T04:02:33.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:02:33 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzAWjIQEi9VqZ+e8LHBkzA1nGYU='} 2024-04-12T04:02:33.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1cbd03fd07e0820-006618b259-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:02:33.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:02:33.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:02:33.971 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:02:34.028 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:34.029 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:34.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:34.081+0000 7efe9b2f8780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:02:34.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:34.083+0000 7efe9b2f8780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:02:34.091 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:02:34.092 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:02:34.094 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:02:34.094 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:34.114 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:02:34.114 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:02:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:02:34.116 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:02:35.117 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:02:35.139 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:02:35.139 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:02:35.140 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:02:35.141 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T04:02:35.141 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-64 2024-04-12T04:02:35.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:35.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:35.142 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:02:35.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:35.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:35.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:35.287+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:35.274 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:35.274 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:02:39.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:02:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:40.289+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:40.275 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:40.275 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:02:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:45.290+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:45.276 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:45.276 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:02:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:50.291+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:50.278 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:50.278 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:02:55.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:02:55.293+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:55.279 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:02:55.279 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:00.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:00.294+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:00.280 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:00.280 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:05.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:05.295+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:05.281 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:05.281 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:10.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:10.296+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:10.282 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:10.282 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:15.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:15.297+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:15.284 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:15.284 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:20.298+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:20.285 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:20.285 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:25.300+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:25.286 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:25.286 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:30.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:30.301+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:30.287 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:30.287 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:35.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:35.303+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:35.288 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:35.288 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:40.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:40.304+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:40.289 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:40.290 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:45.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:45.305+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:45.291 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:45.291 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:50.306+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:50.292 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:50.292 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:03:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:03:55.307+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:55.294 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:03:55.294 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:00.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:00.308+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:00.295 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-12T04:04:00.295 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:05.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:05.310+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:05.296 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , , 00000000001.116.6] 2024-04-12T04:04:05.296 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:10.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:10.312+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:10.297 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:10.297 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:15.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:15.313+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:15.298 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:15.299 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:20.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:20.314+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:20.300 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:20.300 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:25.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:25.315+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:25.301 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:25.301 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:30.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:30.316+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:30.302 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:30.302 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:35.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:35.318+0000 7f351c67d780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:35.303 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , , 00000000002.112.6, 00000000001.116.6] 2024-04-12T04:04:35.303 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:40.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:40.318+0000 7f351c67d780 1 bucket sync caught up with source: 2024-04-12T04:04:40.305 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:40.305 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.98.6, , , 00000000001.73.6, 00000000002.112.6, 00000000002.118.6] 2024-04-12T04:04:40.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:40.318+0000 7f351c67d780 0 bucket checkpoint complete 2024-04-12T04:04:40.323 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-64 2024-04-12T04:04:40.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:40.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:40.324 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:04:40.385 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:40.385 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:40.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:40.437+0000 7f5da4709780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:40.423 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.2477.6, , , , 00000000002.3327.6, 00000000001.1432.6] 2024-04-12T04:04:40.423 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.2477.6, , , 00000000001.1620.6, 00000000002.3327.6, 00000000002.1434.6] 2024-04-12T04:04:45.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:45.438+0000 7f5da4709780 1 bucket sync caught up with source: 2024-04-12T04:04:45.424 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.2477.6, , , 00000000001.1620.6, 00000000002.3327.6, 00000000002.1434.6] 2024-04-12T04:04:45.424 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.2477.6, , , 00000000001.1620.6, 00000000002.3327.6, 00000000002.1434.6] 2024-04-12T04:04:45.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:45.438+0000 7f5da4709780 0 bucket checkpoint complete 2024-04-12T04:04:45.442 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-64 zones={a2, a1} 2024-04-12T04:04:45.442 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.443 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.443 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.443 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:04:45.443 DEBUG:boto:Data: 2024-04-12T04:04:45.443 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.443 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.443 DEBUG:boto:Params: {} 2024-04-12T04:04:45.444 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:04:45.444 DEBUG:boto:Token: None 2024-04-12T04:04:45.444 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.444 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:niD1N+AH7JifkY9ebtdjyXRdHlY= 2024-04-12T04:04:45.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:niD1N+AH7JifkY9ebtdjyXRdHlY=', 'Content-Length': '0'} 2024-04-12T04:04:45.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0432ed494a16d1-006618b2dd-2289-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.459 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.459 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.459 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.459 DEBUG:boto:Path: /dgbpxg-64/ 2024-04-12T04:04:45.459 DEBUG:boto:Data: 2024-04-12T04:04:45.459 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.459 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.459 DEBUG:boto:Params: {} 2024-04-12T04:04:45.460 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:04:45.460 DEBUG:boto:Token: None 2024-04-12T04:04:45.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.460 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:niD1N+AH7JifkY9ebtdjyXRdHlY= 2024-04-12T04:04:45.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:niD1N+AH7JifkY9ebtdjyXRdHlY=', 'Content-Length': '0'} 2024-04-12T04:04:45.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000418e95aae9cecc9f-006618b2dd-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:04:45.473 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.473 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.473 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.473 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.473 DEBUG:boto:Method: GET 2024-04-12T04:04:45.473 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.473 DEBUG:boto:Data: 2024-04-12T04:04:45.474 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.474 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.474 DEBUG:boto:Params: {} 2024-04-12T04:04:45.474 DEBUG:boto:Token: None 2024-04-12T04:04:45.474 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/?versions 2024-04-12T04:04:45.474 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ= 2024-04-12T04:04:45.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e573f032a79d1f-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.477 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:04:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:04:45.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:04:45.479 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.479 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.479 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.479 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.479 DEBUG:boto:Method: GET 2024-04-12T04:04:45.479 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.479 DEBUG:boto:Data: 2024-04-12T04:04:45.479 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.479 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.479 DEBUG:boto:Params: {} 2024-04-12T04:04:45.480 DEBUG:boto:Token: None 2024-04-12T04:04:45.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/?versions 2024-04-12T04:04:45.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ= 2024-04-12T04:04:45.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077414d77e3868f41-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.482 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:04:45.483 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.483 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.483 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.483 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.484 DEBUG:boto:Method: GET 2024-04-12T04:04:45.484 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.484 DEBUG:boto:Data: 2024-04-12T04:04:45.484 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.484 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.484 DEBUG:boto:Params: {} 2024-04-12T04:04:45.484 DEBUG:boto:Token: None 2024-04-12T04:04:45.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/?versions 2024-04-12T04:04:45.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ= 2024-04-12T04:04:45.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009035d4443b30871e-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.487 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.487 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.487 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.487 DEBUG:boto:path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.488 DEBUG:boto:auth_path=/dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.488 DEBUG:boto:Method: GET 2024-04-12T04:04:45.488 DEBUG:boto:Path: /dgbpxg-64/?versions&max-keys=999 2024-04-12T04:04:45.488 DEBUG:boto:Data: 2024-04-12T04:04:45.488 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.488 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.488 DEBUG:boto:Params: {} 2024-04-12T04:04:45.488 DEBUG:boto:Token: None 2024-04-12T04:04:45.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/?versions 2024-04-12T04:04:45.488 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ= 2024-04-12T04:04:45.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FzZ6o5NteAyyDctc79qJhs1JKWQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f4169dec2322da-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.491 DEBUG:boto:b'dgbpxg-64999falsedummynulltrue2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:04:45.492 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.492 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.492 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.492 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.492 DEBUG:boto:Method: GET 2024-04-12T04:04:45.492 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.492 DEBUG:boto:Data: 2024-04-12T04:04:45.492 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.493 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.493 DEBUG:boto:Params: {} 2024-04-12T04:04:45.493 DEBUG:boto:Token: None 2024-04-12T04:04:45.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs= 2024-04-12T04:04:45.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs=', 'Content-Length': '0'} 2024-04-12T04:04:45.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63a9a1d69d11142-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.495 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.495 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.495 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.495 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.495 DEBUG:boto:Method: GET 2024-04-12T04:04:45.495 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.496 DEBUG:boto:Data: 2024-04-12T04:04:45.496 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.496 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.496 DEBUG:boto:Params: {} 2024-04-12T04:04:45.496 DEBUG:boto:Token: None 2024-04-12T04:04:45.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.496 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs= 2024-04-12T04:04:45.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs=', 'Content-Length': '0'} 2024-04-12T04:04:45.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6de00b7568d98d7-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.498 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.498 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.498 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.499 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.499 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.499 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.499 DEBUG:boto:Data: 2024-04-12T04:04:45.499 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.499 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.499 DEBUG:boto:Params: {} 2024-04-12T04:04:45.499 DEBUG:boto:Token: None 2024-04-12T04:04:45.500 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.500 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3vmdhO+R9qBxVlksDKSqRXAQ5CI= 2024-04-12T04:04:45.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3vmdhO+R9qBxVlksDKSqRXAQ5CI=', 'Content-Length': '0'} 2024-04-12T04:04:45.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002852c2dc01a39440-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.501 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.502 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.502 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.502 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.502 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.502 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.502 DEBUG:boto:Data: 2024-04-12T04:04:45.502 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.502 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.502 DEBUG:boto:Params: {} 2024-04-12T04:04:45.503 DEBUG:boto:Token: None 2024-04-12T04:04:45.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3vmdhO+R9qBxVlksDKSqRXAQ5CI= 2024-04-12T04:04:45.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3vmdhO+R9qBxVlksDKSqRXAQ5CI=', 'Content-Length': '0'} 2024-04-12T04:04:45.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf18199f1216d7cf-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:04:45.505 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.505 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.505 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.505 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.505 DEBUG:boto:Method: GET 2024-04-12T04:04:45.505 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.505 DEBUG:boto:Data: 2024-04-12T04:04:45.505 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.506 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.506 DEBUG:boto:Params: {} 2024-04-12T04:04:45.506 DEBUG:boto:Token: None 2024-04-12T04:04:45.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.506 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs= 2024-04-12T04:04:45.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs=', 'Content-Length': '0'} 2024-04-12T04:04:45.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c51b2cc1e1cc3b9-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.554 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.555 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.555 DEBUG:boto:path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.555 DEBUG:boto:auth_path=/dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.555 DEBUG:boto:Method: GET 2024-04-12T04:04:45.555 DEBUG:boto:Path: /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.555 DEBUG:boto:Data: 2024-04-12T04:04:45.555 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.555 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.556 DEBUG:boto:Params: {} 2024-04-12T04:04:45.556 DEBUG:boto:Token: None 2024-04-12T04:04:45.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy?versionId=null 2024-04-12T04:04:45.556 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs= 2024-04-12T04:04:45.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lYqedFK8lIwFhYpmHZLXJAwdjLs=', 'Content-Length': '0'} 2024-04-12T04:04:45.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3ddf051d04b4ec-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.558 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.558 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.559 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.559 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.559 DEBUG:boto:Method: GET 2024-04-12T04:04:45.559 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.559 DEBUG:boto:Data: 2024-04-12T04:04:45.559 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.559 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.559 DEBUG:boto:Params: {} 2024-04-12T04:04:45.559 DEBUG:boto:Token: None 2024-04-12T04:04:45.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.560 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a2c402df5bf4d3-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.562 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.563 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.563 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.563 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.563 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.563 DEBUG:boto:Method: GET 2024-04-12T04:04:45.563 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.563 DEBUG:boto:Data: 2024-04-12T04:04:45.563 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.563 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.563 DEBUG:boto:Params: {} 2024-04-12T04:04:45.564 DEBUG:boto:Token: None 2024-04-12T04:04:45.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9be1825f76add8-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.567 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.567 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.567 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.567 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.567 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:04:45.568 DEBUG:boto:Data: 2024-04-12T04:04:45.568 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.568 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.568 DEBUG:boto:Params: {} 2024-04-12T04:04:45.568 DEBUG:boto:Token: None 2024-04-12T04:04:45.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy 2024-04-12T04:04:45.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s+RBQP3xuF7DOotZv4Mzt/bzdfs= 2024-04-12T04:04:45.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s+RBQP3xuF7DOotZv4Mzt/bzdfs=', 'Content-Length': '0'} 2024-04-12T04:04:45.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e847b4d7d65a409-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.570 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.570 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.570 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.570 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:04:45.571 DEBUG:boto:Data: 2024-04-12T04:04:45.571 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.571 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.571 DEBUG:boto:Params: {} 2024-04-12T04:04:45.571 DEBUG:boto:Token: None 2024-04-12T04:04:45.571 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy 2024-04-12T04:04:45.571 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s+RBQP3xuF7DOotZv4Mzt/bzdfs= 2024-04-12T04:04:45.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s+RBQP3xuF7DOotZv4Mzt/bzdfs=', 'Content-Length': '0'} 2024-04-12T04:04:45.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b3503b27e5129d-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:04:45.573 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.574 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.574 DEBUG:boto:Method: GET 2024-04-12T04:04:45.574 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:04:45.574 DEBUG:boto:Data: 2024-04-12T04:04:45.574 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.574 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.574 DEBUG:boto:Params: {} 2024-04-12T04:04:45.574 DEBUG:boto:Token: None 2024-04-12T04:04:45.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy 2024-04-12T04:04:45.575 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c8OlqgD/lSq+lF9VwoV/oSyiwsM= 2024-04-12T04:04:45.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c8OlqgD/lSq+lF9VwoV/oSyiwsM=', 'Content-Length': '0'} 2024-04-12T04:04:45.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c67cff9f5a9b1a-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.577 DEBUG:boto:path=/dgbpxg-64/dummy 2024-04-12T04:04:45.577 DEBUG:boto:auth_path=/dgbpxg-64/dummy 2024-04-12T04:04:45.577 DEBUG:boto:Method: GET 2024-04-12T04:04:45.577 DEBUG:boto:Path: /dgbpxg-64/dummy 2024-04-12T04:04:45.577 DEBUG:boto:Data: 2024-04-12T04:04:45.577 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.577 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.578 DEBUG:boto:Params: {} 2024-04-12T04:04:45.578 DEBUG:boto:Token: None 2024-04-12T04:04:45.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/dummy 2024-04-12T04:04:45.578 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c8OlqgD/lSq+lF9VwoV/oSyiwsM= 2024-04-12T04:04:45.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c8OlqgD/lSq+lF9VwoV/oSyiwsM=', 'Content-Length': '0'} 2024-04-12T04:04:45.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22b359d1cef9a53-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.622 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.622 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.623 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.623 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.623 DEBUG:boto:Method: GET 2024-04-12T04:04:45.623 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.623 DEBUG:boto:Data: 2024-04-12T04:04:45.623 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.623 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.623 DEBUG:boto:Params: {} 2024-04-12T04:04:45.623 DEBUG:boto:Token: None 2024-04-12T04:04:45.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.624 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a3f48a3587a09b-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.626 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.627 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.627 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.627 DEBUG:boto:path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.627 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.627 DEBUG:boto:Method: GET 2024-04-12T04:04:45.627 DEBUG:boto:Path: /dgbpxg-64/?prefix=dummy 2024-04-12T04:04:45.627 DEBUG:boto:Data: 2024-04-12T04:04:45.628 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.628 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.628 DEBUG:boto:Params: {} 2024-04-12T04:04:45.628 DEBUG:boto:Token: None 2024-04-12T04:04:45.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.628 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ca247b1f1c4122-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.631 DEBUG:boto:b'dgbpxg-64dummy1000falsedummy2024-04-12T04:02:22.355Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:45.631 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.631 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.631 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.631 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.632 DEBUG:boto:Method: GET 2024-04-12T04:04:45.632 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.632 DEBUG:boto:Data: 2024-04-12T04:04:45.632 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.632 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.632 DEBUG:boto:Params: {} 2024-04-12T04:04:45.632 DEBUG:boto:Token: None 2024-04-12T04:04:45.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.633 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw= 2024-04-12T04:04:45.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw=', 'Content-Length': '0'} 2024-04-12T04:04:45.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460c8f3c2cee69ca-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.635 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.635 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.635 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.635 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.636 DEBUG:boto:Method: GET 2024-04-12T04:04:45.636 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.636 DEBUG:boto:Data: 2024-04-12T04:04:45.636 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.636 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.636 DEBUG:boto:Params: {} 2024-04-12T04:04:45.636 DEBUG:boto:Token: None 2024-04-12T04:04:45.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.637 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw= 2024-04-12T04:04:45.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw=', 'Content-Length': '0'} 2024-04-12T04:04:45.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003defb86792a948e4-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.638 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.639 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.639 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.639 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.639 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.639 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.639 DEBUG:boto:Data: 2024-04-12T04:04:45.639 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.639 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.639 DEBUG:boto:Params: {} 2024-04-12T04:04:45.639 DEBUG:boto:Token: None 2024-04-12T04:04:45.640 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.640 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LnP9pGjGgj1qSaylUIW/H+zX6eI= 2024-04-12T04:04:45.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LnP9pGjGgj1qSaylUIW/H+zX6eI=', 'Content-Length': '0'} 2024-04-12T04:04:45.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd50c556ce5803a6-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.642 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.642 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.642 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.643 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.643 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.643 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.643 DEBUG:boto:Data: 2024-04-12T04:04:45.643 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.643 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.643 DEBUG:boto:Params: {} 2024-04-12T04:04:45.643 DEBUG:boto:Token: None 2024-04-12T04:04:45.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.643 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LnP9pGjGgj1qSaylUIW/H+zX6eI= 2024-04-12T04:04:45.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LnP9pGjGgj1qSaylUIW/H+zX6eI=', 'Content-Length': '0'} 2024-04-12T04:04:45.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d3d63ecfde1db35-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:45.645 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.645 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.646 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.646 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.646 DEBUG:boto:Method: GET 2024-04-12T04:04:45.646 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.646 DEBUG:boto:Data: 2024-04-12T04:04:45.646 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.646 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.646 DEBUG:boto:Params: {} 2024-04-12T04:04:45.646 DEBUG:boto:Token: None 2024-04-12T04:04:45.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.647 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw= 2024-04-12T04:04:45.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw=', 'Content-Length': '0'} 2024-04-12T04:04:45.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b3aed7e66975a1-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.690 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.690 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.691 DEBUG:boto:path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.691 DEBUG:boto:auth_path=/dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.691 DEBUG:boto:Method: GET 2024-04-12T04:04:45.691 DEBUG:boto:Path: /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.691 DEBUG:boto:Data: 2024-04-12T04:04:45.691 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.691 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.691 DEBUG:boto:Params: {} 2024-04-12T04:04:45.691 DEBUG:boto:Token: None 2024-04-12T04:04:45.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11?versionId=null 2024-04-12T04:04:45.692 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw= 2024-04-12T04:04:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3DO/s3D8to5tS0EeHkKHugxTmpw=', 'Content-Length': '0'} 2024-04-12T04:04:45.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a892431b92d6cb-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.694 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.694 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.694 DEBUG:boto:path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.694 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.694 DEBUG:boto:Method: GET 2024-04-12T04:04:45.694 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.694 DEBUG:boto:Data: 2024-04-12T04:04:45.694 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.695 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.695 DEBUG:boto:Params: {} 2024-04-12T04:04:45.695 DEBUG:boto:Token: None 2024-04-12T04:04:45.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.695 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000532435538fc28083-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.697 DEBUG:boto:b'dgbpxg-64obj111000falseobj112024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.698 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.698 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.698 DEBUG:boto:path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.698 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.698 DEBUG:boto:Method: GET 2024-04-12T04:04:45.698 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.698 DEBUG:boto:Data: 2024-04-12T04:04:45.698 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.698 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.698 DEBUG:boto:Params: {} 2024-04-12T04:04:45.698 DEBUG:boto:Token: None 2024-04-12T04:04:45.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.699 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a0536f4d0c3c9d-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.701 DEBUG:boto:b'dgbpxg-64obj111000falseobj112024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.701 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.701 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.701 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.702 DEBUG:boto:Path: /dgbpxg-64/obj11 2024-04-12T04:04:45.702 DEBUG:boto:Data: 2024-04-12T04:04:45.702 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.702 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.702 DEBUG:boto:Params: {} 2024-04-12T04:04:45.702 DEBUG:boto:Token: None 2024-04-12T04:04:45.702 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11 2024-04-12T04:04:45.702 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P79NfPIRYsA4CtlwGUAsD97zHRE= 2024-04-12T04:04:45.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P79NfPIRYsA4CtlwGUAsD97zHRE=', 'Content-Length': '0'} 2024-04-12T04:04:45.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca71b3b333d5eae2-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.705 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.705 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.705 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.705 DEBUG:boto:Path: /dgbpxg-64/obj11 2024-04-12T04:04:45.705 DEBUG:boto:Data: 2024-04-12T04:04:45.706 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.706 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.706 DEBUG:boto:Params: {} 2024-04-12T04:04:45.706 DEBUG:boto:Token: None 2024-04-12T04:04:45.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11 2024-04-12T04:04:45.707 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:P79NfPIRYsA4CtlwGUAsD97zHRE= 2024-04-12T04:04:45.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:P79NfPIRYsA4CtlwGUAsD97zHRE=', 'Content-Length': '0'} 2024-04-12T04:04:45.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002633dc4a14b835df-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:45.709 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.709 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.709 DEBUG:boto:Method: GET 2024-04-12T04:04:45.709 DEBUG:boto:Path: /dgbpxg-64/obj11 2024-04-12T04:04:45.709 DEBUG:boto:Data: 2024-04-12T04:04:45.709 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.709 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.709 DEBUG:boto:Params: {} 2024-04-12T04:04:45.709 DEBUG:boto:Token: None 2024-04-12T04:04:45.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11 2024-04-12T04:04:45.710 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r727Rlnwkl0UAusgRY729VnDcO0= 2024-04-12T04:04:45.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r727Rlnwkl0UAusgRY729VnDcO0=', 'Content-Length': '0'} 2024-04-12T04:04:45.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b7f27b744505ea-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.713 DEBUG:boto:path=/dgbpxg-64/obj11 2024-04-12T04:04:45.713 DEBUG:boto:auth_path=/dgbpxg-64/obj11 2024-04-12T04:04:45.713 DEBUG:boto:Method: GET 2024-04-12T04:04:45.713 DEBUG:boto:Path: /dgbpxg-64/obj11 2024-04-12T04:04:45.713 DEBUG:boto:Data: 2024-04-12T04:04:45.713 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.713 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.714 DEBUG:boto:Params: {} 2024-04-12T04:04:45.714 DEBUG:boto:Token: None 2024-04-12T04:04:45.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj11 2024-04-12T04:04:45.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r727Rlnwkl0UAusgRY729VnDcO0= 2024-04-12T04:04:45.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r727Rlnwkl0UAusgRY729VnDcO0=', 'Content-Length': '0'} 2024-04-12T04:04:45.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e51ae476fcb4781-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.716 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.716 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.716 DEBUG:boto:path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.717 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.717 DEBUG:boto:Method: GET 2024-04-12T04:04:45.717 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.717 DEBUG:boto:Data: 2024-04-12T04:04:45.717 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.717 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.717 DEBUG:boto:Params: {} 2024-04-12T04:04:45.717 DEBUG:boto:Token: None 2024-04-12T04:04:45.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.718 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e5be9150ede09e-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.720 DEBUG:boto:b'dgbpxg-64obj111000falseobj112024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.720 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.720 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.720 DEBUG:boto:path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.721 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.721 DEBUG:boto:Method: GET 2024-04-12T04:04:45.721 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj11 2024-04-12T04:04:45.721 DEBUG:boto:Data: 2024-04-12T04:04:45.721 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.721 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.721 DEBUG:boto:Params: {} 2024-04-12T04:04:45.721 DEBUG:boto:Token: None 2024-04-12T04:04:45.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.721 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d9d8a96cdac643-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.724 DEBUG:boto:b'dgbpxg-64obj111000falseobj112024-04-12T04:02:33.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:45.724 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.724 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.724 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.724 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.725 DEBUG:boto:Method: GET 2024-04-12T04:04:45.725 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.725 DEBUG:boto:Data: 2024-04-12T04:04:45.725 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.725 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.725 DEBUG:boto:Params: {} 2024-04-12T04:04:45.725 DEBUG:boto:Token: None 2024-04-12T04:04:45.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o= 2024-04-12T04:04:45.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o=', 'Content-Length': '0'} 2024-04-12T04:04:45.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f210cee492d2034-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.727 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.727 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.728 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.728 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.728 DEBUG:boto:Method: GET 2024-04-12T04:04:45.728 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.728 DEBUG:boto:Data: 2024-04-12T04:04:45.728 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.728 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.728 DEBUG:boto:Params: {} 2024-04-12T04:04:45.729 DEBUG:boto:Token: None 2024-04-12T04:04:45.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o= 2024-04-12T04:04:45.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o=', 'Content-Length': '0'} 2024-04-12T04:04:45.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000131b581e03ff27eb-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.731 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.731 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.731 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.731 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.731 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.731 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.731 DEBUG:boto:Data: 2024-04-12T04:04:45.732 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.732 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.732 DEBUG:boto:Params: {} 2024-04-12T04:04:45.732 DEBUG:boto:Token: None 2024-04-12T04:04:45.732 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wPh3pHk8k7ZJ4u54UEIckl4v2ws= 2024-04-12T04:04:45.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wPh3pHk8k7ZJ4u54UEIckl4v2ws=', 'Content-Length': '0'} 2024-04-12T04:04:45.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a919523e566a3a8f-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.734 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.734 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.734 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.734 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.734 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.734 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.735 DEBUG:boto:Data: 2024-04-12T04:04:45.735 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.735 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.735 DEBUG:boto:Params: {} 2024-04-12T04:04:45.735 DEBUG:boto:Token: None 2024-04-12T04:04:45.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.735 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wPh3pHk8k7ZJ4u54UEIckl4v2ws= 2024-04-12T04:04:45.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wPh3pHk8k7ZJ4u54UEIckl4v2ws=', 'Content-Length': '0'} 2024-04-12T04:04:45.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e792bfaf088da85-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:45.737 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.737 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.737 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.737 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.737 DEBUG:boto:Method: GET 2024-04-12T04:04:45.738 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.738 DEBUG:boto:Data: 2024-04-12T04:04:45.738 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.738 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.738 DEBUG:boto:Params: {} 2024-04-12T04:04:45.738 DEBUG:boto:Token: None 2024-04-12T04:04:45.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o= 2024-04-12T04:04:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o=', 'Content-Length': '0'} 2024-04-12T04:04:45.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed417cef97e0713a-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.740 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.740 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.740 DEBUG:boto:path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.741 DEBUG:boto:auth_path=/dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.741 DEBUG:boto:Method: GET 2024-04-12T04:04:45.741 DEBUG:boto:Path: /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.741 DEBUG:boto:Data: 2024-04-12T04:04:45.741 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.741 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.741 DEBUG:boto:Params: {} 2024-04-12T04:04:45.741 DEBUG:boto:Token: None 2024-04-12T04:04:45.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22?versionId=null 2024-04-12T04:04:45.742 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o= 2024-04-12T04:04:45.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NzI3emDaT2M+DMZOe8CZLkZrJ+o=', 'Content-Length': '0'} 2024-04-12T04:04:45.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c34aa918281f81be-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.744 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.744 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.744 DEBUG:boto:path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.744 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.744 DEBUG:boto:Method: GET 2024-04-12T04:04:45.744 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.744 DEBUG:boto:Data: 2024-04-12T04:04:45.744 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.744 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.745 DEBUG:boto:Params: {} 2024-04-12T04:04:45.745 DEBUG:boto:Token: None 2024-04-12T04:04:45.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.745 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab2306f3e4b19b43-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.747 DEBUG:boto:b'dgbpxg-64obj221000falseobj222024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.748 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.748 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.748 DEBUG:boto:path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.748 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.748 DEBUG:boto:Method: GET 2024-04-12T04:04:45.748 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.748 DEBUG:boto:Data: 2024-04-12T04:04:45.748 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.749 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.749 DEBUG:boto:Params: {} 2024-04-12T04:04:45.749 DEBUG:boto:Token: None 2024-04-12T04:04:45.749 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.749 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b1f52d73e8ebb2-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.798 DEBUG:boto:b'dgbpxg-64obj221000falseobj222024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.799 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.799 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.799 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.799 DEBUG:boto:Path: /dgbpxg-64/obj22 2024-04-12T04:04:45.799 DEBUG:boto:Data: 2024-04-12T04:04:45.799 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.799 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.799 DEBUG:boto:Params: {} 2024-04-12T04:04:45.800 DEBUG:boto:Token: None 2024-04-12T04:04:45.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22 2024-04-12T04:04:45.800 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UnKjZBXagz1AHOY/6HjoseII7K4= 2024-04-12T04:04:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UnKjZBXagz1AHOY/6HjoseII7K4=', 'Content-Length': '0'} 2024-04-12T04:04:45.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca60933beb5aaa9-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.802 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.802 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.802 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.802 DEBUG:boto:Path: /dgbpxg-64/obj22 2024-04-12T04:04:45.803 DEBUG:boto:Data: 2024-04-12T04:04:45.803 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.803 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.803 DEBUG:boto:Params: {} 2024-04-12T04:04:45.803 DEBUG:boto:Token: None 2024-04-12T04:04:45.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22 2024-04-12T04:04:45.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UnKjZBXagz1AHOY/6HjoseII7K4= 2024-04-12T04:04:45.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UnKjZBXagz1AHOY/6HjoseII7K4=', 'Content-Length': '0'} 2024-04-12T04:04:45.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5d9c55301210a8-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:45.805 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.805 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.805 DEBUG:boto:Method: GET 2024-04-12T04:04:45.805 DEBUG:boto:Path: /dgbpxg-64/obj22 2024-04-12T04:04:45.805 DEBUG:boto:Data: 2024-04-12T04:04:45.805 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.806 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.806 DEBUG:boto:Params: {} 2024-04-12T04:04:45.806 DEBUG:boto:Token: None 2024-04-12T04:04:45.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22 2024-04-12T04:04:45.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QCpbrE9VxEr3iI/k+dlG+syUq7c= 2024-04-12T04:04:45.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QCpbrE9VxEr3iI/k+dlG+syUq7c=', 'Content-Length': '0'} 2024-04-12T04:04:45.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d79d010caaef6269-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.808 DEBUG:boto:path=/dgbpxg-64/obj22 2024-04-12T04:04:45.808 DEBUG:boto:auth_path=/dgbpxg-64/obj22 2024-04-12T04:04:45.809 DEBUG:boto:Method: GET 2024-04-12T04:04:45.809 DEBUG:boto:Path: /dgbpxg-64/obj22 2024-04-12T04:04:45.809 DEBUG:boto:Data: 2024-04-12T04:04:45.809 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.809 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.809 DEBUG:boto:Params: {} 2024-04-12T04:04:45.809 DEBUG:boto:Token: None 2024-04-12T04:04:45.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj22 2024-04-12T04:04:45.810 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QCpbrE9VxEr3iI/k+dlG+syUq7c= 2024-04-12T04:04:45.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QCpbrE9VxEr3iI/k+dlG+syUq7c=', 'Content-Length': '0'} 2024-04-12T04:04:45.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09a55ee8699ab5f-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.812 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.812 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.812 DEBUG:boto:path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.812 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.812 DEBUG:boto:Method: GET 2024-04-12T04:04:45.812 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.813 DEBUG:boto:Data: 2024-04-12T04:04:45.813 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.813 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.813 DEBUG:boto:Params: {} 2024-04-12T04:04:45.813 DEBUG:boto:Token: None 2024-04-12T04:04:45.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.813 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3fffc616628530a-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.816 DEBUG:boto:b'dgbpxg-64obj221000falseobj222024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.816 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.816 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.816 DEBUG:boto:path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.816 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.817 DEBUG:boto:Method: GET 2024-04-12T04:04:45.817 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj22 2024-04-12T04:04:45.817 DEBUG:boto:Data: 2024-04-12T04:04:45.817 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.817 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.817 DEBUG:boto:Params: {} 2024-04-12T04:04:45.817 DEBUG:boto:Token: None 2024-04-12T04:04:45.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.817 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113986b16a8e1720-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.820 DEBUG:boto:b'dgbpxg-64obj221000falseobj222024-04-12T04:02:33.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:45.820 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.821 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.821 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.821 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.821 DEBUG:boto:Method: GET 2024-04-12T04:04:45.821 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.821 DEBUG:boto:Data: 2024-04-12T04:04:45.821 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.821 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.821 DEBUG:boto:Params: {} 2024-04-12T04:04:45.821 DEBUG:boto:Token: None 2024-04-12T04:04:45.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.822 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I= 2024-04-12T04:04:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I=', 'Content-Length': '0'} 2024-04-12T04:04:45.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3c9b0336db6f67-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.866 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.867 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.867 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.867 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.867 DEBUG:boto:Method: GET 2024-04-12T04:04:45.868 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.868 DEBUG:boto:Data: 2024-04-12T04:04:45.868 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.868 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.868 DEBUG:boto:Params: {} 2024-04-12T04:04:45.868 DEBUG:boto:Token: None 2024-04-12T04:04:45.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I= 2024-04-12T04:04:45.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I=', 'Content-Length': '0'} 2024-04-12T04:04:45.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9f638f4008ba35-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.870 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.871 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.871 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.871 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.871 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.871 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.871 DEBUG:boto:Data: 2024-04-12T04:04:45.871 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.871 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.871 DEBUG:boto:Params: {} 2024-04-12T04:04:45.872 DEBUG:boto:Token: None 2024-04-12T04:04:45.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.872 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tKyDz6tt3FrthwyQJUzC+xvZrcs= 2024-04-12T04:04:45.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tKyDz6tt3FrthwyQJUzC+xvZrcs=', 'Content-Length': '0'} 2024-04-12T04:04:45.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a8ce2d324df956-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.873 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.874 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.874 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.874 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.874 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.874 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.874 DEBUG:boto:Data: 2024-04-12T04:04:45.874 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.874 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.874 DEBUG:boto:Params: {} 2024-04-12T04:04:45.874 DEBUG:boto:Token: None 2024-04-12T04:04:45.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.875 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tKyDz6tt3FrthwyQJUzC+xvZrcs= 2024-04-12T04:04:45.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tKyDz6tt3FrthwyQJUzC+xvZrcs=', 'Content-Length': '0'} 2024-04-12T04:04:45.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003155d9feb15ef4f7-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:45.877 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.877 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.877 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.877 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.878 DEBUG:boto:Method: GET 2024-04-12T04:04:45.878 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.878 DEBUG:boto:Data: 2024-04-12T04:04:45.878 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.878 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.878 DEBUG:boto:Params: {} 2024-04-12T04:04:45.878 DEBUG:boto:Token: None 2024-04-12T04:04:45.878 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.878 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I= 2024-04-12T04:04:45.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I=', 'Content-Length': '0'} 2024-04-12T04:04:45.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a789b35be498a5a2-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.880 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.880 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.881 DEBUG:boto:path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.881 DEBUG:boto:auth_path=/dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.881 DEBUG:boto:Method: GET 2024-04-12T04:04:45.881 DEBUG:boto:Path: /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.881 DEBUG:boto:Data: 2024-04-12T04:04:45.881 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.881 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.881 DEBUG:boto:Params: {} 2024-04-12T04:04:45.881 DEBUG:boto:Token: None 2024-04-12T04:04:45.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33?versionId=null 2024-04-12T04:04:45.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I= 2024-04-12T04:04:45.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y96yB89fiXqqFL0tAZkRrRhuY3I=', 'Content-Length': '0'} 2024-04-12T04:04:45.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7cb56a56e2502e-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.884 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.884 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.884 DEBUG:boto:path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.884 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.884 DEBUG:boto:Method: GET 2024-04-12T04:04:45.884 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.884 DEBUG:boto:Data: 2024-04-12T04:04:45.884 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.885 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.885 DEBUG:boto:Params: {} 2024-04-12T04:04:45.885 DEBUG:boto:Token: None 2024-04-12T04:04:45.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.885 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082297635577b9b5f-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.887 DEBUG:boto:b'dgbpxg-64obj331000falseobj332024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.888 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.888 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.888 DEBUG:boto:path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.888 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.888 DEBUG:boto:Method: GET 2024-04-12T04:04:45.888 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.888 DEBUG:boto:Data: 2024-04-12T04:04:45.888 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.888 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.888 DEBUG:boto:Params: {} 2024-04-12T04:04:45.889 DEBUG:boto:Token: None 2024-04-12T04:04:45.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052cac947339edf0a-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.891 DEBUG:boto:b'dgbpxg-64obj331000falseobj332024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.891 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.892 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.892 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.892 DEBUG:boto:Path: /dgbpxg-64/obj33 2024-04-12T04:04:45.892 DEBUG:boto:Data: 2024-04-12T04:04:45.892 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.892 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.892 DEBUG:boto:Params: {} 2024-04-12T04:04:45.892 DEBUG:boto:Token: None 2024-04-12T04:04:45.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33 2024-04-12T04:04:45.893 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cGwNYEGtL1yh6hozSRh8sCwRSLQ= 2024-04-12T04:04:45.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cGwNYEGtL1yh6hozSRh8sCwRSLQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec2b98de93fab06-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.894 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.895 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.895 DEBUG:boto:Method: HEAD 2024-04-12T04:04:45.895 DEBUG:boto:Path: /dgbpxg-64/obj33 2024-04-12T04:04:45.895 DEBUG:boto:Data: 2024-04-12T04:04:45.895 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.895 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.895 DEBUG:boto:Params: {} 2024-04-12T04:04:45.895 DEBUG:boto:Token: None 2024-04-12T04:04:45.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33 2024-04-12T04:04:45.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cGwNYEGtL1yh6hozSRh8sCwRSLQ= 2024-04-12T04:04:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cGwNYEGtL1yh6hozSRh8sCwRSLQ=', 'Content-Length': '0'} 2024-04-12T04:04:45.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d768173c88037d0-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:45.898 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.898 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.898 DEBUG:boto:Method: GET 2024-04-12T04:04:45.898 DEBUG:boto:Path: /dgbpxg-64/obj33 2024-04-12T04:04:45.898 DEBUG:boto:Data: 2024-04-12T04:04:45.898 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.899 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.899 DEBUG:boto:Params: {} 2024-04-12T04:04:45.899 DEBUG:boto:Token: None 2024-04-12T04:04:45.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33 2024-04-12T04:04:45.899 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:no0QxDEJSQT2O4o/YoHy4KpJ0PE= 2024-04-12T04:04:45.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:no0QxDEJSQT2O4o/YoHy4KpJ0PE=', 'Content-Length': '0'} 2024-04-12T04:04:45.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076373e7770f0f68b-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.901 DEBUG:boto:path=/dgbpxg-64/obj33 2024-04-12T04:04:45.901 DEBUG:boto:auth_path=/dgbpxg-64/obj33 2024-04-12T04:04:45.901 DEBUG:boto:Method: GET 2024-04-12T04:04:45.901 DEBUG:boto:Path: /dgbpxg-64/obj33 2024-04-12T04:04:45.901 DEBUG:boto:Data: 2024-04-12T04:04:45.902 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.902 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.902 DEBUG:boto:Params: {} 2024-04-12T04:04:45.902 DEBUG:boto:Token: None 2024-04-12T04:04:45.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj33 2024-04-12T04:04:45.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:no0QxDEJSQT2O4o/YoHy4KpJ0PE= 2024-04-12T04:04:45.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:no0QxDEJSQT2O4o/YoHy4KpJ0PE=', 'Content-Length': '0'} 2024-04-12T04:04:45.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001511f44e47352174-006618b2dd-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.950 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.951 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.951 DEBUG:boto:path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.951 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.951 DEBUG:boto:Method: GET 2024-04-12T04:04:45.951 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.951 DEBUG:boto:Data: 2024-04-12T04:04:45.951 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.951 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.951 DEBUG:boto:Params: {} 2024-04-12T04:04:45.952 DEBUG:boto:Token: None 2024-04-12T04:04:45.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f176212f3f48138-006618b2dd-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.955 DEBUG:boto:b'dgbpxg-64obj331000falseobj332024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.955 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:45.955 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:45.955 DEBUG:boto:path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.955 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.955 DEBUG:boto:Method: GET 2024-04-12T04:04:45.955 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj33 2024-04-12T04:04:45.956 DEBUG:boto:Data: 2024-04-12T04:04:45.956 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:45.956 DEBUG:boto:Port: 8000 2024-04-12T04:04:45.956 DEBUG:boto:Params: {} 2024-04-12T04:04:45.956 DEBUG:boto:Token: None 2024-04-12T04:04:45.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/ 2024-04-12T04:04:45.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4= 2024-04-12T04:04:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rtQ1cL3FfqdYVKmOxeI61bGBhS4=', 'Content-Length': '0'} 2024-04-12T04:04:45.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b894e7590d70353f-006618b2dd-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:45.959 DEBUG:boto:b'dgbpxg-64obj331000falseobj332024-04-12T04:02:33.878Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:45.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:45.960 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:45.960 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:45.960 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:45.960 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:45.960 DEBUG:boto:Method: GET 2024-04-12T04:04:45.960 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:45.961 DEBUG:boto:Data: 2024-04-12T04:04:45.961 DEBUG:boto:Headers: {} 2024-04-12T04:04:45.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:45.961 DEBUG:boto:Port: 8001 2024-04-12T04:04:45.961 DEBUG:boto:Params: {} 2024-04-12T04:04:45.961 DEBUG:boto:Token: None 2024-04-12T04:04:45.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:45 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:45.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QcovgbOpvPFlL9gi771+GOSLte8= 2024-04-12T04:04:45.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:45 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QcovgbOpvPFlL9gi771+GOSLte8=', 'Content-Length': '0'} 2024-04-12T04:04:45.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000032759760c2847f-006618b2dd-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.006 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.006 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.007 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.007 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.007 DEBUG:boto:Method: GET 2024-04-12T04:04:46.007 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.007 DEBUG:boto:Data: 2024-04-12T04:04:46.007 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.007 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.008 DEBUG:boto:Params: {} 2024-04-12T04:04:46.008 DEBUG:boto:Token: None 2024-04-12T04:04:46.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg= 2024-04-12T04:04:46.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg=', 'Content-Length': '0'} 2024-04-12T04:04:46.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c438e23fd7d317-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.010 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.011 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.011 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.011 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.011 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.011 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.011 DEBUG:boto:Data: 2024-04-12T04:04:46.011 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.012 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.012 DEBUG:boto:Params: {} 2024-04-12T04:04:46.012 DEBUG:boto:Token: None 2024-04-12T04:04:46.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.012 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GHYBJLAEMNWTlemYlA71jOomWfA= 2024-04-12T04:04:46.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GHYBJLAEMNWTlemYlA71jOomWfA=', 'Content-Length': '0'} 2024-04-12T04:04:46.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d067a41546f8cea-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.015 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.015 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.015 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.015 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.015 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.015 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.015 DEBUG:boto:Data: 2024-04-12T04:04:46.015 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.016 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.016 DEBUG:boto:Params: {} 2024-04-12T04:04:46.016 DEBUG:boto:Token: None 2024-04-12T04:04:46.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.016 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GHYBJLAEMNWTlemYlA71jOomWfA= 2024-04-12T04:04:46.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GHYBJLAEMNWTlemYlA71jOomWfA=', 'Content-Length': '0'} 2024-04-12T04:04:46.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771aa92e124933b5-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:46.018 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.018 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.018 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.019 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.019 DEBUG:boto:Method: GET 2024-04-12T04:04:46.019 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.019 DEBUG:boto:Data: 2024-04-12T04:04:46.019 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.019 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.019 DEBUG:boto:Params: {} 2024-04-12T04:04:46.019 DEBUG:boto:Token: None 2024-04-12T04:04:46.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg= 2024-04-12T04:04:46.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg=', 'Content-Length': '0'} 2024-04-12T04:04:46.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000384322139dd89124-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.022 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.022 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.022 DEBUG:boto:path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.022 DEBUG:boto:auth_path=/dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.023 DEBUG:boto:Method: GET 2024-04-12T04:04:46.023 DEBUG:boto:Path: /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.023 DEBUG:boto:Data: 2024-04-12T04:04:46.023 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.023 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.023 DEBUG:boto:Params: {} 2024-04-12T04:04:46.023 DEBUG:boto:Token: None 2024-04-12T04:04:46.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44?versionId=null 2024-04-12T04:04:46.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg= 2024-04-12T04:04:46.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+Ln/IK7ebM4iL+TPkfikl5d4DZg=', 'Content-Length': '0'} 2024-04-12T04:04:46.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c1feb66f6b125dd-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.026 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.026 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.026 DEBUG:boto:path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.026 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.026 DEBUG:boto:Method: GET 2024-04-12T04:04:46.026 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.026 DEBUG:boto:Data: 2024-04-12T04:04:46.026 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.027 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.027 DEBUG:boto:Params: {} 2024-04-12T04:04:46.027 DEBUG:boto:Token: None 2024-04-12T04:04:46.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2be6a107b87dd0-006618b2de-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.029 DEBUG:boto:b'dgbpxg-64obj441000falseobj442024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.030 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.030 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.030 DEBUG:boto:path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.030 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.030 DEBUG:boto:Method: GET 2024-04-12T04:04:46.030 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.031 DEBUG:boto:Data: 2024-04-12T04:04:46.031 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.031 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.031 DEBUG:boto:Params: {} 2024-04-12T04:04:46.031 DEBUG:boto:Token: None 2024-04-12T04:04:46.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.031 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e40503587a4ed6b-006618b2de-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.034 DEBUG:boto:b'dgbpxg-64obj441000falseobj442024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.034 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.035 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.035 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.035 DEBUG:boto:Path: /dgbpxg-64/obj44 2024-04-12T04:04:46.035 DEBUG:boto:Data: 2024-04-12T04:04:46.035 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.035 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.036 DEBUG:boto:Params: {} 2024-04-12T04:04:46.036 DEBUG:boto:Token: None 2024-04-12T04:04:46.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44 2024-04-12T04:04:46.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B338xxFitmJJeaCGlzF2BmbWccY= 2024-04-12T04:04:46.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B338xxFitmJJeaCGlzF2BmbWccY=', 'Content-Length': '0'} 2024-04-12T04:04:46.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac0027e3cc833a0-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.038 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.038 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.038 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.038 DEBUG:boto:Path: /dgbpxg-64/obj44 2024-04-12T04:04:46.038 DEBUG:boto:Data: 2024-04-12T04:04:46.038 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.038 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.039 DEBUG:boto:Params: {} 2024-04-12T04:04:46.039 DEBUG:boto:Token: None 2024-04-12T04:04:46.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44 2024-04-12T04:04:46.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B338xxFitmJJeaCGlzF2BmbWccY= 2024-04-12T04:04:46.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B338xxFitmJJeaCGlzF2BmbWccY=', 'Content-Length': '0'} 2024-04-12T04:04:46.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628a61d34c10ed4f-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:46.041 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.041 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.041 DEBUG:boto:Method: GET 2024-04-12T04:04:46.042 DEBUG:boto:Path: /dgbpxg-64/obj44 2024-04-12T04:04:46.042 DEBUG:boto:Data: 2024-04-12T04:04:46.042 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.042 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.042 DEBUG:boto:Params: {} 2024-04-12T04:04:46.042 DEBUG:boto:Token: None 2024-04-12T04:04:46.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44 2024-04-12T04:04:46.043 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3jwZwLR17doPFhFOBcpq7vuO7BI= 2024-04-12T04:04:46.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3jwZwLR17doPFhFOBcpq7vuO7BI=', 'Content-Length': '0'} 2024-04-12T04:04:46.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2fb9e35bee99a1-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.045 DEBUG:boto:path=/dgbpxg-64/obj44 2024-04-12T04:04:46.045 DEBUG:boto:auth_path=/dgbpxg-64/obj44 2024-04-12T04:04:46.045 DEBUG:boto:Method: GET 2024-04-12T04:04:46.045 DEBUG:boto:Path: /dgbpxg-64/obj44 2024-04-12T04:04:46.045 DEBUG:boto:Data: 2024-04-12T04:04:46.046 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.046 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.046 DEBUG:boto:Params: {} 2024-04-12T04:04:46.046 DEBUG:boto:Token: None 2024-04-12T04:04:46.046 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj44 2024-04-12T04:04:46.046 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3jwZwLR17doPFhFOBcpq7vuO7BI= 2024-04-12T04:04:46.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3jwZwLR17doPFhFOBcpq7vuO7BI=', 'Content-Length': '0'} 2024-04-12T04:04:46.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b749799b962ba2-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.049 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.049 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.049 DEBUG:boto:path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.049 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.049 DEBUG:boto:Method: GET 2024-04-12T04:04:46.049 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.049 DEBUG:boto:Data: 2024-04-12T04:04:46.049 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.050 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.050 DEBUG:boto:Params: {} 2024-04-12T04:04:46.050 DEBUG:boto:Token: None 2024-04-12T04:04:46.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.050 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc9b8fcebfb4c16-006618b2de-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.053 DEBUG:boto:b'dgbpxg-64obj441000falseobj442024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.053 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.054 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.054 DEBUG:boto:path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.054 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.054 DEBUG:boto:Method: GET 2024-04-12T04:04:46.055 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj44 2024-04-12T04:04:46.055 DEBUG:boto:Data: 2024-04-12T04:04:46.055 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.055 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.056 DEBUG:boto:Params: {} 2024-04-12T04:04:46.056 DEBUG:boto:Token: None 2024-04-12T04:04:46.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.056 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c508b07c66e5b8a-006618b2de-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.059 DEBUG:boto:b'dgbpxg-64obj441000falseobj442024-04-12T04:02:33.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:04:46.060 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.060 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.060 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.060 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.061 DEBUG:boto:Method: GET 2024-04-12T04:04:46.061 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.061 DEBUG:boto:Data: 2024-04-12T04:04:46.061 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.061 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.061 DEBUG:boto:Params: {} 2024-04-12T04:04:46.062 DEBUG:boto:Token: None 2024-04-12T04:04:46.062 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.062 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E= 2024-04-12T04:04:46.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E=', 'Content-Length': '0'} 2024-04-12T04:04:46.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c64c5a84648c82b-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.064 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.064 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.065 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.065 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.065 DEBUG:boto:Method: GET 2024-04-12T04:04:46.065 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.065 DEBUG:boto:Data: 2024-04-12T04:04:46.065 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.065 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.065 DEBUG:boto:Params: {} 2024-04-12T04:04:46.066 DEBUG:boto:Token: None 2024-04-12T04:04:46.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.066 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E= 2024-04-12T04:04:46.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E=', 'Content-Length': '0'} 2024-04-12T04:04:46.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000926a2a78a66e520e-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.068 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.068 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.069 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.069 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.069 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.069 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.069 DEBUG:boto:Data: 2024-04-12T04:04:46.069 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.069 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.069 DEBUG:boto:Params: {} 2024-04-12T04:04:46.069 DEBUG:boto:Token: None 2024-04-12T04:04:46.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.070 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7zqAROHZfYWFv7FpZxzrv7NCDQU= 2024-04-12T04:04:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7zqAROHZfYWFv7FpZxzrv7NCDQU=', 'Content-Length': '0'} 2024-04-12T04:04:46.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c97ceab32d7499-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.072 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.072 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.072 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.072 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.072 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.072 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.072 DEBUG:boto:Data: 2024-04-12T04:04:46.072 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.073 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.073 DEBUG:boto:Params: {} 2024-04-12T04:04:46.073 DEBUG:boto:Token: None 2024-04-12T04:04:46.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.073 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7zqAROHZfYWFv7FpZxzrv7NCDQU= 2024-04-12T04:04:46.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7zqAROHZfYWFv7FpZxzrv7NCDQU=', 'Content-Length': '0'} 2024-04-12T04:04:46.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5528021a195382-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:04:46.075 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.076 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.076 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.076 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.076 DEBUG:boto:Method: GET 2024-04-12T04:04:46.076 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.076 DEBUG:boto:Data: 2024-04-12T04:04:46.076 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.077 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.077 DEBUG:boto:Params: {} 2024-04-12T04:04:46.077 DEBUG:boto:Token: None 2024-04-12T04:04:46.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.078 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E= 2024-04-12T04:04:46.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E=', 'Content-Length': '0'} 2024-04-12T04:04:46.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5c7f0467bd59b9-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.080 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.080 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.080 DEBUG:boto:path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.081 DEBUG:boto:auth_path=/dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.081 DEBUG:boto:Method: GET 2024-04-12T04:04:46.081 DEBUG:boto:Path: /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.081 DEBUG:boto:Data: 2024-04-12T04:04:46.082 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.082 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.082 DEBUG:boto:Params: {} 2024-04-12T04:04:46.083 DEBUG:boto:Token: None 2024-04-12T04:04:46.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55?versionId=null 2024-04-12T04:04:46.083 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E= 2024-04-12T04:04:46.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CwaF2JmdBy+MTdoLQHOOHx3X34E=', 'Content-Length': '0'} 2024-04-12T04:04:46.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b41705c856a00b5f-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.126 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.127 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.127 DEBUG:boto:path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.127 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.127 DEBUG:boto:Method: GET 2024-04-12T04:04:46.127 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.127 DEBUG:boto:Data: 2024-04-12T04:04:46.127 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.127 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.127 DEBUG:boto:Params: {} 2024-04-12T04:04:46.128 DEBUG:boto:Token: None 2024-04-12T04:04:46.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.128 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000623a4ce23a338c53-006618b2de-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.130 DEBUG:boto:b'dgbpxg-64obj551000falseobj552024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.131 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.131 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.131 DEBUG:boto:path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.131 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.131 DEBUG:boto:Method: GET 2024-04-12T04:04:46.131 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.131 DEBUG:boto:Data: 2024-04-12T04:04:46.131 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.132 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.132 DEBUG:boto:Params: {} 2024-04-12T04:04:46.132 DEBUG:boto:Token: None 2024-04-12T04:04:46.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.132 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff3338ccfb1d605-006618b2de-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.134 DEBUG:boto:b'dgbpxg-64obj551000falseobj552024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.135 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.135 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.135 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.135 DEBUG:boto:Path: /dgbpxg-64/obj55 2024-04-12T04:04:46.135 DEBUG:boto:Data: 2024-04-12T04:04:46.135 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.135 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.135 DEBUG:boto:Params: {} 2024-04-12T04:04:46.135 DEBUG:boto:Token: None 2024-04-12T04:04:46.136 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55 2024-04-12T04:04:46.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tiCXQNk+iTSoHjEDsmHuyGdeXAo= 2024-04-12T04:04:46.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tiCXQNk+iTSoHjEDsmHuyGdeXAo=', 'Content-Length': '0'} 2024-04-12T04:04:46.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042724918b071f555-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.137 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.138 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.138 DEBUG:boto:Method: HEAD 2024-04-12T04:04:46.138 DEBUG:boto:Path: /dgbpxg-64/obj55 2024-04-12T04:04:46.138 DEBUG:boto:Data: 2024-04-12T04:04:46.138 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.138 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.138 DEBUG:boto:Params: {} 2024-04-12T04:04:46.138 DEBUG:boto:Token: None 2024-04-12T04:04:46.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55 2024-04-12T04:04:46.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tiCXQNk+iTSoHjEDsmHuyGdeXAo= 2024-04-12T04:04:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tiCXQNk+iTSoHjEDsmHuyGdeXAo=', 'Content-Length': '0'} 2024-04-12T04:04:46.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4ea3d7209313f1-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:04:46.140 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.141 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.141 DEBUG:boto:Method: GET 2024-04-12T04:04:46.141 DEBUG:boto:Path: /dgbpxg-64/obj55 2024-04-12T04:04:46.141 DEBUG:boto:Data: 2024-04-12T04:04:46.141 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.141 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.141 DEBUG:boto:Params: {} 2024-04-12T04:04:46.141 DEBUG:boto:Token: None 2024-04-12T04:04:46.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55 2024-04-12T04:04:46.142 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0SVGQXnWWCFYKxfduAuWjN8669g= 2024-04-12T04:04:46.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0SVGQXnWWCFYKxfduAuWjN8669g=', 'Content-Length': '0'} 2024-04-12T04:04:46.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-64:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47eec26ff8bf4c9-006618b2de-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.143 DEBUG:boto:path=/dgbpxg-64/obj55 2024-04-12T04:04:46.144 DEBUG:boto:auth_path=/dgbpxg-64/obj55 2024-04-12T04:04:46.144 DEBUG:boto:Method: GET 2024-04-12T04:04:46.144 DEBUG:boto:Path: /dgbpxg-64/obj55 2024-04-12T04:04:46.144 DEBUG:boto:Data: 2024-04-12T04:04:46.144 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.144 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.144 DEBUG:boto:Params: {} 2024-04-12T04:04:46.144 DEBUG:boto:Token: None 2024-04-12T04:04:46.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/obj55 2024-04-12T04:04:46.145 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0SVGQXnWWCFYKxfduAuWjN8669g= 2024-04-12T04:04:46.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0SVGQXnWWCFYKxfduAuWjN8669g=', 'Content-Length': '0'} 2024-04-12T04:04:46.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000739f69fbff531aa-006618b2de-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.146 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.146 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.147 DEBUG:boto:path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.147 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.147 DEBUG:boto:Method: GET 2024-04-12T04:04:46.147 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.147 DEBUG:boto:Data: 2024-04-12T04:04:46.147 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:04:46.147 DEBUG:boto:Port: 8001 2024-04-12T04:04:46.147 DEBUG:boto:Params: {} 2024-04-12T04:04:46.147 DEBUG:boto:Token: None 2024-04-12T04:04:46.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba696613d1bbab2-006618b2de-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.190 DEBUG:boto:b'dgbpxg-64obj551000falseobj552024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.191 DEBUG:boto:path=/dgbpxg-64/ 2024-04-12T04:04:46.191 DEBUG:boto:auth_path=/dgbpxg-64/ 2024-04-12T04:04:46.191 DEBUG:boto:path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.191 DEBUG:boto:auth_path=/dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.191 DEBUG:boto:Method: GET 2024-04-12T04:04:46.191 DEBUG:boto:Path: /dgbpxg-64/?prefix=obj55 2024-04-12T04:04:46.191 DEBUG:boto:Data: 2024-04-12T04:04:46.191 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.191 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.191 DEBUG:boto:Params: {} 2024-04-12T04:04:46.192 DEBUG:boto:Token: None 2024-04-12T04:04:46.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-64/ 2024-04-12T04:04:46.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8= 2024-04-12T04:04:46.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gn07dH6uFOwq2CKEIlJA0/4dJN8=', 'Content-Length': '0'} 2024-04-12T04:04:46.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c832e531ea0e001-006618b2de-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.194 DEBUG:boto:b'dgbpxg-64obj551000falseobj552024-04-12T04:02:33.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:46.195 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-64 zones={a2, a1} 2024-04-12T04:04:46.195 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T04:04:46.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.195 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.300 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034609", 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:02:35.137367Z" 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000023988", 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T03:58:55.133471Z" 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000039644", 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:02:33.932283Z" 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000035398", 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:02:35.138938Z" 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.302 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:04:46.303 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000034609', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039644', 3: '00000000000000000000:00000000000000035398'} 2024-04-12T04:04:46.303 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:04:46.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.304 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.417 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:04:46.417 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 9381735839232845819 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.418 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000034609", 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:02:35.137367Z" 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000023988", 2024-04-12T04:04:46.419 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T03:58:55.133471Z" 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.420 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000039644", 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:02:33.932283Z" 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000035398", 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:02:35.138938Z" 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.422 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:04:46.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2024-04-12T04:04:46.423 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2024-04-12T04:04:46.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.423 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:04:46Z" 2024-04-12T04:04:46.423 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:04:46.424 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9381735839232845819 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:02:35.137367Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023988", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:55.133471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000039644", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2024-04-12T04:02:33.932283Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000035398", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:02:35.138938Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2024-04-12T04:04:46Z" } 2024-04-12T04:04:46.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:02:35.137367Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023988', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:55.133471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000039644', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2024-04-12T04:02:33.932283Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035398', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:02:35.138938Z'}}] 2024-04-12T04:04:46.424 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034609', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039644', 3: '00000000000000000000:00000000000000035398'} 2024-04-12T04:04:46.424 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034609', 1: '00000000000000000000:00000000000000023988', 2: '00000000000000000000:00000000000000039644', 3: '00000000000000000000:00000000000000035398'} 2024-04-12T04:04:46.424 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:04:46.424 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T04:04:46.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.425 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012808", 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:04:39.598689Z" 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006432", 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:00:31.103654Z" 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016293", 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:04:39.598738Z" 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016860", 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:04:24.242052Z" 2024-04-12T04:04:46.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.524 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:04:46.524 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000012808', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016293', 3: '00000000000000000000:00000000000000016860'} 2024-04-12T04:04:46.525 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:04:46.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.619 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:04:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:04:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:04:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:04:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012808", 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:04:39.598689Z" 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006432", 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:00:31.103654Z" 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016293", 2024-04-12T04:04:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:04:39.598738Z" 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016860", 2024-04-12T04:04:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:04:24.242052Z" 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T04:04:46.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.625 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:04:46Z" 2024-04-12T04:04:46.625 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:04:46.626 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012808", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:04:39.598689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006432", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:00:31.103654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016293", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:04:39.598738Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016860", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:04:24.242052Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:04:46Z" } 2024-04-12T04:04:46.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012808', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:39.598689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006432', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:00:31.103654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016293', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:39.598738Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016860', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:24.242052Z'}}] 2024-04-12T04:04:46.626 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012808', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016293', 3: '00000000000000000000:00000000000000016860'} 2024-04-12T04:04:46.626 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012808', 1: '00000000000000000000:00000000000000006432', 2: '00000000000000000000:00000000000000016293', 3: '00000000000000000000:00000000000000016860'} 2024-04-12T04:04:46.626 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:04:46.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2024-04-12T04:04:46.628 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:46.628 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:46.628 DEBUG:boto:Method: PUT 2024-04-12T04:04:46.628 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:46.628 DEBUG:boto:Data: 2024-04-12T04:04:46.628 DEBUG:boto:Headers: {} 2024-04-12T04:04:46.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:46.628 DEBUG:boto:Port: 8000 2024-04-12T04:04:46.628 DEBUG:boto:Params: {} 2024-04-12T04:04:46.629 DEBUG:boto:Token: None 2024-04-12T04:04:46.629 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:04:46 GMT /dgbpxg-65/ 2024-04-12T04:04:46.629 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+pwJqKgH2A49Zu+jTTlafJW2t1I= 2024-04-12T04:04:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+pwJqKgH2A49Zu+jTTlafJW2t1I=', 'Content-Length': '0'} 2024-04-12T04:04:46.644 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040c1de4f6ccdf8da-006618b2de-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:46.644 DEBUG:rgw_multi.tests:created bucket=dgbpxg-65 2024-04-12T04:04:46.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.645 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.723 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894686.658536_1657.1", 2024-04-12T04:04:46.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:04:46.658536Z" 2024-04-12T04:04:46.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.739 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:04:46.739 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:04:46.740 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:04:46.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:46.740 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:46.831+0000 7f704f586780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:04:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:46.832+0000 7f704f586780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:04:46.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:46.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:04:46.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894686.652046_1654.1", 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:46.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:04:46.652046Z", 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:46.836 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:04:46Z" 2024-04-12T04:04:46.837 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:04:46.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894686.652046_1654.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:04:46.652046Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:04:46Z" } 2024-04-12T04:04:46.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894686.652046_1654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:46.652046Z', 'realm_epoch': 2}}] 2024-04-12T04:04:46.838 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:04:46.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.652046_1654.1'} 2024-04-12T04:04:46.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894686.658536_1657.1 target=1_1712894686.652046_1654.1 2024-04-12T04:04:51.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:51.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:04:51.839 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:04:51.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:51.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:51.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:51.943+0000 7f725bf50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:04:51.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:51.944+0000 7f725bf50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:04:51.943 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:51.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:51.945 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:04:51.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712894686.658536_1657.1", 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:04:46.658536Z", 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:04:51.948 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:04:51Z" 2024-04-12T04:04:51.949 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:04:51.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894686.658536_1657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:04:46.658536Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:04:51Z" } 2024-04-12T04:04:51.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894686.658536_1657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:46.658536Z', 'realm_epoch': 2}}] 2024-04-12T04:04:51.950 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:04:51.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:04:51.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:04:51.951 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:51.951 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:51.951 DEBUG:boto:Method: HEAD 2024-04-12T04:04:51.951 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:51.951 DEBUG:boto:Data: 2024-04-12T04:04:51.951 DEBUG:boto:Headers: {} 2024-04-12T04:04:51.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:51.952 DEBUG:boto:Port: 8000 2024-04-12T04:04:51.952 DEBUG:boto:Params: {} 2024-04-12T04:04:51.952 DEBUG:boto:Token: None 2024-04-12T04:04:51.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:51 GMT /dgbpxg-65/ 2024-04-12T04:04:51.952 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F6E90/xgIepmHKXSRD0IrbyiUrQ= 2024-04-12T04:04:51.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F6E90/xgIepmHKXSRD0IrbyiUrQ=', 'Content-Length': '0'} 2024-04-12T04:04:51.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039f42862b13f145c-006618b2e3-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:51.955 DEBUG:boto:path=/dgbpxg-65/obj11 2024-04-12T04:04:51.955 DEBUG:boto:auth_path=/dgbpxg-65/obj11 2024-04-12T04:04:51.955 DEBUG:boto:Method: PUT 2024-04-12T04:04:51.955 DEBUG:boto:Path: /dgbpxg-65/obj11 2024-04-12T04:04:51.955 DEBUG:boto:Data: 2024-04-12T04:04:51.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:51.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:51.955 DEBUG:boto:Port: 8000 2024-04-12T04:04:51.955 DEBUG:boto:Params: {} 2024-04-12T04:04:51.956 DEBUG:boto:Token: None 2024-04-12T04:04:51.956 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:51 GMT /dgbpxg-65/obj11 2024-04-12T04:04:51.956 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qboygkJZjuKKzRG0PFw3WOIcyTQ= 2024-04-12T04:04:51.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qboygkJZjuKKzRG0PFw3WOIcyTQ='} 2024-04-12T04:04:51.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5f0b8ed863a6916-006618b2e3-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:51.999 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:51.999 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:51.999 DEBUG:boto:Method: HEAD 2024-04-12T04:04:51.999 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:51.999 DEBUG:boto:Data: 2024-04-12T04:04:51.999 DEBUG:boto:Headers: {} 2024-04-12T04:04:51.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:51.999 DEBUG:boto:Port: 8000 2024-04-12T04:04:51.999 DEBUG:boto:Params: {} 2024-04-12T04:04:52.000 DEBUG:boto:Token: None 2024-04-12T04:04:52.000 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.000 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027a9eb146a6a1d68-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.003 DEBUG:boto:path=/dgbpxg-65/obj22 2024-04-12T04:04:52.003 DEBUG:boto:auth_path=/dgbpxg-65/obj22 2024-04-12T04:04:52.003 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.003 DEBUG:boto:Path: /dgbpxg-65/obj22 2024-04-12T04:04:52.003 DEBUG:boto:Data: 2024-04-12T04:04:52.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.003 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.003 DEBUG:boto:Params: {} 2024-04-12T04:04:52.003 DEBUG:boto:Token: None 2024-04-12T04:04:52.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj22 2024-04-12T04:04:52.004 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7QiQ2AKLx7IowlBU3DPbIZThc5Q= 2024-04-12T04:04:52.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7QiQ2AKLx7IowlBU3DPbIZThc5Q='} 2024-04-12T04:04:52.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3f0fb2c54342cb8-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.008 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.008 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.008 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.008 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.008 DEBUG:boto:Data: 2024-04-12T04:04:52.008 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.009 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.009 DEBUG:boto:Params: {} 2024-04-12T04:04:52.009 DEBUG:boto:Token: None 2024-04-12T04:04:52.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.009 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064898fa2f4459d7c-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.012 DEBUG:boto:path=/dgbpxg-65/obj33 2024-04-12T04:04:52.012 DEBUG:boto:auth_path=/dgbpxg-65/obj33 2024-04-12T04:04:52.012 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.012 DEBUG:boto:Path: /dgbpxg-65/obj33 2024-04-12T04:04:52.012 DEBUG:boto:Data: 2024-04-12T04:04:52.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.012 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.013 DEBUG:boto:Params: {} 2024-04-12T04:04:52.013 DEBUG:boto:Token: None 2024-04-12T04:04:52.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj33 2024-04-12T04:04:52.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FGuzPy1CkygeKG6w8Ym1hmbOres= 2024-04-12T04:04:52.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGuzPy1CkygeKG6w8Ym1hmbOres='} 2024-04-12T04:04:52.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026fb33a15008fad1-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.019 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.019 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.019 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.019 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.019 DEBUG:boto:Data: 2024-04-12T04:04:52.020 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.020 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.020 DEBUG:boto:Params: {} 2024-04-12T04:04:52.020 DEBUG:boto:Token: None 2024-04-12T04:04:52.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022dba9c27b5e40c6-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.023 DEBUG:boto:path=/dgbpxg-65/obj44 2024-04-12T04:04:52.023 DEBUG:boto:auth_path=/dgbpxg-65/obj44 2024-04-12T04:04:52.023 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.023 DEBUG:boto:Path: /dgbpxg-65/obj44 2024-04-12T04:04:52.023 DEBUG:boto:Data: 2024-04-12T04:04:52.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.023 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.023 DEBUG:boto:Params: {} 2024-04-12T04:04:52.023 DEBUG:boto:Token: None 2024-04-12T04:04:52.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj44 2024-04-12T04:04:52.024 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:G2ODJxrpz5g7b8ADSUzwHsjuQko= 2024-04-12T04:04:52.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G2ODJxrpz5g7b8ADSUzwHsjuQko='} 2024-04-12T04:04:52.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003038dbfc09023f9a-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.071 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.071 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.071 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.071 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.071 DEBUG:boto:Data: 2024-04-12T04:04:52.071 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.071 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.072 DEBUG:boto:Params: {} 2024-04-12T04:04:52.072 DEBUG:boto:Token: None 2024-04-12T04:04:52.072 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b08537ab3208058-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.074 DEBUG:boto:path=/dgbpxg-65/obj55 2024-04-12T04:04:52.075 DEBUG:boto:auth_path=/dgbpxg-65/obj55 2024-04-12T04:04:52.075 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.075 DEBUG:boto:Path: /dgbpxg-65/obj55 2024-04-12T04:04:52.075 DEBUG:boto:Data: 2024-04-12T04:04:52.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.075 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.075 DEBUG:boto:Params: {} 2024-04-12T04:04:52.075 DEBUG:boto:Token: None 2024-04-12T04:04:52.075 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj55 2024-04-12T04:04:52.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f3XgLyakhzgaT6JBZj0NC47Rm2c= 2024-04-12T04:04:52.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f3XgLyakhzgaT6JBZj0NC47Rm2c='} 2024-04-12T04:04:52.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4b6f8147dd98db1-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.080 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:52.174 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:52.174 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:52.178 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2024-04-12T04:04:52.218 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.218 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.218 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.218 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.218 DEBUG:boto:Data: 2024-04-12T04:04:52.219 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.219 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.219 DEBUG:boto:Params: {} 2024-04-12T04:04:52.219 DEBUG:boto:Token: None 2024-04-12T04:04:52.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000199d8105808d7803-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.222 DEBUG:boto:path=/dgbpxg-65/obj17 2024-04-12T04:04:52.223 DEBUG:boto:auth_path=/dgbpxg-65/obj17 2024-04-12T04:04:52.223 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.223 DEBUG:boto:Path: /dgbpxg-65/obj17 2024-04-12T04:04:52.223 DEBUG:boto:Data: 2024-04-12T04:04:52.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.223 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.223 DEBUG:boto:Params: {} 2024-04-12T04:04:52.223 DEBUG:boto:Token: None 2024-04-12T04:04:52.223 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj17 2024-04-12T04:04:52.224 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JxUuLhg+QyS5Ina7w81CoR6qP1o= 2024-04-12T04:04:52.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JxUuLhg+QyS5Ina7w81CoR6qP1o='} 2024-04-12T04:04:52.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a79c5d4776f14648-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.229 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.229 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.229 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.229 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.229 DEBUG:boto:Data: 2024-04-12T04:04:52.229 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.229 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.229 DEBUG:boto:Params: {} 2024-04-12T04:04:52.230 DEBUG:boto:Token: None 2024-04-12T04:04:52.230 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.230 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006efadaea728b015d-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.232 DEBUG:boto:path=/dgbpxg-65/obj34 2024-04-12T04:04:52.233 DEBUG:boto:auth_path=/dgbpxg-65/obj34 2024-04-12T04:04:52.233 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.233 DEBUG:boto:Path: /dgbpxg-65/obj34 2024-04-12T04:04:52.233 DEBUG:boto:Data: 2024-04-12T04:04:52.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.233 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.233 DEBUG:boto:Params: {} 2024-04-12T04:04:52.233 DEBUG:boto:Token: None 2024-04-12T04:04:52.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj34 2024-04-12T04:04:52.234 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n1ImiBzxzgKU4KMRoOrFJEkgceY= 2024-04-12T04:04:52.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n1ImiBzxzgKU4KMRoOrFJEkgceY='} 2024-04-12T04:04:52.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8f72bb4d2d3c0c5-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.283 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.283 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.283 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.283 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.283 DEBUG:boto:Data: 2024-04-12T04:04:52.283 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.284 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.284 DEBUG:boto:Params: {} 2024-04-12T04:04:52.284 DEBUG:boto:Token: None 2024-04-12T04:04:52.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.284 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006c745dfee5c1b5a-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.287 DEBUG:boto:path=/dgbpxg-65/obj51 2024-04-12T04:04:52.287 DEBUG:boto:auth_path=/dgbpxg-65/obj51 2024-04-12T04:04:52.287 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.287 DEBUG:boto:Path: /dgbpxg-65/obj51 2024-04-12T04:04:52.287 DEBUG:boto:Data: 2024-04-12T04:04:52.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.288 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.288 DEBUG:boto:Params: {} 2024-04-12T04:04:52.288 DEBUG:boto:Token: None 2024-04-12T04:04:52.288 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj51 2024-04-12T04:04:52.288 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xYocXufHz78vLXu6wTvI/Juqbk0= 2024-04-12T04:04:52.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xYocXufHz78vLXu6wTvI/Juqbk0='} 2024-04-12T04:04:52.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007565682bd3a06161-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.293 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.293 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.293 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.293 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.293 DEBUG:boto:Data: 2024-04-12T04:04:52.294 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.294 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.294 DEBUG:boto:Params: {} 2024-04-12T04:04:52.294 DEBUG:boto:Token: None 2024-04-12T04:04:52.294 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e81d75642c52bf-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.297 DEBUG:boto:path=/dgbpxg-65/obj68 2024-04-12T04:04:52.297 DEBUG:boto:auth_path=/dgbpxg-65/obj68 2024-04-12T04:04:52.297 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.297 DEBUG:boto:Path: /dgbpxg-65/obj68 2024-04-12T04:04:52.297 DEBUG:boto:Data: 2024-04-12T04:04:52.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.298 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.298 DEBUG:boto:Params: {} 2024-04-12T04:04:52.298 DEBUG:boto:Token: None 2024-04-12T04:04:52.298 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj68 2024-04-12T04:04:52.298 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/s3kSLEgxd5f35ZR7PIRZqVwL/o= 2024-04-12T04:04:52.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/s3kSLEgxd5f35ZR7PIRZqVwL/o='} 2024-04-12T04:04:52.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac50f57e3d092e5c-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.347 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.347 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.347 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.347 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.347 DEBUG:boto:Data: 2024-04-12T04:04:52.347 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.347 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.347 DEBUG:boto:Params: {} 2024-04-12T04:04:52.348 DEBUG:boto:Token: None 2024-04-12T04:04:52.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.348 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18b3714af70c070-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.351 DEBUG:boto:path=/dgbpxg-65/obj85 2024-04-12T04:04:52.351 DEBUG:boto:auth_path=/dgbpxg-65/obj85 2024-04-12T04:04:52.351 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.351 DEBUG:boto:Path: /dgbpxg-65/obj85 2024-04-12T04:04:52.351 DEBUG:boto:Data: 2024-04-12T04:04:52.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.352 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.352 DEBUG:boto:Params: {} 2024-04-12T04:04:52.352 DEBUG:boto:Token: None 2024-04-12T04:04:52.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj85 2024-04-12T04:04:52.353 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:s9wjo+ZgUIdJooKalphEPYFLZJw= 2024-04-12T04:04:52.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s9wjo+ZgUIdJooKalphEPYFLZJw='} 2024-04-12T04:04:52.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcadd799bf97935d-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:52.452 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:52.452 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:52.458 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2024-04-12T04:04:52.502 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.502 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.502 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.502 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.502 DEBUG:boto:Data: 2024-04-12T04:04:52.503 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.503 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.503 DEBUG:boto:Params: {} 2024-04-12T04:04:52.503 DEBUG:boto:Token: None 2024-04-12T04:04:52.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.503 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003054a51048efdd24-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.507 DEBUG:boto:path=/dgbpxg-65/obj19 2024-04-12T04:04:52.507 DEBUG:boto:auth_path=/dgbpxg-65/obj19 2024-04-12T04:04:52.507 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.507 DEBUG:boto:Path: /dgbpxg-65/obj19 2024-04-12T04:04:52.507 DEBUG:boto:Data: 2024-04-12T04:04:52.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.507 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.507 DEBUG:boto:Params: {} 2024-04-12T04:04:52.508 DEBUG:boto:Token: None 2024-04-12T04:04:52.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj19 2024-04-12T04:04:52.508 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+xsSAieSYp7k0hdCP6bm6rda8eE= 2024-04-12T04:04:52.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+xsSAieSYp7k0hdCP6bm6rda8eE='} 2024-04-12T04:04:52.513 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efe98f2fddccbc19-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.513 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.513 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.513 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.513 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.513 DEBUG:boto:Data: 2024-04-12T04:04:52.514 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.514 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.514 DEBUG:boto:Params: {} 2024-04-12T04:04:52.514 DEBUG:boto:Token: None 2024-04-12T04:04:52.514 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.514 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d440918b6527df07-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.517 DEBUG:boto:path=/dgbpxg-65/obj38 2024-04-12T04:04:52.517 DEBUG:boto:auth_path=/dgbpxg-65/obj38 2024-04-12T04:04:52.517 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.518 DEBUG:boto:Path: /dgbpxg-65/obj38 2024-04-12T04:04:52.518 DEBUG:boto:Data: 2024-04-12T04:04:52.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.518 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.518 DEBUG:boto:Params: {} 2024-04-12T04:04:52.518 DEBUG:boto:Token: None 2024-04-12T04:04:52.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj38 2024-04-12T04:04:52.518 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:eGeYJZWZpOvf/XrU7Rt6mUqqs18= 2024-04-12T04:04:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eGeYJZWZpOvf/XrU7Rt6mUqqs18='} 2024-04-12T04:04:52.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d1456659017ed6f-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.563 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.563 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.563 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.563 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.563 DEBUG:boto:Data: 2024-04-12T04:04:52.563 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.563 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.564 DEBUG:boto:Params: {} 2024-04-12T04:04:52.564 DEBUG:boto:Token: None 2024-04-12T04:04:52.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.564 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c0a2a76aeb0dc0-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.567 DEBUG:boto:path=/dgbpxg-65/obj57 2024-04-12T04:04:52.567 DEBUG:boto:auth_path=/dgbpxg-65/obj57 2024-04-12T04:04:52.567 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.567 DEBUG:boto:Path: /dgbpxg-65/obj57 2024-04-12T04:04:52.567 DEBUG:boto:Data: 2024-04-12T04:04:52.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.567 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.567 DEBUG:boto:Params: {} 2024-04-12T04:04:52.568 DEBUG:boto:Token: None 2024-04-12T04:04:52.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj57 2024-04-12T04:04:52.568 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uPebCRJTfRLMM3m02/3+ooYgFoA= 2024-04-12T04:04:52.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uPebCRJTfRLMM3m02/3+ooYgFoA='} 2024-04-12T04:04:52.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc67baed8aa65f98-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.572 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.572 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.572 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.572 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.572 DEBUG:boto:Data: 2024-04-12T04:04:52.572 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.572 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.572 DEBUG:boto:Params: {} 2024-04-12T04:04:52.573 DEBUG:boto:Token: None 2024-04-12T04:04:52.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.573 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006c0165cbef814cf-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.576 DEBUG:boto:path=/dgbpxg-65/obj76 2024-04-12T04:04:52.576 DEBUG:boto:auth_path=/dgbpxg-65/obj76 2024-04-12T04:04:52.576 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.576 DEBUG:boto:Path: /dgbpxg-65/obj76 2024-04-12T04:04:52.576 DEBUG:boto:Data: 2024-04-12T04:04:52.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.576 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.576 DEBUG:boto:Params: {} 2024-04-12T04:04:52.576 DEBUG:boto:Token: None 2024-04-12T04:04:52.576 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj76 2024-04-12T04:04:52.577 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:S6q8Fg1S7zqLpZHH+Et1nZ2x4rI= 2024-04-12T04:04:52.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S6q8Fg1S7zqLpZHH+Et1nZ2x4rI='} 2024-04-12T04:04:52.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e042cb5f81646046-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.619 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.619 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.619 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.619 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.619 DEBUG:boto:Data: 2024-04-12T04:04:52.619 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.620 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.620 DEBUG:boto:Params: {} 2024-04-12T04:04:52.620 DEBUG:boto:Token: None 2024-04-12T04:04:52.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.620 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a6b58e6f186341c-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.623 DEBUG:boto:path=/dgbpxg-65/obj95 2024-04-12T04:04:52.623 DEBUG:boto:auth_path=/dgbpxg-65/obj95 2024-04-12T04:04:52.623 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.623 DEBUG:boto:Path: /dgbpxg-65/obj95 2024-04-12T04:04:52.623 DEBUG:boto:Data: 2024-04-12T04:04:52.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.624 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.624 DEBUG:boto:Params: {} 2024-04-12T04:04:52.624 DEBUG:boto:Token: None 2024-04-12T04:04:52.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj95 2024-04-12T04:04:52.625 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UK5oxe3mnH8OhM76MsEEToqD/DU= 2024-04-12T04:04:52.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UK5oxe3mnH8OhM76MsEEToqD/DU='} 2024-04-12T04:04:52.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b62c3c6d4bdc70a-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.629 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:52.682 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:52.683 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:52.728 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:52.729 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:52.736 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2024-04-12T04:04:52.783 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.783 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.783 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.783 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.783 DEBUG:boto:Data: 2024-04-12T04:04:52.783 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.784 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.784 DEBUG:boto:Params: {} 2024-04-12T04:04:52.784 DEBUG:boto:Token: None 2024-04-12T04:04:52.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a8df35890906708-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.787 DEBUG:boto:path=/dgbpxg-65/obj23 2024-04-12T04:04:52.787 DEBUG:boto:auth_path=/dgbpxg-65/obj23 2024-04-12T04:04:52.787 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.788 DEBUG:boto:Path: /dgbpxg-65/obj23 2024-04-12T04:04:52.788 DEBUG:boto:Data: 2024-04-12T04:04:52.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.788 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.788 DEBUG:boto:Params: {} 2024-04-12T04:04:52.788 DEBUG:boto:Token: None 2024-04-12T04:04:52.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj23 2024-04-12T04:04:52.789 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zTI6adg9Ny9Tikr66uMcYLZVg58= 2024-04-12T04:04:52.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zTI6adg9Ny9Tikr66uMcYLZVg58='} 2024-04-12T04:04:52.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af31bbae976d4cd5-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.831 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.831 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.831 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.831 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.831 DEBUG:boto:Data: 2024-04-12T04:04:52.831 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.831 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.832 DEBUG:boto:Params: {} 2024-04-12T04:04:52.832 DEBUG:boto:Token: None 2024-04-12T04:04:52.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.832 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb94f18a10aad133-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.835 DEBUG:boto:path=/dgbpxg-65/obj46 2024-04-12T04:04:52.835 DEBUG:boto:auth_path=/dgbpxg-65/obj46 2024-04-12T04:04:52.835 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.835 DEBUG:boto:Path: /dgbpxg-65/obj46 2024-04-12T04:04:52.835 DEBUG:boto:Data: 2024-04-12T04:04:52.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.836 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.836 DEBUG:boto:Params: {} 2024-04-12T04:04:52.836 DEBUG:boto:Token: None 2024-04-12T04:04:52.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj46 2024-04-12T04:04:52.836 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vF1Yo0NtkxcPLTIyGaGBnx2uhPI= 2024-04-12T04:04:52.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vF1Yo0NtkxcPLTIyGaGBnx2uhPI='} 2024-04-12T04:04:52.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e29a109457232e6-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.841 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.841 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.842 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.842 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.842 DEBUG:boto:Data: 2024-04-12T04:04:52.842 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.842 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.842 DEBUG:boto:Params: {} 2024-04-12T04:04:52.842 DEBUG:boto:Token: None 2024-04-12T04:04:52.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988c5ca5ebfe2200-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.845 DEBUG:boto:path=/dgbpxg-65/obj69 2024-04-12T04:04:52.845 DEBUG:boto:auth_path=/dgbpxg-65/obj69 2024-04-12T04:04:52.846 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.846 DEBUG:boto:Path: /dgbpxg-65/obj69 2024-04-12T04:04:52.846 DEBUG:boto:Data: 2024-04-12T04:04:52.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.846 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.846 DEBUG:boto:Params: {} 2024-04-12T04:04:52.846 DEBUG:boto:Token: None 2024-04-12T04:04:52.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj69 2024-04-12T04:04:52.846 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:u+SQGhReEcoPSYFOLCdFmFfY/BM= 2024-04-12T04:04:52.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+SQGhReEcoPSYFOLCdFmFfY/BM='} 2024-04-12T04:04:52.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adf8a3e485c00d30-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.891 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.891 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.891 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.891 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.891 DEBUG:boto:Data: 2024-04-12T04:04:52.891 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.891 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.891 DEBUG:boto:Params: {} 2024-04-12T04:04:52.892 DEBUG:boto:Token: None 2024-04-12T04:04:52.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051bc2837e8f9a29f-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.895 DEBUG:boto:path=/dgbpxg-65/obj92 2024-04-12T04:04:52.895 DEBUG:boto:auth_path=/dgbpxg-65/obj92 2024-04-12T04:04:52.895 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.895 DEBUG:boto:Path: /dgbpxg-65/obj92 2024-04-12T04:04:52.895 DEBUG:boto:Data: 2024-04-12T04:04:52.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.895 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.895 DEBUG:boto:Params: {} 2024-04-12T04:04:52.896 DEBUG:boto:Token: None 2024-04-12T04:04:52.896 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj92 2024-04-12T04:04:52.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VZC3yQQcqNzeD8IISOkB3XkNIx0= 2024-04-12T04:04:52.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VZC3yQQcqNzeD8IISOkB3XkNIx0='} 2024-04-12T04:04:52.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083518037aa2dc8c8-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.901 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:52.901 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:52.901 DEBUG:boto:Method: HEAD 2024-04-12T04:04:52.901 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:52.901 DEBUG:boto:Data: 2024-04-12T04:04:52.901 DEBUG:boto:Headers: {} 2024-04-12T04:04:52.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.902 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.902 DEBUG:boto:Params: {} 2024-04-12T04:04:52.902 DEBUG:boto:Token: None 2024-04-12T04:04:52.902 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/ 2024-04-12T04:04:52.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:04:52.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:04:52.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514e636f1d8be514-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.905 DEBUG:boto:path=/dgbpxg-65/obj115 2024-04-12T04:04:52.905 DEBUG:boto:auth_path=/dgbpxg-65/obj115 2024-04-12T04:04:52.905 DEBUG:boto:Method: PUT 2024-04-12T04:04:52.905 DEBUG:boto:Path: /dgbpxg-65/obj115 2024-04-12T04:04:52.905 DEBUG:boto:Data: 2024-04-12T04:04:52.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:52.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:52.906 DEBUG:boto:Port: 8000 2024-04-12T04:04:52.906 DEBUG:boto:Params: {} 2024-04-12T04:04:52.906 DEBUG:boto:Token: None 2024-04-12T04:04:52.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:52 GMT /dgbpxg-65/obj115 2024-04-12T04:04:52.906 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:X4dtaPKLzQuemllOa+EOUE7aRXk= 2024-04-12T04:04:52.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X4dtaPKLzQuemllOa+EOUE7aRXk='} 2024-04-12T04:04:52.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5339bc5a9a4d4d3-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:52.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:52.955 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:53.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:53.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:53.053 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:53.054 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:53.061 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2024-04-12T04:04:53.113 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.113 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.113 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.113 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.113 DEBUG:boto:Data: 2024-04-12T04:04:53.113 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.113 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.114 DEBUG:boto:Params: {} 2024-04-12T04:04:53.114 DEBUG:boto:Token: None 2024-04-12T04:04:53.114 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.114 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7cd0fc8905df9c5-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.117 DEBUG:boto:path=/dgbpxg-65/obj29 2024-04-12T04:04:53.118 DEBUG:boto:auth_path=/dgbpxg-65/obj29 2024-04-12T04:04:53.118 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.118 DEBUG:boto:Path: /dgbpxg-65/obj29 2024-04-12T04:04:53.118 DEBUG:boto:Data: 2024-04-12T04:04:53.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.118 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.118 DEBUG:boto:Params: {} 2024-04-12T04:04:53.118 DEBUG:boto:Token: None 2024-04-12T04:04:53.119 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj29 2024-04-12T04:04:53.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hdusGen5xw+ealOK06A1YdI1l34= 2024-04-12T04:04:53.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hdusGen5xw+ealOK06A1YdI1l34='} 2024-04-12T04:04:53.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c41e2c14215a39a-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.124 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.124 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.124 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.125 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.125 DEBUG:boto:Data: 2024-04-12T04:04:53.125 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.125 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.125 DEBUG:boto:Params: {} 2024-04-12T04:04:53.125 DEBUG:boto:Token: None 2024-04-12T04:04:53.125 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.125 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f326a8cd11b6298-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.128 DEBUG:boto:path=/dgbpxg-65/obj58 2024-04-12T04:04:53.129 DEBUG:boto:auth_path=/dgbpxg-65/obj58 2024-04-12T04:04:53.129 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.129 DEBUG:boto:Path: /dgbpxg-65/obj58 2024-04-12T04:04:53.129 DEBUG:boto:Data: 2024-04-12T04:04:53.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.129 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.129 DEBUG:boto:Params: {} 2024-04-12T04:04:53.129 DEBUG:boto:Token: None 2024-04-12T04:04:53.129 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj58 2024-04-12T04:04:53.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JGH6giSCLJaoeheodg+DjyFZsV0= 2024-04-12T04:04:53.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JGH6giSCLJaoeheodg+DjyFZsV0='} 2024-04-12T04:04:53.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d32bbe65ed16a494-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.135 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.135 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.135 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.135 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.135 DEBUG:boto:Data: 2024-04-12T04:04:53.135 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.135 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.135 DEBUG:boto:Params: {} 2024-04-12T04:04:53.135 DEBUG:boto:Token: None 2024-04-12T04:04:53.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fe0cf43210b844d-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.138 DEBUG:boto:path=/dgbpxg-65/obj87 2024-04-12T04:04:53.138 DEBUG:boto:auth_path=/dgbpxg-65/obj87 2024-04-12T04:04:53.139 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.139 DEBUG:boto:Path: /dgbpxg-65/obj87 2024-04-12T04:04:53.139 DEBUG:boto:Data: 2024-04-12T04:04:53.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.139 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.139 DEBUG:boto:Params: {} 2024-04-12T04:04:53.139 DEBUG:boto:Token: None 2024-04-12T04:04:53.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj87 2024-04-12T04:04:53.139 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iGMBzuo2V87hUSw0xSXN9RVZheE= 2024-04-12T04:04:53.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGMBzuo2V87hUSw0xSXN9RVZheE='} 2024-04-12T04:04:53.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028852dbfdce8e4b6-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.187 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.187 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.187 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.187 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.187 DEBUG:boto:Data: 2024-04-12T04:04:53.187 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.187 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.187 DEBUG:boto:Params: {} 2024-04-12T04:04:53.188 DEBUG:boto:Token: None 2024-04-12T04:04:53.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd3373c4c520b954-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.191 DEBUG:boto:path=/dgbpxg-65/obj116 2024-04-12T04:04:53.191 DEBUG:boto:auth_path=/dgbpxg-65/obj116 2024-04-12T04:04:53.191 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.191 DEBUG:boto:Path: /dgbpxg-65/obj116 2024-04-12T04:04:53.191 DEBUG:boto:Data: 2024-04-12T04:04:53.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.192 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.192 DEBUG:boto:Params: {} 2024-04-12T04:04:53.192 DEBUG:boto:Token: None 2024-04-12T04:04:53.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj116 2024-04-12T04:04:53.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ga4KeekOtdsXAbicbjfySnx7a68= 2024-04-12T04:04:53.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga4KeekOtdsXAbicbjfySnx7a68='} 2024-04-12T04:04:53.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbd12025edea1e89-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.198 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.198 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.198 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.198 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.198 DEBUG:boto:Data: 2024-04-12T04:04:53.198 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.198 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.199 DEBUG:boto:Params: {} 2024-04-12T04:04:53.199 DEBUG:boto:Token: None 2024-04-12T04:04:53.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079949862aef07a92-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.202 DEBUG:boto:path=/dgbpxg-65/obj145 2024-04-12T04:04:53.202 DEBUG:boto:auth_path=/dgbpxg-65/obj145 2024-04-12T04:04:53.202 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.202 DEBUG:boto:Path: /dgbpxg-65/obj145 2024-04-12T04:04:53.203 DEBUG:boto:Data: 2024-04-12T04:04:53.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.203 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.203 DEBUG:boto:Params: {} 2024-04-12T04:04:53.203 DEBUG:boto:Token: None 2024-04-12T04:04:53.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj145 2024-04-12T04:04:53.203 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7yYVD+SvTSqgRwh45lZ4khvi5e4= 2024-04-12T04:04:53.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7yYVD+SvTSqgRwh45lZ4khvi5e4='} 2024-04-12T04:04:53.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db8613aa90d56072-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:53.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:53.247 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:53.347 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:53.347 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:53.357 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2024-04-12T04:04:53.415 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.415 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.415 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.415 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.415 DEBUG:boto:Data: 2024-04-12T04:04:53.415 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.415 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.416 DEBUG:boto:Params: {} 2024-04-12T04:04:53.416 DEBUG:boto:Token: None 2024-04-12T04:04:53.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c19feb32cd982b31-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.419 DEBUG:boto:path=/dgbpxg-65/obj31 2024-04-12T04:04:53.420 DEBUG:boto:auth_path=/dgbpxg-65/obj31 2024-04-12T04:04:53.420 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.420 DEBUG:boto:Path: /dgbpxg-65/obj31 2024-04-12T04:04:53.420 DEBUG:boto:Data: 2024-04-12T04:04:53.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.420 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.420 DEBUG:boto:Params: {} 2024-04-12T04:04:53.420 DEBUG:boto:Token: None 2024-04-12T04:04:53.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj31 2024-04-12T04:04:53.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:R7LWZhMKyO61vKmtzsWq25lqWvY= 2024-04-12T04:04:53.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R7LWZhMKyO61vKmtzsWq25lqWvY='} 2024-04-12T04:04:53.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000937414ac06246082-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.426 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.426 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.426 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.426 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.426 DEBUG:boto:Data: 2024-04-12T04:04:53.427 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.427 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.427 DEBUG:boto:Params: {} 2024-04-12T04:04:53.427 DEBUG:boto:Token: None 2024-04-12T04:04:53.427 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.427 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddeadc3fd49bde9e-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.431 DEBUG:boto:path=/dgbpxg-65/obj62 2024-04-12T04:04:53.431 DEBUG:boto:auth_path=/dgbpxg-65/obj62 2024-04-12T04:04:53.431 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.431 DEBUG:boto:Path: /dgbpxg-65/obj62 2024-04-12T04:04:53.431 DEBUG:boto:Data: 2024-04-12T04:04:53.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.431 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.431 DEBUG:boto:Params: {} 2024-04-12T04:04:53.432 DEBUG:boto:Token: None 2024-04-12T04:04:53.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj62 2024-04-12T04:04:53.432 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TpLK7n3uWdfWBT3tqGm4gTwwlhc= 2024-04-12T04:04:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpLK7n3uWdfWBT3tqGm4gTwwlhc='} 2024-04-12T04:04:53.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018eef364fb9f656b-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.475 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.475 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.475 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.475 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.475 DEBUG:boto:Data: 2024-04-12T04:04:53.475 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.475 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.475 DEBUG:boto:Params: {} 2024-04-12T04:04:53.476 DEBUG:boto:Token: None 2024-04-12T04:04:53.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.476 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f5ffae58f245572-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.479 DEBUG:boto:path=/dgbpxg-65/obj93 2024-04-12T04:04:53.479 DEBUG:boto:auth_path=/dgbpxg-65/obj93 2024-04-12T04:04:53.480 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.480 DEBUG:boto:Path: /dgbpxg-65/obj93 2024-04-12T04:04:53.480 DEBUG:boto:Data: 2024-04-12T04:04:53.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.480 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.480 DEBUG:boto:Params: {} 2024-04-12T04:04:53.480 DEBUG:boto:Token: None 2024-04-12T04:04:53.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj93 2024-04-12T04:04:53.481 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:YA0TMm3SPLCcOTpMtBdyvmwjWGI= 2024-04-12T04:04:53.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YA0TMm3SPLCcOTpMtBdyvmwjWGI='} 2024-04-12T04:04:53.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5fb41142cf0d3a-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.486 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.486 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.486 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.486 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.486 DEBUG:boto:Data: 2024-04-12T04:04:53.486 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.486 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.486 DEBUG:boto:Params: {} 2024-04-12T04:04:53.487 DEBUG:boto:Token: None 2024-04-12T04:04:53.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.487 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092922d2aab602fab-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.491 DEBUG:boto:path=/dgbpxg-65/obj124 2024-04-12T04:04:53.491 DEBUG:boto:auth_path=/dgbpxg-65/obj124 2024-04-12T04:04:53.491 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.491 DEBUG:boto:Path: /dgbpxg-65/obj124 2024-04-12T04:04:53.491 DEBUG:boto:Data: 2024-04-12T04:04:53.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.491 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.491 DEBUG:boto:Params: {} 2024-04-12T04:04:53.491 DEBUG:boto:Token: None 2024-04-12T04:04:53.492 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj124 2024-04-12T04:04:53.492 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/uFTaaiViUi5y6oVUZuBuG/spLw= 2024-04-12T04:04:53.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/uFTaaiViUi5y6oVUZuBuG/spLw='} 2024-04-12T04:04:53.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1f02b51959aa1e3-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.535 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.535 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.535 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.535 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.535 DEBUG:boto:Data: 2024-04-12T04:04:53.535 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.535 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.535 DEBUG:boto:Params: {} 2024-04-12T04:04:53.536 DEBUG:boto:Token: None 2024-04-12T04:04:53.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.536 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000885c5f0c2f58f4e1-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.539 DEBUG:boto:path=/dgbpxg-65/obj155 2024-04-12T04:04:53.540 DEBUG:boto:auth_path=/dgbpxg-65/obj155 2024-04-12T04:04:53.540 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.540 DEBUG:boto:Path: /dgbpxg-65/obj155 2024-04-12T04:04:53.540 DEBUG:boto:Data: 2024-04-12T04:04:53.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.540 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.540 DEBUG:boto:Params: {} 2024-04-12T04:04:53.540 DEBUG:boto:Token: None 2024-04-12T04:04:53.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj155 2024-04-12T04:04:53.541 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9yNGjbbUG0TTEWkU/SHU2CCh7Po= 2024-04-12T04:04:53.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9yNGjbbUG0TTEWkU/SHU2CCh7Po='} 2024-04-12T04:04:53.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056321dd58fc23f39-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:53.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:04:53.546 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:04:53.597 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:53.598 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:53.652 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:04:53.652 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-65 2024-04-12T04:04:53.664 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2024-04-12T04:04:53.729 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.730 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.730 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.730 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.731 DEBUG:boto:Data: 2024-04-12T04:04:53.731 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.731 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.731 DEBUG:boto:Params: {} 2024-04-12T04:04:53.732 DEBUG:boto:Token: None 2024-04-12T04:04:53.732 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.732 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a98d8845efa7f157-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.737 DEBUG:boto:path=/dgbpxg-65/obj37 2024-04-12T04:04:53.737 DEBUG:boto:auth_path=/dgbpxg-65/obj37 2024-04-12T04:04:53.737 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.738 DEBUG:boto:Path: /dgbpxg-65/obj37 2024-04-12T04:04:53.738 DEBUG:boto:Data: 2024-04-12T04:04:53.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.738 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.739 DEBUG:boto:Params: {} 2024-04-12T04:04:53.739 DEBUG:boto:Token: None 2024-04-12T04:04:53.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj37 2024-04-12T04:04:53.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1VnYHbF1o/8dBUAQBwBdvXVZUog= 2024-04-12T04:04:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1VnYHbF1o/8dBUAQBwBdvXVZUog='} 2024-04-12T04:04:53.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056eef232e7037b82-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.787 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.787 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.787 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.787 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.787 DEBUG:boto:Data: 2024-04-12T04:04:53.787 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.788 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.788 DEBUG:boto:Params: {} 2024-04-12T04:04:53.788 DEBUG:boto:Token: None 2024-04-12T04:04:53.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f31ac122769446f8-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.794 DEBUG:boto:path=/dgbpxg-65/obj74 2024-04-12T04:04:53.794 DEBUG:boto:auth_path=/dgbpxg-65/obj74 2024-04-12T04:04:53.794 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.794 DEBUG:boto:Path: /dgbpxg-65/obj74 2024-04-12T04:04:53.794 DEBUG:boto:Data: 2024-04-12T04:04:53.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.795 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.795 DEBUG:boto:Params: {} 2024-04-12T04:04:53.795 DEBUG:boto:Token: None 2024-04-12T04:04:53.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj74 2024-04-12T04:04:53.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fn0/gy8SZGr6VMTrrCn8AV8R+9I= 2024-04-12T04:04:53.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fn0/gy8SZGr6VMTrrCn8AV8R+9I='} 2024-04-12T04:04:53.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f91b2f3ceb14a9f-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.800 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.800 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.800 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.800 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.800 DEBUG:boto:Data: 2024-04-12T04:04:53.801 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.801 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.801 DEBUG:boto:Params: {} 2024-04-12T04:04:53.801 DEBUG:boto:Token: None 2024-04-12T04:04:53.801 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.801 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b8226b7ed9afd3-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.805 DEBUG:boto:path=/dgbpxg-65/obj111 2024-04-12T04:04:53.805 DEBUG:boto:auth_path=/dgbpxg-65/obj111 2024-04-12T04:04:53.805 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.805 DEBUG:boto:Path: /dgbpxg-65/obj111 2024-04-12T04:04:53.805 DEBUG:boto:Data: 2024-04-12T04:04:53.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.805 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.805 DEBUG:boto:Params: {} 2024-04-12T04:04:53.805 DEBUG:boto:Token: None 2024-04-12T04:04:53.806 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj111 2024-04-12T04:04:53.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:INjG+F92xFDGbPrJ2Io86c5UMmA= 2024-04-12T04:04:53.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:INjG+F92xFDGbPrJ2Io86c5UMmA='} 2024-04-12T04:04:53.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003063e5739b4b22d1-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.851 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.851 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.851 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.851 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.851 DEBUG:boto:Data: 2024-04-12T04:04:53.851 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.852 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.852 DEBUG:boto:Params: {} 2024-04-12T04:04:53.852 DEBUG:boto:Token: None 2024-04-12T04:04:53.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb63bd7654cbc8de-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.856 DEBUG:boto:path=/dgbpxg-65/obj148 2024-04-12T04:04:53.856 DEBUG:boto:auth_path=/dgbpxg-65/obj148 2024-04-12T04:04:53.856 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.856 DEBUG:boto:Path: /dgbpxg-65/obj148 2024-04-12T04:04:53.856 DEBUG:boto:Data: 2024-04-12T04:04:53.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.856 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.856 DEBUG:boto:Params: {} 2024-04-12T04:04:53.857 DEBUG:boto:Token: None 2024-04-12T04:04:53.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj148 2024-04-12T04:04:53.857 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SrV00NV/58yNtc9QYdk1X+WMNGc= 2024-04-12T04:04:53.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SrV00NV/58yNtc9QYdk1X+WMNGc='} 2024-04-12T04:04:53.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e4b201f935456b-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.862 DEBUG:boto:path=/dgbpxg-65/ 2024-04-12T04:04:53.862 DEBUG:boto:auth_path=/dgbpxg-65/ 2024-04-12T04:04:53.862 DEBUG:boto:Method: HEAD 2024-04-12T04:04:53.862 DEBUG:boto:Path: /dgbpxg-65/ 2024-04-12T04:04:53.862 DEBUG:boto:Data: 2024-04-12T04:04:53.862 DEBUG:boto:Headers: {} 2024-04-12T04:04:53.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.863 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.863 DEBUG:boto:Params: {} 2024-04-12T04:04:53.863 DEBUG:boto:Token: None 2024-04-12T04:04:53.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/ 2024-04-12T04:04:53.863 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:04:53.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:04:53.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff23c4850e24eb7-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.866 DEBUG:boto:path=/dgbpxg-65/obj185 2024-04-12T04:04:53.866 DEBUG:boto:auth_path=/dgbpxg-65/obj185 2024-04-12T04:04:53.867 DEBUG:boto:Method: PUT 2024-04-12T04:04:53.867 DEBUG:boto:Path: /dgbpxg-65/obj185 2024-04-12T04:04:53.867 DEBUG:boto:Data: 2024-04-12T04:04:53.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:04:53.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:04:53.867 DEBUG:boto:Port: 8000 2024-04-12T04:04:53.867 DEBUG:boto:Params: {} 2024-04-12T04:04:53.867 DEBUG:boto:Token: None 2024-04-12T04:04:53.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:04:53 GMT /dgbpxg-65/obj185 2024-04-12T04:04:53.867 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+WXK9/669oASWS4AQ9Oe0bVb1dk= 2024-04-12T04:04:53.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+WXK9/669oASWS4AQ9Oe0bVb1dk='} 2024-04-12T04:04:53.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031b8e06efa4a6362-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:53.915 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-65 2024-04-12T04:04:53.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:53.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:53.915 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:04:53.973 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:54.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:54.026+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:04:59.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:04:59.027+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:05:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:04.027+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:05:09.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:09.028+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:05:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:14.032+0000 7efc9436f780 1 bucket sync caught up with source: 2024-04-12T04:05:14.017 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.121.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.104.6, , , , 00000000001.124.6] 2024-04-12T04:05:14.017 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.121.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.104.6, , , , 00000000001.124.6] 2024-04-12T04:05:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:14.032+0000 7efc9436f780 0 bucket checkpoint complete 2024-04-12T04:05:14.035 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-65 2024-04-12T04:05:14.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:05:14.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:05:14.036 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:05:14.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:05:14.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:05:14.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:14.150+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:19.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:19.151+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:24.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:24.151+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:29.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:29.152+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:34.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:34.153+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:39.154+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:44.154+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:49.155+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:54.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:54.156+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:05:59.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:05:59.157+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:04.156+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:09.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:09.157+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:14.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:14.158+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:19.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:19.159+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:24.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:24.160+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:29.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:29.160+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:34.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:34.161+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:39.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:39.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:44.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:44.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:49.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:54.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:54.163+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:06:59.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:06:59.164+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:04.165+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:09.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:09.166+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:14.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:14.166+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:19.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:19.167+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:24.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:24.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:28.529 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:07:28.544+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:07:29.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:29.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:34.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:39.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:39.169+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:44.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:44.170+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:49.171+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:54.171+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:07:59.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:07:59.172+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:04.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:09.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:09.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:14.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:14.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:19.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:19.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:24.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:24.175+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:29.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:29.176+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:34.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:34.176+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:39.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:39.177+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:44.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:44.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:49.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:49.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:54.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:54.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:08:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:08:59.179+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:04.180+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:09.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:09.182+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:14.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:14.182+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:19.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:19.183+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:24.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:24.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:29.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:29.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:34.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:34.185+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:39.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:39.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:44.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:44.185+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:49.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:49.187+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:54.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:54.188+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:09:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:09:59.188+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:10:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:04.189+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:10:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.190+0000 7f52fcf85780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:10:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.190+0000 7f52fcf85780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:10:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.190+0000 7f52fcf85780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:10:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T04:10:09.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-12T04:10:09.196 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.196 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.196 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.196 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.196 DEBUG:boto:Data: 2024-04-12T04:10:09.196 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.196 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.197 DEBUG:boto:Params: {} 2024-04-12T04:10:09.197 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:10:09.197 DEBUG:boto:Token: None 2024-04-12T04:10:09.197 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.198 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Hw5YpAR5wmMu1THEyphTB19Vieo= 2024-04-12T04:10:09.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hw5YpAR5wmMu1THEyphTB19Vieo=', 'Content-Length': '0'} 2024-04-12T04:10:09.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000679b976c9177fd61-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.242 DEBUG:rgw_multi.tests:created bucket=dgbpxg-66 2024-04-12T04:10:09.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:09.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:09.243 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:10:09.297 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:09.297 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:10:09.348 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:10:09.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.348 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895009.257409_1720.1", 2024-04-12T04:10:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:09.257409Z" 2024-04-12T04:10:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.350 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:10:09.351 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:10:09.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:10:09.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:09.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:09.352 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:10:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:09.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.444+0000 7f6491aeb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:10:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.445+0000 7f6491aeb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:10:09.441 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:10:09.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:10:09.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:10:09.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:10:09.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895009.257409_1720.1", 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:10:09.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:09.257409Z", 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:10:09.448 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:10:09.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:10:09.449 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:10:09Z" 2024-04-12T04:10:09.449 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:10:09.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895009.257409_1720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:09.257409Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:10:09Z" } 2024-04-12T04:10:09.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895009.257409_1720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:09.257409Z', 'realm_epoch': 2}}] 2024-04-12T04:10:09.450 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:10:09.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:10:09.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:10:09.451 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.451 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.451 DEBUG:boto:Method: HEAD 2024-04-12T04:10:09.451 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.451 DEBUG:boto:Data: 2024-04-12T04:10:09.451 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.452 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.452 DEBUG:boto:Params: {} 2024-04-12T04:10:09.452 DEBUG:boto:Token: None 2024-04-12T04:10:09.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.452 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:10:09.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:10:09.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b47caf59f9ceabe-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.455 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:09.455 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:09.455 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.455 DEBUG:boto:Path: /dgbpxg-66/obj11 2024-04-12T04:10:09.455 DEBUG:boto:Data: 2024-04-12T04:10:09.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:09.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.456 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.456 DEBUG:boto:Params: {} 2024-04-12T04:10:09.456 DEBUG:boto:Token: None 2024-04-12T04:10:09.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/obj11 2024-04-12T04:10:09.456 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:st99Bhht4r3RbbHkwSDqNOZwzn0= 2024-04-12T04:10:09.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:st99Bhht4r3RbbHkwSDqNOZwzn0='} 2024-04-12T04:10:09.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000265dfd2f2fbae73c-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.462 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.462 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.462 DEBUG:boto:Method: HEAD 2024-04-12T04:10:09.462 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.462 DEBUG:boto:Data: 2024-04-12T04:10:09.462 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.463 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.463 DEBUG:boto:Params: {} 2024-04-12T04:10:09.463 DEBUG:boto:Token: None 2024-04-12T04:10:09.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.463 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:10:09.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:10:09.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b68cd78ad70be27-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.465 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:09.466 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:09.466 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.466 DEBUG:boto:Path: /dgbpxg-66/obj22 2024-04-12T04:10:09.466 DEBUG:boto:Data: 2024-04-12T04:10:09.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:09.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.466 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.466 DEBUG:boto:Params: {} 2024-04-12T04:10:09.466 DEBUG:boto:Token: None 2024-04-12T04:10:09.466 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/obj22 2024-04-12T04:10:09.467 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:wFDnbU4NzR3vQ9n0wRI31htfhDM= 2024-04-12T04:10:09.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wFDnbU4NzR3vQ9n0wRI31htfhDM='} 2024-04-12T04:10:09.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000551bb34dc6c41c7-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.470 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.471 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.471 DEBUG:boto:Method: HEAD 2024-04-12T04:10:09.471 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.471 DEBUG:boto:Data: 2024-04-12T04:10:09.471 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.471 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.471 DEBUG:boto:Params: {} 2024-04-12T04:10:09.471 DEBUG:boto:Token: None 2024-04-12T04:10:09.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.472 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:10:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:10:09.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba5e1a5c86b3c097-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.474 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:09.474 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:09.474 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.474 DEBUG:boto:Path: /dgbpxg-66/obj33 2024-04-12T04:10:09.474 DEBUG:boto:Data: 2024-04-12T04:10:09.474 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:09.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.474 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.474 DEBUG:boto:Params: {} 2024-04-12T04:10:09.475 DEBUG:boto:Token: None 2024-04-12T04:10:09.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/obj33 2024-04-12T04:10:09.475 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gaQbNZ7G+DwWISJj7ZL+TzywMcA= 2024-04-12T04:10:09.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gaQbNZ7G+DwWISJj7ZL+TzywMcA='} 2024-04-12T04:10:09.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059eeb279a455681c-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.519 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.519 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.519 DEBUG:boto:Method: HEAD 2024-04-12T04:10:09.519 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.520 DEBUG:boto:Data: 2024-04-12T04:10:09.520 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.520 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.520 DEBUG:boto:Params: {} 2024-04-12T04:10:09.521 DEBUG:boto:Token: None 2024-04-12T04:10:09.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.521 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:10:09.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:10:09.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000810f01e0fc4efdf6-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.523 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:09.524 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:09.524 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.524 DEBUG:boto:Path: /dgbpxg-66/obj44 2024-04-12T04:10:09.524 DEBUG:boto:Data: 2024-04-12T04:10:09.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:09.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.524 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.524 DEBUG:boto:Params: {} 2024-04-12T04:10:09.525 DEBUG:boto:Token: None 2024-04-12T04:10:09.525 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/obj44 2024-04-12T04:10:09.525 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5lM2xZpSMxFWkCil3doZ6yvdsg8= 2024-04-12T04:10:09.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5lM2xZpSMxFWkCil3doZ6yvdsg8='} 2024-04-12T04:10:09.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b9868b4f7a34146-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.530 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:09.530 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:09.530 DEBUG:boto:Method: HEAD 2024-04-12T04:10:09.531 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:09.531 DEBUG:boto:Data: 2024-04-12T04:10:09.531 DEBUG:boto:Headers: {} 2024-04-12T04:10:09.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.531 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.531 DEBUG:boto:Params: {} 2024-04-12T04:10:09.531 DEBUG:boto:Token: None 2024-04-12T04:10:09.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/ 2024-04-12T04:10:09.531 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:10:09.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:10:09.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d5e16ca42ddacd0-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.533 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:09.534 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:09.534 DEBUG:boto:Method: PUT 2024-04-12T04:10:09.534 DEBUG:boto:Path: /dgbpxg-66/obj55 2024-04-12T04:10:09.534 DEBUG:boto:Data: 2024-04-12T04:10:09.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:09.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:09.534 DEBUG:boto:Port: 8000 2024-04-12T04:10:09.534 DEBUG:boto:Params: {} 2024-04-12T04:10:09.534 DEBUG:boto:Token: None 2024-04-12T04:10:09.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:09 GMT /dgbpxg-66/obj55 2024-04-12T04:10:09.535 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GYR8ng6EBLmcK7Tg4gdAHBBpyeY= 2024-04-12T04:10:09.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYR8ng6EBLmcK7Tg4gdAHBBpyeY='} 2024-04-12T04:10:09.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061aacd8a228eb450-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:09.579 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-66 2024-04-12T04:10:09.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:09.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:09.579 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:10:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:09.689+0000 7f1199207780 1 waiting to reach incremental sync.. 2024-04-12T04:10:14.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:14.691+0000 7f1199207780 1 bucket sync caught up with source: 2024-04-12T04:10:14.675 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.98.6, 00000000002.148.6, 00000000002.147.6] 2024-04-12T04:10:14.675 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.98.6, 00000000002.148.6, 00000000002.147.6] 2024-04-12T04:10:14.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:14.691+0000 7f1199207780 0 bucket checkpoint complete 2024-04-12T04:10:14.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-66 2024-04-12T04:10:14.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:14.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:14.692 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:10:14.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:14.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:14.803+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:10:19.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:19.803+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:10:24.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:24.804+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:10:29.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:29.805+0000 7fc0de56c780 1 bucket sync caught up with source: 2024-04-12T04:10:29.790 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.3243.6, 00000000002.1972.6, 00000000002.1467.6] 2024-04-12T04:10:29.790 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.3243.6, 00000000002.1972.6, 00000000002.1467.6] 2024-04-12T04:10:29.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:29.805+0000 7fc0de56c780 0 bucket checkpoint complete 2024-04-12T04:10:29.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dgbpxg-66 zones={a2, a1} 2024-04-12T04:10:29.810 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.810 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.811 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.811 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:29.811 DEBUG:boto:Data: 2024-04-12T04:10:29.811 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.811 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.811 DEBUG:boto:Params: {} 2024-04-12T04:10:29.811 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:10:29.812 DEBUG:boto:Token: None 2024-04-12T04:10:29.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.812 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE= 2024-04-12T04:10:29.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE=', 'Content-Length': '0'} 2024-04-12T04:10:29.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e20dae285818d86-006618b435-2289-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.825 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.825 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.825 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.825 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:29.825 DEBUG:boto:Data: 2024-04-12T04:10:29.825 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.826 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.826 DEBUG:boto:Params: {} 2024-04-12T04:10:29.826 DEBUG:boto:Token: None 2024-04-12T04:10:29.826 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.826 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE= 2024-04-12T04:10:29.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE=', 'Content-Length': '0'} 2024-04-12T04:10:29.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cea0678dcf5efd7b-006618b435-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.830 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:10:29.830 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.830 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.830 DEBUG:boto:path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.830 DEBUG:boto:auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.830 DEBUG:boto:Method: GET 2024-04-12T04:10:29.830 DEBUG:boto:Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.830 DEBUG:boto:Data: 2024-04-12T04:10:29.830 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.831 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.831 DEBUG:boto:Params: {} 2024-04-12T04:10:29.831 DEBUG:boto:Token: None 2024-04-12T04:10:29.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/?versions 2024-04-12T04:10:29.831 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:10:29.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:10:29.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d55d587c7145efe-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.834 DEBUG:boto:b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:10:29.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:10:29.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:10:29.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:10:29.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:10:29.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:10:29.836 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.836 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.836 DEBUG:boto:path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.836 DEBUG:boto:auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.837 DEBUG:boto:Method: GET 2024-04-12T04:10:29.837 DEBUG:boto:Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.837 DEBUG:boto:Data: 2024-04-12T04:10:29.837 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.837 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.837 DEBUG:boto:Params: {} 2024-04-12T04:10:29.837 DEBUG:boto:Token: None 2024-04-12T04:10:29.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/?versions 2024-04-12T04:10:29.837 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:10:29.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:10:29.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d89c5dd0a13ea7a-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.840 DEBUG:boto:b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:10:29.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:10:29.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:10:29.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:10:29.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:10:29.841 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.841 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.841 DEBUG:boto:path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.841 DEBUG:boto:auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.842 DEBUG:boto:Method: GET 2024-04-12T04:10:29.842 DEBUG:boto:Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.842 DEBUG:boto:Data: 2024-04-12T04:10:29.842 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.842 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.843 DEBUG:boto:Params: {} 2024-04-12T04:10:29.843 DEBUG:boto:Token: None 2024-04-12T04:10:29.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/?versions 2024-04-12T04:10:29.843 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:10:29.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:10:29.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756086cad2e29852-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.845 DEBUG:boto:b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.846 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.846 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.846 DEBUG:boto:path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.846 DEBUG:boto:auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.846 DEBUG:boto:Method: GET 2024-04-12T04:10:29.846 DEBUG:boto:Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:10:29.846 DEBUG:boto:Data: 2024-04-12T04:10:29.847 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.847 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.847 DEBUG:boto:Params: {} 2024-04-12T04:10:29.847 DEBUG:boto:Token: None 2024-04-12T04:10:29.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/?versions 2024-04-12T04:10:29.847 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:10:29.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:10:29.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2267355cc256ed9-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.850 DEBUG:boto:b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:29.850 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.851 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.851 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.851 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.851 DEBUG:boto:Method: GET 2024-04-12T04:10:29.851 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.851 DEBUG:boto:Data: 2024-04-12T04:10:29.851 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.851 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.851 DEBUG:boto:Params: {} 2024-04-12T04:10:29.851 DEBUG:boto:Token: None 2024-04-12T04:10:29.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.852 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:10:29.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:10:29.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab57b3fbe876b7c9-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.854 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.854 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.854 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.854 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.854 DEBUG:boto:Method: GET 2024-04-12T04:10:29.854 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.854 DEBUG:boto:Data: 2024-04-12T04:10:29.854 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.855 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.855 DEBUG:boto:Params: {} 2024-04-12T04:10:29.855 DEBUG:boto:Token: None 2024-04-12T04:10:29.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.855 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:10:29.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:10:29.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d1a73aeb3051cb-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.857 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.857 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.857 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.857 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.857 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.858 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.858 DEBUG:boto:Data: 2024-04-12T04:10:29.858 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.858 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.858 DEBUG:boto:Params: {} 2024-04-12T04:10:29.858 DEBUG:boto:Token: None 2024-04-12T04:10:29.858 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.858 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo= 2024-04-12T04:10:29.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo=', 'Content-Length': '0'} 2024-04-12T04:10:29.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b3a7c5e2c9346d-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.860 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.860 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.860 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.861 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.861 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.861 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.861 DEBUG:boto:Data: 2024-04-12T04:10:29.861 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.861 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.861 DEBUG:boto:Params: {} 2024-04-12T04:10:29.861 DEBUG:boto:Token: None 2024-04-12T04:10:29.861 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.862 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo= 2024-04-12T04:10:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo=', 'Content-Length': '0'} 2024-04-12T04:10:29.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5fc3d3e0e7ea3c-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:29.864 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.864 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.864 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.864 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.864 DEBUG:boto:Method: GET 2024-04-12T04:10:29.864 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.864 DEBUG:boto:Data: 2024-04-12T04:10:29.864 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.864 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.864 DEBUG:boto:Params: {} 2024-04-12T04:10:29.865 DEBUG:boto:Token: None 2024-04-12T04:10:29.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.865 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:10:29.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:10:29.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc89b53d778faadb-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.867 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.867 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.867 DEBUG:boto:path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.867 DEBUG:boto:auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.867 DEBUG:boto:Method: GET 2024-04-12T04:10:29.867 DEBUG:boto:Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.867 DEBUG:boto:Data: 2024-04-12T04:10:29.867 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.867 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.868 DEBUG:boto:Params: {} 2024-04-12T04:10:29.868 DEBUG:boto:Token: None 2024-04-12T04:10:29.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11?versionId=null 2024-04-12T04:10:29.868 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:10:29.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:10:29.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c91233106f69e1d-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.870 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.870 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.870 DEBUG:boto:path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.870 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.871 DEBUG:boto:Method: GET 2024-04-12T04:10:29.871 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.871 DEBUG:boto:Data: 2024-04-12T04:10:29.871 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.871 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.871 DEBUG:boto:Params: {} 2024-04-12T04:10:29.871 DEBUG:boto:Token: None 2024-04-12T04:10:29.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.871 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:10:29.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:10:29.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7398519c5c0452e-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.874 DEBUG:boto:b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.874 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.875 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.875 DEBUG:boto:path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.875 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.875 DEBUG:boto:Method: GET 2024-04-12T04:10:29.875 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.875 DEBUG:boto:Data: 2024-04-12T04:10:29.875 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.875 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.875 DEBUG:boto:Params: {} 2024-04-12T04:10:29.876 DEBUG:boto:Token: None 2024-04-12T04:10:29.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:10:29.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:10:29.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb70bb30be7a423-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.878 DEBUG:boto:b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.879 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.879 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.879 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.879 DEBUG:boto:Path: /dgbpxg-66/obj11 2024-04-12T04:10:29.879 DEBUG:boto:Data: 2024-04-12T04:10:29.879 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.879 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.879 DEBUG:boto:Params: {} 2024-04-12T04:10:29.879 DEBUG:boto:Token: None 2024-04-12T04:10:29.880 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11 2024-04-12T04:10:29.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI= 2024-04-12T04:10:29.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI=', 'Content-Length': '0'} 2024-04-12T04:10:29.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000963e7c156790c11-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.882 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.882 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.882 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.882 DEBUG:boto:Path: /dgbpxg-66/obj11 2024-04-12T04:10:29.882 DEBUG:boto:Data: 2024-04-12T04:10:29.882 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.882 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.883 DEBUG:boto:Params: {} 2024-04-12T04:10:29.883 DEBUG:boto:Token: None 2024-04-12T04:10:29.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11 2024-04-12T04:10:29.883 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI= 2024-04-12T04:10:29.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI=', 'Content-Length': '0'} 2024-04-12T04:10:29.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097cd2a71de090234-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:29.885 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.885 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.885 DEBUG:boto:Method: GET 2024-04-12T04:10:29.885 DEBUG:boto:Path: /dgbpxg-66/obj11 2024-04-12T04:10:29.885 DEBUG:boto:Data: 2024-04-12T04:10:29.885 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.885 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.885 DEBUG:boto:Params: {} 2024-04-12T04:10:29.885 DEBUG:boto:Token: None 2024-04-12T04:10:29.886 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11 2024-04-12T04:10:29.886 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38= 2024-04-12T04:10:29.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38=', 'Content-Length': '0'} 2024-04-12T04:10:29.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009501a8936bc09985-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.888 DEBUG:boto:path=/dgbpxg-66/obj11 2024-04-12T04:10:29.888 DEBUG:boto:auth_path=/dgbpxg-66/obj11 2024-04-12T04:10:29.888 DEBUG:boto:Method: GET 2024-04-12T04:10:29.888 DEBUG:boto:Path: /dgbpxg-66/obj11 2024-04-12T04:10:29.888 DEBUG:boto:Data: 2024-04-12T04:10:29.888 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.888 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.888 DEBUG:boto:Params: {} 2024-04-12T04:10:29.888 DEBUG:boto:Token: None 2024-04-12T04:10:29.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj11 2024-04-12T04:10:29.889 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38= 2024-04-12T04:10:29.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38=', 'Content-Length': '0'} 2024-04-12T04:10:29.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc34f9cd835bbe1-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.891 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.891 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.891 DEBUG:boto:path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.891 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.891 DEBUG:boto:Method: GET 2024-04-12T04:10:29.891 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.891 DEBUG:boto:Data: 2024-04-12T04:10:29.891 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.892 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.892 DEBUG:boto:Params: {} 2024-04-12T04:10:29.892 DEBUG:boto:Token: None 2024-04-12T04:10:29.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.892 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:10:29.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:10:29.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b728d684d7b2093-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.894 DEBUG:boto:b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.894 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.895 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.895 DEBUG:boto:path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.895 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.895 DEBUG:boto:Method: GET 2024-04-12T04:10:29.895 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:10:29.895 DEBUG:boto:Data: 2024-04-12T04:10:29.895 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.895 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.895 DEBUG:boto:Params: {} 2024-04-12T04:10:29.896 DEBUG:boto:Token: None 2024-04-12T04:10:29.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.896 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:10:29.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:10:29.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000461569c1374fa345-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.942 DEBUG:boto:b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:29.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:29.943 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.943 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.943 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.943 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.943 DEBUG:boto:Method: GET 2024-04-12T04:10:29.943 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.943 DEBUG:boto:Data: 2024-04-12T04:10:29.944 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.944 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.944 DEBUG:boto:Params: {} 2024-04-12T04:10:29.944 DEBUG:boto:Token: None 2024-04-12T04:10:29.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.944 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:10:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:10:29.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3abd3425cc9e4cd-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.946 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.947 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.947 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.947 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.947 DEBUG:boto:Method: GET 2024-04-12T04:10:29.947 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.947 DEBUG:boto:Data: 2024-04-12T04:10:29.947 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.947 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.947 DEBUG:boto:Params: {} 2024-04-12T04:10:29.947 DEBUG:boto:Token: None 2024-04-12T04:10:29.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.948 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:10:29.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:10:29.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab75a4cb435840bd-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.950 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.950 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.950 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.950 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.950 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.950 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.950 DEBUG:boto:Data: 2024-04-12T04:10:29.950 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.951 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.951 DEBUG:boto:Params: {} 2024-04-12T04:10:29.951 DEBUG:boto:Token: None 2024-04-12T04:10:29.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.951 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM= 2024-04-12T04:10:29.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM=', 'Content-Length': '0'} 2024-04-12T04:10:29.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000486202be93bd9768-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.953 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.953 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.953 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.953 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.953 DEBUG:boto:Method: HEAD 2024-04-12T04:10:29.953 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.953 DEBUG:boto:Data: 2024-04-12T04:10:29.954 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.954 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.954 DEBUG:boto:Params: {} 2024-04-12T04:10:29.954 DEBUG:boto:Token: None 2024-04-12T04:10:29.954 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.954 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM= 2024-04-12T04:10:29.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM=', 'Content-Length': '0'} 2024-04-12T04:10:29.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47184937fab3789-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:29.956 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.956 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.956 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.956 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.957 DEBUG:boto:Method: GET 2024-04-12T04:10:29.957 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.957 DEBUG:boto:Data: 2024-04-12T04:10:29.957 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.957 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.957 DEBUG:boto:Params: {} 2024-04-12T04:10:29.957 DEBUG:boto:Token: None 2024-04-12T04:10:29.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.957 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:10:29.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:10:29.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df74829936a75a8-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.959 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:29.959 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:29.960 DEBUG:boto:path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.960 DEBUG:boto:auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.960 DEBUG:boto:Method: GET 2024-04-12T04:10:29.960 DEBUG:boto:Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.960 DEBUG:boto:Data: 2024-04-12T04:10:29.960 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:29.960 DEBUG:boto:Port: 8000 2024-04-12T04:10:29.960 DEBUG:boto:Params: {} 2024-04-12T04:10:29.960 DEBUG:boto:Token: None 2024-04-12T04:10:29.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/obj22?versionId=null 2024-04-12T04:10:29.961 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:10:29.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:10:29.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3b73bfd36c0d5e-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:29.963 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:29.963 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:29.963 DEBUG:boto:path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:29.963 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:29.963 DEBUG:boto:Method: GET 2024-04-12T04:10:29.963 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:10:29.963 DEBUG:boto:Data: 2024-04-12T04:10:29.963 DEBUG:boto:Headers: {} 2024-04-12T04:10:29.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:29.963 DEBUG:boto:Port: 8001 2024-04-12T04:10:29.963 DEBUG:boto:Params: {} 2024-04-12T04:10:29.964 DEBUG:boto:Token: None 2024-04-12T04:10:29.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:29 GMT /dgbpxg-66/ 2024-04-12T04:10:29.964 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:10:29.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:10:29.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5363bf862729c4-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.006 DEBUG:boto:b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.007 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.007 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.007 DEBUG:boto:path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.007 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.007 DEBUG:boto:Method: GET 2024-04-12T04:10:30.007 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.007 DEBUG:boto:Data: 2024-04-12T04:10:30.008 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.008 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.008 DEBUG:boto:Params: {} 2024-04-12T04:10:30.008 DEBUG:boto:Token: None 2024-04-12T04:10:30.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.008 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428e9e17b8210428-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.011 DEBUG:boto:b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.012 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:30.012 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:30.012 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.012 DEBUG:boto:Path: /dgbpxg-66/obj22 2024-04-12T04:10:30.012 DEBUG:boto:Data: 2024-04-12T04:10:30.012 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.012 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.012 DEBUG:boto:Params: {} 2024-04-12T04:10:30.012 DEBUG:boto:Token: None 2024-04-12T04:10:30.013 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj22 2024-04-12T04:10:30.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY= 2024-04-12T04:10:30.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY=', 'Content-Length': '0'} 2024-04-12T04:10:30.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1b2b6f6716f7fc-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.019 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:30.019 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:30.019 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.019 DEBUG:boto:Path: /dgbpxg-66/obj22 2024-04-12T04:10:30.019 DEBUG:boto:Data: 2024-04-12T04:10:30.019 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.020 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.020 DEBUG:boto:Params: {} 2024-04-12T04:10:30.020 DEBUG:boto:Token: None 2024-04-12T04:10:30.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj22 2024-04-12T04:10:30.020 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY= 2024-04-12T04:10:30.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY=', 'Content-Length': '0'} 2024-04-12T04:10:30.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004887da127bdf9d6-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:30.022 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:30.022 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:30.022 DEBUG:boto:Method: GET 2024-04-12T04:10:30.022 DEBUG:boto:Path: /dgbpxg-66/obj22 2024-04-12T04:10:30.023 DEBUG:boto:Data: 2024-04-12T04:10:30.023 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.023 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.023 DEBUG:boto:Params: {} 2024-04-12T04:10:30.023 DEBUG:boto:Token: None 2024-04-12T04:10:30.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj22 2024-04-12T04:10:30.023 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY= 2024-04-12T04:10:30.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY=', 'Content-Length': '0'} 2024-04-12T04:10:30.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa8ed57d84b7538-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.025 DEBUG:boto:path=/dgbpxg-66/obj22 2024-04-12T04:10:30.026 DEBUG:boto:auth_path=/dgbpxg-66/obj22 2024-04-12T04:10:30.026 DEBUG:boto:Method: GET 2024-04-12T04:10:30.026 DEBUG:boto:Path: /dgbpxg-66/obj22 2024-04-12T04:10:30.026 DEBUG:boto:Data: 2024-04-12T04:10:30.026 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.026 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.026 DEBUG:boto:Params: {} 2024-04-12T04:10:30.026 DEBUG:boto:Token: None 2024-04-12T04:10:30.026 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj22 2024-04-12T04:10:30.027 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY= 2024-04-12T04:10:30.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY=', 'Content-Length': '0'} 2024-04-12T04:10:30.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fd1567b31dd63f-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.029 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.029 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.029 DEBUG:boto:path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.029 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.029 DEBUG:boto:Method: GET 2024-04-12T04:10:30.029 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.029 DEBUG:boto:Data: 2024-04-12T04:10:30.029 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.029 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.030 DEBUG:boto:Params: {} 2024-04-12T04:10:30.030 DEBUG:boto:Token: None 2024-04-12T04:10:30.030 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.030 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b76f02f2d868ea-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.033 DEBUG:boto:b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.033 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.033 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.033 DEBUG:boto:path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.033 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.033 DEBUG:boto:Method: GET 2024-04-12T04:10:30.033 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:10:30.034 DEBUG:boto:Data: 2024-04-12T04:10:30.034 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.034 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.034 DEBUG:boto:Params: {} 2024-04-12T04:10:30.034 DEBUG:boto:Token: None 2024-04-12T04:10:30.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.034 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000261ea897b522ecb0-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.037 DEBUG:boto:b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:30.037 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.037 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.037 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.037 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.038 DEBUG:boto:Method: GET 2024-04-12T04:10:30.038 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.038 DEBUG:boto:Data: 2024-04-12T04:10:30.038 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.038 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.038 DEBUG:boto:Params: {} 2024-04-12T04:10:30.038 DEBUG:boto:Token: None 2024-04-12T04:10:30.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:10:30.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:10:30.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c2dbc2feca0311-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.041 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.041 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.041 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.041 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.041 DEBUG:boto:Method: GET 2024-04-12T04:10:30.041 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.041 DEBUG:boto:Data: 2024-04-12T04:10:30.041 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.041 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.042 DEBUG:boto:Params: {} 2024-04-12T04:10:30.042 DEBUG:boto:Token: None 2024-04-12T04:10:30.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:10:30.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:10:30.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a651a36ada3b7412-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.044 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.044 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.044 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.044 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.044 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.045 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.045 DEBUG:boto:Data: 2024-04-12T04:10:30.045 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.045 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.045 DEBUG:boto:Params: {} 2024-04-12T04:10:30.045 DEBUG:boto:Token: None 2024-04-12T04:10:30.045 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM= 2024-04-12T04:10:30.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM=', 'Content-Length': '0'} 2024-04-12T04:10:30.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b225625ca026468-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.047 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.047 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.047 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.047 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.048 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.048 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.048 DEBUG:boto:Data: 2024-04-12T04:10:30.048 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.048 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.048 DEBUG:boto:Params: {} 2024-04-12T04:10:30.048 DEBUG:boto:Token: None 2024-04-12T04:10:30.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.049 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM= 2024-04-12T04:10:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM=', 'Content-Length': '0'} 2024-04-12T04:10:30.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538004dc257f57d6-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:30.051 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.051 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.051 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.051 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.051 DEBUG:boto:Method: GET 2024-04-12T04:10:30.051 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.051 DEBUG:boto:Data: 2024-04-12T04:10:30.051 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.051 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.051 DEBUG:boto:Params: {} 2024-04-12T04:10:30.052 DEBUG:boto:Token: None 2024-04-12T04:10:30.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.052 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:10:30.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:10:30.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f927793338723517-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.094 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.094 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.095 DEBUG:boto:path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.095 DEBUG:boto:auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.095 DEBUG:boto:Method: GET 2024-04-12T04:10:30.095 DEBUG:boto:Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.095 DEBUG:boto:Data: 2024-04-12T04:10:30.095 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.095 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.095 DEBUG:boto:Params: {} 2024-04-12T04:10:30.095 DEBUG:boto:Token: None 2024-04-12T04:10:30.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33?versionId=null 2024-04-12T04:10:30.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:10:30.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:10:30.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0709b91802c24e9-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.098 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.098 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.098 DEBUG:boto:path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.098 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.098 DEBUG:boto:Method: GET 2024-04-12T04:10:30.099 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.099 DEBUG:boto:Data: 2024-04-12T04:10:30.099 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.099 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.099 DEBUG:boto:Params: {} 2024-04-12T04:10:30.099 DEBUG:boto:Token: None 2024-04-12T04:10:30.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.099 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce10d58f2e294c31-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.101 DEBUG:boto:b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.102 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.102 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.102 DEBUG:boto:path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.102 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.102 DEBUG:boto:Method: GET 2024-04-12T04:10:30.102 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.102 DEBUG:boto:Data: 2024-04-12T04:10:30.103 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.103 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.103 DEBUG:boto:Params: {} 2024-04-12T04:10:30.103 DEBUG:boto:Token: None 2024-04-12T04:10:30.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e164bb0732d65e5a-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.105 DEBUG:boto:b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.106 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.106 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.106 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.106 DEBUG:boto:Path: /dgbpxg-66/obj33 2024-04-12T04:10:30.106 DEBUG:boto:Data: 2024-04-12T04:10:30.106 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.106 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.106 DEBUG:boto:Params: {} 2024-04-12T04:10:30.107 DEBUG:boto:Token: None 2024-04-12T04:10:30.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33 2024-04-12T04:10:30.107 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q= 2024-04-12T04:10:30.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q=', 'Content-Length': '0'} 2024-04-12T04:10:30.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001774901aed6fd283-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.109 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.109 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.109 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.109 DEBUG:boto:Path: /dgbpxg-66/obj33 2024-04-12T04:10:30.109 DEBUG:boto:Data: 2024-04-12T04:10:30.109 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.109 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.110 DEBUG:boto:Params: {} 2024-04-12T04:10:30.110 DEBUG:boto:Token: None 2024-04-12T04:10:30.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33 2024-04-12T04:10:30.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q= 2024-04-12T04:10:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q=', 'Content-Length': '0'} 2024-04-12T04:10:30.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3617ef93e2b25cf-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:30.112 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.112 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.112 DEBUG:boto:Method: GET 2024-04-12T04:10:30.112 DEBUG:boto:Path: /dgbpxg-66/obj33 2024-04-12T04:10:30.112 DEBUG:boto:Data: 2024-04-12T04:10:30.112 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.112 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.113 DEBUG:boto:Params: {} 2024-04-12T04:10:30.113 DEBUG:boto:Token: None 2024-04-12T04:10:30.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33 2024-04-12T04:10:30.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI= 2024-04-12T04:10:30.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI=', 'Content-Length': '0'} 2024-04-12T04:10:30.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007106be7b9f6dfe60-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.115 DEBUG:boto:path=/dgbpxg-66/obj33 2024-04-12T04:10:30.115 DEBUG:boto:auth_path=/dgbpxg-66/obj33 2024-04-12T04:10:30.115 DEBUG:boto:Method: GET 2024-04-12T04:10:30.115 DEBUG:boto:Path: /dgbpxg-66/obj33 2024-04-12T04:10:30.115 DEBUG:boto:Data: 2024-04-12T04:10:30.115 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.116 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.116 DEBUG:boto:Params: {} 2024-04-12T04:10:30.116 DEBUG:boto:Token: None 2024-04-12T04:10:30.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj33 2024-04-12T04:10:30.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI= 2024-04-12T04:10:30.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI=', 'Content-Length': '0'} 2024-04-12T04:10:30.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c50106c878ea054-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.118 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.118 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.118 DEBUG:boto:path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.118 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.118 DEBUG:boto:Method: GET 2024-04-12T04:10:30.119 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.119 DEBUG:boto:Data: 2024-04-12T04:10:30.119 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.119 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.119 DEBUG:boto:Params: {} 2024-04-12T04:10:30.119 DEBUG:boto:Token: None 2024-04-12T04:10:30.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.119 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce01f2ec8bed253-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.122 DEBUG:boto:b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.122 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.122 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.122 DEBUG:boto:path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.122 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.122 DEBUG:boto:Method: GET 2024-04-12T04:10:30.123 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:10:30.123 DEBUG:boto:Data: 2024-04-12T04:10:30.123 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.123 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.123 DEBUG:boto:Params: {} 2024-04-12T04:10:30.123 DEBUG:boto:Token: None 2024-04-12T04:10:30.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd1e92148fc5814-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.126 DEBUG:boto:b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:30.126 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.126 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.126 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.126 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.127 DEBUG:boto:Method: GET 2024-04-12T04:10:30.127 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.127 DEBUG:boto:Data: 2024-04-12T04:10:30.127 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.127 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.127 DEBUG:boto:Params: {} 2024-04-12T04:10:30.127 DEBUG:boto:Token: None 2024-04-12T04:10:30.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.127 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:10:30.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:10:30.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49ebf4fd1cbb186-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.129 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.130 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.130 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.130 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.130 DEBUG:boto:Method: GET 2024-04-12T04:10:30.130 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.130 DEBUG:boto:Data: 2024-04-12T04:10:30.130 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.130 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.130 DEBUG:boto:Params: {} 2024-04-12T04:10:30.131 DEBUG:boto:Token: None 2024-04-12T04:10:30.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.131 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:10:30.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:10:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71cd9569191d5b3-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.133 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.133 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.133 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.133 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.133 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.133 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.133 DEBUG:boto:Data: 2024-04-12T04:10:30.133 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.133 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.134 DEBUG:boto:Params: {} 2024-04-12T04:10:30.134 DEBUG:boto:Token: None 2024-04-12T04:10:30.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.134 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw= 2024-04-12T04:10:30.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw=', 'Content-Length': '0'} 2024-04-12T04:10:30.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6ad901e3ef1f60-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.136 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.136 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.136 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.136 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.136 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.136 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.136 DEBUG:boto:Data: 2024-04-12T04:10:30.136 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.137 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.137 DEBUG:boto:Params: {} 2024-04-12T04:10:30.137 DEBUG:boto:Token: None 2024-04-12T04:10:30.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.137 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw= 2024-04-12T04:10:30.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw=', 'Content-Length': '0'} 2024-04-12T04:10:30.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983b995480b2f79e-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:30.139 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.139 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.139 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.139 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.139 DEBUG:boto:Method: GET 2024-04-12T04:10:30.139 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.139 DEBUG:boto:Data: 2024-04-12T04:10:30.140 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.140 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.140 DEBUG:boto:Params: {} 2024-04-12T04:10:30.140 DEBUG:boto:Token: None 2024-04-12T04:10:30.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:10:30.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:10:30.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebe0e01441e564f-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.142 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.142 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.142 DEBUG:boto:path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.143 DEBUG:boto:auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.143 DEBUG:boto:Method: GET 2024-04-12T04:10:30.143 DEBUG:boto:Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.143 DEBUG:boto:Data: 2024-04-12T04:10:30.143 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.143 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.143 DEBUG:boto:Params: {} 2024-04-12T04:10:30.143 DEBUG:boto:Token: None 2024-04-12T04:10:30.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44?versionId=null 2024-04-12T04:10:30.143 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:10:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:10:30.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a6640e5b4c5f8a-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.186 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.187 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.187 DEBUG:boto:path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.187 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.187 DEBUG:boto:Method: GET 2024-04-12T04:10:30.187 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.187 DEBUG:boto:Data: 2024-04-12T04:10:30.187 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.187 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.187 DEBUG:boto:Params: {} 2024-04-12T04:10:30.188 DEBUG:boto:Token: None 2024-04-12T04:10:30.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54e2b8188df32ce-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.190 DEBUG:boto:b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.191 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.191 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.191 DEBUG:boto:path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.191 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.191 DEBUG:boto:Method: GET 2024-04-12T04:10:30.191 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.191 DEBUG:boto:Data: 2024-04-12T04:10:30.191 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.191 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.192 DEBUG:boto:Params: {} 2024-04-12T04:10:30.192 DEBUG:boto:Token: None 2024-04-12T04:10:30.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.192 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ba670f549fc67a-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.194 DEBUG:boto:b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.195 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.195 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.195 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.195 DEBUG:boto:Path: /dgbpxg-66/obj44 2024-04-12T04:10:30.195 DEBUG:boto:Data: 2024-04-12T04:10:30.195 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.195 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.195 DEBUG:boto:Params: {} 2024-04-12T04:10:30.196 DEBUG:boto:Token: None 2024-04-12T04:10:30.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44 2024-04-12T04:10:30.196 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo= 2024-04-12T04:10:30.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo=', 'Content-Length': '0'} 2024-04-12T04:10:30.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049792ab73de34fdb-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.198 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.198 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.198 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.198 DEBUG:boto:Path: /dgbpxg-66/obj44 2024-04-12T04:10:30.198 DEBUG:boto:Data: 2024-04-12T04:10:30.198 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.198 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.198 DEBUG:boto:Params: {} 2024-04-12T04:10:30.199 DEBUG:boto:Token: None 2024-04-12T04:10:30.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44 2024-04-12T04:10:30.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo= 2024-04-12T04:10:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo=', 'Content-Length': '0'} 2024-04-12T04:10:30.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff4fd3a5a66e74f-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:30.201 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.201 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.201 DEBUG:boto:Method: GET 2024-04-12T04:10:30.201 DEBUG:boto:Path: /dgbpxg-66/obj44 2024-04-12T04:10:30.201 DEBUG:boto:Data: 2024-04-12T04:10:30.201 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.202 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.202 DEBUG:boto:Params: {} 2024-04-12T04:10:30.202 DEBUG:boto:Token: None 2024-04-12T04:10:30.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44 2024-04-12T04:10:30.202 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc= 2024-04-12T04:10:30.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc=', 'Content-Length': '0'} 2024-04-12T04:10:30.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d308110a225935-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.246 DEBUG:boto:path=/dgbpxg-66/obj44 2024-04-12T04:10:30.247 DEBUG:boto:auth_path=/dgbpxg-66/obj44 2024-04-12T04:10:30.247 DEBUG:boto:Method: GET 2024-04-12T04:10:30.247 DEBUG:boto:Path: /dgbpxg-66/obj44 2024-04-12T04:10:30.247 DEBUG:boto:Data: 2024-04-12T04:10:30.247 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.247 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.247 DEBUG:boto:Params: {} 2024-04-12T04:10:30.247 DEBUG:boto:Token: None 2024-04-12T04:10:30.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj44 2024-04-12T04:10:30.248 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc= 2024-04-12T04:10:30.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc=', 'Content-Length': '0'} 2024-04-12T04:10:30.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4337e4d10b257e-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.250 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.250 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.250 DEBUG:boto:path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.250 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.251 DEBUG:boto:Method: GET 2024-04-12T04:10:30.251 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.251 DEBUG:boto:Data: 2024-04-12T04:10:30.251 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.251 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.251 DEBUG:boto:Params: {} 2024-04-12T04:10:30.251 DEBUG:boto:Token: None 2024-04-12T04:10:30.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.251 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bacc5ca692a3742-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.254 DEBUG:boto:b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.254 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.254 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.254 DEBUG:boto:path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.255 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.255 DEBUG:boto:Method: GET 2024-04-12T04:10:30.255 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:10:30.255 DEBUG:boto:Data: 2024-04-12T04:10:30.255 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.255 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.255 DEBUG:boto:Params: {} 2024-04-12T04:10:30.255 DEBUG:boto:Token: None 2024-04-12T04:10:30.255 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.256 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a17a947069cc52-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.258 DEBUG:boto:b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:30.258 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.258 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.259 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.259 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.259 DEBUG:boto:Method: GET 2024-04-12T04:10:30.259 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.259 DEBUG:boto:Data: 2024-04-12T04:10:30.259 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.259 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.259 DEBUG:boto:Params: {} 2024-04-12T04:10:30.259 DEBUG:boto:Token: None 2024-04-12T04:10:30.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.260 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:10:30.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:10:30.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8d9be9ebb98948-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.261 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.262 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.262 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.262 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.262 DEBUG:boto:Method: GET 2024-04-12T04:10:30.262 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.262 DEBUG:boto:Data: 2024-04-12T04:10:30.262 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.262 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.262 DEBUG:boto:Params: {} 2024-04-12T04:10:30.263 DEBUG:boto:Token: None 2024-04-12T04:10:30.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.263 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:10:30.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:10:30.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e200658308131ad2-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.265 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.265 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.265 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.265 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.265 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.265 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.265 DEBUG:boto:Data: 2024-04-12T04:10:30.266 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.266 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.266 DEBUG:boto:Params: {} 2024-04-12T04:10:30.266 DEBUG:boto:Token: None 2024-04-12T04:10:30.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.266 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE= 2024-04-12T04:10:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE=', 'Content-Length': '0'} 2024-04-12T04:10:30.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ef435734a25289-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.268 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.268 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.268 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.269 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.269 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.269 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.269 DEBUG:boto:Data: 2024-04-12T04:10:30.269 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.269 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.269 DEBUG:boto:Params: {} 2024-04-12T04:10:30.269 DEBUG:boto:Token: None 2024-04-12T04:10:30.269 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.270 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE= 2024-04-12T04:10:30.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE=', 'Content-Length': '0'} 2024-04-12T04:10:30.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbed53554442abc3-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:30.272 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.272 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.272 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.272 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.272 DEBUG:boto:Method: GET 2024-04-12T04:10:30.272 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.272 DEBUG:boto:Data: 2024-04-12T04:10:30.272 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.272 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.272 DEBUG:boto:Params: {} 2024-04-12T04:10:30.272 DEBUG:boto:Token: None 2024-04-12T04:10:30.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.273 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:10:30.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:10:30.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000495935e5da5f985-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.275 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.275 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.275 DEBUG:boto:path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.275 DEBUG:boto:auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.275 DEBUG:boto:Method: GET 2024-04-12T04:10:30.275 DEBUG:boto:Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.275 DEBUG:boto:Data: 2024-04-12T04:10:30.275 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.275 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.275 DEBUG:boto:Params: {} 2024-04-12T04:10:30.276 DEBUG:boto:Token: None 2024-04-12T04:10:30.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55?versionId=null 2024-04-12T04:10:30.276 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:10:30.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:10:30.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973ffc78f6b34ca9-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.318 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.318 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.318 DEBUG:boto:path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.319 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.319 DEBUG:boto:Method: GET 2024-04-12T04:10:30.319 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.319 DEBUG:boto:Data: 2024-04-12T04:10:30.319 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.319 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.319 DEBUG:boto:Params: {} 2024-04-12T04:10:30.319 DEBUG:boto:Token: None 2024-04-12T04:10:30.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.320 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce67c8d145c6639e-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.322 DEBUG:boto:b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.323 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.323 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.323 DEBUG:boto:path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.323 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.323 DEBUG:boto:Method: GET 2024-04-12T04:10:30.323 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.323 DEBUG:boto:Data: 2024-04-12T04:10:30.323 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.323 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.323 DEBUG:boto:Params: {} 2024-04-12T04:10:30.324 DEBUG:boto:Token: None 2024-04-12T04:10:30.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4589d60da4b33b-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.326 DEBUG:boto:b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.326 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.327 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.327 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.327 DEBUG:boto:Path: /dgbpxg-66/obj55 2024-04-12T04:10:30.327 DEBUG:boto:Data: 2024-04-12T04:10:30.327 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.327 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.327 DEBUG:boto:Params: {} 2024-04-12T04:10:30.327 DEBUG:boto:Token: None 2024-04-12T04:10:30.327 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55 2024-04-12T04:10:30.328 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM= 2024-04-12T04:10:30.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM=', 'Content-Length': '0'} 2024-04-12T04:10:30.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecae9f7046142b48-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.329 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.329 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.329 DEBUG:boto:Method: HEAD 2024-04-12T04:10:30.329 DEBUG:boto:Path: /dgbpxg-66/obj55 2024-04-12T04:10:30.330 DEBUG:boto:Data: 2024-04-12T04:10:30.330 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.330 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.330 DEBUG:boto:Params: {} 2024-04-12T04:10:30.330 DEBUG:boto:Token: None 2024-04-12T04:10:30.330 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55 2024-04-12T04:10:30.330 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM= 2024-04-12T04:10:30.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM=', 'Content-Length': '0'} 2024-04-12T04:10:30.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2de37a56abce70-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:30.332 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.332 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.332 DEBUG:boto:Method: GET 2024-04-12T04:10:30.332 DEBUG:boto:Path: /dgbpxg-66/obj55 2024-04-12T04:10:30.332 DEBUG:boto:Data: 2024-04-12T04:10:30.333 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.333 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.333 DEBUG:boto:Params: {} 2024-04-12T04:10:30.333 DEBUG:boto:Token: None 2024-04-12T04:10:30.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55 2024-04-12T04:10:30.333 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI= 2024-04-12T04:10:30.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI=', 'Content-Length': '0'} 2024-04-12T04:10:30.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a29b2bcc6846c0f-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.335 DEBUG:boto:path=/dgbpxg-66/obj55 2024-04-12T04:10:30.335 DEBUG:boto:auth_path=/dgbpxg-66/obj55 2024-04-12T04:10:30.335 DEBUG:boto:Method: GET 2024-04-12T04:10:30.335 DEBUG:boto:Path: /dgbpxg-66/obj55 2024-04-12T04:10:30.335 DEBUG:boto:Data: 2024-04-12T04:10:30.335 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.336 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.336 DEBUG:boto:Params: {} 2024-04-12T04:10:30.336 DEBUG:boto:Token: None 2024-04-12T04:10:30.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/obj55 2024-04-12T04:10:30.336 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI= 2024-04-12T04:10:30.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI=', 'Content-Length': '0'} 2024-04-12T04:10:30.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004980eff2ba0fc596-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.338 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.338 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.338 DEBUG:boto:path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.338 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.339 DEBUG:boto:Method: GET 2024-04-12T04:10:30.339 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.339 DEBUG:boto:Data: 2024-04-12T04:10:30.339 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:30.339 DEBUG:boto:Port: 8001 2024-04-12T04:10:30.339 DEBUG:boto:Params: {} 2024-04-12T04:10:30.339 DEBUG:boto:Token: None 2024-04-12T04:10:30.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01f2c215c9aa807-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.342 DEBUG:boto:b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.342 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:30.342 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:30.342 DEBUG:boto:path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.342 DEBUG:boto:auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.343 DEBUG:boto:Method: GET 2024-04-12T04:10:30.343 DEBUG:boto:Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:10:30.343 DEBUG:boto:Data: 2024-04-12T04:10:30.343 DEBUG:boto:Headers: {} 2024-04-12T04:10:30.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:30.343 DEBUG:boto:Port: 8000 2024-04-12T04:10:30.343 DEBUG:boto:Params: {} 2024-04-12T04:10:30.343 DEBUG:boto:Token: None 2024-04-12T04:10:30.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:30 GMT /dgbpxg-66/ 2024-04-12T04:10:30.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:10:30.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:10:30.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3c12b824f1e276-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:30.346 DEBUG:boto:b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:30.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dgbpxg-66 zones={a2, a1} 2024-04-12T04:10:30.346 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:10:30.346 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:10:30.347 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:10:30.364+0000 7fc17bf9f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 112588) UID: 0 2024-04-12T04:10:30.347 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:10:30.364+0000 7fc1811747c0 -1 shutting down 2024-04-12T04:10:32.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:10:36.446 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:10:36.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.447 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:36.552 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:36.553 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:36.557 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2024-04-12T04:10:36.597 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.597 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.598 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.598 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.598 DEBUG:boto:Data: 2024-04-12T04:10:36.598 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.598 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.598 DEBUG:boto:Params: {} 2024-04-12T04:10:36.598 DEBUG:boto:Token: None 2024-04-12T04:10:36.598 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.599 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061bdec6070dd203f-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.602 DEBUG:boto:path=/dgbpxg-66/obj17 2024-04-12T04:10:36.602 DEBUG:boto:auth_path=/dgbpxg-66/obj17 2024-04-12T04:10:36.602 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.602 DEBUG:boto:Path: /dgbpxg-66/obj17 2024-04-12T04:10:36.602 DEBUG:boto:Data: 2024-04-12T04:10:36.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.603 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.603 DEBUG:boto:Params: {} 2024-04-12T04:10:36.603 DEBUG:boto:Token: None 2024-04-12T04:10:36.603 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj17 2024-04-12T04:10:36.603 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uqr2jhTFyf+TQprgX15ua4KOdTE= 2024-04-12T04:10:36.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqr2jhTFyf+TQprgX15ua4KOdTE='} 2024-04-12T04:10:36.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb695bdb9678ef05-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.608 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.608 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.609 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.609 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.609 DEBUG:boto:Data: 2024-04-12T04:10:36.609 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.609 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.609 DEBUG:boto:Params: {} 2024-04-12T04:10:36.609 DEBUG:boto:Token: None 2024-04-12T04:10:36.609 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.609 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1394b976c41d7cd-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.612 DEBUG:boto:path=/dgbpxg-66/obj34 2024-04-12T04:10:36.612 DEBUG:boto:auth_path=/dgbpxg-66/obj34 2024-04-12T04:10:36.612 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.612 DEBUG:boto:Path: /dgbpxg-66/obj34 2024-04-12T04:10:36.612 DEBUG:boto:Data: 2024-04-12T04:10:36.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.613 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.613 DEBUG:boto:Params: {} 2024-04-12T04:10:36.613 DEBUG:boto:Token: None 2024-04-12T04:10:36.613 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj34 2024-04-12T04:10:36.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pEy1fx+Sgu7e/xxnYWy9akq/xwY= 2024-04-12T04:10:36.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pEy1fx+Sgu7e/xxnYWy9akq/xwY='} 2024-04-12T04:10:36.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9834779ad899747-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.654 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.655 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.655 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.655 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.655 DEBUG:boto:Data: 2024-04-12T04:10:36.655 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.655 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.655 DEBUG:boto:Params: {} 2024-04-12T04:10:36.655 DEBUG:boto:Token: None 2024-04-12T04:10:36.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.656 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c686d7b6643ef2-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.658 DEBUG:boto:path=/dgbpxg-66/obj51 2024-04-12T04:10:36.658 DEBUG:boto:auth_path=/dgbpxg-66/obj51 2024-04-12T04:10:36.658 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.658 DEBUG:boto:Path: /dgbpxg-66/obj51 2024-04-12T04:10:36.658 DEBUG:boto:Data: 2024-04-12T04:10:36.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.659 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.659 DEBUG:boto:Params: {} 2024-04-12T04:10:36.659 DEBUG:boto:Token: None 2024-04-12T04:10:36.659 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj51 2024-04-12T04:10:36.659 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:apOJYnItgo3FoIXIUMQQQTN6HWo= 2024-04-12T04:10:36.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:apOJYnItgo3FoIXIUMQQQTN6HWo='} 2024-04-12T04:10:36.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040cb8925e2051e65-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.664 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.665 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.665 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.665 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.665 DEBUG:boto:Data: 2024-04-12T04:10:36.665 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.665 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.665 DEBUG:boto:Params: {} 2024-04-12T04:10:36.665 DEBUG:boto:Token: None 2024-04-12T04:10:36.665 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.665 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5f258c77b7e7793-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.668 DEBUG:boto:path=/dgbpxg-66/obj68 2024-04-12T04:10:36.668 DEBUG:boto:auth_path=/dgbpxg-66/obj68 2024-04-12T04:10:36.668 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.668 DEBUG:boto:Path: /dgbpxg-66/obj68 2024-04-12T04:10:36.668 DEBUG:boto:Data: 2024-04-12T04:10:36.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.669 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.669 DEBUG:boto:Params: {} 2024-04-12T04:10:36.669 DEBUG:boto:Token: None 2024-04-12T04:10:36.669 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj68 2024-04-12T04:10:36.669 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bK3d4x7MvzQ1Gwb5gpSA9aUAwhw= 2024-04-12T04:10:36.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bK3d4x7MvzQ1Gwb5gpSA9aUAwhw='} 2024-04-12T04:10:36.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cca05c7cb5b1d3de-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.715 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.715 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.715 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.715 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.715 DEBUG:boto:Data: 2024-04-12T04:10:36.715 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.715 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.715 DEBUG:boto:Params: {} 2024-04-12T04:10:36.715 DEBUG:boto:Token: None 2024-04-12T04:10:36.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.716 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e55f34c7058ad3ef-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.718 DEBUG:boto:path=/dgbpxg-66/obj85 2024-04-12T04:10:36.718 DEBUG:boto:auth_path=/dgbpxg-66/obj85 2024-04-12T04:10:36.718 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.718 DEBUG:boto:Path: /dgbpxg-66/obj85 2024-04-12T04:10:36.719 DEBUG:boto:Data: 2024-04-12T04:10:36.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.719 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.719 DEBUG:boto:Params: {} 2024-04-12T04:10:36.719 DEBUG:boto:Token: None 2024-04-12T04:10:36.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj85 2024-04-12T04:10:36.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rQRWhOcqmQsb38YpN6CgrcAzb9w= 2024-04-12T04:10:36.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rQRWhOcqmQsb38YpN6CgrcAzb9w='} 2024-04-12T04:10:36.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009acaaab8f3f46ecb-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.725 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:36.823 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:36.823 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:36.829 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2024-04-12T04:10:36.874 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.874 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.875 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.875 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.875 DEBUG:boto:Data: 2024-04-12T04:10:36.875 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.875 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.875 DEBUG:boto:Params: {} 2024-04-12T04:10:36.875 DEBUG:boto:Token: None 2024-04-12T04:10:36.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af970a8d0c11f50e-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.878 DEBUG:boto:path=/dgbpxg-66/obj19 2024-04-12T04:10:36.879 DEBUG:boto:auth_path=/dgbpxg-66/obj19 2024-04-12T04:10:36.879 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.879 DEBUG:boto:Path: /dgbpxg-66/obj19 2024-04-12T04:10:36.879 DEBUG:boto:Data: 2024-04-12T04:10:36.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.879 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.879 DEBUG:boto:Params: {} 2024-04-12T04:10:36.879 DEBUG:boto:Token: None 2024-04-12T04:10:36.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj19 2024-04-12T04:10:36.880 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FYbnQbMIVPgapU+FKhP0v+nCFgQ= 2024-04-12T04:10:36.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FYbnQbMIVPgapU+FKhP0v+nCFgQ='} 2024-04-12T04:10:36.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa2adeb5205d37c-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.885 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.885 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.886 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.886 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.886 DEBUG:boto:Data: 2024-04-12T04:10:36.886 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.886 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.886 DEBUG:boto:Params: {} 2024-04-12T04:10:36.886 DEBUG:boto:Token: None 2024-04-12T04:10:36.886 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.886 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e73e68df47b82cca-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.889 DEBUG:boto:path=/dgbpxg-66/obj38 2024-04-12T04:10:36.889 DEBUG:boto:auth_path=/dgbpxg-66/obj38 2024-04-12T04:10:36.889 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.889 DEBUG:boto:Path: /dgbpxg-66/obj38 2024-04-12T04:10:36.889 DEBUG:boto:Data: 2024-04-12T04:10:36.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.890 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.890 DEBUG:boto:Params: {} 2024-04-12T04:10:36.890 DEBUG:boto:Token: None 2024-04-12T04:10:36.890 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj38 2024-04-12T04:10:36.890 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:unABRh5bTOl1w8TPqoG5c/T4awM= 2024-04-12T04:10:36.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:unABRh5bTOl1w8TPqoG5c/T4awM='} 2024-04-12T04:10:36.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000738dd4687ab29e7a-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.893 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.893 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.894 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.894 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.894 DEBUG:boto:Data: 2024-04-12T04:10:36.894 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.894 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.894 DEBUG:boto:Params: {} 2024-04-12T04:10:36.894 DEBUG:boto:Token: None 2024-04-12T04:10:36.894 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.894 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad241aac06187881-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.897 DEBUG:boto:path=/dgbpxg-66/obj57 2024-04-12T04:10:36.897 DEBUG:boto:auth_path=/dgbpxg-66/obj57 2024-04-12T04:10:36.897 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.897 DEBUG:boto:Path: /dgbpxg-66/obj57 2024-04-12T04:10:36.897 DEBUG:boto:Data: 2024-04-12T04:10:36.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.897 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.897 DEBUG:boto:Params: {} 2024-04-12T04:10:36.898 DEBUG:boto:Token: None 2024-04-12T04:10:36.898 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj57 2024-04-12T04:10:36.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:88Hmh0M1Gy0CHxGa8y/eybsAhF0= 2024-04-12T04:10:36.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88Hmh0M1Gy0CHxGa8y/eybsAhF0='} 2024-04-12T04:10:36.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008428d6cd19bb515c-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.901 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.901 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.902 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.902 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.902 DEBUG:boto:Data: 2024-04-12T04:10:36.902 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.902 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.902 DEBUG:boto:Params: {} 2024-04-12T04:10:36.902 DEBUG:boto:Token: None 2024-04-12T04:10:36.902 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.902 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002df48289e77b0c53-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.905 DEBUG:boto:path=/dgbpxg-66/obj76 2024-04-12T04:10:36.906 DEBUG:boto:auth_path=/dgbpxg-66/obj76 2024-04-12T04:10:36.906 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.906 DEBUG:boto:Path: /dgbpxg-66/obj76 2024-04-12T04:10:36.906 DEBUG:boto:Data: 2024-04-12T04:10:36.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.906 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.906 DEBUG:boto:Params: {} 2024-04-12T04:10:36.906 DEBUG:boto:Token: None 2024-04-12T04:10:36.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj76 2024-04-12T04:10:36.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1SrFk0DQ9YayBxGt2kMLxf7qUCs= 2024-04-12T04:10:36.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SrFk0DQ9YayBxGt2kMLxf7qUCs='} 2024-04-12T04:10:36.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a46f6c248dca53-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.912 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:36.912 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:36.912 DEBUG:boto:Method: HEAD 2024-04-12T04:10:36.912 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:36.912 DEBUG:boto:Data: 2024-04-12T04:10:36.912 DEBUG:boto:Headers: {} 2024-04-12T04:10:36.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.912 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.913 DEBUG:boto:Params: {} 2024-04-12T04:10:36.913 DEBUG:boto:Token: None 2024-04-12T04:10:36.913 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/ 2024-04-12T04:10:36.913 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:10:36.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:10:36.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fea01e8789ee917-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.916 DEBUG:boto:path=/dgbpxg-66/obj95 2024-04-12T04:10:36.916 DEBUG:boto:auth_path=/dgbpxg-66/obj95 2024-04-12T04:10:36.916 DEBUG:boto:Method: PUT 2024-04-12T04:10:36.916 DEBUG:boto:Path: /dgbpxg-66/obj95 2024-04-12T04:10:36.916 DEBUG:boto:Data: 2024-04-12T04:10:36.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:36.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:36.916 DEBUG:boto:Port: 8000 2024-04-12T04:10:36.917 DEBUG:boto:Params: {} 2024-04-12T04:10:36.917 DEBUG:boto:Token: None 2024-04-12T04:10:36.917 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:36 GMT /dgbpxg-66/obj95 2024-04-12T04:10:36.917 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4+1pZqfV01VJK5OGfHEGb/STL9g= 2024-04-12T04:10:36.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+1pZqfV01VJK5OGfHEGb/STL9g='} 2024-04-12T04:10:36.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ece6dfd09f4828e7-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:36.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:36.959 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:37.058 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:37.058 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:37.065 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2024-04-12T04:10:37.114 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.114 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.115 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.115 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.115 DEBUG:boto:Data: 2024-04-12T04:10:37.115 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.115 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.115 DEBUG:boto:Params: {} 2024-04-12T04:10:37.115 DEBUG:boto:Token: None 2024-04-12T04:10:37.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed70d8d4e1977242-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.119 DEBUG:boto:path=/dgbpxg-66/obj23 2024-04-12T04:10:37.119 DEBUG:boto:auth_path=/dgbpxg-66/obj23 2024-04-12T04:10:37.119 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.119 DEBUG:boto:Path: /dgbpxg-66/obj23 2024-04-12T04:10:37.119 DEBUG:boto:Data: 2024-04-12T04:10:37.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.120 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.120 DEBUG:boto:Params: {} 2024-04-12T04:10:37.120 DEBUG:boto:Token: None 2024-04-12T04:10:37.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj23 2024-04-12T04:10:37.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5e9elyVxMGYCRkvIIcS7MYnaSgY= 2024-04-12T04:10:37.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5e9elyVxMGYCRkvIIcS7MYnaSgY='} 2024-04-12T04:10:37.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026a9932c7354fb51-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.125 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.125 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.125 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.125 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.125 DEBUG:boto:Data: 2024-04-12T04:10:37.125 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.126 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.126 DEBUG:boto:Params: {} 2024-04-12T04:10:37.126 DEBUG:boto:Token: None 2024-04-12T04:10:37.126 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.126 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000512805d1f08d7e8b-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.129 DEBUG:boto:path=/dgbpxg-66/obj46 2024-04-12T04:10:37.129 DEBUG:boto:auth_path=/dgbpxg-66/obj46 2024-04-12T04:10:37.129 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.129 DEBUG:boto:Path: /dgbpxg-66/obj46 2024-04-12T04:10:37.129 DEBUG:boto:Data: 2024-04-12T04:10:37.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.130 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.130 DEBUG:boto:Params: {} 2024-04-12T04:10:37.130 DEBUG:boto:Token: None 2024-04-12T04:10:37.130 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj46 2024-04-12T04:10:37.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:D47hnpgTuGWH7yHDDJgpLMhwHik= 2024-04-12T04:10:37.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D47hnpgTuGWH7yHDDJgpLMhwHik='} 2024-04-12T04:10:37.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9a82e5eba5bd6ae-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.135 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.135 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.136 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.136 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.136 DEBUG:boto:Data: 2024-04-12T04:10:37.136 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.136 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.136 DEBUG:boto:Params: {} 2024-04-12T04:10:37.136 DEBUG:boto:Token: None 2024-04-12T04:10:37.136 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.136 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dab56b520e5fefd-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.139 DEBUG:boto:path=/dgbpxg-66/obj69 2024-04-12T04:10:37.139 DEBUG:boto:auth_path=/dgbpxg-66/obj69 2024-04-12T04:10:37.139 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.140 DEBUG:boto:Path: /dgbpxg-66/obj69 2024-04-12T04:10:37.140 DEBUG:boto:Data: 2024-04-12T04:10:37.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.140 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.140 DEBUG:boto:Params: {} 2024-04-12T04:10:37.140 DEBUG:boto:Token: None 2024-04-12T04:10:37.140 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj69 2024-04-12T04:10:37.140 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rwBBJygixEXzAL0CjgHE5RDPosw= 2024-04-12T04:10:37.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwBBJygixEXzAL0CjgHE5RDPosw='} 2024-04-12T04:10:37.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016250765597d6c34-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.183 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.183 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.183 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.183 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.183 DEBUG:boto:Data: 2024-04-12T04:10:37.183 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.183 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.183 DEBUG:boto:Params: {} 2024-04-12T04:10:37.183 DEBUG:boto:Token: None 2024-04-12T04:10:37.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7f05196a53aa483-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.186 DEBUG:boto:path=/dgbpxg-66/obj92 2024-04-12T04:10:37.186 DEBUG:boto:auth_path=/dgbpxg-66/obj92 2024-04-12T04:10:37.187 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.187 DEBUG:boto:Path: /dgbpxg-66/obj92 2024-04-12T04:10:37.187 DEBUG:boto:Data: 2024-04-12T04:10:37.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.187 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.187 DEBUG:boto:Params: {} 2024-04-12T04:10:37.187 DEBUG:boto:Token: None 2024-04-12T04:10:37.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj92 2024-04-12T04:10:37.187 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r7mr2Mi/NI4F88wuPuGORfpkc/I= 2024-04-12T04:10:37.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r7mr2Mi/NI4F88wuPuGORfpkc/I='} 2024-04-12T04:10:37.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c097850834e2e259-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.193 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.193 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.193 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.193 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.193 DEBUG:boto:Data: 2024-04-12T04:10:37.193 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.193 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.193 DEBUG:boto:Params: {} 2024-04-12T04:10:37.193 DEBUG:boto:Token: None 2024-04-12T04:10:37.194 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.194 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f46aa8103ad8d31d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.197 DEBUG:boto:path=/dgbpxg-66/obj115 2024-04-12T04:10:37.197 DEBUG:boto:auth_path=/dgbpxg-66/obj115 2024-04-12T04:10:37.197 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.197 DEBUG:boto:Path: /dgbpxg-66/obj115 2024-04-12T04:10:37.198 DEBUG:boto:Data: 2024-04-12T04:10:37.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.198 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.198 DEBUG:boto:Params: {} 2024-04-12T04:10:37.199 DEBUG:boto:Token: None 2024-04-12T04:10:37.199 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj115 2024-04-12T04:10:37.199 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:o9G4Cm6Ea3y4hylE0z5VHM+zxOI= 2024-04-12T04:10:37.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o9G4Cm6Ea3y4hylE0z5VHM+zxOI='} 2024-04-12T04:10:37.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff4ada091939bb3f-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.205 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:37.304 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:37.304 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:37.312 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2024-04-12T04:10:37.366 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.367 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.367 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.367 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.367 DEBUG:boto:Data: 2024-04-12T04:10:37.367 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.367 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.367 DEBUG:boto:Params: {} 2024-04-12T04:10:37.368 DEBUG:boto:Token: None 2024-04-12T04:10:37.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.368 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002504a63f946b361f-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.371 DEBUG:boto:path=/dgbpxg-66/obj29 2024-04-12T04:10:37.372 DEBUG:boto:auth_path=/dgbpxg-66/obj29 2024-04-12T04:10:37.372 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.372 DEBUG:boto:Path: /dgbpxg-66/obj29 2024-04-12T04:10:37.372 DEBUG:boto:Data: 2024-04-12T04:10:37.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.372 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.372 DEBUG:boto:Params: {} 2024-04-12T04:10:37.372 DEBUG:boto:Token: None 2024-04-12T04:10:37.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj29 2024-04-12T04:10:37.373 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qz/qg6KUhlnbap8uF52v/nOqomo= 2024-04-12T04:10:37.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qz/qg6KUhlnbap8uF52v/nOqomo='} 2024-04-12T04:10:37.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a6f0b71b6f66ac8-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.419 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.419 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.419 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.419 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.419 DEBUG:boto:Data: 2024-04-12T04:10:37.419 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.419 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.420 DEBUG:boto:Params: {} 2024-04-12T04:10:37.420 DEBUG:boto:Token: None 2024-04-12T04:10:37.420 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.420 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021a7c34b936ebc24-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.423 DEBUG:boto:path=/dgbpxg-66/obj58 2024-04-12T04:10:37.423 DEBUG:boto:auth_path=/dgbpxg-66/obj58 2024-04-12T04:10:37.423 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.424 DEBUG:boto:Path: /dgbpxg-66/obj58 2024-04-12T04:10:37.424 DEBUG:boto:Data: 2024-04-12T04:10:37.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.424 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.424 DEBUG:boto:Params: {} 2024-04-12T04:10:37.424 DEBUG:boto:Token: None 2024-04-12T04:10:37.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj58 2024-04-12T04:10:37.424 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JqYfvpM7Cq++4lw701aUdRK6GpY= 2024-04-12T04:10:37.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JqYfvpM7Cq++4lw701aUdRK6GpY='} 2024-04-12T04:10:37.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e2cadcb55db28fd-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.430 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.430 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.430 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.430 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.430 DEBUG:boto:Data: 2024-04-12T04:10:37.430 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.430 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.430 DEBUG:boto:Params: {} 2024-04-12T04:10:37.430 DEBUG:boto:Token: None 2024-04-12T04:10:37.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.431 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fd5eb20cb288ee7-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.434 DEBUG:boto:path=/dgbpxg-66/obj87 2024-04-12T04:10:37.434 DEBUG:boto:auth_path=/dgbpxg-66/obj87 2024-04-12T04:10:37.434 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.434 DEBUG:boto:Path: /dgbpxg-66/obj87 2024-04-12T04:10:37.434 DEBUG:boto:Data: 2024-04-12T04:10:37.434 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.435 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.435 DEBUG:boto:Params: {} 2024-04-12T04:10:37.435 DEBUG:boto:Token: None 2024-04-12T04:10:37.435 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj87 2024-04-12T04:10:37.435 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:649sxG1smdjDaXqneK3ATr5yzNs= 2024-04-12T04:10:37.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:649sxG1smdjDaXqneK3ATr5yzNs='} 2024-04-12T04:10:37.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c8e7c388ac32d8c-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.479 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.479 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.479 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.479 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.479 DEBUG:boto:Data: 2024-04-12T04:10:37.479 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.479 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.479 DEBUG:boto:Params: {} 2024-04-12T04:10:37.479 DEBUG:boto:Token: None 2024-04-12T04:10:37.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001477ecf848edfffa-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.483 DEBUG:boto:path=/dgbpxg-66/obj116 2024-04-12T04:10:37.483 DEBUG:boto:auth_path=/dgbpxg-66/obj116 2024-04-12T04:10:37.483 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.483 DEBUG:boto:Path: /dgbpxg-66/obj116 2024-04-12T04:10:37.483 DEBUG:boto:Data: 2024-04-12T04:10:37.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.483 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.483 DEBUG:boto:Params: {} 2024-04-12T04:10:37.483 DEBUG:boto:Token: None 2024-04-12T04:10:37.483 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj116 2024-04-12T04:10:37.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:psnAcLcgvVrZi/x5QvnLwVisuYc= 2024-04-12T04:10:37.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:psnAcLcgvVrZi/x5QvnLwVisuYc='} 2024-04-12T04:10:37.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037b318073f118519-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.488 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.489 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.489 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.489 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.489 DEBUG:boto:Data: 2024-04-12T04:10:37.489 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.489 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.489 DEBUG:boto:Params: {} 2024-04-12T04:10:37.489 DEBUG:boto:Token: None 2024-04-12T04:10:37.489 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.490 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001230c3cb3649cdfc-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.492 DEBUG:boto:path=/dgbpxg-66/obj145 2024-04-12T04:10:37.492 DEBUG:boto:auth_path=/dgbpxg-66/obj145 2024-04-12T04:10:37.493 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.493 DEBUG:boto:Path: /dgbpxg-66/obj145 2024-04-12T04:10:37.493 DEBUG:boto:Data: 2024-04-12T04:10:37.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.493 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.493 DEBUG:boto:Params: {} 2024-04-12T04:10:37.493 DEBUG:boto:Token: None 2024-04-12T04:10:37.493 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj145 2024-04-12T04:10:37.493 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Es56RVxGOU0J9mbArFKDSJAp4/8= 2024-04-12T04:10:37.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Es56RVxGOU0J9mbArFKDSJAp4/8='} 2024-04-12T04:10:37.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035740b3bd54bad78-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.539 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:37.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:37.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:37.638 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:37.638 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:37.650 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2024-04-12T04:10:37.712 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.712 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.713 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.713 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.713 DEBUG:boto:Data: 2024-04-12T04:10:37.713 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.713 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.713 DEBUG:boto:Params: {} 2024-04-12T04:10:37.713 DEBUG:boto:Token: None 2024-04-12T04:10:37.713 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.714 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000013a1c90976a762d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.718 DEBUG:boto:path=/dgbpxg-66/obj31 2024-04-12T04:10:37.718 DEBUG:boto:auth_path=/dgbpxg-66/obj31 2024-04-12T04:10:37.718 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.718 DEBUG:boto:Path: /dgbpxg-66/obj31 2024-04-12T04:10:37.718 DEBUG:boto:Data: 2024-04-12T04:10:37.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.718 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.718 DEBUG:boto:Params: {} 2024-04-12T04:10:37.718 DEBUG:boto:Token: None 2024-04-12T04:10:37.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj31 2024-04-12T04:10:37.719 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zLHRNvVDRlBQn3jsaa+pC4FRTi0= 2024-04-12T04:10:37.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLHRNvVDRlBQn3jsaa+pC4FRTi0='} 2024-04-12T04:10:37.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e16302e9d1200d51-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.724 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.724 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.724 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.724 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.724 DEBUG:boto:Data: 2024-04-12T04:10:37.724 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.725 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.725 DEBUG:boto:Params: {} 2024-04-12T04:10:37.725 DEBUG:boto:Token: None 2024-04-12T04:10:37.725 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.725 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a55d3e7887b991-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.728 DEBUG:boto:path=/dgbpxg-66/obj62 2024-04-12T04:10:37.728 DEBUG:boto:auth_path=/dgbpxg-66/obj62 2024-04-12T04:10:37.728 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.729 DEBUG:boto:Path: /dgbpxg-66/obj62 2024-04-12T04:10:37.729 DEBUG:boto:Data: 2024-04-12T04:10:37.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.729 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.729 DEBUG:boto:Params: {} 2024-04-12T04:10:37.729 DEBUG:boto:Token: None 2024-04-12T04:10:37.729 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj62 2024-04-12T04:10:37.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5/LLMLxE3CFJhMdghaxciLPbs+8= 2024-04-12T04:10:37.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5/LLMLxE3CFJhMdghaxciLPbs+8='} 2024-04-12T04:10:37.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040059dc5615cb235-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.735 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.735 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.735 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.735 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.735 DEBUG:boto:Data: 2024-04-12T04:10:37.735 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.735 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.735 DEBUG:boto:Params: {} 2024-04-12T04:10:37.736 DEBUG:boto:Token: None 2024-04-12T04:10:37.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3305f97cce9d30d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.739 DEBUG:boto:path=/dgbpxg-66/obj93 2024-04-12T04:10:37.739 DEBUG:boto:auth_path=/dgbpxg-66/obj93 2024-04-12T04:10:37.739 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.740 DEBUG:boto:Path: /dgbpxg-66/obj93 2024-04-12T04:10:37.740 DEBUG:boto:Data: 2024-04-12T04:10:37.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.740 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.740 DEBUG:boto:Params: {} 2024-04-12T04:10:37.740 DEBUG:boto:Token: None 2024-04-12T04:10:37.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj93 2024-04-12T04:10:37.740 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9zjVWyHaapzFgdzeCZewmU6BWwQ= 2024-04-12T04:10:37.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9zjVWyHaapzFgdzeCZewmU6BWwQ='} 2024-04-12T04:10:37.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ed559ce16a20910-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.783 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.783 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.783 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.783 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.783 DEBUG:boto:Data: 2024-04-12T04:10:37.783 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.783 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.783 DEBUG:boto:Params: {} 2024-04-12T04:10:37.783 DEBUG:boto:Token: None 2024-04-12T04:10:37.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.784 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f5e110b93ea213c-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.787 DEBUG:boto:path=/dgbpxg-66/obj124 2024-04-12T04:10:37.787 DEBUG:boto:auth_path=/dgbpxg-66/obj124 2024-04-12T04:10:37.787 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.787 DEBUG:boto:Path: /dgbpxg-66/obj124 2024-04-12T04:10:37.787 DEBUG:boto:Data: 2024-04-12T04:10:37.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.788 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.788 DEBUG:boto:Params: {} 2024-04-12T04:10:37.788 DEBUG:boto:Token: None 2024-04-12T04:10:37.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj124 2024-04-12T04:10:37.788 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kiSSjjRP48dv4Nrqrm/JvucxQgE= 2024-04-12T04:10:37.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kiSSjjRP48dv4Nrqrm/JvucxQgE='} 2024-04-12T04:10:37.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbe21b0f43383303-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.794 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:37.794 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:37.794 DEBUG:boto:Method: HEAD 2024-04-12T04:10:37.794 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:37.794 DEBUG:boto:Data: 2024-04-12T04:10:37.794 DEBUG:boto:Headers: {} 2024-04-12T04:10:37.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.794 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.794 DEBUG:boto:Params: {} 2024-04-12T04:10:37.795 DEBUG:boto:Token: None 2024-04-12T04:10:37.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/ 2024-04-12T04:10:37.795 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:10:37.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:10:37.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031b7b3185fc004e0-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.798 DEBUG:boto:path=/dgbpxg-66/obj155 2024-04-12T04:10:37.798 DEBUG:boto:auth_path=/dgbpxg-66/obj155 2024-04-12T04:10:37.798 DEBUG:boto:Method: PUT 2024-04-12T04:10:37.798 DEBUG:boto:Path: /dgbpxg-66/obj155 2024-04-12T04:10:37.798 DEBUG:boto:Data: 2024-04-12T04:10:37.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:37.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:37.799 DEBUG:boto:Port: 8000 2024-04-12T04:10:37.799 DEBUG:boto:Params: {} 2024-04-12T04:10:37.799 DEBUG:boto:Token: None 2024-04-12T04:10:37.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:37 GMT /dgbpxg-66/obj155 2024-04-12T04:10:37.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:b6PjyC0CKKVmZGJ7Ut3Md+U+FLY= 2024-04-12T04:10:37.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b6PjyC0CKKVmZGJ7Ut3Md+U+FLY='} 2024-04-12T04:10:37.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004759fe3fad22a21-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:37.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:37.843 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:37.944 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2024-04-12T04:10:37.944 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: dgbpxg-66 2024-04-12T04:10:37.956 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2024-04-12T04:10:38.023 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:38.023 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:38.023 DEBUG:boto:Method: HEAD 2024-04-12T04:10:38.024 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:38.024 DEBUG:boto:Data: 2024-04-12T04:10:38.024 DEBUG:boto:Headers: {} 2024-04-12T04:10:38.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.024 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.024 DEBUG:boto:Params: {} 2024-04-12T04:10:38.025 DEBUG:boto:Token: None 2024-04-12T04:10:38.025 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/ 2024-04-12T04:10:38.025 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:10:38.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:10:38.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa5609ae43d59234-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.029 DEBUG:boto:path=/dgbpxg-66/obj37 2024-04-12T04:10:38.029 DEBUG:boto:auth_path=/dgbpxg-66/obj37 2024-04-12T04:10:38.029 DEBUG:boto:Method: PUT 2024-04-12T04:10:38.029 DEBUG:boto:Path: /dgbpxg-66/obj37 2024-04-12T04:10:38.029 DEBUG:boto:Data: 2024-04-12T04:10:38.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:38.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.029 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.029 DEBUG:boto:Params: {} 2024-04-12T04:10:38.030 DEBUG:boto:Token: None 2024-04-12T04:10:38.030 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/obj37 2024-04-12T04:10:38.030 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:WmDLsvUOD7A2zhszllLz6OHAcX8= 2024-04-12T04:10:38.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WmDLsvUOD7A2zhszllLz6OHAcX8='} 2024-04-12T04:10:38.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058df0aabd2f07b79-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.035 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:38.035 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:38.035 DEBUG:boto:Method: HEAD 2024-04-12T04:10:38.036 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:38.036 DEBUG:boto:Data: 2024-04-12T04:10:38.036 DEBUG:boto:Headers: {} 2024-04-12T04:10:38.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.036 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.036 DEBUG:boto:Params: {} 2024-04-12T04:10:38.037 DEBUG:boto:Token: None 2024-04-12T04:10:38.037 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/ 2024-04-12T04:10:38.037 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:10:38.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:10:38.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f151d1451fc0969c-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.040 DEBUG:boto:path=/dgbpxg-66/obj74 2024-04-12T04:10:38.041 DEBUG:boto:auth_path=/dgbpxg-66/obj74 2024-04-12T04:10:38.041 DEBUG:boto:Method: PUT 2024-04-12T04:10:38.041 DEBUG:boto:Path: /dgbpxg-66/obj74 2024-04-12T04:10:38.041 DEBUG:boto:Data: 2024-04-12T04:10:38.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:38.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.041 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.041 DEBUG:boto:Params: {} 2024-04-12T04:10:38.042 DEBUG:boto:Token: None 2024-04-12T04:10:38.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/obj74 2024-04-12T04:10:38.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Rd3GLe1qNSCaCqKb7XQSR95dQUM= 2024-04-12T04:10:38.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rd3GLe1qNSCaCqKb7XQSR95dQUM='} 2024-04-12T04:10:38.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a088c4a06442065-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.087 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:38.087 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:38.087 DEBUG:boto:Method: HEAD 2024-04-12T04:10:38.087 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:38.087 DEBUG:boto:Data: 2024-04-12T04:10:38.087 DEBUG:boto:Headers: {} 2024-04-12T04:10:38.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.087 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.087 DEBUG:boto:Params: {} 2024-04-12T04:10:38.088 DEBUG:boto:Token: None 2024-04-12T04:10:38.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/ 2024-04-12T04:10:38.088 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:10:38.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:10:38.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005545d66562e747e7-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.092 DEBUG:boto:path=/dgbpxg-66/obj111 2024-04-12T04:10:38.093 DEBUG:boto:auth_path=/dgbpxg-66/obj111 2024-04-12T04:10:38.093 DEBUG:boto:Method: PUT 2024-04-12T04:10:38.093 DEBUG:boto:Path: /dgbpxg-66/obj111 2024-04-12T04:10:38.093 DEBUG:boto:Data: 2024-04-12T04:10:38.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:38.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.093 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.093 DEBUG:boto:Params: {} 2024-04-12T04:10:38.094 DEBUG:boto:Token: None 2024-04-12T04:10:38.094 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/obj111 2024-04-12T04:10:38.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zxN8Zcq61D69+Mmf3WIqbPxUb7w= 2024-04-12T04:10:38.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxN8Zcq61D69+Mmf3WIqbPxUb7w='} 2024-04-12T04:10:38.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a913c0487b1d6913-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.099 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:38.099 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:38.099 DEBUG:boto:Method: HEAD 2024-04-12T04:10:38.099 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:38.099 DEBUG:boto:Data: 2024-04-12T04:10:38.099 DEBUG:boto:Headers: {} 2024-04-12T04:10:38.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.100 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.100 DEBUG:boto:Params: {} 2024-04-12T04:10:38.100 DEBUG:boto:Token: None 2024-04-12T04:10:38.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/ 2024-04-12T04:10:38.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:10:38.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:10:38.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006316d501d39b04d0-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.104 DEBUG:boto:path=/dgbpxg-66/obj148 2024-04-12T04:10:38.104 DEBUG:boto:auth_path=/dgbpxg-66/obj148 2024-04-12T04:10:38.104 DEBUG:boto:Method: PUT 2024-04-12T04:10:38.104 DEBUG:boto:Path: /dgbpxg-66/obj148 2024-04-12T04:10:38.104 DEBUG:boto:Data: 2024-04-12T04:10:38.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:38.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.105 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.105 DEBUG:boto:Params: {} 2024-04-12T04:10:38.105 DEBUG:boto:Token: None 2024-04-12T04:10:38.105 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/obj148 2024-04-12T04:10:38.105 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:K7meYlMiznH9EE2Bj94GCJIfx+I= 2024-04-12T04:10:38.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K7meYlMiznH9EE2Bj94GCJIfx+I='} 2024-04-12T04:10:38.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba54cf9742c1549c-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.147 DEBUG:boto:path=/dgbpxg-66/ 2024-04-12T04:10:38.147 DEBUG:boto:auth_path=/dgbpxg-66/ 2024-04-12T04:10:38.147 DEBUG:boto:Method: HEAD 2024-04-12T04:10:38.147 DEBUG:boto:Path: /dgbpxg-66/ 2024-04-12T04:10:38.147 DEBUG:boto:Data: 2024-04-12T04:10:38.147 DEBUG:boto:Headers: {} 2024-04-12T04:10:38.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.147 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.147 DEBUG:boto:Params: {} 2024-04-12T04:10:38.147 DEBUG:boto:Token: None 2024-04-12T04:10:38.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/ 2024-04-12T04:10:38.148 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:10:38.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:10:38.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd7085ae970c4a0-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.151 DEBUG:boto:path=/dgbpxg-66/obj185 2024-04-12T04:10:38.151 DEBUG:boto:auth_path=/dgbpxg-66/obj185 2024-04-12T04:10:38.151 DEBUG:boto:Method: PUT 2024-04-12T04:10:38.151 DEBUG:boto:Path: /dgbpxg-66/obj185 2024-04-12T04:10:38.152 DEBUG:boto:Data: 2024-04-12T04:10:38.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:10:38.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:10:38.152 DEBUG:boto:Port: 8000 2024-04-12T04:10:38.152 DEBUG:boto:Params: {} 2024-04-12T04:10:38.152 DEBUG:boto:Token: None 2024-04-12T04:10:38.152 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:10:38 GMT /dgbpxg-66/obj185 2024-04-12T04:10:38.152 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:mvpIjQrZmRW0z4g76yS3tiLV9oI= 2024-04-12T04:10:38.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mvpIjQrZmRW0z4g76yS3tiLV9oI='} 2024-04-12T04:10:38.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3ea89bb8649886f-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:38.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:38.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:38.158 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:38.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:38.255+0000 7fdb2b0f8780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:10:38.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:38.257+0000 7fdb2b0f8780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:10:38.260 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:10:38.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:10:38.263 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:10:38.263 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:10:38.284 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:10:38.284 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:10:38.284 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:10:38.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:10:39.285 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:10:39.306 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:10:39.306 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:10:39.307 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:10:39.308 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2024-04-12T04:10:39.308 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-66 2024-04-12T04:10:39.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:39.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:39.309 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:10:39.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:39.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:39.456+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:10:44.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:44.457+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:10:49.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:49.458+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:10:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:54.458+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:10:59.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:10:59.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:04.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:09.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:09.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:14.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:14.460+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:19.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:19.461+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:24.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:24.462+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:29.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:29.463+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:34.463+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:39.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:44.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:49.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:49.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:54.465+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:11:59.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:11:59.466+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:04.467+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:09.467+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:14.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:14.468+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:19.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:19.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:24.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:24.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:29.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:29.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:34.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:34.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:39.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:39.471+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:12:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:44.472+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:12:45.977 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:12:45.994+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:12:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:49.473+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:12:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:54.473+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:12:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:12:59.474+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:04.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:09.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:09.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:14.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:14.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:19.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:19.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:24.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:24.477+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:29.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:29.477+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:34.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:34.478+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:39.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:39.478+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:44.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:44.479+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:49.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:49.480+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:54.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:54.479+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:13:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:13:59.480+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:04.481+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:09.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:09.482+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:14.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:14.483+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:19.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:19.483+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:24.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:24.484+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:29.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:29.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:34.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:34.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:39.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:39.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:44.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:44.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:49.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:49.487+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:54.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:54.487+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:14:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:14:59.488+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:04.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:04.489+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:09.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:09.489+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:14.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:14.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:19.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:19.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:24.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:24.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:29.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:29.491+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:15:34.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:34.493+0000 7f22bbbf5780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-12T04:15:34.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:34.493+0000 7f22bbbf5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:15:34.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:34.493+0000 7f22bbbf5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:15:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T04:15:34.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-12T04:15:34.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:34.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:34.499 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:15:34.598 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:15:34.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.598 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:15:34.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:34.600 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:37.981797Z" 2024-04-12T04:15:34.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.600 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:15:34.601 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:34.601 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:15:34.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:34.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:34.602 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:34.650 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:34.650 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:34.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:34.697+0000 7ff0c73ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:34.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:34.698+0000 7ff0c73ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:34.693 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:34.693 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:34.693 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:34.693 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:34.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:34.695 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:34Z" 2024-04-12T04:15:34.699 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:34.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:34Z" } 2024-04-12T04:15:34.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:34.701 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:34.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:34.701 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:15:34.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.701 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:34.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:34.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:34.789 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:15:34.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.790 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:34.883 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:15:34.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.884 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:34.931 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:34.931 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:34.977 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:34.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:34.977 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.068 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-12T04:15:35.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.069 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.113 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.113 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.159 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:35.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.160 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.211 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.212 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.257 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:35.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.258 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.357 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:35.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.357 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.409 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.409 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.454 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:15:35.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.455 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.501 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.546 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-12T04:15:35.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:35.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:35.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:35.644 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13, 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.645 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:15:35.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:15:35.647 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:15:35.648 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:15:35.649 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:15:35.650 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.651 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.652 INFO:teuthology.orchestra.run.smithi099.stdout: "directional": [ 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "source_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "dest_zone": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:15:35.653 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:15:35.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.656 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:15:35.657 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:15:35.658 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:15:35.660 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:15:35.661 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:15:35.662 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.663 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:15:35.664 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:15:35.664 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:15:35.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.666 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:15:35.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:15:35.668 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:15:35.669 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:15:35.669 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:15:35.669 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:15:35.669 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:15:35.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:15:35.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:35.671 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:35.672 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-67 2024-04-12T04:15:35.672 DEBUG:boto:path=/dgbpxg-67/ 2024-04-12T04:15:35.672 DEBUG:boto:auth_path=/dgbpxg-67/ 2024-04-12T04:15:35.672 DEBUG:boto:Method: PUT 2024-04-12T04:15:35.673 DEBUG:boto:Path: /dgbpxg-67/ 2024-04-12T04:15:35.673 DEBUG:boto:Data: 2024-04-12T04:15:35.673 DEBUG:boto:Headers: {} 2024-04-12T04:15:35.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:15:35.673 DEBUG:boto:Port: 8000 2024-04-12T04:15:35.673 DEBUG:boto:Params: {} 2024-04-12T04:15:35.673 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:15:35.674 DEBUG:boto:Token: None 2024-04-12T04:15:35.674 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:15:35 GMT /dgbpxg-67/ 2024-04-12T04:15:35.674 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uEkODwW++VitlW6KWuXHU5OiE4I= 2024-04-12T04:15:35.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:35 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uEkODwW++VitlW6KWuXHU5OiE4I=', 'Content-Length': '0'} 2024-04-12T04:15:35.998 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000067862eaa96ac6f37-006618b567-23e0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:35.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:35.998 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.248 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:15:36.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.249 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.327 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.327 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.418 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:36.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.419 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.549 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-12T04:15:36.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.550 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.649 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:36.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.650 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.754 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:36.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.755 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.859 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:15:36.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.860 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:36.963 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:15:36.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:36.964 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:37.062 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-04-12T04:15:37.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:37.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:37.063 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:37.141 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:15:37.154 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:15:36.966601Z" 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.156 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:15:37.157 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:37.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:15:37.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:37.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:37.158 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:37.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:37.252+0000 7f812561f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:37.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:37.253+0000 7f812561f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:37Z" 2024-04-12T04:15:37.254 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:37.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:37Z" } 2024-04-12T04:15:37.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:37.255 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:37.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:37.255 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:15:42.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:42.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:42.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:42.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:42.362+0000 7f4ef29d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:42.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:42.363+0000 7f4ef29d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:42.359 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:42.359 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:42.359 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:42.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:42.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:42.362 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:42.363 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:42.364 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:42Z" 2024-04-12T04:15:42.365 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:42.366 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:42Z" } 2024-04-12T04:15:42.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:42.366 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:42.366 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:42.367 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:15:47.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:47.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:47.367 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:47.474+0000 7fde89c5b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:47.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:47.475+0000 7fde89c5b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:47Z" 2024-04-12T04:15:47.477 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:47.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:47Z" } 2024-04-12T04:15:47.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:47.478 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:47.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:47.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:15:52.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:52.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:52.479 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:52.585+0000 7f8baf457780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:52.586+0000 7f8baf457780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:52.582 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:52.582 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:52.582 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:52.582 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:52.582 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:52.584 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:52.585 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:52.586 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.587 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:52.588 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:52.588 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:52.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:52.588 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:52Z" 2024-04-12T04:15:52.588 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:52.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:52Z" } 2024-04-12T04:15:52.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:52.589 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:52.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:52.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:15:57.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:57.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:57.591 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:57.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:57.701+0000 7f03383d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:15:57.702+0000 7f03383d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:57.698 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:15:57.698 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:15:57.698 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:57.699 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:57.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:57.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:15:57.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:15:57.704 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:15:57.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:15:57.705 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:15:57Z" 2024-04-12T04:15:57.705 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:15:57.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:57Z" } 2024-04-12T04:15:57.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:15:57.706 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:15:57.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:15:57.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:02.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:02.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:02.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:02.764 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:02.764 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:02.814+0000 7fe04156f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:02.815+0000 7fe04156f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:02.811 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:02Z" 2024-04-12T04:16:02.817 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:02.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:02Z" } 2024-04-12T04:16:02.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:02.819 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:02.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:02.819 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:07.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:07.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:07.820 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:07.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:07.925+0000 7fb4d612a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:07.926+0000 7fb4d612a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:07.923 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:07.923 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:07.923 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:07.923 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:07.924 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:07.925 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:07.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:07.927 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:07.928 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:07.930 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:07Z" 2024-04-12T04:16:07.930 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:07.931 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:07Z" } 2024-04-12T04:16:07.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:07.931 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:07.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:07.931 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:12.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:12.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:12.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:13.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:13.040+0000 7f4d28806780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:13.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:13.041+0000 7f4d28806780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:13.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:13.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:13.039 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:13.040 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:13.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:13Z" 2024-04-12T04:16:13.042 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:13.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:13Z" } 2024-04-12T04:16:13.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:13.044 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:13.044 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:13.044 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:18.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:18.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:18.045 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:18.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:18.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:18.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:18.155+0000 7fc94e5ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:18.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:18.156+0000 7fc94e5ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:18Z" 2024-04-12T04:16:18.159 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:18.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:18Z" } 2024-04-12T04:16:18.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:18.161 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:18.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:18.161 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:23.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:23.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:23.162 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:23.269+0000 7eff15bab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:23.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:23.270+0000 7eff15bab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:23.266 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:23.267 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:23.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:23.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:23.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:23.271 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:23Z" 2024-04-12T04:16:23.272 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:23.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:23Z" } 2024-04-12T04:16:23.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:23.273 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:23.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:23.274 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:28.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:28.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:28.276 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:28.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:28.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:28.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:28.385+0000 7fecc4ab8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:28.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:28.387+0000 7fecc4ab8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:28.383 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:28.384 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:28.385 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:28.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:28.388 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:28Z" 2024-04-12T04:16:28.389 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:28.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:28Z" } 2024-04-12T04:16:28.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:28.391 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:28.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:28.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:33.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:33.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:33.393 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:33.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:33.450 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:33.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:33.501+0000 7f9e2aec0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:33.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:33.502+0000 7f9e2aec0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:33.498 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:33.498 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:33.498 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:33.499 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.501 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:33.502 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:33.503 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:33.504 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:33Z" 2024-04-12T04:16:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:33.506 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:33Z" } 2024-04-12T04:16:33.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:33.507 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:33.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:33.507 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:38.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:38.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:38.509 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:38.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:38.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:38.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:38.617+0000 7f158d23f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:38.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:38.618+0000 7f158d23f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:38.613 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:38.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:38.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:38.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:38.618 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:38Z" 2024-04-12T04:16:38.619 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:38.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:38Z" } 2024-04-12T04:16:38.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:38.621 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:38.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:38.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:43.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:43.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:43.622 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:43.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:43.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:43.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:43.729+0000 7f272ac15780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:43.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:43.731+0000 7f272ac15780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:43.726 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:43.726 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:43.727 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:43.728 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:43.729 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:43.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:43.731 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:43Z" 2024-04-12T04:16:43.732 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:43.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:43Z" } 2024-04-12T04:16:43.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:43.733 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:43.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:43.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:48.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:48.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:48.735 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:48.791 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:48.791 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:48.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:48.841+0000 7fc954751780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:48.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:48.842+0000 7fc954751780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:48.839 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:48.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:48.841 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:48.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:48.843 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:48.844 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:48.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:48.845 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:48Z" 2024-04-12T04:16:48.845 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:48.846 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:48Z" } 2024-04-12T04:16:48.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:48.846 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:48.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:48.846 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:53.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:53.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:53.848 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:53.903 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:53.903 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:53.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:53.953+0000 7f1bb9892780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:53.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:53.954+0000 7f1bb9892780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:53Z" 2024-04-12T04:16:53.956 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:53.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:53Z" } 2024-04-12T04:16:53.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:53.957 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:53.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:53.958 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:16:58.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:58.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:16:58.959 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:16:59.015 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:16:59.015 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:16:59.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:59.066+0000 7f7b0e754780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:59.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:16:59.067+0000 7f7b0e754780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:16:59.063 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:16:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:16:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:16:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:16:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:16:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:16:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:16:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:16:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:16:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:16:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:16:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:16:59Z" 2024-04-12T04:16:59.069 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:16:59.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:16:59Z" } 2024-04-12T04:16:59.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:16:59.070 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:16:59.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:16:59.070 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:17:04.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:04.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:04.072 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:04.127 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:04.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:04.179+0000 7f544d2eb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:04.180+0000 7f544d2eb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:04.176 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:04.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:04.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:04.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:04.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:04.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:04.182 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:04Z" 2024-04-12T04:17:04.183 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:04.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:04Z" } 2024-04-12T04:17:04.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:17:04.184 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:04.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:17:04.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:17:09.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:09.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:09.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:09.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:09.294+0000 7fb4fde28780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:09.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:09.295+0000 7fb4fde28780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:09.294 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:09Z" 2024-04-12T04:17:09.300 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:09.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:09Z" } 2024-04-12T04:17:09.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:17:09.301 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:09.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:17:09.301 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:17:14.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:14.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:14.303 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:14.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:14.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:14.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:14.410+0000 7fe94e9c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:14.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:14.411+0000 7fe94e9c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:14.406 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:14.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:14.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:14.409 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:14.411 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:14.412 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895037.981797_1748.1", 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:37.981797Z", 2024-04-12T04:17:14.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:14.414 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:14Z" 2024-04-12T04:17:14.415 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:14.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895037.981797_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:37.981797Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:14Z" } 2024-04-12T04:17:14.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895037.981797_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:37.981797Z', 'realm_epoch': 2}}] 2024-04-12T04:17:14.416 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:14.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895037.981797_1748.1'} 2024-04-12T04:17:14.416 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895037.981797_1748.1 2024-04-12T04:17:19.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:19.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:19.418 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:19.473 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:19.473 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:19.524+0000 7fa222085780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:19.525+0000 7fa222085780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:19.520 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:19.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:19.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:19.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.755487_1825.1", 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.755487Z", 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:19.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:19Z" 2024-04-12T04:17:19.526 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:19.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.755487_1825.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.755487Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:19Z" } 2024-04-12T04:17:19.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.755487_1825.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.755487Z', 'realm_epoch': 2}}] 2024-04-12T04:17:19.528 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:19.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.755487_1825.1'} 2024-04-12T04:17:19.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895336.755487_1825.1 2024-04-12T04:17:24.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:24.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:24.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:24.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:24.635+0000 7ff052a29780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:24.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:24.636+0000 7ff052a29780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:24.632 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895335.973007_1782.1", 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:35.973007Z", 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:24Z" 2024-04-12T04:17:24.639 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:24.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895335.973007_1782.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:35.973007Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:24Z" } 2024-04-12T04:17:24.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895335.973007_1782.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:35.973007Z', 'realm_epoch': 2}}] 2024-04-12T04:17:24.640 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:24.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895335.973007_1782.1'} 2024-04-12T04:17:24.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895336.966601_1830.1 target=1_1712895335.973007_1782.1 2024-04-12T04:17:29.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:29.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:29.642 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:29.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:29.751+0000 7f64becef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:29.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:29.752+0000 7f64becef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:29.749 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:29.752 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:17:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.966601Z", 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:29.755 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:29Z" 2024-04-12T04:17:29.755 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:29.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.966601_1830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.966601Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:29Z" } 2024-04-12T04:17:29.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.966601_1830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.966601Z', 'realm_epoch': 2}}] 2024-04-12T04:17:29.756 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:29.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:29.756 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:17:29.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.757 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:29.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:29.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:29.859 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:29.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.860 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:29.964 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:29.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:29.965 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:30.018 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:30.019 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:30.069 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:30.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:30.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:30.070 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:30.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:30.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:30.167 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:17:30.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:30.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:30.168 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:30.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:30.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:30.264 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:17:30.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:30.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:30.264 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:30.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:30.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:30.341 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:17:30.354 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.355 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:17:30.171194Z" 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.356 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:17:30.357 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:30.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:17:30.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:30.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:30.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:30.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:30.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:30.452+0000 7f60baa3a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:30.453+0000 7f60baa3a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:30.448 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:30.448 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:30.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:30.450 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.966601Z", 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:30.453 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:30Z" 2024-04-12T04:17:30.454 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:30.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.966601_1830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.966601Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:30Z" } 2024-04-12T04:17:30.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.966601_1830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.966601Z', 'realm_epoch': 2}}] 2024-04-12T04:17:30.455 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:30.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:30.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895450.171194_1845.1 target=1_1712895336.966601_1830.1 2024-04-12T04:17:35.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:35.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:35.457 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:35.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:35.563+0000 7f123ad2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:35.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:35.564+0000 7f123ad2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.561 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:35.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:35.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:35.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.966601Z", 2024-04-12T04:17:35.565 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:35Z" 2024-04-12T04:17:35.566 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:35.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.966601_1830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.966601Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:35Z" } 2024-04-12T04:17:35.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.966601_1830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.966601Z', 'realm_epoch': 2}}] 2024-04-12T04:17:35.568 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:35.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:35.568 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895450.171194_1845.1 target=1_1712895336.966601_1830.1 2024-04-12T04:17:40.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:40.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:40.570 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:40.627 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:40.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:40.678+0000 7fb92ea29780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:40.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:40.679+0000 7fb92ea29780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:40.674 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:40.676 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:40.677 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.966601Z", 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:40.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:40Z" 2024-04-12T04:17:40.680 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:40.681 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.966601_1830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.966601Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:40Z" } 2024-04-12T04:17:40.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.966601_1830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.966601Z', 'realm_epoch': 2}}] 2024-04-12T04:17:40.681 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:40.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:40.682 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895450.171194_1845.1 target=1_1712895336.966601_1830.1 2024-04-12T04:17:45.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:45.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:45.683 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:45.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:45.793+0000 7fd66335c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:45.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:45.794+0000 7fd66335c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:45.790 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895336.966601_1830.1", 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:15:36.966601Z", 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:45Z" 2024-04-12T04:17:45.796 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:45.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895336.966601_1830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:36.966601Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:45Z" } 2024-04-12T04:17:45.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895336.966601_1830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:36.966601Z', 'realm_epoch': 2}}] 2024-04-12T04:17:45.798 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:45.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895336.966601_1830.1'} 2024-04-12T04:17:45.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895450.171194_1845.1 target=1_1712895336.966601_1830.1 2024-04-12T04:17:50.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:50.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:50.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:50.907+0000 7f4d95737780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:50.908+0000 7f4d95737780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:50.904 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:50Z" 2024-04-12T04:17:50.910 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:50.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:50Z" } 2024-04-12T04:17:50.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:17:50.912 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:50.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:50.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:17:50.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:50.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:50.913 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:50.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:50.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.011 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:51.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.012 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.112 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:51.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.113 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.212 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:17:51.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.212 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.306 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:17:51.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.307 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.350 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.396 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:17:51.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.397 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.496 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:51.496 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:51.496 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14, 2024-04-12T04:17:51.496 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:17:51.496 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:17:51.497 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.498 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.499 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:51.500 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.502 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:17:51.503 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.504 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:51.505 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:51.506 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:51.506 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:51.506 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:17:51.509 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:51.510 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.514 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:17:51.515 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:51.516 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:17:51.517 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.518 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:51.520 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-04-12T04:17:51.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:51.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:51.522 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:17:51.639 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:51.640 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:17:30.171194Z" 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.641 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:17:51.642 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:51.642 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:17:51.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:51.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:51.643 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:51.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:51.832+0000 7f7fb73dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:51.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:51.835+0000 7f7fb73dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:51.865 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:51.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:51.867 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:51.868 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:51.869 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:51.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:51Z" 2024-04-12T04:17:51.871 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:51.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:51Z" } 2024-04-12T04:17:51.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:17:51.872 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:51.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:51.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:17:51.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:51.873 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:51.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:51.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.108 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:17:52.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.108 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.179 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.279 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.280 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:17:52.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.410 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:17:52.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.410 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.510 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:17:52.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.511 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.606 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:17:52.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.657 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.657 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.707 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:52.708 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:52.708 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15, 2024-04-12T04:17:52.708 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:17:52.709 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:17:52.709 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:17:52.709 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:52.709 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:17:52.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.710 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:17:52.710 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:17:52.710 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:17:52.710 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:17:52.710 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.711 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:17:52.711 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.711 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:17:52.711 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:17:52.711 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.712 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:52.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.712 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.712 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:17:52.712 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.713 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:17:52.713 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.713 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:52.713 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:52.713 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:52.714 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:52.714 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:52.714 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:52.714 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:52.714 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:52.715 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:52.715 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:52.715 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.715 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.716 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:17:52.716 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:17:52.716 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.717 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:17:52.717 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.717 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:52.717 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:52.717 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:52.718 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:52.718 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:52.718 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:52.718 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:52.718 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.719 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:17:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:17:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:17:52.720 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:17:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:17:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.721 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:17:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:17:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:17:52.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:17:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:17:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:17:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.723 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.724 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:17:52.725 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:17:52.726 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:17:52.727 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:17:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:17:52.729 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:17:52.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.729 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.729 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.730 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.731 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:17:52.732 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.733 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:17:52.734 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:17:52.735 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:17:52.736 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:17:52.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.739 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:17:52.739 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:17:52.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:17:52.741 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:52.742 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:52.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:17:52.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:17:52.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:52.745 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:52.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:17:52.745 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:17:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:52.874 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:17:52.874 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-68 2024-04-12T04:17:52.874 DEBUG:boto:path=/dgbpxg-68/ 2024-04-12T04:17:52.874 DEBUG:boto:auth_path=/dgbpxg-68/ 2024-04-12T04:17:52.875 DEBUG:boto:Method: PUT 2024-04-12T04:17:52.875 DEBUG:boto:Path: /dgbpxg-68/ 2024-04-12T04:17:52.875 DEBUG:boto:Data: 2024-04-12T04:17:52.875 DEBUG:boto:Headers: {} 2024-04-12T04:17:52.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:17:52.875 DEBUG:boto:Port: 8000 2024-04-12T04:17:52.875 DEBUG:boto:Params: {} 2024-04-12T04:17:52.876 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:17:52.876 DEBUG:boto:Token: None 2024-04-12T04:17:52.876 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:17:52 GMT /dgbpxg-68/ 2024-04-12T04:17:52.876 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9Mph+az2Hr6KsjLddB7GmlPS1Js= 2024-04-12T04:17:52.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Mph+az2Hr6KsjLddB7GmlPS1Js=', 'Content-Length': '0'} 2024-04-12T04:17:52.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ddcc5cc7b0909dcf-006618b5f0-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:52.918 DEBUG:boto:path=/dgbpxg-68/ 2024-04-12T04:17:52.918 DEBUG:boto:auth_path=/dgbpxg-68/ 2024-04-12T04:17:52.918 DEBUG:boto:Method: HEAD 2024-04-12T04:17:52.918 DEBUG:boto:Path: /dgbpxg-68/ 2024-04-12T04:17:52.918 DEBUG:boto:Data: 2024-04-12T04:17:52.918 DEBUG:boto:Headers: {} 2024-04-12T04:17:52.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:17:52.919 DEBUG:boto:Port: 8000 2024-04-12T04:17:52.919 DEBUG:boto:Params: {} 2024-04-12T04:17:52.919 DEBUG:boto:Token: None 2024-04-12T04:17:52.919 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:17:52 GMT /dgbpxg-68/ 2024-04-12T04:17:52.919 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XQwe9IO8KAeXmPbkMVr3Jlu8zYo= 2024-04-12T04:17:52.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XQwe9IO8KAeXmPbkMVr3Jlu8zYo=', 'Content-Length': '0'} 2024-04-12T04:17:52.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a448229c66da58-006618b5f0-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:52.922 DEBUG:boto:path=/dgbpxg-68/obj1 2024-04-12T04:17:52.922 DEBUG:boto:auth_path=/dgbpxg-68/obj1 2024-04-12T04:17:52.922 DEBUG:boto:Method: PUT 2024-04-12T04:17:52.922 DEBUG:boto:Path: /dgbpxg-68/obj1 2024-04-12T04:17:52.923 DEBUG:boto:Data: 2024-04-12T04:17:52.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:17:52.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:17:52.923 DEBUG:boto:Port: 8000 2024-04-12T04:17:52.923 DEBUG:boto:Params: {} 2024-04-12T04:17:52.923 DEBUG:boto:Token: None 2024-04-12T04:17:52.923 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:17:52 GMT /dgbpxg-68/obj1 2024-04-12T04:17:52.923 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:45Aa49m0Oz3dZnQUteL5Zv1sbnk= 2024-04-12T04:17:52.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:45Aa49m0Oz3dZnQUteL5Zv1sbnk='} 2024-04-12T04:17:52.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000847efa78c6e263c7-006618b5f0-25b5-a1'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:52.938 INFO:rgw_multi.tests:create bucket zone=a2 name=dgbpxg-69 2024-04-12T04:17:52.938 DEBUG:boto:path=/dgbpxg-69/ 2024-04-12T04:17:52.939 DEBUG:boto:auth_path=/dgbpxg-69/ 2024-04-12T04:17:52.939 DEBUG:boto:Method: PUT 2024-04-12T04:17:52.939 DEBUG:boto:Path: /dgbpxg-69/ 2024-04-12T04:17:52.939 DEBUG:boto:Data: 2024-04-12T04:17:52.939 DEBUG:boto:Headers: {} 2024-04-12T04:17:52.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:17:52.939 DEBUG:boto:Port: 8001 2024-04-12T04:17:52.939 DEBUG:boto:Params: {} 2024-04-12T04:17:52.939 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:17:52.940 DEBUG:boto:Token: None 2024-04-12T04:17:52.940 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:17:52 GMT /dgbpxg-69/ 2024-04-12T04:17:52.940 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KixI0QcTNQ8i3DsVkaj+H3bwYXQ= 2024-04-12T04:17:52.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KixI0QcTNQ8i3DsVkaj+H3bwYXQ=', 'Content-Length': '0'} 2024-04-12T04:17:53.114 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bcb036023695b2f5-006618b5f1-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:53.114 DEBUG:boto:path=/dgbpxg-69/ 2024-04-12T04:17:53.114 DEBUG:boto:auth_path=/dgbpxg-69/ 2024-04-12T04:17:53.115 DEBUG:boto:Method: HEAD 2024-04-12T04:17:53.115 DEBUG:boto:Path: /dgbpxg-69/ 2024-04-12T04:17:53.115 DEBUG:boto:Data: 2024-04-12T04:17:53.115 DEBUG:boto:Headers: {} 2024-04-12T04:17:53.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:17:53.115 DEBUG:boto:Port: 8001 2024-04-12T04:17:53.115 DEBUG:boto:Params: {} 2024-04-12T04:17:53.115 DEBUG:boto:Token: None 2024-04-12T04:17:53.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:17:53 GMT /dgbpxg-69/ 2024-04-12T04:17:53.116 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:f7Bvs4aZ2r50vv9w7XFhPYYdvco= 2024-04-12T04:17:53.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f7Bvs4aZ2r50vv9w7XFhPYYdvco=', 'Content-Length': '0'} 2024-04-12T04:17:53.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000367bb8380f08a35e-006618b5f1-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:53.120 DEBUG:boto:path=/dgbpxg-69/obj2 2024-04-12T04:17:53.120 DEBUG:boto:auth_path=/dgbpxg-69/obj2 2024-04-12T04:17:53.120 DEBUG:boto:Method: PUT 2024-04-12T04:17:53.120 DEBUG:boto:Path: /dgbpxg-69/obj2 2024-04-12T04:17:53.120 DEBUG:boto:Data: 2024-04-12T04:17:53.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:17:53.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:17:53.121 DEBUG:boto:Port: 8001 2024-04-12T04:17:53.121 DEBUG:boto:Params: {} 2024-04-12T04:17:53.121 DEBUG:boto:Token: None 2024-04-12T04:17:53.121 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:17:53 GMT /dgbpxg-69/obj2 2024-04-12T04:17:53.121 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:F7dcn+SSe1DYPErdWjaHAq3nJ4k= 2024-04-12T04:17:53.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:17:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F7dcn+SSe1DYPErdWjaHAq3nJ4k='} 2024-04-12T04:17:53.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f652871d6f02f5d3-006618b5f1-25c1-a2'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:17:53.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:53.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:53.140 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:53.293 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:17:53.314 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:17:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:17:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:17:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:17:53.109714Z" 2024-04-12T04:17:53.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.316 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:17:53.316 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:17:53.317 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:17:53.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:53.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:53.317 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:53.381 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:53.382 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:53.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:53.442+0000 7f41a6d2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:53.444+0000 7f41a6d2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:53Z" 2024-04-12T04:17:53.448 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:53.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:53Z" } 2024-04-12T04:17:53.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:17:53.449 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:17:53.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:53.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:17:58.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:58.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:17:58.451 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:17:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:17:58.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:17:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:58.556+0000 7fdbb9005780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:17:58.557+0000 7fdbb9005780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:17:58.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:17:58.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:58.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:17:58.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:17:58.556 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:17:58.557 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:17:58.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:17:58.558 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:17:58Z" 2024-04-12T04:17:58.558 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:17:58.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:17:58Z" } 2024-04-12T04:17:58.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:17:58.559 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:17:58.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:17:58.559 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:03.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:03.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:03.561 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:03.617 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:03.617 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:03.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:03.668+0000 7fb8332ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:03.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:03.669+0000 7fb8332ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:03.664 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:03.664 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:03.664 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:03.664 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:03.670 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:03Z" 2024-04-12T04:18:03.670 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:03.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:03Z" } 2024-04-12T04:18:03.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:03.671 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:03.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:03.671 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:08.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:08.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:08.672 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:08.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:08.777+0000 7fbb5b4fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:08.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:08.778+0000 7fbb5b4fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:08.773 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:08.773 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:08.773 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:08.773 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:08.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:08.775 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:08.776 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:08.777 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.778 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:08.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:08.779 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:08.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:08.779 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:08Z" 2024-04-12T04:18:08.779 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:08.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:08Z" } 2024-04-12T04:18:08.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:08.780 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:08.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:08.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:13.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:13.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:13.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:13.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:13.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:13.888+0000 7f705842f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:13.889+0000 7f705842f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:13.885 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:13.885 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:13.886 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:13.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:13.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:13.890 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:13Z" 2024-04-12T04:18:13.891 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:13.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:13Z" } 2024-04-12T04:18:13.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:13.893 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:13.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:13.893 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:18.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:18.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:18.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:18.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:19.000+0000 7f0fbc0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:18.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:19.001+0000 7f0fbc0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:18.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:18.998 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:18.999 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:19.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:19.001 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:19Z" 2024-04-12T04:18:19.002 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:19.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:19Z" } 2024-04-12T04:18:19.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:19.003 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:19.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:19.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:24.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:24.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:24.005 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:24.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:24.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:24.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:24.113+0000 7f4696e21780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:24.114+0000 7f4696e21780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:24.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:24Z" 2024-04-12T04:18:24.115 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:24.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:24Z" } 2024-04-12T04:18:24.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:24.116 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:24.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:24.117 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:29.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:29.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:29.118 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:29.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:29.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:29.225+0000 7f1ef517c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:29.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:29.226+0000 7f1ef517c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:29.222 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:29.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:29.222 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:29.223 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:29.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:29.226 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:29.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:29.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:29.229 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:29.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:29.229 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:29Z" 2024-04-12T04:18:29.229 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:29.230 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:29Z" } 2024-04-12T04:18:29.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:29.230 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:29.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:29.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:34.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:34.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:34.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:34.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:34.338+0000 7fdc54566780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:34.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:34.339+0000 7fdc54566780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:34.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:34.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:34.336 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:34.337 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:34.338 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:34.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:34.340 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:34Z" 2024-04-12T04:18:34.340 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:34.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:34Z" } 2024-04-12T04:18:34.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:34.341 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:34.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:34.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:39.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:39.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:39.343 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:39.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:39.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:39.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:39.451+0000 7f7897db0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:39.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:39.452+0000 7f7897db0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.448 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:39.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:39.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:39.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:39.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:39Z" 2024-04-12T04:18:39.453 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:39.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:39Z" } 2024-04-12T04:18:39.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:39.455 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:39.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:39.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:44.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:44.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:44.457 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:44.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:44.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:44.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:44.563+0000 7f09db0ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:44.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:44.564+0000 7f09db0ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:44.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:44.561 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:44.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:44.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:44.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:44.565 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:44Z" 2024-04-12T04:18:44.566 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:44.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:44Z" } 2024-04-12T04:18:44.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:44.567 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:44.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:44.567 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:49.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:49.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:49.569 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:49.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:49.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:49.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:49.676+0000 7fa450256780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:49.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:49.677+0000 7fa450256780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.673 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:49.674 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:49.675 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:49.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:49.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:49.678 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:49Z" 2024-04-12T04:18:49.679 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:49.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:49Z" } 2024-04-12T04:18:49.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:49.680 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:49.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:49.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:54.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:54.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:54.681 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:54.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:54.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:54.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:54.786+0000 7fe625d31780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:54.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:54.787+0000 7fe625d31780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:54.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:54.786 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:54.787 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:54.788 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:54.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:54.790 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:54Z" 2024-04-12T04:18:54.790 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:54.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:54Z" } 2024-04-12T04:18:54.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:54.791 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:54.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:54.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:18:59.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:59.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:18:59.793 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:18:59.847 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:18:59.847 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:18:59.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:59.896+0000 7f503d915780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:18:59.897+0000 7f503d915780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:18:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:18:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:18:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:18:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:18:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:18:59Z" 2024-04-12T04:18:59.900 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:18:59.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:18:59Z" } 2024-04-12T04:18:59.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:18:59.901 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:18:59.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:18:59.901 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:04.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:04.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:04.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:04.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:04.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:05.009+0000 7f428a2c3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:04.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:05.010+0000 7f428a2c3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:05.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:05.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:05.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:05.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:19:05.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:05Z" 2024-04-12T04:19:05.011 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:05.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:05Z" } 2024-04-12T04:19:05.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:19:05.013 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:05.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:19:05.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:10.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:10.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:10.014 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:10.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:10.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:10.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:10.121+0000 7f3011060780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:10.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:10.122+0000 7f3011060780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:10.117 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:10.117 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:10.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:10.119 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:10.120 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:10.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:19:10.123 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:10Z" 2024-04-12T04:19:10.124 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:10.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:10Z" } 2024-04-12T04:19:10.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:19:10.126 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:10.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:19:10.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:15.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:15.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:15.128 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:15.184 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:15.184 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:15.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:15.235+0000 7f92d24bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:15.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:15.236+0000 7f92d24bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:15.231 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.232 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:15.233 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:15.234 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:15.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:15.236 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:15.237 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:15Z" 2024-04-12T04:19:15.238 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:15.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:15Z" } 2024-04-12T04:19:15.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:19:15.240 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:15.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:19:15.240 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:20.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:20.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:20.241 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:20.345+0000 7f9fd7cde780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:20.346+0000 7f9fd7cde780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:20.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:20.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:20.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:20.344 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:19:20.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:20Z" 2024-04-12T04:19:20.346 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:20.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:20Z" } 2024-04-12T04:19:20.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:19:20.348 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:20.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:19:20.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:24.115 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:19:24.133+0000 7ff19438b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:19:25.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:25.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:25.349 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:25.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:25.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:25.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:25.457+0000 7f49cd338780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:25.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:25.458+0000 7f49cd338780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:25.453 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:19:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:25Z" 2024-04-12T04:19:25.459 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:25.460 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895450.171194_1845.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:30.171194Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:25Z" } 2024-04-12T04:19:25.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:19:25.461 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:25.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:19:25.461 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:19:30.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.462 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:30.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:30.569+0000 7f12f4838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:30.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:30.570+0000 7f12f4838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:30.566 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:30.566 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:30.567 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:30.568 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:30.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:53.109714Z", 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:19:30.572 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:30.573 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895473.109714_1924.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:53.109714Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:30Z" } 2024-04-12T04:19:30.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895473.109714_1924.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:53.109714Z', 'realm_epoch': 2}}] 2024-04-12T04:19:30.573 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:30.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:30.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:19:30.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.574 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:30.622 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:30.622 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:30.677 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:19:30.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:30.678 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.679 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:19:30.680 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:30.680 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:19:30.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.681 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:30.731 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:30.732 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:30.776 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:19:30.781 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:30.782 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046713", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:10:38.051655Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:30Z" } 2024-04-12T04:19:30.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:19:30.783 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:30.783 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:30.783 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:19:30.783 DEBUG:boto:path=/dgbpxg-68/ 2024-04-12T04:19:30.783 DEBUG:boto:auth_path=/dgbpxg-68/ 2024-04-12T04:19:30.783 DEBUG:boto:Method: HEAD 2024-04-12T04:19:30.783 DEBUG:boto:Path: /dgbpxg-68/ 2024-04-12T04:19:30.784 DEBUG:boto:Data: 2024-04-12T04:19:30.784 DEBUG:boto:Headers: {} 2024-04-12T04:19:30.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:19:30.784 DEBUG:boto:Port: 8001 2024-04-12T04:19:30.784 DEBUG:boto:Params: {} 2024-04-12T04:19:30.784 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:19:30.785 DEBUG:boto:Token: None 2024-04-12T04:19:30.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:30 GMT /dgbpxg-68/ 2024-04-12T04:19:30.785 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aD8NnjQAPwH6HY3SkB7yQ+cb8pM= 2024-04-12T04:19:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aD8NnjQAPwH6HY3SkB7yQ+cb8pM=', 'Content-Length': '0'} 2024-04-12T04:19:30.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dd769ccbb688a75-006618b652-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:30.798 DEBUG:boto:path=/dgbpxg-68/obj1 2024-04-12T04:19:30.798 DEBUG:boto:auth_path=/dgbpxg-68/obj1 2024-04-12T04:19:30.798 DEBUG:boto:Method: HEAD 2024-04-12T04:19:30.799 DEBUG:boto:Path: /dgbpxg-68/obj1 2024-04-12T04:19:30.799 DEBUG:boto:Data: 2024-04-12T04:19:30.799 DEBUG:boto:Headers: {} 2024-04-12T04:19:30.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:19:30.799 DEBUG:boto:Port: 8001 2024-04-12T04:19:30.799 DEBUG:boto:Params: {} 2024-04-12T04:19:30.799 DEBUG:boto:Token: None 2024-04-12T04:19:30.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:30 GMT /dgbpxg-68/obj1 2024-04-12T04:19:30.799 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8rvvpH/Bs6x98+4PcH3b0MsHtP8= 2024-04-12T04:19:30.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8rvvpH/Bs6x98+4PcH3b0MsHtP8=', 'Content-Length': '0'} 2024-04-12T04:19:30.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-68:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b7115a52cc3fe14-006618b652-25c1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:30.801 DEBUG:boto:path=/dgbpxg-68/obj1 2024-04-12T04:19:30.802 DEBUG:boto:auth_path=/dgbpxg-68/obj1 2024-04-12T04:19:30.802 DEBUG:boto:Method: GET 2024-04-12T04:19:30.802 DEBUG:boto:Path: /dgbpxg-68/obj1 2024-04-12T04:19:30.802 DEBUG:boto:Data: 2024-04-12T04:19:30.802 DEBUG:boto:Headers: {} 2024-04-12T04:19:30.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:19:30.802 DEBUG:boto:Port: 8001 2024-04-12T04:19:30.802 DEBUG:boto:Params: {} 2024-04-12T04:19:30.802 DEBUG:boto:Token: None 2024-04-12T04:19:30.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:30 GMT /dgbpxg-68/obj1 2024-04-12T04:19:30.803 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6guTUZqQHsZMrnsBP+UIPhXM+gI= 2024-04-12T04:19:30.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6guTUZqQHsZMrnsBP+UIPhXM+gI=', 'Content-Length': '0'} 2024-04-12T04:19:30.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-68:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d92f63b207bada0-006618b652-25c1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:30.805 DEBUG:boto:path=/dgbpxg-69/ 2024-04-12T04:19:30.805 DEBUG:boto:auth_path=/dgbpxg-69/ 2024-04-12T04:19:30.805 DEBUG:boto:Method: HEAD 2024-04-12T04:19:30.805 DEBUG:boto:Path: /dgbpxg-69/ 2024-04-12T04:19:30.805 DEBUG:boto:Data: 2024-04-12T04:19:30.805 DEBUG:boto:Headers: {} 2024-04-12T04:19:30.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:30.805 DEBUG:boto:Port: 8000 2024-04-12T04:19:30.805 DEBUG:boto:Params: {} 2024-04-12T04:19:30.806 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:19:30.806 DEBUG:boto:Token: None 2024-04-12T04:19:30.806 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:30 GMT /dgbpxg-69/ 2024-04-12T04:19:30.806 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NBmw9rd5Tu5fvwVq75Bp9yjfUJk= 2024-04-12T04:19:30.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBmw9rd5Tu5fvwVq75Bp9yjfUJk=', 'Content-Length': '0'} 2024-04-12T04:19:30.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000339e9357113cd464-006618b652-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:30.818 DEBUG:boto:path=/dgbpxg-69/obj2 2024-04-12T04:19:30.818 DEBUG:boto:auth_path=/dgbpxg-69/obj2 2024-04-12T04:19:30.818 DEBUG:boto:Method: HEAD 2024-04-12T04:19:30.818 DEBUG:boto:Path: /dgbpxg-69/obj2 2024-04-12T04:19:30.818 DEBUG:boto:Data: 2024-04-12T04:19:30.818 DEBUG:boto:Headers: {} 2024-04-12T04:19:30.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:30.818 DEBUG:boto:Port: 8000 2024-04-12T04:19:30.819 DEBUG:boto:Params: {} 2024-04-12T04:19:30.819 DEBUG:boto:Token: None 2024-04-12T04:19:30.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:30 GMT /dgbpxg-69/obj2 2024-04-12T04:19:30.819 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8Vcz+TeT10ROJNpcIj10623OB20= 2024-04-12T04:19:30.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Vcz+TeT10ROJNpcIj10623OB20=', 'Content-Length': '0'} 2024-04-12T04:19:30.820 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008d079799a6c7bb12-006618b652-25b5-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:30.823 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-04-12T04:19:30.824 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T04:19:30.824 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T04:19:30.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.825 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:30.902 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:19:30.916 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:19:30.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:19:30.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:30.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:30.918 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:19:30.918 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:17:53.109714Z" 2024-04-12T04:19:30.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:30.918 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:19:30.919 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:30.919 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:19:30.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:30.919 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:31.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:31.024+0000 7f368fa66780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:31.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:31.025+0000 7f368fa66780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:31.026 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:31.027 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:53.109714Z", 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:31Z" 2024-04-12T04:19:31.032 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:31.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895473.109714_1924.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:53.109714Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:31Z" } 2024-04-12T04:19:31.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895473.109714_1924.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:53.109714Z', 'realm_epoch': 2}}] 2024-04-12T04:19:31.033 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:31.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:19:31.033 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:19:31.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:31.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:31.034 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:31.133 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:19:31.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:31.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:31.133 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:31.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:31.177 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:31.225 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:19:31.226 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-70 2024-04-12T04:19:31.226 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:19:31.226 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:19:31.226 DEBUG:boto:Method: PUT 2024-04-12T04:19:31.226 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:19:31.226 DEBUG:boto:Data: 2024-04-12T04:19:31.226 DEBUG:boto:Headers: {} 2024-04-12T04:19:31.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:31.227 DEBUG:boto:Port: 8000 2024-04-12T04:19:31.227 DEBUG:boto:Params: {} 2024-04-12T04:19:31.227 DEBUG:boto:Token: None 2024-04-12T04:19:31.227 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:19:31 GMT /dgbpxg-70/ 2024-04-12T04:19:31.227 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+RaAmXyJ1joB5ywFiJ5G0/Ckaa4= 2024-04-12T04:19:31.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+RaAmXyJ1joB5ywFiJ5G0/Ckaa4=', 'Content-Length': '0'} 2024-04-12T04:19:31.243 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005657b7ab5859393c-006618b653-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:31.243 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-71 2024-04-12T04:19:31.243 DEBUG:boto:path=/dgbpxg-71/ 2024-04-12T04:19:31.243 DEBUG:boto:auth_path=/dgbpxg-71/ 2024-04-12T04:19:31.243 DEBUG:boto:Method: PUT 2024-04-12T04:19:31.243 DEBUG:boto:Path: /dgbpxg-71/ 2024-04-12T04:19:31.244 DEBUG:boto:Data: 2024-04-12T04:19:31.244 DEBUG:boto:Headers: {} 2024-04-12T04:19:31.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:31.244 DEBUG:boto:Port: 8000 2024-04-12T04:19:31.244 DEBUG:boto:Params: {} 2024-04-12T04:19:31.244 DEBUG:boto:Token: None 2024-04-12T04:19:31.244 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:19:31 GMT /dgbpxg-71/ 2024-04-12T04:19:31.244 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4ixHUCnoZdOsorbBBY16y/tFKa0= 2024-04-12T04:19:31.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:31 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4ixHUCnoZdOsorbBBY16y/tFKa0=', 'Content-Length': '0'} 2024-04-12T04:19:31.263 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b4ab633fed69adb-006618b653-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:31.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:31.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:31.264 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:31.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:31.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:31.367 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:19:31.380 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:31.281159Z" 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.382 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:19:31.383 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:31.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:19:31.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:31.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:31.384 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:31.481+0000 7fe488e86780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:31.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:31.482+0000 7fe488e86780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:31.484 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.485 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.486 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:31.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.254269_1935.1", 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.254269Z", 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:31.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:31Z" 2024-04-12T04:19:31.491 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:31.492 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.254269_1935.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.254269Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:31Z" } 2024-04-12T04:19:31.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.254269_1935.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.254269Z', 'realm_epoch': 2}}] 2024-04-12T04:19:31.493 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:31.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.254269_1935.1'} 2024-04-12T04:19:31.493 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895571.281159_1949.1 target=1_1712895571.254269_1935.1 2024-04-12T04:19:36.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:36.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:36.494 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:36.548 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:36.548 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:36.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:36.600+0000 7fc8dae74780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:36.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:36.601+0000 7fc8dae74780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:36.597 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:36.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:36.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:36.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:36.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:36.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:36Z" 2024-04-12T04:19:36.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:36.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:36Z" } 2024-04-12T04:19:36.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:19:36.605 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:36.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:36.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:19:36.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.605 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:36.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:36.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:36.715 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:19:36.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.716 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:36.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:36.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:36.814 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:19:36.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dgbpxg-70', '--dest-bucket=dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dgbpxg-70', '--dest-bucket=dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.815 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=dgbpxg-70 --dest-bucket=dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:36.868 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:36.868 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:36.935 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:19:36.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:36.936 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:36.984 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:36.984 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:37.056 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:19:37.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.057 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:37.146 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-70","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:19:37.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.147 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:37.245 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:37.245 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:37.245 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 16, 2024-04-12T04:19:37.245 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.246 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.247 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:19:37.248 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:19:37.249 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.250 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:19:37.251 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.252 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-70", 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:19:37.253 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-70", 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:19:37.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.256 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:19:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:19:37.259 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.260 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:19:37.261 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.262 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:19:37.263 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:19:37.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.263 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:19:37.265 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:19:37.267 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:19:37.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:19:37.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:37.269 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:37.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:19:37.269 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:19:37.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:37.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:37.395 INFO:teuthology.orchestra.run.smithi099.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"dgbpxg-70"},"dest":{"zone":"a1","bucket":"dgbpxg-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"dgbpxg-70"},"dest":{"zone":"a2","bucket":"dgbpxg-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-12T04:19:37.395 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:19:37.395 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:19:37.395 DEBUG:boto:Method: HEAD 2024-04-12T04:19:37.396 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:19:37.396 DEBUG:boto:Data: 2024-04-12T04:19:37.396 DEBUG:boto:Headers: {} 2024-04-12T04:19:37.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.396 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.396 DEBUG:boto:Params: {} 2024-04-12T04:19:37.396 DEBUG:boto:Token: None 2024-04-12T04:19:37.397 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/ 2024-04-12T04:19:37.397 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI= 2024-04-12T04:19:37.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI=', 'Content-Length': '0'} 2024-04-12T04:19:37.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008072e335eb7c5d42-006618b659-26e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.483 DEBUG:boto:path=/dgbpxg-70/obj1 2024-04-12T04:19:37.483 DEBUG:boto:auth_path=/dgbpxg-70/obj1 2024-04-12T04:19:37.483 DEBUG:boto:Method: PUT 2024-04-12T04:19:37.484 DEBUG:boto:Path: /dgbpxg-70/obj1 2024-04-12T04:19:37.484 DEBUG:boto:Data: 2024-04-12T04:19:37.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:19:37.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.484 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.484 DEBUG:boto:Params: {} 2024-04-12T04:19:37.484 DEBUG:boto:Token: None 2024-04-12T04:19:37.484 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/obj1 2024-04-12T04:19:37.484 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:izpftYb4KQ9MiK+CISjBk+f97Sg= 2024-04-12T04:19:37.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:izpftYb4KQ9MiK+CISjBk+f97Sg='} 2024-04-12T04:19:37.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013a987076f1a347c-006618b659-26e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.518 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:19:37.518 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:19:37.518 DEBUG:boto:Method: HEAD 2024-04-12T04:19:37.519 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:19:37.519 DEBUG:boto:Data: 2024-04-12T04:19:37.519 DEBUG:boto:Headers: {} 2024-04-12T04:19:37.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.519 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.519 DEBUG:boto:Params: {} 2024-04-12T04:19:37.519 DEBUG:boto:Token: None 2024-04-12T04:19:37.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/ 2024-04-12T04:19:37.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI= 2024-04-12T04:19:37.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI=', 'Content-Length': '0'} 2024-04-12T04:19:37.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071518715f9b30ad6-006618b659-26e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.526 DEBUG:boto:path=/dgbpxg-70/obj2 2024-04-12T04:19:37.526 DEBUG:boto:auth_path=/dgbpxg-70/obj2 2024-04-12T04:19:37.526 DEBUG:boto:Method: PUT 2024-04-12T04:19:37.527 DEBUG:boto:Path: /dgbpxg-70/obj2 2024-04-12T04:19:37.527 DEBUG:boto:Data: 2024-04-12T04:19:37.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:19:37.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.527 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.527 DEBUG:boto:Params: {} 2024-04-12T04:19:37.527 DEBUG:boto:Token: None 2024-04-12T04:19:37.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/obj2 2024-04-12T04:19:37.528 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dOCo+rKvC1ZPyzyOcdhHpbAn2kU= 2024-04-12T04:19:37.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dOCo+rKvC1ZPyzyOcdhHpbAn2kU='} 2024-04-12T04:19:37.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000446e2c01a85ec9b8-006618b659-26e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.570 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:19:37.571 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:19:37.571 DEBUG:boto:Method: HEAD 2024-04-12T04:19:37.571 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:19:37.571 DEBUG:boto:Data: 2024-04-12T04:19:37.571 DEBUG:boto:Headers: {} 2024-04-12T04:19:37.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.571 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.571 DEBUG:boto:Params: {} 2024-04-12T04:19:37.571 DEBUG:boto:Token: None 2024-04-12T04:19:37.571 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/ 2024-04-12T04:19:37.572 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI= 2024-04-12T04:19:37.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HO8IsoLRnYicyzPU7W9pIeyqoGI=', 'Content-Length': '0'} 2024-04-12T04:19:37.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000da63710003d430-006618b659-26e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.577 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:19:37.578 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:19:37.578 DEBUG:boto:Method: PUT 2024-04-12T04:19:37.578 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:19:37.578 DEBUG:boto:Data: 2024-04-12T04:19:37.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:19:37.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.578 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.578 DEBUG:boto:Params: {} 2024-04-12T04:19:37.578 DEBUG:boto:Token: None 2024-04-12T04:19:37.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-70/obj3 2024-04-12T04:19:37.579 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aZBbqKYTUCnbF3gVPX76fhD+yRg= 2024-04-12T04:19:37.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aZBbqKYTUCnbF3gVPX76fhD+yRg='} 2024-04-12T04:19:37.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076ccbb181aed1f09-006618b659-26e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.596 DEBUG:boto:path=/dgbpxg-71/ 2024-04-12T04:19:37.596 DEBUG:boto:auth_path=/dgbpxg-71/ 2024-04-12T04:19:37.596 DEBUG:boto:Method: HEAD 2024-04-12T04:19:37.596 DEBUG:boto:Path: /dgbpxg-71/ 2024-04-12T04:19:37.596 DEBUG:boto:Data: 2024-04-12T04:19:37.597 DEBUG:boto:Headers: {} 2024-04-12T04:19:37.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.597 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.597 DEBUG:boto:Params: {} 2024-04-12T04:19:37.597 DEBUG:boto:Token: None 2024-04-12T04:19:37.597 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-71/ 2024-04-12T04:19:37.597 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:VJtzGOs2WwPCnXo0alhnH7+8QV4= 2024-04-12T04:19:37.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VJtzGOs2WwPCnXo0alhnH7+8QV4=', 'Content-Length': '0'} 2024-04-12T04:19:37.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ece5cb6c1188d659-006618b659-26e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.606 DEBUG:boto:path=/dgbpxg-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:19:37.606 DEBUG:boto:auth_path=/dgbpxg-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:19:37.606 DEBUG:boto:Method: PUT 2024-04-12T04:19:37.606 DEBUG:boto:Path: /dgbpxg-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:19:37.606 DEBUG:boto:Data: 2024-04-12T04:19:37.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:19:37.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:19:37.606 DEBUG:boto:Port: 8000 2024-04-12T04:19:37.606 DEBUG:boto:Params: {} 2024-04-12T04:19:37.606 DEBUG:boto:Token: None 2024-04-12T04:19:37.606 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:19:37 GMT /dgbpxg-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:19:37.607 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yO+G2tx9vzgsgaTNqhkdUQI0SHk= 2024-04-12T04:19:37.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:19:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yO+G2tx9vzgsgaTNqhkdUQI0SHk='} 2024-04-12T04:19:37.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b54c39e069c2c6b-006618b659-26e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:37.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:37.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:37.651 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:37.835 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:31.281159Z" 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:37.892 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:19:37.893 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:37.893 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:19:37.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:37.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:37.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:37.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:37.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:38.077+0000 7f635fa06780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:19:38.078+0000 7f635fa06780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:19:38.102 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:38.102 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:38.102 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:38.102 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:19:38.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:19:38.104 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:38.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.106 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.107 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:19:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:19:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:38Z" 2024-04-12T04:19:38.108 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:38.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:19:38Z" } 2024-04-12T04:19:38.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:19:38.109 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:38.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:19:38.109 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:19:38.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:38.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:38.110 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:38.239 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:19:38.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053365", 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.612612Z" 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:19:38.241 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:19:38.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.241 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:19:38.242 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:38.242 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:19:38.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:38.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:38.242 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:38.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:38.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:38.391 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.392 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.394 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:38.395 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:38.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:38.397 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:38Z" 2024-04-12T04:19:38.397 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:38.398 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046713", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:10:38.051655Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:38Z" } 2024-04-12T04:19:38.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:19:38.398 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:38.398 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:38.398 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085, shard=3 master=00000000000000000000:00000000000000046855 target=00000000000000000000:00000000000000046713 2024-04-12T04:19:43.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:43.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:43.400 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:43.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:43.525 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.526 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:43Z" 2024-04-12T04:19:43.527 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:43.528 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046713", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:10:38.051655Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:43Z" } 2024-04-12T04:19:43.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:19:43.529 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:43.529 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:43.529 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085, shard=3 master=00000000000000000000:00000000000000046855 target=00000000000000000000:00000000000000046713 2024-04-12T04:19:48.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:48.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:48.530 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:48.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:48.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:48.638 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:48Z" 2024-04-12T04:19:48.645 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:48.646 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046713", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:10:38.051655Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:48Z" } 2024-04-12T04:19:48.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:19:48.646 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:48.646 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:48.647 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085, shard=3 master=00000000000000000000:00000000000000046855 target=00000000000000000000:00000000000000046713 2024-04-12T04:19:53.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:53.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:53.648 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:53.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:53.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:53.777 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:53.777 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:53Z" 2024-04-12T04:19:53.783 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:53.784 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046713", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:10:38.051655Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:53Z" } 2024-04-12T04:19:53.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:19:53.784 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:53.784 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:19:53.784 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085, shard=3 master=00000000000000000000:00000000000000046855 target=00000000000000000000:00000000000000046713 2024-04-12T04:19:58.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:58.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:19:58.786 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:19:58.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:58.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:19:58.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:58.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:19:58.899 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:19:58Z" 2024-04-12T04:19:58.900 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:19:58.901 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:19:58Z" } 2024-04-12T04:19:58.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:19:58.901 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:58.901 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:19:58.901 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:03.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:03.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:03.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:03.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:03.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:04.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:04.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:04.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:04.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:04Z" 2024-04-12T04:20:04.019 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:04.020 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:04Z" } 2024-04-12T04:20:04.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:04.020 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:04.021 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:04.021 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:09.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:09.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:09.022 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:09.082 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:09.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:09.150 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:09.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:09.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:09.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:09.154 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:09Z" 2024-04-12T04:20:09.155 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:09.155 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:09Z" } 2024-04-12T04:20:09.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:09.156 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:09.156 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:09.156 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:14.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:14.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:14.157 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:14.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:14.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:14.273 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:14.276 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:14.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:14Z" 2024-04-12T04:20:14.279 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:14.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:14Z" } 2024-04-12T04:20:14.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:14.280 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:14.281 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:14.281 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:19.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:19.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:19.282 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:19.341 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:19.341 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:19.402 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:19Z" 2024-04-12T04:20:19.407 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:19.408 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:19Z" } 2024-04-12T04:20:19.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:19.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:19.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:19.409 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:24.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:24.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:24.411 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:24.468 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:24.468 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:24.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:24.519 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:24.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:24.521 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:24.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:24.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:24.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:24.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:24Z" 2024-04-12T04:20:24.525 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:24.526 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:24Z" } 2024-04-12T04:20:24.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:24.526 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:24.526 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:24.527 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:29.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:29.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:29.528 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:29.586 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:29.586 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:29Z" 2024-04-12T04:20:29.641 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:29.642 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:29Z" } 2024-04-12T04:20:29.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:29.642 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:29.642 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:29.642 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:34.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:34.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:34.644 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:34.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:34.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:34.776 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:34.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:34.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:34.779 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:34.780 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.781 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:34.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:34.782 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:34.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:34.782 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:34Z" 2024-04-12T04:20:34.782 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:34.783 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:34Z" } 2024-04-12T04:20:34.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:34.783 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:34.783 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:34.783 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:39.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:39.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:39.785 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:39.893 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:39.893 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:39.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:39.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:39.896 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:39.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:39.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:39.899 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:39Z" 2024-04-12T04:20:39.899 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:39.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:39Z" } 2024-04-12T04:20:39.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:39.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:39.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:39.900 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:44.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:44.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:44.902 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:44.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:45.027 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:45.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:45.029 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:45.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:45.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:45.032 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:45.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:45.033 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:45Z" 2024-04-12T04:20:45.033 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:45.034 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:45Z" } 2024-04-12T04:20:45.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:45.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:45.034 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:45.034 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:50.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:50.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:50.036 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:50.093 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:50.093 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:50.152 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:50Z" 2024-04-12T04:20:50.158 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:50.159 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:50Z" } 2024-04-12T04:20:50.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:50.159 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:50.159 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:50.159 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:20:55.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:55.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:55.161 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:55.218 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:55.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:20:55.278 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:55.279 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:20:55.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:20:55.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:20:55Z" 2024-04-12T04:20:55.283 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:20:55.284 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:20:55Z" } 2024-04-12T04:20:55.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:20:55.284 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:55.284 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:20:55.284 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:00.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:00.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:00.285 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:00.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:00.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:00.419 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:00.419 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:00.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:00.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:00.422 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:00.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:00.424 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:00.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:00.425 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:00Z" 2024-04-12T04:21:00.425 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:00.426 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:00Z" } 2024-04-12T04:21:00.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:00.426 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:00.426 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:00.426 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:05.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:05.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:05.428 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:05.551 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:05.552 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:05.552 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:05.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:05.552 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:05.552 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:05.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.553 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:05.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:05.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:05.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:05.554 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:05.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:05.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:05.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:05.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:05.555 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:05.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:05.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:05.556 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:05.556 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:05.556 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:05.556 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:05.557 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:05.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:05.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:05.557 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:05.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:05.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:05.558 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:05.558 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:05.558 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:05.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:05.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:05.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:05.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:05.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:05.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:05.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:05.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:05.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:05.561 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:05.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:05.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:05.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:05.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:05.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:05.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:05.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:05.563 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:05.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.563 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:05.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:05.563 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:05.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:05.564 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:05Z" 2024-04-12T04:21:05.564 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:05.565 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:05Z" } 2024-04-12T04:21:05.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:05.566 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:05.566 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:05.567 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:10.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:10.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:10.568 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:10.680 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:10.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:10Z" 2024-04-12T04:21:10.685 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:10.686 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:10Z" } 2024-04-12T04:21:10.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:10.686 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:10.686 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:10.686 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:15.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:15.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:15.688 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:15.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:15.815 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:15.816 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:15.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:15.818 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:15.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:15.820 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:15Z" 2024-04-12T04:21:15.821 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:15.822 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:15Z" } 2024-04-12T04:21:15.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:15.822 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:15.822 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:15.823 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:20.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:20.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:20.824 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:20Z" 2024-04-12T04:21:20.936 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:20.937 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:20Z" } 2024-04-12T04:21:20.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:20.938 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:20.938 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:20.938 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:25.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:25.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:25.940 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:26.048 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:26.048 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:26.048 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:26.048 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:26.048 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:26.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:26.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:26.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:26.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:26.053 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:26Z" 2024-04-12T04:21:26.054 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:26.054 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:26Z" } 2024-04-12T04:21:26.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:26.055 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:26.055 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:26.055 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:31.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:31.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:31.056 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:31.182 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:31.182 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:31.182 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:31.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:31.184 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:31.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:31.186 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:31.187 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:31Z" 2024-04-12T04:21:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:31.188 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:31Z" } 2024-04-12T04:21:31.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:31.189 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:31.189 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:31.189 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:36.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:36.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:36.191 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:36.294 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:36.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:36.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:36.297 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:36.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.299 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:36.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:36.300 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:36.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:36.300 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:36Z" 2024-04-12T04:21:36.300 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:36.301 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047354", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:38.115982Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053085", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:17:52.955658Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:36Z" } 2024-04-12T04:21:36.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:36.301 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:36.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:36.301 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047495 target=00000000000000000000:00000000000000047354, shard=2 master=00000000000000000000:00000000000000053365 target=00000000000000000000:00000000000000053085 2024-04-12T04:21:41.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:41.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:41.303 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:41.411 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053365", 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.612612Z" 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:41Z" 2024-04-12T04:21:41.417 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:41.418 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053365", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:19:37.612612Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:21:41Z" } 2024-04-12T04:21:41.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053365', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.612612Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:21:41.419 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:41.419 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053365', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:21:41.419 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:21:41.419 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:21:41.419 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:21:41.420 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.420 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:21:41.420 DEBUG:boto:Data: 2024-04-12T04:21:41.420 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.420 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.420 DEBUG:boto:Params: {} 2024-04-12T04:21:41.421 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:21:41.421 DEBUG:boto:Token: None 2024-04-12T04:21:41.421 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/ 2024-04-12T04:21:41.421 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZJTpMQivpQyk/3YOsIf548q9EO0= 2024-04-12T04:21:41.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZJTpMQivpQyk/3YOsIf548q9EO0=', 'Content-Length': '0'} 2024-04-12T04:21:41.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ead2fdb9d8ddb1f-006618b6d5-26eb-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.453 DEBUG:boto:path=/dgbpxg-70/obj1 2024-04-12T04:21:41.453 DEBUG:boto:auth_path=/dgbpxg-70/obj1 2024-04-12T04:21:41.453 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.453 DEBUG:boto:Path: /dgbpxg-70/obj1 2024-04-12T04:21:41.454 DEBUG:boto:Data: 2024-04-12T04:21:41.454 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.454 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.454 DEBUG:boto:Params: {} 2024-04-12T04:21:41.454 DEBUG:boto:Token: None 2024-04-12T04:21:41.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj1 2024-04-12T04:21:41.454 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:POcZGmxdxfrGTu0qqmumXhNXwDI= 2024-04-12T04:21:41.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:POcZGmxdxfrGTu0qqmumXhNXwDI=', 'Content-Length': '0'} 2024-04-12T04:21:41.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098271974fd56a26e-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.457 DEBUG:boto:path=/dgbpxg-70/obj1 2024-04-12T04:21:41.457 DEBUG:boto:auth_path=/dgbpxg-70/obj1 2024-04-12T04:21:41.457 DEBUG:boto:Method: GET 2024-04-12T04:21:41.457 DEBUG:boto:Path: /dgbpxg-70/obj1 2024-04-12T04:21:41.457 DEBUG:boto:Data: 2024-04-12T04:21:41.457 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.457 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.457 DEBUG:boto:Params: {} 2024-04-12T04:21:41.458 DEBUG:boto:Token: None 2024-04-12T04:21:41.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj1 2024-04-12T04:21:41.458 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Bt6oOwxRQYAXmHGOX1YZhGdH3DM= 2024-04-12T04:21:41.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bt6oOwxRQYAXmHGOX1YZhGdH3DM=', 'Content-Length': '0'} 2024-04-12T04:21:41.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5381b0918113944-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.460 DEBUG:boto:path=/dgbpxg-70/obj2 2024-04-12T04:21:41.460 DEBUG:boto:auth_path=/dgbpxg-70/obj2 2024-04-12T04:21:41.460 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.460 DEBUG:boto:Path: /dgbpxg-70/obj2 2024-04-12T04:21:41.460 DEBUG:boto:Data: 2024-04-12T04:21:41.460 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.461 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.461 DEBUG:boto:Params: {} 2024-04-12T04:21:41.461 DEBUG:boto:Token: None 2024-04-12T04:21:41.461 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj2 2024-04-12T04:21:41.461 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jtRZvG+T4p2XXfocy2AVoZseZ8g= 2024-04-12T04:21:41.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jtRZvG+T4p2XXfocy2AVoZseZ8g=', 'Content-Length': '0'} 2024-04-12T04:21:41.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000410b512f97d20f8b-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.463 DEBUG:boto:path=/dgbpxg-70/obj2 2024-04-12T04:21:41.463 DEBUG:boto:auth_path=/dgbpxg-70/obj2 2024-04-12T04:21:41.463 DEBUG:boto:Method: GET 2024-04-12T04:21:41.463 DEBUG:boto:Path: /dgbpxg-70/obj2 2024-04-12T04:21:41.463 DEBUG:boto:Data: 2024-04-12T04:21:41.463 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.463 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.463 DEBUG:boto:Params: {} 2024-04-12T04:21:41.465 DEBUG:boto:Token: None 2024-04-12T04:21:41.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj2 2024-04-12T04:21:41.465 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+Z+tAi9JkhEbCqUggH/ZJtWINSo= 2024-04-12T04:21:41.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+Z+tAi9JkhEbCqUggH/ZJtWINSo=', 'Content-Length': '0'} 2024-04-12T04:21:41.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e534823f1fd7a073-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.467 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:21:41.467 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:21:41.467 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.467 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:21:41.467 DEBUG:boto:Data: 2024-04-12T04:21:41.468 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.468 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.468 DEBUG:boto:Params: {} 2024-04-12T04:21:41.468 DEBUG:boto:Token: None 2024-04-12T04:21:41.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj3 2024-04-12T04:21:41.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DBq6l+P8ckOe7qMtjiu9z50P8R4= 2024-04-12T04:21:41.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DBq6l+P8ckOe7qMtjiu9z50P8R4=', 'Content-Length': '0'} 2024-04-12T04:21:41.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a7a45c5f93a1cf-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.470 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:21:41.470 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:21:41.470 DEBUG:boto:Method: GET 2024-04-12T04:21:41.470 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:21:41.470 DEBUG:boto:Data: 2024-04-12T04:21:41.470 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.470 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.471 DEBUG:boto:Params: {} 2024-04-12T04:21:41.471 DEBUG:boto:Token: None 2024-04-12T04:21:41.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-70/obj3 2024-04-12T04:21:41.471 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DVt9RLOwanXzzrK1e1iUTe04TyU= 2024-04-12T04:21:41.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DVt9RLOwanXzzrK1e1iUTe04TyU=', 'Content-Length': '0'} 2024-04-12T04:21:41.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd3a76749d86ec1c-006618b6d5-26eb-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.514 DEBUG:boto:path=/dgbpxg-71/ 2024-04-12T04:21:41.514 DEBUG:boto:auth_path=/dgbpxg-71/ 2024-04-12T04:21:41.515 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.515 DEBUG:boto:Path: /dgbpxg-71/ 2024-04-12T04:21:41.515 DEBUG:boto:Data: 2024-04-12T04:21:41.515 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.515 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.515 DEBUG:boto:Params: {} 2024-04-12T04:21:41.515 DEBUG:boto:Token: None 2024-04-12T04:21:41.515 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-71/ 2024-04-12T04:21:41.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lDqRoa+e6+mRZb2f84tkjdD1u4k= 2024-04-12T04:21:41.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lDqRoa+e6+mRZb2f84tkjdD1u4k=', 'Content-Length': '0'} 2024-04-12T04:21:41.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3ee91c9f79337bb-006618b6d5-26eb-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.518 DEBUG:boto:path=/dgbpxg-71/obj1 2024-04-12T04:21:41.518 DEBUG:boto:auth_path=/dgbpxg-71/obj1 2024-04-12T04:21:41.518 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.518 DEBUG:boto:Path: /dgbpxg-71/obj1 2024-04-12T04:21:41.518 DEBUG:boto:Data: 2024-04-12T04:21:41.518 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.519 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.519 DEBUG:boto:Params: {} 2024-04-12T04:21:41.519 DEBUG:boto:Token: None 2024-04-12T04:21:41.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-71/obj1 2024-04-12T04:21:41.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:L5oVmHiUGAJEeDvBtiBTC1+zmLU= 2024-04-12T04:21:41.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L5oVmHiUGAJEeDvBtiBTC1+zmLU=', 'Content-Length': '0'} 2024-04-12T04:21:41.520 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d535f4863dbced91-006618b6d5-26eb-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.521 DEBUG:boto:path=/dgbpxg-71/obj2 2024-04-12T04:21:41.521 DEBUG:boto:auth_path=/dgbpxg-71/obj2 2024-04-12T04:21:41.521 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.521 DEBUG:boto:Path: /dgbpxg-71/obj2 2024-04-12T04:21:41.521 DEBUG:boto:Data: 2024-04-12T04:21:41.521 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.521 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.521 DEBUG:boto:Params: {} 2024-04-12T04:21:41.521 DEBUG:boto:Token: None 2024-04-12T04:21:41.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-71/obj2 2024-04-12T04:21:41.522 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:iaHUnMsjHxhe8mtNrigae13URH0= 2024-04-12T04:21:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iaHUnMsjHxhe8mtNrigae13URH0=', 'Content-Length': '0'} 2024-04-12T04:21:41.523 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006aab11f6dfa64c0e-006618b6d5-26eb-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.523 DEBUG:boto:path=/dgbpxg-71/obj3 2024-04-12T04:21:41.523 DEBUG:boto:auth_path=/dgbpxg-71/obj3 2024-04-12T04:21:41.523 DEBUG:boto:Method: HEAD 2024-04-12T04:21:41.523 DEBUG:boto:Path: /dgbpxg-71/obj3 2024-04-12T04:21:41.524 DEBUG:boto:Data: 2024-04-12T04:21:41.524 DEBUG:boto:Headers: {} 2024-04-12T04:21:41.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:21:41.524 DEBUG:boto:Port: 8001 2024-04-12T04:21:41.524 DEBUG:boto:Params: {} 2024-04-12T04:21:41.524 DEBUG:boto:Token: None 2024-04-12T04:21:41.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:41 GMT /dgbpxg-71/obj3 2024-04-12T04:21:41.524 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:SdBgqrSn1UtgIkbfIx+dav3FRKU= 2024-04-12T04:21:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SdBgqrSn1UtgIkbfIx+dav3FRKU=', 'Content-Length': '0'} 2024-04-12T04:21:41.525 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000203af7a67dc62c56-006618b6d5-26eb-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:21:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:41.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.526 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:41.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:41.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:41.627 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:21:41.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.628 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:41.728 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:21:41.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.729 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:41.830 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:21:41.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.831 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:41.878 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:41.878 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:41.940 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 17, 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:21:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:21:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:21:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:21:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:21:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:21:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:21:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:21:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:21:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:21:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:21:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:21:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:21:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:21:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:41.965 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:41.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:41.967 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:42.106 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:21:42.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:42.316 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:21:42.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.317 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.392 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:42.540 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:21:42.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.541 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:42.724 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:21:42.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.725 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:42.882 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:21:42.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:21:42.883 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:21:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:43.034 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:21:43.035 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:21:43.035 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:21:43.035 DEBUG:boto:Method: HEAD 2024-04-12T04:21:43.035 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:21:43.035 DEBUG:boto:Data: 2024-04-12T04:21:43.035 DEBUG:boto:Headers: {} 2024-04-12T04:21:43.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:21:43.035 DEBUG:boto:Port: 8000 2024-04-12T04:21:43.035 DEBUG:boto:Params: {} 2024-04-12T04:21:43.036 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:21:43.036 DEBUG:boto:Token: None 2024-04-12T04:21:43.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:43 GMT /dgbpxg-70/ 2024-04-12T04:21:43.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:PBeG5d/614e/FsYHF4jnEYvgy1E= 2024-04-12T04:21:43.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PBeG5d/614e/FsYHF4jnEYvgy1E=', 'Content-Length': '0'} 2024-04-12T04:21:43.049 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d67e9752480a61da-006618b6d7-27e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:21:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:43.050 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:21:43.050 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:21:43.050 DEBUG:boto:Method: PUT 2024-04-12T04:21:43.050 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:21:43.050 DEBUG:boto:Data: 2024-04-12T04:21:43.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:21:43.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:21:43.050 DEBUG:boto:Port: 8000 2024-04-12T04:21:43.050 DEBUG:boto:Params: {} 2024-04-12T04:21:43.050 DEBUG:boto:Token: None 2024-04-12T04:21:43.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:21:43 GMT /dgbpxg-70/obj3 2024-04-12T04:21:43.051 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JvZH/bSpfz3kl9nEv6ijKfVXgLA= 2024-04-12T04:21:43.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:21:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JvZH/bSpfz3kl9nEv6ijKfVXgLA='} 2024-04-12T04:21:43.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f81dce4cf93cda2-006618b6d7-27e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:21:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:43.071 DEBUG:boto:path=/dgbpxg-71/ 2024-04-12T04:21:43.071 DEBUG:boto:auth_path=/dgbpxg-71/ 2024-04-12T04:21:43.071 DEBUG:boto:Method: HEAD 2024-04-12T04:21:43.071 DEBUG:boto:Path: /dgbpxg-71/ 2024-04-12T04:21:43.071 DEBUG:boto:Data: 2024-04-12T04:21:43.071 DEBUG:boto:Headers: {} 2024-04-12T04:21:43.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:21:43.071 DEBUG:boto:Port: 8000 2024-04-12T04:21:43.071 DEBUG:boto:Params: {} 2024-04-12T04:21:43.071 DEBUG:boto:Token: None 2024-04-12T04:21:43.072 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:21:43 GMT /dgbpxg-71/ 2024-04-12T04:21:43.072 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hnRuIA5YBpB0dMi6riExbjeS/Zg= 2024-04-12T04:21:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:21:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hnRuIA5YBpB0dMi6riExbjeS/Zg=', 'Content-Length': '0'} 2024-04-12T04:21:43.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c029405aa7e9c96-006618b6d7-27e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:21:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:43.075 DEBUG:boto:path=/dgbpxg-71/obj3 2024-04-12T04:21:43.075 DEBUG:boto:auth_path=/dgbpxg-71/obj3 2024-04-12T04:21:43.075 DEBUG:boto:Method: PUT 2024-04-12T04:21:43.075 DEBUG:boto:Path: /dgbpxg-71/obj3 2024-04-12T04:21:43.075 DEBUG:boto:Data: 2024-04-12T04:21:43.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:21:43.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:21:43.076 DEBUG:boto:Port: 8000 2024-04-12T04:21:43.076 DEBUG:boto:Params: {} 2024-04-12T04:21:43.076 DEBUG:boto:Token: None 2024-04-12T04:21:43.076 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:21:43 GMT /dgbpxg-71/obj3 2024-04-12T04:21:43.076 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ooqQIIwi4pTqut2pZXfaQGXT1oI= 2024-04-12T04:21:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:21:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ooqQIIwi4pTqut2pZXfaQGXT1oI='} 2024-04-12T04:21:43.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007518e40615e2bc16-006618b6d7-27e4-a1'), ('Date', 'Fri, 12 Apr 2024 04:21:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:21:43.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:43.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:43.085 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:43.136 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:43.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:21:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:21:42.875482Z" 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.186 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:21:43.187 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:21:43.187 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:21:43.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:43.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:43.188 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:43.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:43.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:43.290+0000 7f5bc1d84780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:43.291+0000 7f5bc1d84780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:43.293 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:43.293 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:43.293 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:43.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:43.294 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:21:43.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:21:43.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:21:43.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:21:43.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:43.299 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:43Z" 2024-04-12T04:21:43.299 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:43.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:21:43Z" } 2024-04-12T04:21:43.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:21:43.300 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:21:43.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:21:43.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:21:48.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:48.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:48.301 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:48.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:48.405+0000 7f4b405b3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:48.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:48.406+0000 7f4b405b3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:48.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:48.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:48.410 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:48.411 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:48.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:48.413 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:48Z" 2024-04-12T04:21:48.414 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:48.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:21:48Z" } 2024-04-12T04:21:48.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:21:48.415 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:21:48.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:21:48.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:21:53.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:53.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:53.417 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:53.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:53.473 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:53.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:53.536+0000 7f5bf8174780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:53.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:53.537+0000 7f5bf8174780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:21:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:21:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:53Z" 2024-04-12T04:21:53.544 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:53.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:21:53Z" } 2024-04-12T04:21:53.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:21:53.545 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:21:53.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:21:53.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:21:58.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:58.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:21:58.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:21:58.601 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:21:58.601 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:21:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:58.650+0000 7f679d8a4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:21:58.651+0000 7f679d8a4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:21:58.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:58.653 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:58.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:58.655 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:21:58.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:58.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:58.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:58.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:21:58.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:21:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:21:58Z" 2024-04-12T04:21:58.660 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:21:58.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:21:58Z" } 2024-04-12T04:21:58.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:21:58.661 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:21:58.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:21:58.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:03.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:03.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:03.663 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:03.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:03.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:03.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:03.780+0000 7f69954a0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:03.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:03.781+0000 7f69954a0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:03.789 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:03.790 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:03.791 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:03.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:03.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:03.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:03.795 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:03.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:03.795 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:03Z" 2024-04-12T04:22:03.795 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:03.796 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:03Z" } 2024-04-12T04:22:03.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:03.796 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:03.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:03.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:08.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:08.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:08.797 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:08.853 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:08.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:08.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:08.917+0000 7fa4880bd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:08.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:08.918+0000 7fa4880bd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:08.918 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:08.918 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:08.918 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:08.919 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:08.920 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:08.921 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:08.922 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:08.922 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:08.922 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:08.922 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:08.922 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:08.923 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:08.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:08.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:08.926 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:08Z" 2024-04-12T04:22:08.926 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:08.928 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:08Z" } 2024-04-12T04:22:08.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:08.928 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:08.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:08.928 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:13.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:13.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:13.930 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:13.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:13.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:14.032+0000 7f641bf32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:14.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:14.033+0000 7f641bf32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:14.034 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:14.035 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:14.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:14.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:14.038 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:14.039 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:14Z" 2024-04-12T04:22:14.040 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:14.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:14Z" } 2024-04-12T04:22:14.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:14.041 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:14.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:14.042 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:19.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:19.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:19.044 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:19.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:19.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:19.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:19.154+0000 7f25f609c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:19.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:19.155+0000 7f25f609c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.164 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:19.165 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:19.166 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:19.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:19.168 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:19.169 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:19.170 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:19.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:19.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:19.171 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:19.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.171 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:19.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:19.172 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:19.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:19.172 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:19Z" 2024-04-12T04:22:19.172 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:19.173 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:19Z" } 2024-04-12T04:22:19.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:19.173 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:19.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:19.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:24.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:24.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:24.175 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:24.291+0000 7f0db16e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:24.292+0000 7f0db16e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:24.292 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:24.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:24.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:24.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:24.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:24.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:24Z" 2024-04-12T04:22:24.298 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:24.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:24Z" } 2024-04-12T04:22:24.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:24.299 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:24.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:24.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:29.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:29.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:29.301 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:29.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:29.404+0000 7f59663e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:29.405+0000 7f59663e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:29.407 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:29.407 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:29.407 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:29.407 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:29.408 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:29.409 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:29.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:29.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:29Z" 2024-04-12T04:22:29.413 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:29.414 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:29Z" } 2024-04-12T04:22:29.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:29.415 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:29.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:29.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:34.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:34.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:34.416 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:34.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:34.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:34.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:34.516+0000 7fd49642e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:34.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:34.517+0000 7fd49642e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:34.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:34.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:34.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:34.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:34.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:34.526 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:34Z" 2024-04-12T04:22:34.526 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:34.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:34Z" } 2024-04-12T04:22:34.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:34.527 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:34.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:34.527 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:39.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:39.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:39.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:39.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:39.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:39.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:39.630+0000 7fa184da8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:39.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:39.631+0000 7fa184da8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:39.633 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:39.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:39.635 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:39.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:39.638 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:39Z" 2024-04-12T04:22:39.639 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:39.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:39Z" } 2024-04-12T04:22:39.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:39.640 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:39.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:39.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:44.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:44.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:44.642 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:44.698 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:44.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:44.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:44.754+0000 7f7c36e25780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:44.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:44.755+0000 7f7c36e25780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:44.756 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:44Z" 2024-04-12T04:22:44.764 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:44.765 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:44Z" } 2024-04-12T04:22:44.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:44.766 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:44.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:44.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:49.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:49.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:49.768 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:49.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:49.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:49.868+0000 7f4d653ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:49.869+0000 7f4d653ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:49Z" 2024-04-12T04:22:49.877 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:49.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:49Z" } 2024-04-12T04:22:49.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:49.878 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:49.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:49.878 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:54.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:54.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:54.880 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:54.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:54.939 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:54.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:54.984+0000 7fc5772e3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:22:54.985+0000 7fc5772e3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:54.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:22:54.987 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:22:54.987 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:54.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:54.989 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:22:54.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:22:54.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.992 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:22:54.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:22:54.993 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:22:54.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:22:54.993 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:22:54Z" 2024-04-12T04:22:54.993 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:22:54.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:54Z" } 2024-04-12T04:22:54.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:22:54.994 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:22:54.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:22:54.994 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:22:59.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:59.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:59.996 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:00.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:00.096+0000 7f115b804780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:00.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:00.097+0000 7f115b804780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:00.104 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:00Z" 2024-04-12T04:23:00.110 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:00.111 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:00Z" } 2024-04-12T04:23:00.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:00.112 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:00.112 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:00.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:05.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:05.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:05.113 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:05.169 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:05.170 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:05.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:05.215+0000 7ff272f70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:05.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:05.216+0000 7ff272f70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:05.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:05.222 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:05.223 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:05.224 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:05.225 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:05.226 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:05.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:05.227 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:05Z" 2024-04-12T04:23:05.227 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:05.228 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:05Z" } 2024-04-12T04:23:05.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:05.228 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:05.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:05.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:10.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:10.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:10.230 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:10.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:10.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:10.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:10.348+0000 7f98cc176780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:10.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:10.349+0000 7f98cc176780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:10.355 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:10.355 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:10.355 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:10.356 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:10.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:10.358 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:10.359 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:10.360 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:10.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:10.362 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:10.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:10.362 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:10Z" 2024-04-12T04:23:10.362 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:10.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:10Z" } 2024-04-12T04:23:10.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:10.363 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:10.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:10.363 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:15.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:15.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:15.365 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:15.420 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:15.421 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:15.475+0000 7fbaf99fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:15.476+0000 7fbaf99fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:15.486 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:15.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:15.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:15.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:15Z" 2024-04-12T04:23:15.492 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:15.493 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:15Z" } 2024-04-12T04:23:15.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:15.494 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:15.494 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:15.494 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:20.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:20.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:20.496 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:20.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:20.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:20.596+0000 7fabe3905780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:20.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:20.597+0000 7fabe3905780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:20.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:20.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:20.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:20.604 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:20.605 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:20Z" 2024-04-12T04:23:20.606 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:20.607 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:20Z" } 2024-04-12T04:23:20.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:20.608 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:20.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:20.608 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:25.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:25.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:25.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:25.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:25.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:25.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:25.728+0000 7f39a9fcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:25.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:25.729+0000 7f39a9fcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:25.740 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:25.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:25.742 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:25.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:25.746 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:25.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:25.746 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:25Z" 2024-04-12T04:23:25.746 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:25.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:25Z" } 2024-04-12T04:23:25.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:25.747 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:25.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:25.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:30.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:30.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:30.749 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:30.805 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:30.805 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:30.868+0000 7f745a4e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:30.869+0000 7f745a4e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:30.871 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:30.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:30.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:30.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:30.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:30Z" 2024-04-12T04:23:30.877 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:30.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:30Z" } 2024-04-12T04:23:30.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:30.879 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:30.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:30.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:35.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:35.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:35.880 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:35.982+0000 7f583ec47780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:35.983+0000 7f583ec47780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:35.989 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:35.989 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:35.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:35.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:35.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:35Z" 2024-04-12T04:23:35.995 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:35.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:35Z" } 2024-04-12T04:23:35.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:35.996 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:35.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:35.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:40.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:40.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:40.998 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:41.055 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:41.055 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:41.106+0000 7efd9dafb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:41.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:41.107+0000 7efd9dafb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:41.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:41.111 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:41.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:41.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895571.281159_1949.1", 2024-04-12T04:23:41.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:31.281159Z", 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:41.115 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:41.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:41.116 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:41Z" 2024-04-12T04:23:41.116 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:41.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895571.281159_1949.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:31.281159Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:41Z" } 2024-04-12T04:23:41.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895571.281159_1949.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:31.281159Z', 'realm_epoch': 2}}] 2024-04-12T04:23:41.117 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:41.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895571.281159_1949.1'} 2024-04-12T04:23:41.117 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895702.875482_2034.1 target=1_1712895571.281159_1949.1 2024-04-12T04:23:46.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.119 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:46.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:46.236+0000 7fcfbfeb0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:46.237+0000 7fcfbfeb0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:46.250 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.251 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.252 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:42.875482Z", 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.254 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:46Z" 2024-04-12T04:23:46.255 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:46.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895702.875482_2034.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:21:42.875482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:46Z" } 2024-04-12T04:23:46.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895702.875482_2034.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:21:42.875482Z', 'realm_epoch': 2}}] 2024-04-12T04:23:46.256 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:46.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:46.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:23:46.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:46.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.368 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:23:46.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:23:46.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.370 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:23:46.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:23:46.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.370 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:23:46.371 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:23:46.371 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:23:46.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.371 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:46.421 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.421 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.469 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:23:46.471 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:23:46.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:23:46.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:23:46.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.475 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:46Z" 2024-04-12T04:23:46.475 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:46.476 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:23:46Z" } 2024-04-12T04:23:46.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:23:46.476 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:23:46.476 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:23:46.476 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:23:46.477 DEBUG:boto:path=/dgbpxg-70/ 2024-04-12T04:23:46.477 DEBUG:boto:auth_path=/dgbpxg-70/ 2024-04-12T04:23:46.477 DEBUG:boto:Method: HEAD 2024-04-12T04:23:46.477 DEBUG:boto:Path: /dgbpxg-70/ 2024-04-12T04:23:46.477 DEBUG:boto:Data: 2024-04-12T04:23:46.477 DEBUG:boto:Headers: {} 2024-04-12T04:23:46.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:23:46.477 DEBUG:boto:Port: 8001 2024-04-12T04:23:46.477 DEBUG:boto:Params: {} 2024-04-12T04:23:46.479 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:23:46.479 DEBUG:boto:Token: None 2024-04-12T04:23:46.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:46 GMT /dgbpxg-70/ 2024-04-12T04:23:46.480 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Uw1Lt62D7xzS7cJc6H+INXKblY= 2024-04-12T04:23:46.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Uw1Lt62D7xzS7cJc6H+INXKblY=', 'Content-Length': '0'} 2024-04-12T04:23:46.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d586eab1c7124e-006618b752-27d8-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:46.493 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:23:46.493 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:23:46.494 DEBUG:boto:Method: HEAD 2024-04-12T04:23:46.494 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:23:46.494 DEBUG:boto:Data: 2024-04-12T04:23:46.494 DEBUG:boto:Headers: {} 2024-04-12T04:23:46.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:23:46.494 DEBUG:boto:Port: 8001 2024-04-12T04:23:46.494 DEBUG:boto:Params: {} 2024-04-12T04:23:46.494 DEBUG:boto:Token: None 2024-04-12T04:23:46.494 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:46 GMT /dgbpxg-70/obj3 2024-04-12T04:23:46.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Unnyg2f4DG2/3bkvJtC3skZuIiA= 2024-04-12T04:23:46.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Unnyg2f4DG2/3bkvJtC3skZuIiA=', 'Content-Length': '0'} 2024-04-12T04:23:46.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb01f40d8e764169-006618b752-27d8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:23:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:46.497 DEBUG:boto:path=/dgbpxg-70/obj3 2024-04-12T04:23:46.497 DEBUG:boto:auth_path=/dgbpxg-70/obj3 2024-04-12T04:23:46.497 DEBUG:boto:Method: GET 2024-04-12T04:23:46.497 DEBUG:boto:Path: /dgbpxg-70/obj3 2024-04-12T04:23:46.497 DEBUG:boto:Data: 2024-04-12T04:23:46.497 DEBUG:boto:Headers: {} 2024-04-12T04:23:46.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:23:46.498 DEBUG:boto:Port: 8001 2024-04-12T04:23:46.498 DEBUG:boto:Params: {} 2024-04-12T04:23:46.498 DEBUG:boto:Token: None 2024-04-12T04:23:46.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:23:46 GMT /dgbpxg-70/obj3 2024-04-12T04:23:46.498 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:n+rq0KwMDATtbbGoeYRbcp8w1oc= 2024-04-12T04:23:46.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n+rq0KwMDATtbbGoeYRbcp8w1oc=', 'Content-Length': '0'} 2024-04-12T04:23:46.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-70:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f33207886654d28c-006618b752-27d8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:23:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:46.500 DEBUG:boto:path=/dgbpxg-71/ 2024-04-12T04:23:46.500 DEBUG:boto:auth_path=/dgbpxg-71/ 2024-04-12T04:23:46.500 DEBUG:boto:Method: HEAD 2024-04-12T04:23:46.500 DEBUG:boto:Path: /dgbpxg-71/ 2024-04-12T04:23:46.500 DEBUG:boto:Data: 2024-04-12T04:23:46.500 DEBUG:boto:Headers: {} 2024-04-12T04:23:46.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:23:46.501 DEBUG:boto:Port: 8001 2024-04-12T04:23:46.501 DEBUG:boto:Params: {} 2024-04-12T04:23:46.501 DEBUG:boto:Token: None 2024-04-12T04:23:46.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:46 GMT /dgbpxg-71/ 2024-04-12T04:23:46.501 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UCpiRqnDFdvv6M15vMfMPVQH1p8= 2024-04-12T04:23:46.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UCpiRqnDFdvv6M15vMfMPVQH1p8=', 'Content-Length': '0'} 2024-04-12T04:23:46.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000222625b9645538cc-006618b752-27d8-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:46.503 DEBUG:boto:path=/dgbpxg-71/obj3 2024-04-12T04:23:46.503 DEBUG:boto:auth_path=/dgbpxg-71/obj3 2024-04-12T04:23:46.504 DEBUG:boto:Method: HEAD 2024-04-12T04:23:46.504 DEBUG:boto:Path: /dgbpxg-71/obj3 2024-04-12T04:23:46.504 DEBUG:boto:Data: 2024-04-12T04:23:46.504 DEBUG:boto:Headers: {} 2024-04-12T04:23:46.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:23:46.504 DEBUG:boto:Port: 8001 2024-04-12T04:23:46.504 DEBUG:boto:Params: {} 2024-04-12T04:23:46.504 DEBUG:boto:Token: None 2024-04-12T04:23:46.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:46 GMT /dgbpxg-71/obj3 2024-04-12T04:23:46.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:TXEPl43Eg8b2pflI/SPeKIad09I= 2024-04-12T04:23:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TXEPl43Eg8b2pflI/SPeKIad09I=', 'Content-Length': '0'} 2024-04-12T04:23:46.506 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000040455cbe12835171-006618b752-27d8-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:23:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:46.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:46.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:46.506 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.609 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:23:46.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:46.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:46.609 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.706 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:23:46.706 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2024-04-12T04:23:46.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:23:46.801 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:23:46.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.803 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895826.605584_2051.1", 2024-04-12T04:23:46.803 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:23:46.605584Z" 2024-04-12T04:23:46.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.803 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:23:46.804 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:23:46.804 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:23:46.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:46.804 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:46.852 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:46.852 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:46.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:46.892+0000 7f39d4bdb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:46.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:46.893+0000 7f39d4bdb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.898 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:46.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:46.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:42.875482Z", 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:46.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:46.904 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:46Z" 2024-04-12T04:23:46.904 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:46.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895702.875482_2034.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:21:42.875482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:46Z" } 2024-04-12T04:23:46.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895702.875482_2034.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:21:42.875482Z', 'realm_epoch': 2}}] 2024-04-12T04:23:46.905 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:23:46.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:46.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895826.605584_2051.1 target=1_1712895702.875482_2034.1 2024-04-12T04:23:51.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:51.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:51.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:51.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:51.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:52.033+0000 7f2ffaa09780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:52.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:52.034+0000 7f2ffaa09780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:52.041 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:52.041 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:52.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:52.043 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:52.044 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:52.045 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:42.875482Z", 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:52.046 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:52Z" 2024-04-12T04:23:52.047 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:52.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895702.875482_2034.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:21:42.875482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:52Z" } 2024-04-12T04:23:52.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895702.875482_2034.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:21:42.875482Z', 'realm_epoch': 2}}] 2024-04-12T04:23:52.049 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:23:52.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:52.049 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895826.605584_2051.1 target=1_1712895702.875482_2034.1 2024-04-12T04:23:57.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:57.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:57.051 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:57.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:57.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:57.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:57.161+0000 7fbae94f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:57.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:23:57.162+0000 7fbae94f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.182 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:57.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:23:57.184 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:23:57.185 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:23:57.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:42.875482Z", 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:23:57Z" 2024-04-12T04:23:57.187 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:23:57.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895702.875482_2034.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:21:42.875482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:57Z" } 2024-04-12T04:23:57.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895702.875482_2034.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:21:42.875482Z', 'realm_epoch': 2}}] 2024-04-12T04:23:57.189 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:23:57.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:23:57.189 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895826.605584_2051.1 target=1_1712895702.875482_2034.1 2024-04-12T04:24:02.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:02.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:02.191 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:02.247 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:02.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:02.304+0000 7f6160f7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:02.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:02.305+0000 7f6160f7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:02.312 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:24:02.313 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:02.314 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:02.315 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895702.875482_2034.1", 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:02.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:42.875482Z", 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:02Z" 2024-04-12T04:24:02.317 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:02.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895702.875482_2034.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:21:42.875482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:24:02Z" } 2024-04-12T04:24:02.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895702.875482_2034.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:21:42.875482Z', 'realm_epoch': 2}}] 2024-04-12T04:24:02.319 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:24:02.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895702.875482_2034.1'} 2024-04-12T04:24:02.319 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895826.605584_2051.1 target=1_1712895702.875482_2034.1 2024-04-12T04:24:07.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.320 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:07.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:07.440+0000 7f5b8a56e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:07.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:07.442+0000 7f5b8a56e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:07.453 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:07.453 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:07.453 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.454 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.455 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:24:07.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:24:07.457 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895826.605584_2051.1", 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:23:46.605584Z", 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:07.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:24:07.460 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:24:07.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.460 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:07Z" 2024-04-12T04:24:07.460 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:07.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895826.605584_2051.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:23:46.605584Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:24:07Z" } 2024-04-12T04:24:07.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895826.605584_2051.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:23:46.605584Z', 'realm_epoch': 2}}] 2024-04-12T04:24:07.461 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:24:07.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895826.605584_2051.1'} 2024-04-12T04:24:07.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:24:07.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:07.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:07.462 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:07.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:07.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:07.579 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:24:07.580 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-72 2024-04-12T04:24:07.580 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:24:07.580 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:24:07.580 DEBUG:boto:Method: PUT 2024-04-12T04:24:07.581 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:24:07.581 DEBUG:boto:Data: 2024-04-12T04:24:07.581 DEBUG:boto:Headers: {} 2024-04-12T04:24:07.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:07.581 DEBUG:boto:Port: 8000 2024-04-12T04:24:07.581 DEBUG:boto:Params: {} 2024-04-12T04:24:07.582 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:24:07.582 DEBUG:boto:Token: None 2024-04-12T04:24:07.582 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:24:07 GMT /dgbpxg-72/ 2024-04-12T04:24:07.582 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:df+ZKJoUmtxnojL6jlCGf/ZBC9E= 2024-04-12T04:24:07.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:24:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:df+ZKJoUmtxnojL6jlCGf/ZBC9E=', 'Content-Length': '0'} 2024-04-12T04:24:07.611 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf255dcddfd058cd-006618b767-27e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:24:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:07.612 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-73 2024-04-12T04:24:07.612 DEBUG:boto:path=/dgbpxg-73/ 2024-04-12T04:24:07.612 DEBUG:boto:auth_path=/dgbpxg-73/ 2024-04-12T04:24:07.612 DEBUG:boto:Method: PUT 2024-04-12T04:24:07.612 DEBUG:boto:Path: /dgbpxg-73/ 2024-04-12T04:24:07.612 DEBUG:boto:Data: 2024-04-12T04:24:07.612 DEBUG:boto:Headers: {} 2024-04-12T04:24:07.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:07.613 DEBUG:boto:Port: 8000 2024-04-12T04:24:07.613 DEBUG:boto:Params: {} 2024-04-12T04:24:07.613 DEBUG:boto:Token: None 2024-04-12T04:24:07.613 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:24:07 GMT /dgbpxg-73/ 2024-04-12T04:24:07.613 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9LVanj573Vhx5MgC9Leh1KbLH0E= 2024-04-12T04:24:07.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:24:07 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9LVanj573Vhx5MgC9Leh1KbLH0E=', 'Content-Length': '0'} 2024-04-12T04:24:07.629 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000004598558911d1b31-006618b767-27e4-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:24:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:07.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.629 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:07.721 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:24:07.734 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:24:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:24:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:24:07.638556Z" 2024-04-12T04:24:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.736 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:24:07.737 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:07.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:24:07.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:07.737 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:07.785 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:07.786 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:07.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:07.828+0000 7fe5364ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:07.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:07.828+0000 7fe5364ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:07.831 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:07.831 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:07.831 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:07.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:07.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:07.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.834 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.614414_2052.1", 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.614414Z", 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:24:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:24:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:07Z" 2024-04-12T04:24:07.837 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:07.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.614414_2052.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.614414Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:24:07Z" } 2024-04-12T04:24:07.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.614414_2052.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.614414Z', 'realm_epoch': 2}}] 2024-04-12T04:24:07.838 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:07.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.614414_2052.1'} 2024-04-12T04:24:07.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895847.638556_2059.1 target=1_1712895847.614414_2052.1 2024-04-12T04:24:12.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:12.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:12.840 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:12.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:12.950+0000 7f1d6ad8d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:12.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:12.952+0000 7f1d6ad8d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:12.955 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:12.956 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:12.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:24:12.959 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:24:12.960 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:12Z" 2024-04-12T04:24:12.961 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:12.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:24:12Z" } 2024-04-12T04:24:12.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:24:12.963 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:12.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:12.963 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:24:12.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:12.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:12.963 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.012 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.012 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.058 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:24:13.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.058 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.165 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:24:13.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-72', '--dest-bucket=dgbpxg-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-72', '--dest-bucket=dgbpxg-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.166 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-72 --dest-bucket=dgbpxg-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.286 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-72","zones":["a1"]},"dest":{"bucket":"dgbpxg-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:24:13.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.287 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.335 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.335 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.391 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-72","zones":["a1"]},"dest":{"bucket":"dgbpxg-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:24:13.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.391 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 18, 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:13.496 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:24:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:24:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:24:13.499 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:24:13.500 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:24:13.501 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.502 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.503 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-72", 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:24:13.504 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-73", 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.505 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.506 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.507 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:24:13.508 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:24:13.509 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:24:13.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.511 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:24:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:24:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:24:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:24:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:24:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:24:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:13.527 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:13.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:24:13.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:24:13.572 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:13.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:13.663 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-72","zones":["a1"]},"dest":{"bucket":"dgbpxg-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:24:13.663 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:24:13.663 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:24:13.663 DEBUG:boto:Method: HEAD 2024-04-12T04:24:13.663 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:24:13.663 DEBUG:boto:Data: 2024-04-12T04:24:13.664 DEBUG:boto:Headers: {} 2024-04-12T04:24:13.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:13.664 DEBUG:boto:Port: 8000 2024-04-12T04:24:13.664 DEBUG:boto:Params: {} 2024-04-12T04:24:13.664 DEBUG:boto:Token: None 2024-04-12T04:24:13.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:24:13 GMT /dgbpxg-72/ 2024-04-12T04:24:13.664 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+wECL2UZBffwh8LGHsFG7NLtvhc= 2024-04-12T04:24:13.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:24:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+wECL2UZBffwh8LGHsFG7NLtvhc=', 'Content-Length': '0'} 2024-04-12T04:24:13.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf8c049984d6f42f-006618b76d-2984-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:13.883 DEBUG:boto:path=/dgbpxg-72/obj1 2024-04-12T04:24:13.883 DEBUG:boto:auth_path=/dgbpxg-72/obj1 2024-04-12T04:24:13.883 DEBUG:boto:Method: PUT 2024-04-12T04:24:13.883 DEBUG:boto:Path: /dgbpxg-72/obj1 2024-04-12T04:24:13.883 DEBUG:boto:Data: 2024-04-12T04:24:13.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:24:13.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:13.883 DEBUG:boto:Port: 8000 2024-04-12T04:24:13.884 DEBUG:boto:Params: {} 2024-04-12T04:24:13.884 DEBUG:boto:Token: None 2024-04-12T04:24:13.884 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:24:13 GMT /dgbpxg-72/obj1 2024-04-12T04:24:13.884 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:RJXK2I83cfGZ6Ltxh82I418Kz7c= 2024-04-12T04:24:13.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:24:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RJXK2I83cfGZ6Ltxh82I418Kz7c='} 2024-04-12T04:24:13.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000619f83c8588d1c30-006618b76d-2984-a1'), ('Date', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:13.913 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:24:13.914 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:24:13.914 DEBUG:boto:Method: HEAD 2024-04-12T04:24:13.914 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:24:13.914 DEBUG:boto:Data: 2024-04-12T04:24:13.914 DEBUG:boto:Headers: {} 2024-04-12T04:24:13.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:13.914 DEBUG:boto:Port: 8000 2024-04-12T04:24:13.914 DEBUG:boto:Params: {} 2024-04-12T04:24:13.914 DEBUG:boto:Token: None 2024-04-12T04:24:13.914 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:24:13 GMT /dgbpxg-72/ 2024-04-12T04:24:13.915 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:+wECL2UZBffwh8LGHsFG7NLtvhc= 2024-04-12T04:24:13.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:24:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+wECL2UZBffwh8LGHsFG7NLtvhc=', 'Content-Length': '0'} 2024-04-12T04:24:13.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec1822c018c60fce-006618b76d-2984-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:13.920 DEBUG:boto:path=/dgbpxg-72/obj2 2024-04-12T04:24:13.920 DEBUG:boto:auth_path=/dgbpxg-72/obj2 2024-04-12T04:24:13.920 DEBUG:boto:Method: PUT 2024-04-12T04:24:13.920 DEBUG:boto:Path: /dgbpxg-72/obj2 2024-04-12T04:24:13.920 DEBUG:boto:Data: 2024-04-12T04:24:13.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:24:13.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:24:13.920 DEBUG:boto:Port: 8000 2024-04-12T04:24:13.920 DEBUG:boto:Params: {} 2024-04-12T04:24:13.921 DEBUG:boto:Token: None 2024-04-12T04:24:13.921 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:24:13 GMT /dgbpxg-72/obj2 2024-04-12T04:24:13.921 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:dZLMEiCGVvu2M7vgXJZR3Uy0gcg= 2024-04-12T04:24:13.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:24:13 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dZLMEiCGVvu2M7vgXJZR3Uy0gcg='} 2024-04-12T04:24:13.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085611105b9c71071-006618b76d-2984-a1'), ('Date', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:24:13.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:13.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:13.967 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:14.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:14.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:14.170 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:24:14.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:24:07.638556Z" 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.236 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:24:14.237 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:14.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:24:14.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.238 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:14.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:14.313 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:14.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:14.406+0000 7fb29876b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:14.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:24:14.407+0000 7fb29876b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:24:14.430 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.433 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:14Z" 2024-04-12T04:24:14.436 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:14.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:24:14Z" } 2024-04-12T04:24:14.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:24:14.438 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:14.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:24:14.438 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:24:14.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.438 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:14.595 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:24:14.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.595 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:24:14.595 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:24:14.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:24:14.596 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.597 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:24:14.598 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:14.599 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:24:14.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:14.600 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:14.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:14.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:14.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:14.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:14.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:14.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:14Z" 2024-04-12T04:24:14.770 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:14.771 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:14Z" } 2024-04-12T04:24:14.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:14.771 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:14.771 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:14.771 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:19.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:19.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:19.773 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:19.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:19.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:19.907 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:19.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:19.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:19.910 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:19.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:19.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:19Z" 2024-04-12T04:24:19.913 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:19.914 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:19Z" } 2024-04-12T04:24:19.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:19.915 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:19.915 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:19.915 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:24.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:24.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:24.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:24.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:24.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:25Z" 2024-04-12T04:24:25.033 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:25.034 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:25Z" } 2024-04-12T04:24:25.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:25.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:25.035 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:25.035 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:30.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:30.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:30.037 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:30.167 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:30.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:30.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:30.170 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:30.171 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:30Z" 2024-04-12T04:24:30.172 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:30.173 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:30Z" } 2024-04-12T04:24:30.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:30.174 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:30.174 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:30.174 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:35.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:35.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:35.176 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:35.234 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:35.234 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:35.295 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:35.295 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:35.295 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:35.296 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:35.298 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:35.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:35.300 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:35.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:35.301 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:35Z" 2024-04-12T04:24:35.301 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:35.302 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:35Z" } 2024-04-12T04:24:35.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:35.302 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:35.302 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:35.302 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:40.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:40.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:40.304 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:40.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:40.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:40Z" 2024-04-12T04:24:40.433 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:40.434 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:40Z" } 2024-04-12T04:24:40.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:40.435 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:40.435 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:40.435 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:45.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:45.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:45.436 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:45.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:45.547 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:45.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:45Z" 2024-04-12T04:24:45.549 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:45.550 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:45Z" } 2024-04-12T04:24:45.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:45.550 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:45.550 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:45.550 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:50.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:50.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:50.551 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:50.609 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:50.609 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:50Z" 2024-04-12T04:24:50.673 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:50.673 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:50Z" } 2024-04-12T04:24:50.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:50.674 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:50.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:50.674 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:24:55.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:55.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:24:55.675 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:24:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:24:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:24:55.792 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:24:55.792 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:24:55.792 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:24:55.792 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:24:55.792 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:24:55.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:55.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:24:55.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:24:55Z" 2024-04-12T04:24:55.797 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:24:55.798 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:24:55Z" } 2024-04-12T04:24:55.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:24:55.799 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:55.799 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:24:55.799 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:00.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:00.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:00.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:00.924 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:00.924 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:00.924 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:00Z" 2024-04-12T04:25:00.930 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:00.931 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:00Z" } 2024-04-12T04:25:00.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:00.931 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:00.931 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:00.931 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:05.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:05.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:05.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:06.063 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:06.069 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:06.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:06.069 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:06Z" 2024-04-12T04:25:06.069 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:06.070 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:06Z" } 2024-04-12T04:25:06.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:06.070 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:06.070 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:06.070 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:11.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:11.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:11.071 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:11.129 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:11.129 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:11.204 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:11.204 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:11.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:11.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:11.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:11.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:11.210 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:11.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:11.210 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:11Z" 2024-04-12T04:25:11.210 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:11.211 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:11Z" } 2024-04-12T04:25:11.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:11.211 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:11.211 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:11.211 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:16.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:16.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:16.213 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:16.321 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:16.321 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:16.321 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:16.321 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:16.322 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:16.323 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:16.324 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:16.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:16Z" 2024-04-12T04:25:16.326 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:16.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:16Z" } 2024-04-12T04:25:16.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:16.328 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:16.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:16.328 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:21.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:21.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:21.330 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:21.464 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:21.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:21.471 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:21Z" 2024-04-12T04:25:21.471 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:21.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:21Z" } 2024-04-12T04:25:21.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:21.472 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:21.472 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:21.472 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:26.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:26.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:26.474 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:26.531 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:26.532 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:26.601 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:26.606 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:26.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:26.607 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:26Z" 2024-04-12T04:25:26.607 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:26.608 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:26Z" } 2024-04-12T04:25:26.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:26.608 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:26.608 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:26.608 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:31.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:31.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:31.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:31.667 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:31.741 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:31Z" 2024-04-12T04:25:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:31.748 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:31Z" } 2024-04-12T04:25:31.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:31.748 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:31.748 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:31.749 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:36.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:36.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:36.750 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:36.807 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:36.807 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:36.859 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:36.860 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:36.861 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:36.862 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:36.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:36.864 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:36.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:36.865 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:36Z" 2024-04-12T04:25:36.865 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:36.866 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:36Z" } 2024-04-12T04:25:36.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:36.866 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:36.866 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:36.866 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:41.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:41.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:41.868 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047495", 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.573504Z" 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:21:43.098058Z" 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:41Z" 2024-04-12T04:25:42.015 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:42.016 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047495", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:19:37.573504Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:21:43.098058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:41Z" } 2024-04-12T04:25:42.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047495', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.573504Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:21:43.098058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:42.017 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:42.017 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047495', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053645', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:42.017 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047635 target=00000000000000000000:00000000000000047495, shard=2 master=00000000000000000000:00000000000000053785 target=00000000000000000000:00000000000000053645 2024-04-12T04:25:47.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:47.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:47.019 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.136 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.137 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.138 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.139 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:47.140 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:47.141 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:25:47.142 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:25:47.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.142 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:47Z" 2024-04-12T04:25:47.142 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:47.143 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:25:47Z" } 2024-04-12T04:25:47.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:25:47.143 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:47.143 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:25:47.144 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:25:47.144 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:25:47.144 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:25:47.144 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.144 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:25:47.144 DEBUG:boto:Data: 2024-04-12T04:25:47.144 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.144 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.145 DEBUG:boto:Params: {} 2024-04-12T04:25:47.145 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:25:47.145 DEBUG:boto:Token: None 2024-04-12T04:25:47.146 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-72/ 2024-04-12T04:25:47.146 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OnG8RFQSpJ5PltMztS0gS6H0FqY= 2024-04-12T04:25:47.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OnG8RFQSpJ5PltMztS0gS6H0FqY=', 'Content-Length': '0'} 2024-04-12T04:25:47.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a850c469fbff981-006618b7cb-2980-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.178 DEBUG:boto:path=/dgbpxg-72/obj1 2024-04-12T04:25:47.178 DEBUG:boto:auth_path=/dgbpxg-72/obj1 2024-04-12T04:25:47.178 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.178 DEBUG:boto:Path: /dgbpxg-72/obj1 2024-04-12T04:25:47.178 DEBUG:boto:Data: 2024-04-12T04:25:47.178 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.178 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.179 DEBUG:boto:Params: {} 2024-04-12T04:25:47.179 DEBUG:boto:Token: None 2024-04-12T04:25:47.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-72/obj1 2024-04-12T04:25:47.179 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xLraZdFVWPoBc7kOqLlnASycb80= 2024-04-12T04:25:47.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xLraZdFVWPoBc7kOqLlnASycb80=', 'Content-Length': '0'} 2024-04-12T04:25:47.180 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e6ea8bd7455f3024-006618b7cb-2980-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.180 DEBUG:boto:path=/dgbpxg-72/obj2 2024-04-12T04:25:47.181 DEBUG:boto:auth_path=/dgbpxg-72/obj2 2024-04-12T04:25:47.181 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.181 DEBUG:boto:Path: /dgbpxg-72/obj2 2024-04-12T04:25:47.181 DEBUG:boto:Data: 2024-04-12T04:25:47.181 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.181 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.181 DEBUG:boto:Params: {} 2024-04-12T04:25:47.181 DEBUG:boto:Token: None 2024-04-12T04:25:47.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-72/obj2 2024-04-12T04:25:47.182 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aUKMt4QVj9f3bZIarpY4NZOx6Kw= 2024-04-12T04:25:47.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aUKMt4QVj9f3bZIarpY4NZOx6Kw=', 'Content-Length': '0'} 2024-04-12T04:25:47.183 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d8836cdbd6b57a2b-006618b7cb-2980-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.183 DEBUG:boto:path=/dgbpxg-73/ 2024-04-12T04:25:47.183 DEBUG:boto:auth_path=/dgbpxg-73/ 2024-04-12T04:25:47.183 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.183 DEBUG:boto:Path: /dgbpxg-73/ 2024-04-12T04:25:47.183 DEBUG:boto:Data: 2024-04-12T04:25:47.184 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.184 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.184 DEBUG:boto:Params: {} 2024-04-12T04:25:47.184 DEBUG:boto:Token: None 2024-04-12T04:25:47.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-73/ 2024-04-12T04:25:47.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nTfbaV+DkUdz3m1KWNS/Axyp6Q4= 2024-04-12T04:25:47.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nTfbaV+DkUdz3m1KWNS/Axyp6Q4=', 'Content-Length': '0'} 2024-04-12T04:25:47.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015887c95113499ce-006618b7cb-2980-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.187 DEBUG:boto:path=/dgbpxg-73/obj1 2024-04-12T04:25:47.187 DEBUG:boto:auth_path=/dgbpxg-73/obj1 2024-04-12T04:25:47.187 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.187 DEBUG:boto:Path: /dgbpxg-73/obj1 2024-04-12T04:25:47.187 DEBUG:boto:Data: 2024-04-12T04:25:47.187 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.187 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.188 DEBUG:boto:Params: {} 2024-04-12T04:25:47.188 DEBUG:boto:Token: None 2024-04-12T04:25:47.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-73/obj1 2024-04-12T04:25:47.188 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:3FzCN5I9YXSU1eN1U6nzg/1ERtg= 2024-04-12T04:25:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3FzCN5I9YXSU1eN1U6nzg/1ERtg=', 'Content-Length': '0'} 2024-04-12T04:25:47.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c71df7ade46aff07-006618b7cb-2980-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.190 DEBUG:boto:path=/dgbpxg-73/obj1 2024-04-12T04:25:47.190 DEBUG:boto:auth_path=/dgbpxg-73/obj1 2024-04-12T04:25:47.190 DEBUG:boto:Method: GET 2024-04-12T04:25:47.190 DEBUG:boto:Path: /dgbpxg-73/obj1 2024-04-12T04:25:47.190 DEBUG:boto:Data: 2024-04-12T04:25:47.191 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.191 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.191 DEBUG:boto:Params: {} 2024-04-12T04:25:47.191 DEBUG:boto:Token: None 2024-04-12T04:25:47.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-73/obj1 2024-04-12T04:25:47.191 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KXtYhgqEZvN3AJN6oLDNE5AfHs0= 2024-04-12T04:25:47.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KXtYhgqEZvN3AJN6oLDNE5AfHs0=', 'Content-Length': '0'} 2024-04-12T04:25:47.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a49fdc2b63ee274-006618b7cb-2980-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.193 DEBUG:boto:path=/dgbpxg-73/obj2 2024-04-12T04:25:47.193 DEBUG:boto:auth_path=/dgbpxg-73/obj2 2024-04-12T04:25:47.194 DEBUG:boto:Method: HEAD 2024-04-12T04:25:47.194 DEBUG:boto:Path: /dgbpxg-73/obj2 2024-04-12T04:25:47.194 DEBUG:boto:Data: 2024-04-12T04:25:47.194 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.194 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.194 DEBUG:boto:Params: {} 2024-04-12T04:25:47.194 DEBUG:boto:Token: None 2024-04-12T04:25:47.194 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-73/obj2 2024-04-12T04:25:47.195 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zhnc04cr/GDmH/tTlcqLu+6VQ4k= 2024-04-12T04:25:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zhnc04cr/GDmH/tTlcqLu+6VQ4k=', 'Content-Length': '0'} 2024-04-12T04:25:47.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007320db2c97a8deb2-006618b7cb-2980-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.196 DEBUG:boto:path=/dgbpxg-73/obj2 2024-04-12T04:25:47.196 DEBUG:boto:auth_path=/dgbpxg-73/obj2 2024-04-12T04:25:47.197 DEBUG:boto:Method: GET 2024-04-12T04:25:47.197 DEBUG:boto:Path: /dgbpxg-73/obj2 2024-04-12T04:25:47.197 DEBUG:boto:Data: 2024-04-12T04:25:47.197 DEBUG:boto:Headers: {} 2024-04-12T04:25:47.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:25:47.197 DEBUG:boto:Port: 8001 2024-04-12T04:25:47.197 DEBUG:boto:Params: {} 2024-04-12T04:25:47.197 DEBUG:boto:Token: None 2024-04-12T04:25:47.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:25:47 GMT /dgbpxg-73/obj2 2024-04-12T04:25:47.197 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZTu8XofSeUMOXQMug8J0Z/KEC7M= 2024-04-12T04:25:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:25:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZTu8XofSeUMOXQMug8J0Z/KEC7M=', 'Content-Length': '0'} 2024-04-12T04:25:47.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:24:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063e4a615e116b16b-006618b7cb-2980-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:25:47.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.200 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.249 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.250 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.314 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:25:47.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.314 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.365 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.365 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.423 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:25:47.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.424 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.526 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:25:47.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.527 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.574 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.574 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.629 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:47.629 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 19, 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:25:47.630 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.631 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:25:47.632 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:25:47.633 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:25:47.634 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.636 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:25:47.637 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:25:47.638 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:25:47.639 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:25:47.640 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:25:47.641 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:25:47.642 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:25:47.644 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:25:47.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:25:47.646 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:25:47.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:25:47.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:25:47.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:25:47.651 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:25:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:25:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:25:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:25:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:47.655 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:47.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.656 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.703 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.704 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:47.811 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:25:47.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:47.811 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:47.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:47.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.049 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:25:48.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.050 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:48.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:48.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.293 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:25:48.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-72', '--source-bucket=*', '--dest-bucket=dgbpxg-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-72', '--source-bucket=*', '--dest-bucket=dgbpxg-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.294 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dgbpxg-72 '--source-bucket=*' --dest-bucket=dgbpxg-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.493 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:25:48.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.493 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:48.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:48.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.667 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:25:48.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:25:48.668 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:25:48.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:48.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.833 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:25:48.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:48.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:48.833 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:48.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:48.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:48.949 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:25:48.973 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:25:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:25:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:25:48.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:25:48.974 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:25:48.633068Z" 2024-04-12T04:25:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:48.975 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:25:48.976 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:25:48.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:25:48.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:48.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:48.976 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:49.025 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:49.025 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:49.074+0000 7ff0f50dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:49.075+0000 7ff0f50dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:49.080 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:49.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:49.084 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:49.085 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:49Z" 2024-04-12T04:25:49.086 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:49.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:25:49Z" } 2024-04-12T04:25:49.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:25:49.087 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:25:49.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:25:49.088 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:25:54.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:54.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:54.089 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:54.145 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:54.145 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:54.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:54.204+0000 7f33dc264780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:54.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:54.205+0000 7f33dc264780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:54.214 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:25:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:25:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:54Z" 2024-04-12T04:25:54.221 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:54.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:25:54Z" } 2024-04-12T04:25:54.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:25:54.222 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:25:54.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:25:54.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:25:59.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:59.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:25:59.224 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:25:59.280 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:25:59.280 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:25:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:59.322+0000 7f48fe2f6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:59.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:25:59.323+0000 7f48fe2f6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:25:59.337 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:59.338 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:59.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:59.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:25:59.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:25:59.342 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:25:59Z" 2024-04-12T04:25:59.343 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:25:59.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:25:59Z" } 2024-04-12T04:25:59.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:25:59.344 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:25:59.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:25:59.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:26:04.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:04.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:04.345 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:04.463+0000 7f2ba1cdd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:04.464+0000 7f2ba1cdd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:04.477 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:04.479 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:04.480 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:04.481 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:04.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:04Z" 2024-04-12T04:26:04.483 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:04.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:26:04Z" } 2024-04-12T04:26:04.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:26:04.484 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:04.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:26:04.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:26:09.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:09.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:09.486 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:09.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:09.542 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:09.589+0000 7f40cb448780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:09.590+0000 7f40cb448780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:09.597 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:09.597 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:09.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:09.599 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:09.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:09.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:09.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:09Z" 2024-04-12T04:26:09.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:09.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:26:09Z" } 2024-04-12T04:26:09.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:26:09.604 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:09.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:26:09.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:26:14.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:14.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:14.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:14.662 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:14.663 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:14.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:14.711+0000 7f70ada7d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:14.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:14.712+0000 7f70ada7d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:14.724 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:14.724 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:14.724 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:14.724 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:14.725 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:14.727 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895847.638556_2059.1", 2024-04-12T04:26:14.728 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:07.638556Z", 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:26:14.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:14.730 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:14Z" 2024-04-12T04:26:14.730 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:14.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895847.638556_2059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:07.638556Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:26:14Z" } 2024-04-12T04:26:14.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895847.638556_2059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:07.638556Z', 'realm_epoch': 2}}] 2024-04-12T04:26:14.731 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:14.731 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895847.638556_2059.1'} 2024-04-12T04:26:14.731 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895948.633068_2142.1 target=1_1712895847.638556_2059.1 2024-04-12T04:26:19.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:19.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:19.732 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:19.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:19.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:19.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:19.832+0000 7fd5fc9e3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:19.833+0000 7fd5fc9e3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:19.839 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:25:48.633068Z", 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:19Z" 2024-04-12T04:26:19.845 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:19.846 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895948.633068_2142.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:25:48.633068Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:26:19Z" } 2024-04-12T04:26:19.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895948.633068_2142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:25:48.633068Z', 'realm_epoch': 2}}] 2024-04-12T04:26:19.847 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:19.847 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:19.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:26:19.847 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:26:19.847 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:26:19.847 DEBUG:boto:Method: HEAD 2024-04-12T04:26:19.847 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:26:19.847 DEBUG:boto:Data: 2024-04-12T04:26:19.848 DEBUG:boto:Headers: {} 2024-04-12T04:26:19.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:26:19.848 DEBUG:boto:Port: 8000 2024-04-12T04:26:19.848 DEBUG:boto:Params: {} 2024-04-12T04:26:19.848 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:26:19.848 DEBUG:boto:Token: None 2024-04-12T04:26:19.849 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:26:19 GMT /dgbpxg-72/ 2024-04-12T04:26:19.849 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UUvmfEUtPaX3cQUceMF9+XdvvdM= 2024-04-12T04:26:19.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:26:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UUvmfEUtPaX3cQUceMF9+XdvvdM=', 'Content-Length': '0'} 2024-04-12T04:26:19.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f248d9fa509dfc91-006618b7eb-2a61-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:26:19.881 DEBUG:boto:path=/dgbpxg-72/obj5 2024-04-12T04:26:19.881 DEBUG:boto:auth_path=/dgbpxg-72/obj5 2024-04-12T04:26:19.881 DEBUG:boto:Method: PUT 2024-04-12T04:26:19.881 DEBUG:boto:Path: /dgbpxg-72/obj5 2024-04-12T04:26:19.881 DEBUG:boto:Data: 2024-04-12T04:26:19.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:26:19.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:26:19.881 DEBUG:boto:Port: 8000 2024-04-12T04:26:19.881 DEBUG:boto:Params: {} 2024-04-12T04:26:19.882 DEBUG:boto:Token: None 2024-04-12T04:26:19.882 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:26:19 GMT /dgbpxg-72/obj5 2024-04-12T04:26:19.882 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:7rD5qWr02exRbjzyXsE26D+pl/4= 2024-04-12T04:26:19.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:26:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7rD5qWr02exRbjzyXsE26D+pl/4='} 2024-04-12T04:26:19.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000143a21d8981fe87a-006618b7eb-2a61-a1'), ('Date', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:26:19.896 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:26:19.897 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:26:19.897 DEBUG:boto:Method: HEAD 2024-04-12T04:26:19.897 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:26:19.897 DEBUG:boto:Data: 2024-04-12T04:26:19.897 DEBUG:boto:Headers: {} 2024-04-12T04:26:19.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:26:19.897 DEBUG:boto:Port: 8000 2024-04-12T04:26:19.897 DEBUG:boto:Params: {} 2024-04-12T04:26:19.897 DEBUG:boto:Token: None 2024-04-12T04:26:19.898 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:26:19 GMT /dgbpxg-72/ 2024-04-12T04:26:19.898 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UUvmfEUtPaX3cQUceMF9+XdvvdM= 2024-04-12T04:26:19.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:26:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UUvmfEUtPaX3cQUceMF9+XdvvdM=', 'Content-Length': '0'} 2024-04-12T04:26:19.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d9a6d2f14833e3f-006618b7eb-2a61-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:26:19.900 DEBUG:boto:path=/dgbpxg-72/obj6 2024-04-12T04:26:19.900 DEBUG:boto:auth_path=/dgbpxg-72/obj6 2024-04-12T04:26:19.900 DEBUG:boto:Method: PUT 2024-04-12T04:26:19.901 DEBUG:boto:Path: /dgbpxg-72/obj6 2024-04-12T04:26:19.901 DEBUG:boto:Data: 2024-04-12T04:26:19.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:26:19.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:26:19.901 DEBUG:boto:Port: 8000 2024-04-12T04:26:19.901 DEBUG:boto:Params: {} 2024-04-12T04:26:19.901 DEBUG:boto:Token: None 2024-04-12T04:26:19.901 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:26:19 GMT /dgbpxg-72/obj6 2024-04-12T04:26:19.901 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Qxu0RMDaVKMVQLL1+hRiIGxwVkU= 2024-04-12T04:26:19.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:26:19 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qxu0RMDaVKMVQLL1+hRiIGxwVkU='} 2024-04-12T04:26:19.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1ffa891272ca6dd-006618b7eb-2a61-a1'), ('Date', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:26:19.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:19.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:19.914 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:19.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:19.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:20.010 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:26:20.034 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:26:20.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.034 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:20.034 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:26:20.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:26:20.036 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:25:48.633068Z" 2024-04-12T04:26:20.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.036 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:26:20.037 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:20.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:26:20.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.037 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:20.085 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:20.085 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:20.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:20.135+0000 7efedd6dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:20.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:26:20.136+0000 7efedd6dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:26:20.147 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:20.147 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:20.147 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.148 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:20.149 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:25:48.633068Z", 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:20Z" 2024-04-12T04:26:20.153 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:20.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895948.633068_2142.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:25:48.633068Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:26:20Z" } 2024-04-12T04:26:20.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895948.633068_2142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:25:48.633068Z', 'realm_epoch': 2}}] 2024-04-12T04:26:20.154 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:20.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:26:20.155 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:26:20.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.155 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:20.202 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:20.202 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:20.267 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:26:20.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:26:20.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:20.269 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:20.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.269 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:26:20.270 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:20.270 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:26:20.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:20.270 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.385 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:20.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:20Z" 2024-04-12T04:26:20.390 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:20.391 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:20Z" } 2024-04-12T04:26:20.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:20.392 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:20.392 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:20.392 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:25.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:25.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:25.394 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:25.452 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:25.452 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:25.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:25.530 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:25.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:25.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:25.533 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:25.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:25.535 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:25Z" 2024-04-12T04:26:25.535 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:25.535 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:25Z" } 2024-04-12T04:26:25.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:25.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:25.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:25.536 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:30.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:30.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:30.538 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:30.649 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:30.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:30.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:30.652 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:30.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.654 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:30.655 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:30.655 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:30.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:30.655 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:30Z" 2024-04-12T04:26:30.655 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:30.656 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:30Z" } 2024-04-12T04:26:30.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:30.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:30.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:30.656 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:35.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:35.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:35.658 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:35.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:35.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:35.793 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:35.793 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:35.793 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:35.793 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:35.793 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:35.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:35.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:35.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:35.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:35Z" 2024-04-12T04:26:35.798 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:35.799 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:35Z" } 2024-04-12T04:26:35.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:35.800 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:35.800 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:35.800 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:40.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:40.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:40.802 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:40.927 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:40.928 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:40.929 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:40.930 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:40.931 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.932 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:40.933 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:40.933 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:40.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:40.933 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:40Z" 2024-04-12T04:26:40.933 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:40.934 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:40Z" } 2024-04-12T04:26:40.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:40.934 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:40.934 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:40.934 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:45.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:45.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:45.936 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:46.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:46.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:46.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:46Z" 2024-04-12T04:26:46.051 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:46.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:46Z" } 2024-04-12T04:26:46.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:46.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:46.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:46.053 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:51.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:51.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:51.054 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:51.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:51.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:51.175 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:51Z" 2024-04-12T04:26:51.181 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:51.183 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:51Z" } 2024-04-12T04:26:51.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:51.183 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:51.183 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:51.183 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:26:56.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:56.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:26:56.184 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:26:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:26:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:26:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:26:56Z" 2024-04-12T04:26:56.304 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:26:56.306 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:26:56Z" } 2024-04-12T04:26:56.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:26:56.306 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:56.306 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:26:56.306 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:01.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:01.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:01.308 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:01.365 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:01.365 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:01.432 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:01.432 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:01.432 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:01.432 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:01.433 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:01.434 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:01.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:01.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:01Z" 2024-04-12T04:27:01.437 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:01.438 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:01Z" } 2024-04-12T04:27:01.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:01.439 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:01.439 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:01.439 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:06.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:06.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:06.440 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:06.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:06.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:06.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.569 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:06.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:06.570 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:06.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:06.570 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:06Z" 2024-04-12T04:27:06.570 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:06.571 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:06Z" } 2024-04-12T04:27:06.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:06.571 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:06.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:06.571 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:11.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:11.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:11.573 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:11.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:11.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:11Z" 2024-04-12T04:27:11.686 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:11.687 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:11Z" } 2024-04-12T04:27:11.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:11.687 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:11.687 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:11.688 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:16.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:16.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:16.689 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:16.747 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:16.747 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:16.807 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:16.807 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:16.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:16.809 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:16Z" 2024-04-12T04:27:16.813 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:16.813 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:16Z" } 2024-04-12T04:27:16.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:16.814 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:16.814 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:16.814 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:21.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:21.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:21.815 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:21.931 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:21.931 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:21.932 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:21.933 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:21.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:21.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:21.936 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:21Z" 2024-04-12T04:27:21.937 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:21.937 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:21Z" } 2024-04-12T04:27:21.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:21.938 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:21.938 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:21.938 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:26.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:26.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:26.940 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:26.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:26.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:27.051 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:27Z" 2024-04-12T04:27:27.057 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:27.058 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:27Z" } 2024-04-12T04:27:27.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:27.058 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:27.058 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:27.058 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:32.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:32.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:32.060 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:32.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:32.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:32.190 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:32.190 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:32.190 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:32.190 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:32.190 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:32.192 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:32.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:32.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:32Z" 2024-04-12T04:27:32.195 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:32.196 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:32Z" } 2024-04-12T04:27:32.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:32.196 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:32.197 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:32.197 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:37.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:37.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:37.198 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:37.310 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:37.310 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:37.310 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:37.310 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:37.311 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:37.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:37.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:37.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:37.315 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:37.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:37.316 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:37Z" 2024-04-12T04:27:37.316 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:37.317 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:37Z" } 2024-04-12T04:27:37.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:37.317 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:37.317 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:37.317 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:42.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:42.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:42.319 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:42.375 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:42.375 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:42.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:42.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:42.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:42.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:42.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:42Z" 2024-04-12T04:27:42.442 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:42.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:42Z" } 2024-04-12T04:27:42.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:42.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:42.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:42.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:47.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:47.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:47.445 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:47.571 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:47.571 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:47.572 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047635", 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.957536Z" 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:47.574 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053785", 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:24:13.920698Z" 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:47.575 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:47.576 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:47.577 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:47.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:47.577 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:47Z" 2024-04-12T04:27:47.577 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:47.578 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047635", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:24:13.957536Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053785", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:24:13.920698Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:47Z" } 2024-04-12T04:27:47.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047635', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.957536Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053785', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:24:13.920698Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:47.578 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:47.578 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047635', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053785', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:47.578 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047776 target=00000000000000000000:00000000000000047635, shard=2 master=00000000000000000000:00000000000000053926 target=00000000000000000000:00000000000000053785 2024-04-12T04:27:52.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:52.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:52.579 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:52.702 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:52.702 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:52.702 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:52.702 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:52.702 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:52.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:52.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:52.705 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046855", 2024-04-12T04:27:52.706 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:19:37.643892Z" 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:52.707 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:52Z" 2024-04-12T04:27:52.708 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:52.708 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:10:39.150359Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046855", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:19:37.643892Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:27:52Z" } 2024-04-12T04:27:52.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046855', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:19:37.643892Z'}}] 2024-04-12T04:27:52.709 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:52.709 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046855'} 2024-04-12T04:27:52.709 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:27:52.709 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:27:52.709 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:27:52.710 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.710 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:27:52.710 DEBUG:boto:Data: 2024-04-12T04:27:52.710 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.710 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.710 DEBUG:boto:Params: {} 2024-04-12T04:27:52.711 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:27:52.711 DEBUG:boto:Token: None 2024-04-12T04:27:52.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-72/ 2024-04-12T04:27:52.711 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4C0B9tzNXpUfndkYdx6ik6kBb6U= 2024-04-12T04:27:52.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4C0B9tzNXpUfndkYdx6ik6kBb6U=', 'Content-Length': '0'} 2024-04-12T04:27:52.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006b4bfe5ee727303-006618b848-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.725 DEBUG:boto:path=/dgbpxg-72/obj5 2024-04-12T04:27:52.725 DEBUG:boto:auth_path=/dgbpxg-72/obj5 2024-04-12T04:27:52.725 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.726 DEBUG:boto:Path: /dgbpxg-72/obj5 2024-04-12T04:27:52.726 DEBUG:boto:Data: 2024-04-12T04:27:52.726 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.726 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.726 DEBUG:boto:Params: {} 2024-04-12T04:27:52.726 DEBUG:boto:Token: None 2024-04-12T04:27:52.726 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-72/obj5 2024-04-12T04:27:52.726 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Cb1QwImYghu/8imI+avcOY8rA/g= 2024-04-12T04:27:52.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Cb1QwImYghu/8imI+avcOY8rA/g=', 'Content-Length': '0'} 2024-04-12T04:27:52.728 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000010392f13739ee86d-006618b848-2a56-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.728 DEBUG:boto:path=/dgbpxg-72/obj6 2024-04-12T04:27:52.728 DEBUG:boto:auth_path=/dgbpxg-72/obj6 2024-04-12T04:27:52.728 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.728 DEBUG:boto:Path: /dgbpxg-72/obj6 2024-04-12T04:27:52.728 DEBUG:boto:Data: 2024-04-12T04:27:52.728 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.729 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.729 DEBUG:boto:Params: {} 2024-04-12T04:27:52.729 DEBUG:boto:Token: None 2024-04-12T04:27:52.729 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-72/obj6 2024-04-12T04:27:52.729 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ms7BipPWuu6ZxZ8aqqnJ7Oqwvwo= 2024-04-12T04:27:52.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ms7BipPWuu6ZxZ8aqqnJ7Oqwvwo=', 'Content-Length': '0'} 2024-04-12T04:27:52.730 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f6ee513ad4787b7a-006618b848-2a56-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.730 DEBUG:boto:path=/dgbpxg-73/ 2024-04-12T04:27:52.730 DEBUG:boto:auth_path=/dgbpxg-73/ 2024-04-12T04:27:52.731 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.731 DEBUG:boto:Path: /dgbpxg-73/ 2024-04-12T04:27:52.731 DEBUG:boto:Data: 2024-04-12T04:27:52.731 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.731 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.731 DEBUG:boto:Params: {} 2024-04-12T04:27:52.731 DEBUG:boto:Token: None 2024-04-12T04:27:52.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-73/ 2024-04-12T04:27:52.731 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UCswRJcRDGW0EXiPMXMr2WY50kE= 2024-04-12T04:27:52.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UCswRJcRDGW0EXiPMXMr2WY50kE=', 'Content-Length': '0'} 2024-04-12T04:27:52.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000894236998f16ad34-006618b848-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.733 DEBUG:boto:path=/dgbpxg-73/obj5 2024-04-12T04:27:52.734 DEBUG:boto:auth_path=/dgbpxg-73/obj5 2024-04-12T04:27:52.734 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.734 DEBUG:boto:Path: /dgbpxg-73/obj5 2024-04-12T04:27:52.734 DEBUG:boto:Data: 2024-04-12T04:27:52.734 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.734 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.734 DEBUG:boto:Params: {} 2024-04-12T04:27:52.734 DEBUG:boto:Token: None 2024-04-12T04:27:52.737 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-73/obj5 2024-04-12T04:27:52.738 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:6MRGagw/BySluTfOdeovOGfPDyQ= 2024-04-12T04:27:52.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6MRGagw/BySluTfOdeovOGfPDyQ=', 'Content-Length': '0'} 2024-04-12T04:27:52.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000994910fc32d28743-006618b848-2a56-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.740 DEBUG:boto:path=/dgbpxg-73/obj5 2024-04-12T04:27:52.740 DEBUG:boto:auth_path=/dgbpxg-73/obj5 2024-04-12T04:27:52.740 DEBUG:boto:Method: GET 2024-04-12T04:27:52.740 DEBUG:boto:Path: /dgbpxg-73/obj5 2024-04-12T04:27:52.740 DEBUG:boto:Data: 2024-04-12T04:27:52.740 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.740 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.740 DEBUG:boto:Params: {} 2024-04-12T04:27:52.741 DEBUG:boto:Token: None 2024-04-12T04:27:52.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-73/obj5 2024-04-12T04:27:52.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:39BHdKDYWvcriS8M0lbTBWEEKKk= 2024-04-12T04:27:52.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:39BHdKDYWvcriS8M0lbTBWEEKKk=', 'Content-Length': '0'} 2024-04-12T04:27:52.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd5f931fa5edbd48-006618b848-2a56-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.743 DEBUG:boto:path=/dgbpxg-73/obj6 2024-04-12T04:27:52.743 DEBUG:boto:auth_path=/dgbpxg-73/obj6 2024-04-12T04:27:52.743 DEBUG:boto:Method: HEAD 2024-04-12T04:27:52.743 DEBUG:boto:Path: /dgbpxg-73/obj6 2024-04-12T04:27:52.743 DEBUG:boto:Data: 2024-04-12T04:27:52.743 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.743 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.743 DEBUG:boto:Params: {} 2024-04-12T04:27:52.744 DEBUG:boto:Token: None 2024-04-12T04:27:52.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-73/obj6 2024-04-12T04:27:52.744 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9vKNhFTPt743165PDVDF62TVYCQ= 2024-04-12T04:27:52.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9vKNhFTPt743165PDVDF62TVYCQ=', 'Content-Length': '0'} 2024-04-12T04:27:52.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e686795172f807f-006618b848-2a56-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.746 DEBUG:boto:path=/dgbpxg-73/obj6 2024-04-12T04:27:52.746 DEBUG:boto:auth_path=/dgbpxg-73/obj6 2024-04-12T04:27:52.746 DEBUG:boto:Method: GET 2024-04-12T04:27:52.746 DEBUG:boto:Path: /dgbpxg-73/obj6 2024-04-12T04:27:52.746 DEBUG:boto:Data: 2024-04-12T04:27:52.746 DEBUG:boto:Headers: {} 2024-04-12T04:27:52.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:27:52.746 DEBUG:boto:Port: 8001 2024-04-12T04:27:52.746 DEBUG:boto:Params: {} 2024-04-12T04:27:52.746 DEBUG:boto:Token: None 2024-04-12T04:27:52.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:27:52 GMT /dgbpxg-73/obj6 2024-04-12T04:27:52.747 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:kHpcJk278rrKBNbp2RKhir2rDEs= 2024-04-12T04:27:52.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:27:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kHpcJk278rrKBNbp2RKhir2rDEs=', 'Content-Length': '0'} 2024-04-12T04:27:52.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:26:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-72:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0fcc85c83ac2ad-006618b848-2a56-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:27:52.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:27:52.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:27:52.791 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:27:52.839 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:52.839 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:52.928 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:27:52.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:52.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:52.929 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:52.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:52.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:53.018 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896072.908482_2152.1", 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:27:52.908482Z" 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.032 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:27:53.033 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:27:53.033 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:27:53.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:53.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:53.034 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:53.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:27:53.119+0000 7f50c733e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:27:53.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:27:53.120+0000 7f50c733e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:27:53.132 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:53.133 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:27:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:27:53.135 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:25:48.633068Z", 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:27:53.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:53.137 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:53Z" 2024-04-12T04:27:53.137 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:53.137 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895948.633068_2142.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:25:48.633068Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:27:53Z" } 2024-04-12T04:27:53.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895948.633068_2142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:25:48.633068Z', 'realm_epoch': 2}}] 2024-04-12T04:27:53.138 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:27:53.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:27:53.138 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896072.908482_2152.1 target=1_1712895948.633068_2142.1 2024-04-12T04:27:58.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:58.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:27:58.140 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:27:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:27:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:27:58.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:27:58.251+0000 7f234627d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:27:58.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:27:58.252+0000 7f234627d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:27:58.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:27:58.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:58.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:58.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712895948.633068_2142.1", 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:27:58.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:25:48.633068Z", 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:27:58.273 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:27:58Z" 2024-04-12T04:27:58.274 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:27:58.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895948.633068_2142.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:25:48.633068Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:27:58Z" } 2024-04-12T04:27:58.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895948.633068_2142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:25:48.633068Z', 'realm_epoch': 2}}] 2024-04-12T04:27:58.275 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:27:58.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895948.633068_2142.1'} 2024-04-12T04:27:58.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896072.908482_2152.1 target=1_1712895948.633068_2142.1 2024-04-12T04:28:03.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:03.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:03.276 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:03.375+0000 7f825cd62780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:03.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:03.376+0000 7f825cd62780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:03.384 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:03.385 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:03.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:03.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896072.908482_2152.1", 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:27:52.908482Z", 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:03.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:03Z" 2024-04-12T04:28:03.390 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:03.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896072.908482_2152.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:27:52.908482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:03Z" } 2024-04-12T04:28:03.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896072.908482_2152.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:27:52.908482Z', 'realm_epoch': 2}}] 2024-04-12T04:28:03.391 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:28:03.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:28:03.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:03.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.392 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:03.505 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:28:03.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.505 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:03.555 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.555 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:03.641 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:28:03.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.642 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:03.756 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:28:03.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-72', '--source-bucket=dgbpxg-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dgbpxg-72', '--source-bucket=dgbpxg-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.757 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dgbpxg-72 --source-bucket=dgbpxg-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:03.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:03.879 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:28:03.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:03.880 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:04.004 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:28:04.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:04.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:04.004 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:04.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:04.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:04.107 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dgbpxg-73","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:28:04.107 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:28:04.107 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:28:04.108 DEBUG:boto:Method: HEAD 2024-04-12T04:28:04.108 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:28:04.108 DEBUG:boto:Data: 2024-04-12T04:28:04.108 DEBUG:boto:Headers: {} 2024-04-12T04:28:04.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.108 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.108 DEBUG:boto:Params: {} 2024-04-12T04:28:04.108 DEBUG:boto:Token: None 2024-04-12T04:28:04.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-72/ 2024-04-12T04:28:04.109 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c6sYfvd6SdBMMEwoI1ILaVM3GI8= 2024-04-12T04:28:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c6sYfvd6SdBMMEwoI1ILaVM3GI8=', 'Content-Length': '0'} 2024-04-12T04:28:04.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000935f371652b8d2b6-006618b854-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.111 DEBUG:boto:path=/dgbpxg-72/obj7 2024-04-12T04:28:04.111 DEBUG:boto:auth_path=/dgbpxg-72/obj7 2024-04-12T04:28:04.112 DEBUG:boto:Method: PUT 2024-04-12T04:28:04.112 DEBUG:boto:Path: /dgbpxg-72/obj7 2024-04-12T04:28:04.112 DEBUG:boto:Data: 2024-04-12T04:28:04.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:04.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.112 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.112 DEBUG:boto:Params: {} 2024-04-12T04:28:04.112 DEBUG:boto:Token: None 2024-04-12T04:28:04.112 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-72/obj7 2024-04-12T04:28:04.113 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8rAkb2JIkEZ1VLxPKvpYVy1xxSw= 2024-04-12T04:28:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8rAkb2JIkEZ1VLxPKvpYVy1xxSw='} 2024-04-12T04:28:04.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ef8ea06b3b54a62-006618b854-2a56-a2'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.119 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:28:04.119 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:28:04.119 DEBUG:boto:Method: HEAD 2024-04-12T04:28:04.119 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:28:04.119 DEBUG:boto:Data: 2024-04-12T04:28:04.120 DEBUG:boto:Headers: {} 2024-04-12T04:28:04.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.120 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.120 DEBUG:boto:Params: {} 2024-04-12T04:28:04.120 DEBUG:boto:Token: None 2024-04-12T04:28:04.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-72/ 2024-04-12T04:28:04.120 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:c6sYfvd6SdBMMEwoI1ILaVM3GI8= 2024-04-12T04:28:04.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:c6sYfvd6SdBMMEwoI1ILaVM3GI8=', 'Content-Length': '0'} 2024-04-12T04:28:04.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ad18fde7d34dc29-006618b854-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.122 DEBUG:boto:path=/dgbpxg-72/obj8 2024-04-12T04:28:04.122 DEBUG:boto:auth_path=/dgbpxg-72/obj8 2024-04-12T04:28:04.122 DEBUG:boto:Method: PUT 2024-04-12T04:28:04.123 DEBUG:boto:Path: /dgbpxg-72/obj8 2024-04-12T04:28:04.123 DEBUG:boto:Data: 2024-04-12T04:28:04.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:04.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.123 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.123 DEBUG:boto:Params: {} 2024-04-12T04:28:04.123 DEBUG:boto:Token: None 2024-04-12T04:28:04.123 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-72/obj8 2024-04-12T04:28:04.123 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bgM89ubDxX2YaPww3HDnJv/zV24= 2024-04-12T04:28:04.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bgM89ubDxX2YaPww3HDnJv/zV24='} 2024-04-12T04:28:04.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a75837a303fb36f5-006618b854-2a56-a2'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.128 DEBUG:boto:path=/dgbpxg-73/ 2024-04-12T04:28:04.129 DEBUG:boto:auth_path=/dgbpxg-73/ 2024-04-12T04:28:04.129 DEBUG:boto:Method: HEAD 2024-04-12T04:28:04.129 DEBUG:boto:Path: /dgbpxg-73/ 2024-04-12T04:28:04.129 DEBUG:boto:Data: 2024-04-12T04:28:04.129 DEBUG:boto:Headers: {} 2024-04-12T04:28:04.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.129 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.129 DEBUG:boto:Params: {} 2024-04-12T04:28:04.129 DEBUG:boto:Token: None 2024-04-12T04:28:04.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-73/ 2024-04-12T04:28:04.130 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:loulrHod+30xYT3L2jRMA3Bv1W4= 2024-04-12T04:28:04.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:loulrHod+30xYT3L2jRMA3Bv1W4=', 'Content-Length': '0'} 2024-04-12T04:28:04.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a83d5599848b04a1-006618b854-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.132 DEBUG:boto:path=/dgbpxg-73/obj9 2024-04-12T04:28:04.132 DEBUG:boto:auth_path=/dgbpxg-73/obj9 2024-04-12T04:28:04.132 DEBUG:boto:Method: PUT 2024-04-12T04:28:04.132 DEBUG:boto:Path: /dgbpxg-73/obj9 2024-04-12T04:28:04.132 DEBUG:boto:Data: 2024-04-12T04:28:04.132 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:04.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.132 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.133 DEBUG:boto:Params: {} 2024-04-12T04:28:04.133 DEBUG:boto:Token: None 2024-04-12T04:28:04.133 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-73/obj9 2024-04-12T04:28:04.133 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Zr+hp3yKvPrUcAfMXiIlRRmHe6g= 2024-04-12T04:28:04.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Zr+hp3yKvPrUcAfMXiIlRRmHe6g='} 2024-04-12T04:28:04.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006be377f296237a81-006618b854-2a56-a2'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.179 DEBUG:boto:path=/dgbpxg-73/ 2024-04-12T04:28:04.179 DEBUG:boto:auth_path=/dgbpxg-73/ 2024-04-12T04:28:04.179 DEBUG:boto:Method: HEAD 2024-04-12T04:28:04.179 DEBUG:boto:Path: /dgbpxg-73/ 2024-04-12T04:28:04.179 DEBUG:boto:Data: 2024-04-12T04:28:04.180 DEBUG:boto:Headers: {} 2024-04-12T04:28:04.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.180 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.180 DEBUG:boto:Params: {} 2024-04-12T04:28:04.180 DEBUG:boto:Token: None 2024-04-12T04:28:04.180 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-73/ 2024-04-12T04:28:04.180 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:loulrHod+30xYT3L2jRMA3Bv1W4= 2024-04-12T04:28:04.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:loulrHod+30xYT3L2jRMA3Bv1W4=', 'Content-Length': '0'} 2024-04-12T04:28:04.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db75a967a5b6a1a9-006618b854-2a56-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.183 DEBUG:boto:path=/dgbpxg-73/obj10 2024-04-12T04:28:04.183 DEBUG:boto:auth_path=/dgbpxg-73/obj10 2024-04-12T04:28:04.183 DEBUG:boto:Method: PUT 2024-04-12T04:28:04.183 DEBUG:boto:Path: /dgbpxg-73/obj10 2024-04-12T04:28:04.183 DEBUG:boto:Data: 2024-04-12T04:28:04.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:04.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:28:04.183 DEBUG:boto:Port: 8001 2024-04-12T04:28:04.183 DEBUG:boto:Params: {} 2024-04-12T04:28:04.183 DEBUG:boto:Token: None 2024-04-12T04:28:04.184 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:04 GMT /dgbpxg-73/obj10 2024-04-12T04:28:04.184 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:brgX6ud1HUtY7X1sXvKatlIa8FQ= 2024-04-12T04:28:04.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:brgX6ud1HUtY7X1sXvKatlIa8FQ='} 2024-04-12T04:28:04.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a13449b7dad8e815-006618b854-2a56-a2'), ('Date', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:04.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:04.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:04.194 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:04.241 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:04.242 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:03.990630Z" 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.304 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:04.305 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:04.305 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:28:04.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:04.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:04.306 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:04.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:04.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:04.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:04.413+0000 7f49d10b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:04.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:04.414+0000 7f49d10b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:04.429 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:04.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896072.908482_2152.1", 2024-04-12T04:28:04.434 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:27:52.908482Z", 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:04.435 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:04.436 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:04Z" 2024-04-12T04:28:04.436 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:04.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896072.908482_2152.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:27:52.908482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:04Z" } 2024-04-12T04:28:04.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896072.908482_2152.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:27:52.908482Z', 'realm_epoch': 2}}] 2024-04-12T04:28:04.437 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:04.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:28:04.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896083.990630_2160.1 target=1_1712896072.908482_2152.1 2024-04-12T04:28:09.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:09.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:09.440 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:09.548+0000 7fb2b7c85780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:09.549+0000 7fb2b7c85780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:09.566 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896072.908482_2152.1", 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:27:52.908482Z", 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:09Z" 2024-04-12T04:28:09.572 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:09.573 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896072.908482_2152.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:27:52.908482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:09Z" } 2024-04-12T04:28:09.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896072.908482_2152.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:27:52.908482Z', 'realm_epoch': 2}}] 2024-04-12T04:28:09.573 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:09.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:28:09.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896083.990630_2160.1 target=1_1712896072.908482_2152.1 2024-04-12T04:28:14.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.575 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:14.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:14.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:14.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:14.672+0000 7fac90afe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:14.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:14.673+0000 7fac90afe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:14.682 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896072.908482_2152.1", 2024-04-12T04:28:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:27:52.908482Z", 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:14Z" 2024-04-12T04:28:14.688 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:14.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896072.908482_2152.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:27:52.908482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:14Z" } 2024-04-12T04:28:14.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896072.908482_2152.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:27:52.908482Z', 'realm_epoch': 2}}] 2024-04-12T04:28:14.689 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:14.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896072.908482_2152.1'} 2024-04-12T04:28:14.689 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896083.990630_2160.1 target=1_1712896072.908482_2152.1 2024-04-12T04:28:19.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.690 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:19.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:19.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:19.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:19.806+0000 7fe797be9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:19.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:19.807+0000 7fe797be9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:19.825 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:19.826 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:19.827 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:03.990630Z", 2024-04-12T04:28:19.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:19Z" 2024-04-12T04:28:19.829 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:19.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896083.990630_2160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:03.990630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:19Z" } 2024-04-12T04:28:19.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896083.990630_2160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:03.990630Z', 'realm_epoch': 2}}] 2024-04-12T04:28:19.831 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:19.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:19.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:19.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.831 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:19.878 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:19.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015069", 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:27:49.140802Z" 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007419", 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:04.195195Z" 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018264", 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:04.124609Z" 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018975", 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:04.144546Z" 2024-04-12T04:28:19.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:19.939 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:19.939 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015069', 1: '00000000000000000000:00000000000000007419', 2: '00000000000000000000:00000000000000018264', 3: '00000000000000000000:00000000000000018975'} 2024-04-12T04:28:19.940 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:28:19.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:19.940 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 797626959077608812 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000015069", 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:27:49.140802Z" 2024-04-12T04:28:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007419", 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:04.195195Z" 2024-04-12T04:28:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018264", 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:04.124609Z" 2024-04-12T04:28:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018975", 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:04.144546Z" 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:20Z" 2024-04-12T04:28:20.054 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:20.055 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 797626959077608812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015069", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:27:49.140802Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007419", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:04.195195Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018264", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:04.124609Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018975", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:04.144546Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:28:20Z" } 2024-04-12T04:28:20.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015069', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:27:49.140802Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007419', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:04.195195Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018264', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:04.124609Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018975', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:04.144546Z'}}] 2024-04-12T04:28:20.056 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015069', 1: '00000000000000000000:00000000000000007419', 2: '00000000000000000000:00000000000000018264', 3: '00000000000000000000:00000000000000018975'} 2024-04-12T04:28:20.056 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015069', 1: '00000000000000000000:00000000000000007419', 2: '00000000000000000000:00000000000000018264', 3: '00000000000000000000:00000000000000018975'} 2024-04-12T04:28:20.056 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:28:20.056 DEBUG:boto:path=/dgbpxg-72/ 2024-04-12T04:28:20.056 DEBUG:boto:auth_path=/dgbpxg-72/ 2024-04-12T04:28:20.056 DEBUG:boto:Method: HEAD 2024-04-12T04:28:20.056 DEBUG:boto:Path: /dgbpxg-72/ 2024-04-12T04:28:20.057 DEBUG:boto:Data: 2024-04-12T04:28:20.057 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.057 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.057 DEBUG:boto:Params: {} 2024-04-12T04:28:20.057 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:28:20.058 DEBUG:boto:Token: None 2024-04-12T04:28:20.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/ 2024-04-12T04:28:20.058 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1QuRrpsE6vYm39EfN7dbdYfGhiQ= 2024-04-12T04:28:20.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1QuRrpsE6vYm39EfN7dbdYfGhiQ=', 'Content-Length': '0'} 2024-04-12T04:28:20.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000798a2496d2a99d8f-006618b864-2a61-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.090 DEBUG:boto:path=/dgbpxg-72/obj7 2024-04-12T04:28:20.091 DEBUG:boto:auth_path=/dgbpxg-72/obj7 2024-04-12T04:28:20.091 DEBUG:boto:Method: HEAD 2024-04-12T04:28:20.091 DEBUG:boto:Path: /dgbpxg-72/obj7 2024-04-12T04:28:20.091 DEBUG:boto:Data: 2024-04-12T04:28:20.091 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.091 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.091 DEBUG:boto:Params: {} 2024-04-12T04:28:20.091 DEBUG:boto:Token: None 2024-04-12T04:28:20.092 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj7 2024-04-12T04:28:20.092 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:E0uKNeo0qKs7K8dmmU1pQc37I6g= 2024-04-12T04:28:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:E0uKNeo0qKs7K8dmmU1pQc37I6g=', 'Content-Length': '0'} 2024-04-12T04:28:20.093 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cc0f09442e19db74-006618b864-2a61-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.093 DEBUG:boto:path=/dgbpxg-72/obj8 2024-04-12T04:28:20.093 DEBUG:boto:auth_path=/dgbpxg-72/obj8 2024-04-12T04:28:20.093 DEBUG:boto:Method: HEAD 2024-04-12T04:28:20.093 DEBUG:boto:Path: /dgbpxg-72/obj8 2024-04-12T04:28:20.094 DEBUG:boto:Data: 2024-04-12T04:28:20.094 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.094 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.094 DEBUG:boto:Params: {} 2024-04-12T04:28:20.094 DEBUG:boto:Token: None 2024-04-12T04:28:20.094 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj8 2024-04-12T04:28:20.094 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Y0OT+PHtL4IAjRiBFcYEKH+MNKE= 2024-04-12T04:28:20.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Y0OT+PHtL4IAjRiBFcYEKH+MNKE=', 'Content-Length': '0'} 2024-04-12T04:28:20.095 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000aa6b12540d570446-006618b864-2a61-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.096 DEBUG:boto:path=/dgbpxg-72/obj9 2024-04-12T04:28:20.096 DEBUG:boto:auth_path=/dgbpxg-72/obj9 2024-04-12T04:28:20.096 DEBUG:boto:Method: HEAD 2024-04-12T04:28:20.096 DEBUG:boto:Path: /dgbpxg-72/obj9 2024-04-12T04:28:20.096 DEBUG:boto:Data: 2024-04-12T04:28:20.096 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.096 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.096 DEBUG:boto:Params: {} 2024-04-12T04:28:20.097 DEBUG:boto:Token: None 2024-04-12T04:28:20.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj9 2024-04-12T04:28:20.097 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DFP2NHKaCYI1XiXTNTvITIvr6qI= 2024-04-12T04:28:20.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DFP2NHKaCYI1XiXTNTvITIvr6qI=', 'Content-Length': '0'} 2024-04-12T04:28:20.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-73:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d5c1bcefedc4cc8-006618b864-2a61-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.099 DEBUG:boto:path=/dgbpxg-72/obj9 2024-04-12T04:28:20.099 DEBUG:boto:auth_path=/dgbpxg-72/obj9 2024-04-12T04:28:20.099 DEBUG:boto:Method: GET 2024-04-12T04:28:20.099 DEBUG:boto:Path: /dgbpxg-72/obj9 2024-04-12T04:28:20.099 DEBUG:boto:Data: 2024-04-12T04:28:20.099 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.099 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.100 DEBUG:boto:Params: {} 2024-04-12T04:28:20.100 DEBUG:boto:Token: None 2024-04-12T04:28:20.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj9 2024-04-12T04:28:20.100 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zw/rFnvLCopgzQJA3S+zhmV+MgY= 2024-04-12T04:28:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zw/rFnvLCopgzQJA3S+zhmV+MgY=', 'Content-Length': '0'} 2024-04-12T04:28:20.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-73:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000336dbcad85460ab3-006618b864-2a61-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.102 DEBUG:boto:path=/dgbpxg-72/obj10 2024-04-12T04:28:20.102 DEBUG:boto:auth_path=/dgbpxg-72/obj10 2024-04-12T04:28:20.102 DEBUG:boto:Method: HEAD 2024-04-12T04:28:20.102 DEBUG:boto:Path: /dgbpxg-72/obj10 2024-04-12T04:28:20.102 DEBUG:boto:Data: 2024-04-12T04:28:20.102 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.102 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.103 DEBUG:boto:Params: {} 2024-04-12T04:28:20.103 DEBUG:boto:Token: None 2024-04-12T04:28:20.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj10 2024-04-12T04:28:20.103 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:FiwODjRfELZN91pszYI9CvQMDDg= 2024-04-12T04:28:20.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FiwODjRfELZN91pszYI9CvQMDDg=', 'Content-Length': '0'} 2024-04-12T04:28:20.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-73:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041f684d5373caed3-006618b864-2a61-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.105 DEBUG:boto:path=/dgbpxg-72/obj10 2024-04-12T04:28:20.105 DEBUG:boto:auth_path=/dgbpxg-72/obj10 2024-04-12T04:28:20.105 DEBUG:boto:Method: GET 2024-04-12T04:28:20.105 DEBUG:boto:Path: /dgbpxg-72/obj10 2024-04-12T04:28:20.105 DEBUG:boto:Data: 2024-04-12T04:28:20.105 DEBUG:boto:Headers: {} 2024-04-12T04:28:20.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:20.105 DEBUG:boto:Port: 8000 2024-04-12T04:28:20.105 DEBUG:boto:Params: {} 2024-04-12T04:28:20.106 DEBUG:boto:Token: None 2024-04-12T04:28:20.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:20 GMT /dgbpxg-72/obj10 2024-04-12T04:28:20.106 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:DqD3cGRsORgdNOdphcP6iXT4zRQ= 2024-04-12T04:28:20.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:20 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DqD3cGRsORgdNOdphcP6iXT4zRQ=', 'Content-Length': '0'} 2024-04-12T04:28:20.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd800b3e1-d3eb-4250-a3c6-cce8e78e25b6:dgbpxg-73:b7d2416c-9d6f-4780-9e14-1f66614caba8.10220.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cd58bf8fc5c3662-006618b864-2a61-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:20.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:20.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:20.108 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:20.243 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:28:20.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:20.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:20.244 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:20.362 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:28:20.363 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2024-04-12T04:28:20.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:20.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:20.365 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:20.446 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896100.218385_2165.1", 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:20.218385Z" 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.463 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:20.464 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:20.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:28:20.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:20.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:20.465 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:20.512 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:20.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:20.564+0000 7fa8b4e69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:20.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:20.565+0000 7fa8b4e69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:20.575 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:20.575 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:20.576 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.577 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:20.579 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:20.580 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:03.990630Z", 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:20Z" 2024-04-12T04:28:20.581 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:20.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896083.990630_2160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:03.990630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:20Z" } 2024-04-12T04:28:20.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896083.990630_2160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:03.990630Z', 'realm_epoch': 2}}] 2024-04-12T04:28:20.583 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:20.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:20.583 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896100.218385_2165.1 target=1_1712896083.990630_2160.1 2024-04-12T04:28:25.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:25.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:25.584 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:25.641 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:25.641 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:25.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:25.683+0000 7fe99e9e9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:25.684+0000 7fe99e9e9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:25.694 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:25.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:25.696 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:25.697 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:25.698 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:03.990630Z", 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:25.699 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:25.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:25.700 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:25Z" 2024-04-12T04:28:25.700 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:25.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896083.990630_2160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:03.990630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:25Z" } 2024-04-12T04:28:25.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896083.990630_2160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:03.990630Z', 'realm_epoch': 2}}] 2024-04-12T04:28:25.701 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:25.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:25.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896100.218385_2165.1 target=1_1712896083.990630_2160.1 2024-04-12T04:28:30.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:30.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:30.703 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:30.758 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:30.759 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:30.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:30.818+0000 7f4901be1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:30.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:30.819+0000 7f4901be1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:30.832 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:30.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:30.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:03.990630Z", 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:30.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:30Z" 2024-04-12T04:28:30.838 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:30.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896083.990630_2160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:03.990630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:30Z" } 2024-04-12T04:28:30.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896083.990630_2160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:03.990630Z', 'realm_epoch': 2}}] 2024-04-12T04:28:30.840 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:30.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:30.840 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896100.218385_2165.1 target=1_1712896083.990630_2160.1 2024-04-12T04:28:35.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:35.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:35.841 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:35.940+0000 7f4a94fc2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:35.941+0000 7f4a94fc2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:35.949 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:35.950 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:35.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:35.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896083.990630_2160.1", 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:03.990630Z", 2024-04-12T04:28:35.954 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:35Z" 2024-04-12T04:28:35.955 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:35.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896083.990630_2160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:03.990630Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:35Z" } 2024-04-12T04:28:35.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896083.990630_2160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:03.990630Z', 'realm_epoch': 2}}] 2024-04-12T04:28:35.956 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:35.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896083.990630_2160.1'} 2024-04-12T04:28:35.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896100.218385_2165.1 target=1_1712896083.990630_2160.1 2024-04-12T04:28:40.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:40.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:40.958 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:41.067+0000 7f5e70e68780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:41.068+0000 7f5e70e68780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:41.078 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:41.078 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:41.078 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:41.078 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:41.078 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.079 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:41.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:41.081 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.082 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896100.218385_2165.1", 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:20.218385Z", 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.083 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:41Z" 2024-04-12T04:28:41.084 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:41.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896100.218385_2165.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:20.218385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:41Z" } 2024-04-12T04:28:41.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896100.218385_2165.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:20.218385Z', 'realm_epoch': 2}}] 2024-04-12T04:28:41.086 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:41.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896100.218385_2165.1'} 2024-04-12T04:28:41.086 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:41.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:41.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:41.087 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:41.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:41.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:41.197 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:28:41.197 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-74 2024-04-12T04:28:41.197 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:28:41.197 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:28:41.198 DEBUG:boto:Method: PUT 2024-04-12T04:28:41.198 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:28:41.199 DEBUG:boto:Data: 2024-04-12T04:28:41.199 DEBUG:boto:Headers: {} 2024-04-12T04:28:41.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:41.199 DEBUG:boto:Port: 8000 2024-04-12T04:28:41.199 DEBUG:boto:Params: {} 2024-04-12T04:28:41.199 DEBUG:boto:Token: None 2024-04-12T04:28:41.199 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:28:41 GMT /dgbpxg-74/ 2024-04-12T04:28:41.200 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5f7akVOFk8bbyTewD8eZnScODI0= 2024-04-12T04:28:41.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5f7akVOFk8bbyTewD8eZnScODI0=', 'Content-Length': '0'} 2024-04-12T04:28:41.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002508bdf9adbe3fe7-006618b879-2a61-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:41.217 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-75 2024-04-12T04:28:41.218 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:28:41.218 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:28:41.218 DEBUG:boto:Method: PUT 2024-04-12T04:28:41.218 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:28:41.218 DEBUG:boto:Data: 2024-04-12T04:28:41.218 DEBUG:boto:Headers: {} 2024-04-12T04:28:41.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:41.218 DEBUG:boto:Port: 8000 2024-04-12T04:28:41.218 DEBUG:boto:Params: {} 2024-04-12T04:28:41.219 DEBUG:boto:Token: None 2024-04-12T04:28:41.219 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:28:41 GMT /dgbpxg-75/ 2024-04-12T04:28:41.219 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:OSe3keBsicRSIq8Hh9c3OJXwHO0= 2024-04-12T04:28:41.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OSe3keBsicRSIq8Hh9c3OJXwHO0=', 'Content-Length': '0'} 2024-04-12T04:28:41.235 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ee5077841074415-006618b879-2a61-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:41.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:41.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:41.236 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:41.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:41.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:41.330 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:28:41.366 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:41.239851Z" 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.368 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:41.368 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:41.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:28:41.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:41.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:41.369 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:41.418 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:41.418 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:41.475+0000 7f480e23e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:41.476+0000 7f480e23e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:41.487 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:41.487 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:41.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.215501_2166.1", 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.215501Z", 2024-04-12T04:28:41.492 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:41Z" 2024-04-12T04:28:41.493 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:41.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.215501_2166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.215501Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:41Z" } 2024-04-12T04:28:41.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.215501_2166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.215501Z', 'realm_epoch': 2}}] 2024-04-12T04:28:41.495 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:41.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.215501_2166.1'} 2024-04-12T04:28:41.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896121.239851_2173.1 target=1_1712896121.215501_2166.1 2024-04-12T04:28:46.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:46.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:46.498 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:46.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:46.603+0000 7f2f24921780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:46.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:46.604+0000 7f2f24921780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:46.617 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:46.618 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:46.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:46.620 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:46.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:28:46.622 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:46Z" 2024-04-12T04:28:46.623 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:46.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:46Z" } 2024-04-12T04:28:46.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:28:46.625 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:46.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:46.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:46.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.625 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:46.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:46.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:46.760 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:28:46.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:46.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:46.814 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:46.864 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:28:46.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-74', '--dest-bucket=dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-74', '--dest-bucket=dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.865 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dgbpxg-74 --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-74 --dest-bucket=dgbpxg-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:46.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:46.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:46.980 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:28:46.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-75', '--dest-bucket=dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-75', '--dest-bucket=dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.981 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dgbpxg-75 --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-75 --dest-bucket=dgbpxg-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.114 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:28:47.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.115 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.235 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:28:47.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.236 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.342 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:47.342 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:47.342 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 20, 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.343 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:28:47.344 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:28:47.345 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:28:47.346 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:28:47.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:28:47.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "directional": [ 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "source_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "dest_zone": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:28:47.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-74", 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-74", 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-75", 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:28:47.350 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:28:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-75", 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-75", 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-75", 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:28:47.356 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:28:47.357 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:28:47.358 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:28:47.359 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.360 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:28:47.361 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:28:47.362 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:47.363 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:28:47.364 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.365 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:28:47.366 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:47.367 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:28:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:28:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:47.368 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:47.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.369 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.494 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:28:47.494 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:28:47.494 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:28:47.494 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.494 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:28:47.494 DEBUG:boto:Data: 2024-04-12T04:28:47.495 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.495 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.495 DEBUG:boto:Params: {} 2024-04-12T04:28:47.495 DEBUG:boto:Token: None 2024-04-12T04:28:47.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-74/ 2024-04-12T04:28:47.495 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qeEh1sYm10SKH7ch0goxvgLstbU= 2024-04-12T04:28:47.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qeEh1sYm10SKH7ch0goxvgLstbU=', 'Content-Length': '0'} 2024-04-12T04:28:47.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003750300005be9c08-006618b87f-2ca5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.688 DEBUG:boto:path=/dgbpxg-74/obj1 2024-04-12T04:28:47.688 DEBUG:boto:auth_path=/dgbpxg-74/obj1 2024-04-12T04:28:47.688 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.688 DEBUG:boto:Path: /dgbpxg-74/obj1 2024-04-12T04:28:47.688 DEBUG:boto:Data: 2024-04-12T04:28:47.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:47.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.689 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.689 DEBUG:boto:Params: {} 2024-04-12T04:28:47.689 DEBUG:boto:Token: None 2024-04-12T04:28:47.689 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-74/obj1 2024-04-12T04:28:47.689 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:r2X4R5Jt1Vj1QJmiLRxWvCGdVWI= 2024-04-12T04:28:47.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r2X4R5Jt1Vj1QJmiLRxWvCGdVWI='} 2024-04-12T04:28:47.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7f345bec9060226-006618b87f-2ca5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.735 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:28:47.735 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:28:47.735 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.735 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:28:47.735 DEBUG:boto:Data: 2024-04-12T04:28:47.735 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.735 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.735 DEBUG:boto:Params: {} 2024-04-12T04:28:47.736 DEBUG:boto:Token: None 2024-04-12T04:28:47.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-74/ 2024-04-12T04:28:47.736 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qeEh1sYm10SKH7ch0goxvgLstbU= 2024-04-12T04:28:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qeEh1sYm10SKH7ch0goxvgLstbU=', 'Content-Length': '0'} 2024-04-12T04:28:47.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eeb9bfaa0ecf955-006618b87f-2ca5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.740 DEBUG:boto:path=/dgbpxg-74/obj2 2024-04-12T04:28:47.740 DEBUG:boto:auth_path=/dgbpxg-74/obj2 2024-04-12T04:28:47.740 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.740 DEBUG:boto:Path: /dgbpxg-74/obj2 2024-04-12T04:28:47.740 DEBUG:boto:Data: 2024-04-12T04:28:47.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:47.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.741 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.741 DEBUG:boto:Params: {} 2024-04-12T04:28:47.741 DEBUG:boto:Token: None 2024-04-12T04:28:47.741 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-74/obj2 2024-04-12T04:28:47.741 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uBBepJ8m7qkB7V4jOVbDag22kGE= 2024-04-12T04:28:47.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uBBepJ8m7qkB7V4jOVbDag22kGE='} 2024-04-12T04:28:47.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8adf918c8c444ca-006618b87f-2ca5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.769 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:28:47.769 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:28:47.769 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.770 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:28:47.770 DEBUG:boto:Data: 2024-04-12T04:28:47.770 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.770 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.770 DEBUG:boto:Params: {} 2024-04-12T04:28:47.770 DEBUG:boto:Token: None 2024-04-12T04:28:47.770 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-75/ 2024-04-12T04:28:47.770 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dtc8mBVwgJXKVONLsMf/j2am/1A= 2024-04-12T04:28:47.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dtc8mBVwgJXKVONLsMf/j2am/1A=', 'Content-Length': '0'} 2024-04-12T04:28:47.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000772aaf0e49ecc82b-006618b87f-2ca5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.780 DEBUG:boto:path=/dgbpxg-75/obj3 2024-04-12T04:28:47.781 DEBUG:boto:auth_path=/dgbpxg-75/obj3 2024-04-12T04:28:47.781 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.781 DEBUG:boto:Path: /dgbpxg-75/obj3 2024-04-12T04:28:47.781 DEBUG:boto:Data: 2024-04-12T04:28:47.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:47.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.781 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.781 DEBUG:boto:Params: {} 2024-04-12T04:28:47.781 DEBUG:boto:Token: None 2024-04-12T04:28:47.782 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-75/obj3 2024-04-12T04:28:47.782 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1Z1O67Adg94FHzDFqT46jmERCTw= 2024-04-12T04:28:47.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Z1O67Adg94FHzDFqT46jmERCTw='} 2024-04-12T04:28:47.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a74d522d6d69c273-006618b87f-2ca5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.827 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:28:47.827 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:28:47.827 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.827 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:28:47.827 DEBUG:boto:Data: 2024-04-12T04:28:47.827 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.828 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.828 DEBUG:boto:Params: {} 2024-04-12T04:28:47.828 DEBUG:boto:Token: None 2024-04-12T04:28:47.828 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-75/ 2024-04-12T04:28:47.828 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Dtc8mBVwgJXKVONLsMf/j2am/1A= 2024-04-12T04:28:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dtc8mBVwgJXKVONLsMf/j2am/1A=', 'Content-Length': '0'} 2024-04-12T04:28:47.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005f49fde668517ed-006618b87f-2ca5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.833 DEBUG:boto:path=/dgbpxg-75/obj4 2024-04-12T04:28:47.833 DEBUG:boto:auth_path=/dgbpxg-75/obj4 2024-04-12T04:28:47.833 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.833 DEBUG:boto:Path: /dgbpxg-75/obj4 2024-04-12T04:28:47.833 DEBUG:boto:Data: 2024-04-12T04:28:47.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:28:47.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:28:47.834 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.834 DEBUG:boto:Params: {} 2024-04-12T04:28:47.834 DEBUG:boto:Token: None 2024-04-12T04:28:47.834 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dgbpxg-75/obj4 2024-04-12T04:28:47.834 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:qj8k5xbPF510wSpjBcnbkQSD9rE= 2024-04-12T04:28:47.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qj8k5xbPF510wSpjBcnbkQSD9rE='} 2024-04-12T04:28:47.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fba084fa60767426-006618b87f-2ca5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:47.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:47.864 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:47.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.048 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:28:48.118 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:48.119 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:41.239851Z" 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.120 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:48.121 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:48.121 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:28:48.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.122 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:48.191 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.191 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:48.286+0000 7f4d714a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:48.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:28:48.288+0000 7f4d714a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:48.323 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:48.323 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:48.323 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:48.323 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:48.324 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:48.324 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:28:48.324 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:48.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.324 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:48.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:48.325 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.325 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:48.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:28:48.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:28:48.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:28:48.326 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:48.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:28:48.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:48.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:28:48.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:48.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:48.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:48.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:28:48.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:48.332 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.332 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:28:48.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:28:48.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:48.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:28:48.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:28:48.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.335 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:48.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.335 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:48.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:28:48.336 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:28:48.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.336 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:48Z" 2024-04-12T04:28:48.336 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:48.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:48Z" } 2024-04-12T04:28:48.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:28:48.338 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:48.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:28:48.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:48.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.340 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:48.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.514 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:28:48.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.516 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:28:48.517 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:48.518 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:28:48.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:48.518 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:48.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.574 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.672 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:48.672 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:48.672 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:48.673 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:48.674 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:48.675 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:48.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:48.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:48Z" 2024-04-12T04:28:48.678 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:48.679 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:28:48Z" } 2024-04-12T04:28:48.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:28:48.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:48.680 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:48.680 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:28:53.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:53.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:53.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:53.805 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:53.805 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:53Z" 2024-04-12T04:28:53.811 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:53.812 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:28:53Z" } 2024-04-12T04:28:53.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:28:53.812 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:53.812 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:53.813 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:28:58.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:58.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:58.814 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:58.926 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:28:58.927 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:58.928 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:28:58.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:58.930 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:58.931 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:28:58.932 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:28:58.933 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:28:58Z" 2024-04-12T04:28:58.934 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:28:58.935 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:28:58Z" } 2024-04-12T04:28:58.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:28:58.935 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:58.935 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:28:58.935 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:03.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:03.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:03.937 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:03.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:03.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:04.070 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:04.070 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:04.070 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:04.070 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:04.070 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:04.071 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:04.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:04.073 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:04.074 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:04.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:04.076 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:04Z" 2024-04-12T04:29:04.076 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:04.077 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:04Z" } 2024-04-12T04:29:04.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:04.077 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:04.077 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:04.077 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:09.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:09.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:09.079 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:09.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:09.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:09.195 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:09Z" 2024-04-12T04:29:09.202 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:09.203 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:09Z" } 2024-04-12T04:29:09.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:09.203 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:09.204 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:09.204 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:14.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:14.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:14.205 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:14.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:14.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:14.336 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:14.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:14.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:14.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.341 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:14.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:14.342 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:14.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:14.342 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:14Z" 2024-04-12T04:29:14.342 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:14.343 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:14Z" } 2024-04-12T04:29:14.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:14.343 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:14.343 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:14.344 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:19.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:19.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:19.345 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:19.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:19.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:19.462 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:19Z" 2024-04-12T04:29:19.468 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:19.468 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:19Z" } 2024-04-12T04:29:19.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:19.469 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:19.469 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:19.469 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:24.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:24.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:24.470 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:24.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:24.528 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:24.593 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:24Z" 2024-04-12T04:29:24.599 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:24.599 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:24Z" } 2024-04-12T04:29:24.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:24.600 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:24.600 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:24.600 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:29.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:29.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:29.601 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:29.729 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:29.729 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:29.729 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:29.729 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:29.729 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:29.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:29.735 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:29Z" 2024-04-12T04:29:29.735 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:29.736 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:29Z" } 2024-04-12T04:29:29.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:29.736 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:29.736 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:29.736 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:34.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:34.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:34.737 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:34.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:34.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:34.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:34.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:34.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.848 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:34.849 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:34.849 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:34.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:34.849 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:34Z" 2024-04-12T04:29:34.849 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:34.850 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:34Z" } 2024-04-12T04:29:34.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:34.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:34.850 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:34.850 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:39.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:39.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:39.852 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:39.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:39.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.985 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:39.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:39.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:39.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:39.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:39Z" 2024-04-12T04:29:39.990 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:39.991 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:39Z" } 2024-04-12T04:29:39.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:39.992 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:39.992 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:39.992 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:44.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:44.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:44.994 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:45.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:45.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:45.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:45.112 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:45.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:45.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:45.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:45Z" 2024-04-12T04:29:45.116 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:45.117 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:45Z" } 2024-04-12T04:29:45.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:45.117 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:45.117 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:45.118 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:50.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:50.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:50.119 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:50.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:50.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:50Z" 2024-04-12T04:29:50.232 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:50.232 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:50Z" } 2024-04-12T04:29:50.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:50.233 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:50.233 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:50.233 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:29:55.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:55.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:29:55.234 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:29:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:29:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:29:55.361 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:29:55.361 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:29:55.361 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:55.362 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:29:55.363 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:55.364 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:29:55.365 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:29:55.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:29:55.367 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:29:55Z" 2024-04-12T04:29:55.367 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:29:55.368 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:29:55Z" } 2024-04-12T04:29:55.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:29:55.368 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:55.368 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:29:55.368 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:30:00.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:00.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:00.373 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:00.431 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:00.431 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.493 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:30:00.494 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:00.495 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:00.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:00.497 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:00Z" 2024-04-12T04:30:00.498 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:00.499 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:00Z" } 2024-04-12T04:30:00.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:00.500 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:00.500 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:00.500 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:30:05.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:05.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:05.502 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:05.629 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:30:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:05Z" 2024-04-12T04:30:05.636 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:05.637 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:05Z" } 2024-04-12T04:30:05.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:05.637 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:05.637 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:05.638 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:30:10.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:10.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:10.639 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:10.757 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:10.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:10.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:10.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:10.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:10.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:10Z" 2024-04-12T04:30:10.763 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:10.764 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:10Z" } 2024-04-12T04:30:10.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:10.764 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:10.764 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:10.765 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:30:15.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:15.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:15.766 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:15.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:15.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.887 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047776", 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.920124Z" 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:15.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.090738Z" 2024-04-12T04:30:15.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000053926", 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:26:19.903224Z" 2024-04-12T04:30:15.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:15.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:15Z" 2024-04-12T04:30:15.892 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:15.893 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047776", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:26:19.920124Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:08.090738Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053926", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:26:19.903224Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:15Z" } 2024-04-12T04:30:15.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047776', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.920124Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.090738Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053926', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:26:19.903224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:15.893 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:15.893 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047776', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000053926', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:15.894 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047917 target=00000000000000000000:00000000000000047776, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000054208 target=00000000000000000000:00000000000000053926 2024-04-12T04:30:20.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:20.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:20.895 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.003 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:21.003 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:21.003 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:30:21.004 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:30:21.005 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:30:21.006 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:21.007 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:20Z" 2024-04-12T04:30:21.009 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:21.009 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:20Z" } 2024-04-12T04:30:21.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:21.010 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:21.010 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:21.010 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:30:21.011 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:30:21.011 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:30:21.011 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.011 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:30:21.011 DEBUG:boto:Data: 2024-04-12T04:30:21.011 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.011 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.011 DEBUG:boto:Params: {} 2024-04-12T04:30:21.012 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:30:21.012 DEBUG:boto:Token: None 2024-04-12T04:30:21.013 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-74/ 2024-04-12T04:30:21.013 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:QdfgW3bO7is5dlDnmpzU0nPiSc8= 2024-04-12T04:30:21.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdfgW3bO7is5dlDnmpzU0nPiSc8=', 'Content-Length': '0'} 2024-04-12T04:30:21.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000875233cbae4671ab-006618b8dd-2ca7-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.029 DEBUG:boto:path=/dgbpxg-74/obj1 2024-04-12T04:30:21.029 DEBUG:boto:auth_path=/dgbpxg-74/obj1 2024-04-12T04:30:21.029 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.030 DEBUG:boto:Path: /dgbpxg-74/obj1 2024-04-12T04:30:21.030 DEBUG:boto:Data: 2024-04-12T04:30:21.030 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.030 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.030 DEBUG:boto:Params: {} 2024-04-12T04:30:21.030 DEBUG:boto:Token: None 2024-04-12T04:30:21.030 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-74/obj1 2024-04-12T04:30:21.030 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zKHkcs4ub9TzWy39SFVmSjBypDU= 2024-04-12T04:30:21.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zKHkcs4ub9TzWy39SFVmSjBypDU=', 'Content-Length': '0'} 2024-04-12T04:30:21.032 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000024100f9ea4a3894-006618b8dd-2ca7-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.032 DEBUG:boto:path=/dgbpxg-74/obj2 2024-04-12T04:30:21.032 DEBUG:boto:auth_path=/dgbpxg-74/obj2 2024-04-12T04:30:21.032 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.032 DEBUG:boto:Path: /dgbpxg-74/obj2 2024-04-12T04:30:21.032 DEBUG:boto:Data: 2024-04-12T04:30:21.033 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.033 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.033 DEBUG:boto:Params: {} 2024-04-12T04:30:21.033 DEBUG:boto:Token: None 2024-04-12T04:30:21.033 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-74/obj2 2024-04-12T04:30:21.033 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zfQ3VAYY2ZOpbi6XVon1WfJMuug= 2024-04-12T04:30:21.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zfQ3VAYY2ZOpbi6XVon1WfJMuug=', 'Content-Length': '0'} 2024-04-12T04:30:21.035 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cf339f754f889d7d-006618b8dd-2ca7-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.035 DEBUG:boto:path=/dgbpxg-74/obj3 2024-04-12T04:30:21.035 DEBUG:boto:auth_path=/dgbpxg-74/obj3 2024-04-12T04:30:21.035 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.035 DEBUG:boto:Path: /dgbpxg-74/obj3 2024-04-12T04:30:21.035 DEBUG:boto:Data: 2024-04-12T04:30:21.035 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.035 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.035 DEBUG:boto:Params: {} 2024-04-12T04:30:21.036 DEBUG:boto:Token: None 2024-04-12T04:30:21.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-74/obj3 2024-04-12T04:30:21.036 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:hm/vjnzhIZD/ZiMENeF/tBKuOm8= 2024-04-12T04:30:21.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hm/vjnzhIZD/ZiMENeF/tBKuOm8=', 'Content-Length': '0'} 2024-04-12T04:30:21.037 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008fbd158aceb0ba29-006618b8dd-2ca7-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.037 DEBUG:boto:path=/dgbpxg-74/obj4 2024-04-12T04:30:21.037 DEBUG:boto:auth_path=/dgbpxg-74/obj4 2024-04-12T04:30:21.038 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.038 DEBUG:boto:Path: /dgbpxg-74/obj4 2024-04-12T04:30:21.038 DEBUG:boto:Data: 2024-04-12T04:30:21.038 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.038 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.038 DEBUG:boto:Params: {} 2024-04-12T04:30:21.038 DEBUG:boto:Token: None 2024-04-12T04:30:21.038 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-74/obj4 2024-04-12T04:30:21.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nhq2fvZW+SV635Scg2XtefZf+DI= 2024-04-12T04:30:21.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nhq2fvZW+SV635Scg2XtefZf+DI=', 'Content-Length': '0'} 2024-04-12T04:30:21.040 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000071b0a6af425dba17-006618b8dd-2ca7-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.040 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:30:21.040 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:30:21.040 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.040 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:30:21.040 DEBUG:boto:Data: 2024-04-12T04:30:21.040 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.041 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.041 DEBUG:boto:Params: {} 2024-04-12T04:30:21.041 DEBUG:boto:Token: None 2024-04-12T04:30:21.041 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/ 2024-04-12T04:30:21.041 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8HSZZ4qBe33CMKUyOsEByqpL8/E= 2024-04-12T04:30:21.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8HSZZ4qBe33CMKUyOsEByqpL8/E=', 'Content-Length': '0'} 2024-04-12T04:30:21.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e921808f887c593f-006618b8dd-2ca7-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.043 DEBUG:boto:path=/dgbpxg-75/obj1 2024-04-12T04:30:21.044 DEBUG:boto:auth_path=/dgbpxg-75/obj1 2024-04-12T04:30:21.044 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.044 DEBUG:boto:Path: /dgbpxg-75/obj1 2024-04-12T04:30:21.044 DEBUG:boto:Data: 2024-04-12T04:30:21.044 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.044 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.044 DEBUG:boto:Params: {} 2024-04-12T04:30:21.044 DEBUG:boto:Token: None 2024-04-12T04:30:21.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj1 2024-04-12T04:30:21.045 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:O8wdd/NkhEMwtIY9wC0UvYe5G5U= 2024-04-12T04:30:21.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O8wdd/NkhEMwtIY9wC0UvYe5G5U=', 'Content-Length': '0'} 2024-04-12T04:30:21.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-74:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c839679bc09fe350-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.047 DEBUG:boto:path=/dgbpxg-75/obj1 2024-04-12T04:30:21.047 DEBUG:boto:auth_path=/dgbpxg-75/obj1 2024-04-12T04:30:21.047 DEBUG:boto:Method: GET 2024-04-12T04:30:21.047 DEBUG:boto:Path: /dgbpxg-75/obj1 2024-04-12T04:30:21.047 DEBUG:boto:Data: 2024-04-12T04:30:21.047 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.047 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.047 DEBUG:boto:Params: {} 2024-04-12T04:30:21.048 DEBUG:boto:Token: None 2024-04-12T04:30:21.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj1 2024-04-12T04:30:21.048 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:2dpJs/5Un15iC4KbF5EOOhnAClI= 2024-04-12T04:30:21.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dpJs/5Un15iC4KbF5EOOhnAClI=', 'Content-Length': '0'} 2024-04-12T04:30:21.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-74:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cf53a827bd5693a-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.094 DEBUG:boto:path=/dgbpxg-75/obj2 2024-04-12T04:30:21.095 DEBUG:boto:auth_path=/dgbpxg-75/obj2 2024-04-12T04:30:21.095 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.095 DEBUG:boto:Path: /dgbpxg-75/obj2 2024-04-12T04:30:21.095 DEBUG:boto:Data: 2024-04-12T04:30:21.095 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.095 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.095 DEBUG:boto:Params: {} 2024-04-12T04:30:21.095 DEBUG:boto:Token: None 2024-04-12T04:30:21.095 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj2 2024-04-12T04:30:21.096 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:GPEFuuY1Gnk5qsuBZM8zoLSR6Ns= 2024-04-12T04:30:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GPEFuuY1Gnk5qsuBZM8zoLSR6Ns=', 'Content-Length': '0'} 2024-04-12T04:30:21.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-74:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e4ac8c8265d1497-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.097 DEBUG:boto:path=/dgbpxg-75/obj2 2024-04-12T04:30:21.097 DEBUG:boto:auth_path=/dgbpxg-75/obj2 2024-04-12T04:30:21.097 DEBUG:boto:Method: GET 2024-04-12T04:30:21.098 DEBUG:boto:Path: /dgbpxg-75/obj2 2024-04-12T04:30:21.098 DEBUG:boto:Data: 2024-04-12T04:30:21.098 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.098 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.098 DEBUG:boto:Params: {} 2024-04-12T04:30:21.098 DEBUG:boto:Token: None 2024-04-12T04:30:21.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj2 2024-04-12T04:30:21.098 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:gl+v1nTvAA0QU+OiEjDqR0R2wT4= 2024-04-12T04:30:21.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gl+v1nTvAA0QU+OiEjDqR0R2wT4=', 'Content-Length': '0'} 2024-04-12T04:30:21.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-74:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc22afe9a20f6c67-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.100 DEBUG:boto:path=/dgbpxg-75/obj3 2024-04-12T04:30:21.100 DEBUG:boto:auth_path=/dgbpxg-75/obj3 2024-04-12T04:30:21.101 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.101 DEBUG:boto:Path: /dgbpxg-75/obj3 2024-04-12T04:30:21.101 DEBUG:boto:Data: 2024-04-12T04:30:21.101 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.101 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.101 DEBUG:boto:Params: {} 2024-04-12T04:30:21.101 DEBUG:boto:Token: None 2024-04-12T04:30:21.101 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj3 2024-04-12T04:30:21.102 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:lsAgU0LGIk2rsgqnRVdGFyfsoqg= 2024-04-12T04:30:21.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lsAgU0LGIk2rsgqnRVdGFyfsoqg=', 'Content-Length': '0'} 2024-04-12T04:30:21.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf733aa0ee52bad1-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.103 DEBUG:boto:path=/dgbpxg-75/obj3 2024-04-12T04:30:21.103 DEBUG:boto:auth_path=/dgbpxg-75/obj3 2024-04-12T04:30:21.103 DEBUG:boto:Method: GET 2024-04-12T04:30:21.104 DEBUG:boto:Path: /dgbpxg-75/obj3 2024-04-12T04:30:21.104 DEBUG:boto:Data: 2024-04-12T04:30:21.104 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.104 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.104 DEBUG:boto:Params: {} 2024-04-12T04:30:21.104 DEBUG:boto:Token: None 2024-04-12T04:30:21.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj3 2024-04-12T04:30:21.104 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8MR6Hg+Bjg8Bd4Lhp6EGV7UG1fQ= 2024-04-12T04:30:21.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8MR6Hg+Bjg8Bd4Lhp6EGV7UG1fQ=', 'Content-Length': '0'} 2024-04-12T04:30:21.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2d9ba1c3d2f7872-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.106 DEBUG:boto:path=/dgbpxg-75/obj4 2024-04-12T04:30:21.106 DEBUG:boto:auth_path=/dgbpxg-75/obj4 2024-04-12T04:30:21.107 DEBUG:boto:Method: HEAD 2024-04-12T04:30:21.107 DEBUG:boto:Path: /dgbpxg-75/obj4 2024-04-12T04:30:21.107 DEBUG:boto:Data: 2024-04-12T04:30:21.107 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.107 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.107 DEBUG:boto:Params: {} 2024-04-12T04:30:21.107 DEBUG:boto:Token: None 2024-04-12T04:30:21.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj4 2024-04-12T04:30:21.108 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:pd0c8BwmUvBWZ8gMJrle/ppdLIo= 2024-04-12T04:30:21.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pd0c8BwmUvBWZ8gMJrle/ppdLIo=', 'Content-Length': '0'} 2024-04-12T04:30:21.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0c4c1ec483460d-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.109 DEBUG:boto:path=/dgbpxg-75/obj4 2024-04-12T04:30:21.109 DEBUG:boto:auth_path=/dgbpxg-75/obj4 2024-04-12T04:30:21.109 DEBUG:boto:Method: GET 2024-04-12T04:30:21.110 DEBUG:boto:Path: /dgbpxg-75/obj4 2024-04-12T04:30:21.110 DEBUG:boto:Data: 2024-04-12T04:30:21.110 DEBUG:boto:Headers: {} 2024-04-12T04:30:21.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:30:21.110 DEBUG:boto:Port: 8001 2024-04-12T04:30:21.110 DEBUG:boto:Params: {} 2024-04-12T04:30:21.110 DEBUG:boto:Token: None 2024-04-12T04:30:21.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:30:21 GMT /dgbpxg-75/obj4 2024-04-12T04:30:21.110 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EtNCJ/SDGgx6T74BYf6FSvul7G8= 2024-04-12T04:30:21.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:21 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EtNCJ/SDGgx6T74BYf6FSvul7G8=', 'Content-Length': '0'} 2024-04-12T04:30:21.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4bd39a813366924-006618b8dd-2ca7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:30:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:21.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.113 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dgbpxg-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.231 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"dgbpxg-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:30:21.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dgbpxg-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.331 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:30:21.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.332 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.452 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:30:21.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.452 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.575 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:30:21.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.576 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.627 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.674 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:30:21.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.675 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.725 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.725 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.781 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:30:21.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.829 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.829 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:21.907 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:21.907 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 21, 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.908 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.909 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.910 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:30:21.911 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.914 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.915 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:30:21.918 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.919 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:30:21.920 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:30:21.921 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.923 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:30:21.924 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:30:21.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:21.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:30:21.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:30:21.928 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.929 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:30:21.930 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:21.931 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:21.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:21.932 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:22.091 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:30:22.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.092 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:22.349 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:30:22.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.350 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:22.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:22.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:22.602 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:30:22.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-74', '--source-bucket=dgbpxg-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-74', '--source-bucket=dgbpxg-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.603 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dgbpxg-74 --source-zones=a1 --dest-zones=a2 --bucket=dgbpxg-74 --source-bucket=dgbpxg-74 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:22.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:22.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:22.816 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:30:22.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-74', '--source-bucket=dgbpxg-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-74', '--source-bucket=dgbpxg-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:22.817 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dgbpxg-75 --source-zones=a1 --dest-zones=a2 --bucket=dgbpxg-74 --source-bucket=dgbpxg-75 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:23.022 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:30:23.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:23.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:23.022 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:23.078 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:23.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:23.196 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-74","source":{"bucket":"dgbpxg-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dgbpxg-75","source":{"bucket":"dgbpxg-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:30:23.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:23.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:30:23.197 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:30:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:23.245 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:23.295 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:30:23.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:23.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:23.295 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:23.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:23.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:23.159827Z" 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.420 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:30:23.421 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:23.421 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:30:23.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:23.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:23.422 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:23.469 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:23.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:23.522+0000 7f1fcbcc8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:23.523+0000 7f1fcbcc8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.535 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:23.536 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:23.537 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:23.538 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:23.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:23.540 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:23.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:23.541 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:23Z" 2024-04-12T04:30:23.541 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:23.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:23Z" } 2024-04-12T04:30:23.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:23.542 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:23.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:23.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:28.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:28.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:28.544 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:28.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:28.600 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:28.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:28.639+0000 7fb108c9e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:28.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:28.641+0000 7fb108c9e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:28.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:28.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:28.662 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:28.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:28.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:28Z" 2024-04-12T04:30:28.665 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:28.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:28Z" } 2024-04-12T04:30:28.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:28.667 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:28.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:28.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:33.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:33.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:33.668 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:33.724 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:33.724 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:33.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:33.779+0000 7f3291e96780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:33.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:33.780+0000 7f3291e96780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.792 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:33.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:33.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:33.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:33.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:33.797 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:33Z" 2024-04-12T04:30:33.798 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:33.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:33Z" } 2024-04-12T04:30:33.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:33.799 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:33.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:33.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:38.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:38.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:38.801 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:38.857 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:38.857 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:38.912+0000 7fe3ce8cb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:38.913+0000 7fe3ce8cb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:38.929 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:38.930 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:38.931 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:38.932 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:38.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.934 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:38.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:38.935 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:38.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:38.935 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:38Z" 2024-04-12T04:30:38.935 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:38.936 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:38Z" } 2024-04-12T04:30:38.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:38.936 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:38.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:38.936 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:43.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:43.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:43.937 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:43.994 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:43.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:44.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:44.034+0000 7f7f7e024780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:44.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:44.035+0000 7f7f7e024780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:44.049 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:44.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:44.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:44.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:44Z" 2024-04-12T04:30:44.055 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:44.056 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:44Z" } 2024-04-12T04:30:44.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:44.057 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:44.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:44.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:49.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:49.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:49.058 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:49.114 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:49.115 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:49.159+0000 7fed5a1d1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:49.160+0000 7fed5a1d1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:49.177 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896121.239851_2173.1", 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:41.239851Z", 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:49Z" 2024-04-12T04:30:49.183 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:49.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896121.239851_2173.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:41.239851Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:49Z" } 2024-04-12T04:30:49.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896121.239851_2173.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:41.239851Z', 'realm_epoch': 2}}] 2024-04-12T04:30:49.184 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:49.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896121.239851_2173.1'} 2024-04-12T04:30:49.184 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896223.159827_2264.1 target=1_1712896121.239851_2173.1 2024-04-12T04:30:54.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:54.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:54.297+0000 7f6d85422780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:54.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:54.298+0000 7f6d85422780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:23.159827Z", 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:54Z" 2024-04-12T04:30:54.320 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:54.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896223.159827_2264.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:23.159827Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:54Z" } 2024-04-12T04:30:54.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896223.159827_2264.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:23.159827Z', 'realm_epoch': 2}}] 2024-04-12T04:30:54.322 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:54.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:54.322 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:30:54.322 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:30:54.322 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:30:54.322 DEBUG:boto:Method: HEAD 2024-04-12T04:30:54.322 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:30:54.323 DEBUG:boto:Data: 2024-04-12T04:30:54.323 DEBUG:boto:Headers: {} 2024-04-12T04:30:54.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.323 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.323 DEBUG:boto:Params: {} 2024-04-12T04:30:54.323 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:30:54.324 DEBUG:boto:Token: None 2024-04-12T04:30:54.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-74/ 2024-04-12T04:30:54.324 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aE3HFYLg/9re1FYf9hChpyrYMYc= 2024-04-12T04:30:54.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aE3HFYLg/9re1FYf9hChpyrYMYc=', 'Content-Length': '0'} 2024-04-12T04:30:54.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065558e001266bc2f-006618b8fe-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.338 DEBUG:boto:path=/dgbpxg-74/obj5 2024-04-12T04:30:54.338 DEBUG:boto:auth_path=/dgbpxg-74/obj5 2024-04-12T04:30:54.338 DEBUG:boto:Method: PUT 2024-04-12T04:30:54.338 DEBUG:boto:Path: /dgbpxg-74/obj5 2024-04-12T04:30:54.338 DEBUG:boto:Data: 2024-04-12T04:30:54.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:30:54.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.338 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.339 DEBUG:boto:Params: {} 2024-04-12T04:30:54.339 DEBUG:boto:Token: None 2024-04-12T04:30:54.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-74/obj5 2024-04-12T04:30:54.339 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Z5w+Y6S/TiqOHP03E+O1m7ibnWI= 2024-04-12T04:30:54.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z5w+Y6S/TiqOHP03E+O1m7ibnWI='} 2024-04-12T04:30:54.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000999fd4d1321d222e-006618b8fe-2d94-a1'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.348 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:30:54.348 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:30:54.348 DEBUG:boto:Method: HEAD 2024-04-12T04:30:54.348 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:30:54.348 DEBUG:boto:Data: 2024-04-12T04:30:54.348 DEBUG:boto:Headers: {} 2024-04-12T04:30:54.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.348 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.349 DEBUG:boto:Params: {} 2024-04-12T04:30:54.349 DEBUG:boto:Token: None 2024-04-12T04:30:54.349 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-74/ 2024-04-12T04:30:54.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:aE3HFYLg/9re1FYf9hChpyrYMYc= 2024-04-12T04:30:54.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aE3HFYLg/9re1FYf9hChpyrYMYc=', 'Content-Length': '0'} 2024-04-12T04:30:54.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d7e59b70e6ee363-006618b8fe-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.351 DEBUG:boto:path=/dgbpxg-74/obj6 2024-04-12T04:30:54.351 DEBUG:boto:auth_path=/dgbpxg-74/obj6 2024-04-12T04:30:54.351 DEBUG:boto:Method: PUT 2024-04-12T04:30:54.352 DEBUG:boto:Path: /dgbpxg-74/obj6 2024-04-12T04:30:54.352 DEBUG:boto:Data: 2024-04-12T04:30:54.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:30:54.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.352 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.352 DEBUG:boto:Params: {} 2024-04-12T04:30:54.352 DEBUG:boto:Token: None 2024-04-12T04:30:54.352 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-74/obj6 2024-04-12T04:30:54.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:NemmMkq8+L3oS2+TBYDQmajYLeU= 2024-04-12T04:30:54.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NemmMkq8+L3oS2+TBYDQmajYLeU='} 2024-04-12T04:30:54.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023d5e98e5259d519-006618b8fe-2d94-a1'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.362 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:30:54.362 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:30:54.362 DEBUG:boto:Method: HEAD 2024-04-12T04:30:54.362 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:30:54.362 DEBUG:boto:Data: 2024-04-12T04:30:54.362 DEBUG:boto:Headers: {} 2024-04-12T04:30:54.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.363 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.363 DEBUG:boto:Params: {} 2024-04-12T04:30:54.363 DEBUG:boto:Token: None 2024-04-12T04:30:54.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-75/ 2024-04-12T04:30:54.363 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vXzeM9iDFjtAuma9m5DtD5FohZg= 2024-04-12T04:30:54.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vXzeM9iDFjtAuma9m5DtD5FohZg=', 'Content-Length': '0'} 2024-04-12T04:30:54.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9659b9fb604f9ac-006618b8fe-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.366 DEBUG:boto:path=/dgbpxg-75/obj7 2024-04-12T04:30:54.366 DEBUG:boto:auth_path=/dgbpxg-75/obj7 2024-04-12T04:30:54.366 DEBUG:boto:Method: PUT 2024-04-12T04:30:54.366 DEBUG:boto:Path: /dgbpxg-75/obj7 2024-04-12T04:30:54.366 DEBUG:boto:Data: 2024-04-12T04:30:54.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:30:54.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.366 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.366 DEBUG:boto:Params: {} 2024-04-12T04:30:54.367 DEBUG:boto:Token: None 2024-04-12T04:30:54.367 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-75/obj7 2024-04-12T04:30:54.367 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:nNzBqoWXhJKx13Dnjwu7hQEvgcg= 2024-04-12T04:30:54.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nNzBqoWXhJKx13Dnjwu7hQEvgcg='} 2024-04-12T04:30:54.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082c19480daf5004e-006618b8fe-2d94-a1'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.411 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:30:54.411 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:30:54.411 DEBUG:boto:Method: HEAD 2024-04-12T04:30:54.411 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:30:54.411 DEBUG:boto:Data: 2024-04-12T04:30:54.411 DEBUG:boto:Headers: {} 2024-04-12T04:30:54.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.411 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.411 DEBUG:boto:Params: {} 2024-04-12T04:30:54.412 DEBUG:boto:Token: None 2024-04-12T04:30:54.412 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-75/ 2024-04-12T04:30:54.412 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:vXzeM9iDFjtAuma9m5DtD5FohZg= 2024-04-12T04:30:54.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vXzeM9iDFjtAuma9m5DtD5FohZg=', 'Content-Length': '0'} 2024-04-12T04:30:54.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060be05d1b3e98e88-006618b8fe-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.415 DEBUG:boto:path=/dgbpxg-75/obj8 2024-04-12T04:30:54.415 DEBUG:boto:auth_path=/dgbpxg-75/obj8 2024-04-12T04:30:54.415 DEBUG:boto:Method: PUT 2024-04-12T04:30:54.415 DEBUG:boto:Path: /dgbpxg-75/obj8 2024-04-12T04:30:54.415 DEBUG:boto:Data: 2024-04-12T04:30:54.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:30:54.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:30:54.415 DEBUG:boto:Port: 8000 2024-04-12T04:30:54.415 DEBUG:boto:Params: {} 2024-04-12T04:30:54.415 DEBUG:boto:Token: None 2024-04-12T04:30:54.415 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:30:54 GMT /dgbpxg-75/obj8 2024-04-12T04:30:54.416 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:e1BfAhuJuyjFnpCSP7dBGA537L4= 2024-04-12T04:30:54.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:30:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e1BfAhuJuyjFnpCSP7dBGA537L4='} 2024-04-12T04:30:54.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000677f90dc3202eaf-006618b8fe-2d94-a1'), ('Date', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:30:54.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.424 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:54.503 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:30:54.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:23.159827Z" 2024-04-12T04:30:54.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.524 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:30:54.525 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:54.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:30:54.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:54.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:54.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:54.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:54.613+0000 7f21542fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:30:54.614+0000 7f21542fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:30:54.632 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:30:54.633 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.634 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.635 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:23.159827Z", 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:54Z" 2024-04-12T04:30:54.637 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:54.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896223.159827_2264.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:23.159827Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:30:54Z" } 2024-04-12T04:30:54.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896223.159827_2264.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:23.159827Z', 'realm_epoch': 2}}] 2024-04-12T04:30:54.639 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:54.639 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:30:54.639 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:30:54.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.639 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:54.686 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:54.686 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:54.745 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:30:54.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.745 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:30:54.745 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:30:54.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:54.747 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:54.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.747 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:30:54.748 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:54.748 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:30:54.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:54.748 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:54Z" 2024-04-12T04:30:54.856 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:54.857 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:54Z" } 2024-04-12T04:30:54.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:54.857 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:54.857 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:54.858 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:30:59.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:59.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:30:59.859 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:30:59.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:30:59.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:30:59.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:30:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:30:59.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:30:59.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:30:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:30:59.994 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:30:59Z" 2024-04-12T04:30:59.994 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:30:59.995 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:30:59Z" } 2024-04-12T04:30:59.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:30:59.995 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:59.995 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:30:59.995 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:04.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:04.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:04.997 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:05.108 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:05.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:05.110 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:05.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:05.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.113 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:05.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:05.114 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:05.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:05.114 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:05Z" 2024-04-12T04:31:05.114 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:05.115 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:05Z" } 2024-04-12T04:31:05.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:05.115 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:05.115 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:05.116 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:10.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:10.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:10.117 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:10.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:10.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:10Z" 2024-04-12T04:31:10.248 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:10.249 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:10Z" } 2024-04-12T04:31:10.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:10.249 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:10.249 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:10.250 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:15.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:15.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:15.250 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:15.306 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:15.306 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:15.369 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:15Z" 2024-04-12T04:31:15.375 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:15.376 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:15Z" } 2024-04-12T04:31:15.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:15.377 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:15.377 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:15.377 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:20.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:20.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:20.379 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:20.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:20.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:20.513 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:20.513 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:20.513 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:20.514 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:20.515 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:20.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:20Z" 2024-04-12T04:31:20.519 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:20.520 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:20Z" } 2024-04-12T04:31:20.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:20.521 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:20.521 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:20.521 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:25.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:25.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:25.522 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:25.581 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:25.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:25.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:25.641 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:25.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:25.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:25.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:25Z" 2024-04-12T04:31:25.645 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:25.646 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:25Z" } 2024-04-12T04:31:25.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:25.647 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:25.647 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:25.647 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:30.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:30.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:30.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:30.707 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:30.707 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:30.779 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:30.779 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:30.780 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:30.781 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:30.782 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:30.783 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:30.784 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:30.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:30.785 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:30Z" 2024-04-12T04:31:30.785 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:30.786 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:30Z" } 2024-04-12T04:31:30.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:30.786 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:30.786 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:30.786 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:35.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:35.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:35.788 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:35.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:35.845 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:35.908 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:35.908 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:35.908 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:35.908 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:35.908 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:35.909 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:35.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:35.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:35.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:35Z" 2024-04-12T04:31:35.913 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:35.914 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:35Z" } 2024-04-12T04:31:35.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:35.914 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:35.915 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:35.915 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:40.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:40.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:40.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:40.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:40.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:41.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:41.042 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:41Z" 2024-04-12T04:31:41.043 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:41.043 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:41Z" } 2024-04-12T04:31:41.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:41.044 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:41.044 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:41.044 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:46.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:46.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:46.045 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:46.103 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:46.189 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:46.189 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:46.190 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:46.191 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:46.192 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:46.193 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:46.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:46.195 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:46Z" 2024-04-12T04:31:46.195 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:46.196 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:46Z" } 2024-04-12T04:31:46.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:46.196 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:46.196 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:46.196 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:51.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:51.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:51.197 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:51.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:51.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:51.314 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:51.314 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:51.320 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:51Z" 2024-04-12T04:31:51.320 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:51.321 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:51Z" } 2024-04-12T04:31:51.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:51.321 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:51.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:51.321 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:31:56.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:56.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:31:56.323 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:31:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:31:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:31:56.439 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:31:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:31:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:31:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.444 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:31:56.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:31:56.445 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:31:56.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:31:56.445 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:31:56Z" 2024-04-12T04:31:56.445 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:31:56.446 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:31:56Z" } 2024-04-12T04:31:56.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:31:56.446 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:56.446 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:31:56.446 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:01.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:01.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:01.448 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:01.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:01.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.571 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:01.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:01.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:01Z" 2024-04-12T04:32:01.576 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:01.577 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:01Z" } 2024-04-12T04:32:01.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:01.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:01.577 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:01.578 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:06.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:06.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:06.579 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:06.636 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:06.636 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:06.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:32:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:06.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:06Z" 2024-04-12T04:32:06.705 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:06.706 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:06Z" } 2024-04-12T04:32:06.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:06.707 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:06.707 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:06.707 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:11.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:11.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:11.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:11.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:11.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:11.822 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:11.822 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:11.822 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:11.822 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:11.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:11.824 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:11Z" 2024-04-12T04:32:11.827 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:11.828 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:11Z" } 2024-04-12T04:32:11.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:11.829 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:11.829 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:11.829 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:16.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:16.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:16.831 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:16.953 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:16Z" 2024-04-12T04:32:16.959 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:16.960 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:16Z" } 2024-04-12T04:32:16.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:16.960 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:16.960 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:16.960 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:21.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:21.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:21.962 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:22.021 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:22.021 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:22.088 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:22.088 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:22.088 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:22.088 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000047917", 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:22.089 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.770057Z" 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:22.090 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.865125Z" 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054208", 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:22.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:47.807225Z" 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:22.092 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:22Z" 2024-04-12T04:32:22.093 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:22.094 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047917", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.770057Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:47.865125Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054208", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:28:47.807225Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:22Z" } 2024-04-12T04:32:22.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047917', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.770057Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.865125Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054208', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:28:47.807225Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:22.095 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:22.095 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047917', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000054208', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:22.095 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047917, shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054208 2024-04-12T04:32:27.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.096 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:27.154 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:27.231 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:27Z" 2024-04-12T04:32:27.237 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:27.238 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:27Z" } 2024-04-12T04:32:27.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:27.238 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:27.239 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:27.239 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:32:27.239 DEBUG:boto:path=/dgbpxg-75/ 2024-04-12T04:32:27.239 DEBUG:boto:auth_path=/dgbpxg-75/ 2024-04-12T04:32:27.239 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.240 DEBUG:boto:Path: /dgbpxg-75/ 2024-04-12T04:32:27.240 DEBUG:boto:Data: 2024-04-12T04:32:27.240 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.240 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.240 DEBUG:boto:Params: {} 2024-04-12T04:32:27.240 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:32:27.241 DEBUG:boto:Token: None 2024-04-12T04:32:27.241 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-75/ 2024-04-12T04:32:27.241 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:is2eOv+Aa7v4psMtENGYgMBMZNc= 2024-04-12T04:32:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:is2eOv+Aa7v4psMtENGYgMBMZNc=', 'Content-Length': '0'} 2024-04-12T04:32:27.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d7381587e9609b7-006618b95b-2d9e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.274 DEBUG:boto:path=/dgbpxg-75/obj5 2024-04-12T04:32:27.274 DEBUG:boto:auth_path=/dgbpxg-75/obj5 2024-04-12T04:32:27.274 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.274 DEBUG:boto:Path: /dgbpxg-75/obj5 2024-04-12T04:32:27.274 DEBUG:boto:Data: 2024-04-12T04:32:27.274 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.274 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.274 DEBUG:boto:Params: {} 2024-04-12T04:32:27.275 DEBUG:boto:Token: None 2024-04-12T04:32:27.275 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-75/obj5 2024-04-12T04:32:27.275 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:I0ipV08hw2rsoQYlKkfaIjCMMog= 2024-04-12T04:32:27.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:I0ipV08hw2rsoQYlKkfaIjCMMog=', 'Content-Length': '0'} 2024-04-12T04:32:27.276 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f95305368dae3721-006618b95b-2d9e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.277 DEBUG:boto:path=/dgbpxg-75/obj6 2024-04-12T04:32:27.277 DEBUG:boto:auth_path=/dgbpxg-75/obj6 2024-04-12T04:32:27.277 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.277 DEBUG:boto:Path: /dgbpxg-75/obj6 2024-04-12T04:32:27.277 DEBUG:boto:Data: 2024-04-12T04:32:27.277 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.277 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.277 DEBUG:boto:Params: {} 2024-04-12T04:32:27.277 DEBUG:boto:Token: None 2024-04-12T04:32:27.277 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-75/obj6 2024-04-12T04:32:27.278 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:EmkRJXJ1jWn1ZomjqHuGAQGjUoA= 2024-04-12T04:32:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:EmkRJXJ1jWn1ZomjqHuGAQGjUoA=', 'Content-Length': '0'} 2024-04-12T04:32:27.279 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000939d0ce689021115-006618b95b-2d9e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.279 DEBUG:boto:path=/dgbpxg-75/obj7 2024-04-12T04:32:27.279 DEBUG:boto:auth_path=/dgbpxg-75/obj7 2024-04-12T04:32:27.279 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.280 DEBUG:boto:Path: /dgbpxg-75/obj7 2024-04-12T04:32:27.280 DEBUG:boto:Data: 2024-04-12T04:32:27.280 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.280 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.280 DEBUG:boto:Params: {} 2024-04-12T04:32:27.280 DEBUG:boto:Token: None 2024-04-12T04:32:27.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-75/obj7 2024-04-12T04:32:27.280 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JmiVsgX9K80RgEwDjj7epVJs+tY= 2024-04-12T04:32:27.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JmiVsgX9K80RgEwDjj7epVJs+tY=', 'Content-Length': '0'} 2024-04-12T04:32:27.281 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000093e5c54d54641426-006618b95b-2d9e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.282 DEBUG:boto:path=/dgbpxg-75/obj8 2024-04-12T04:32:27.282 DEBUG:boto:auth_path=/dgbpxg-75/obj8 2024-04-12T04:32:27.282 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.282 DEBUG:boto:Path: /dgbpxg-75/obj8 2024-04-12T04:32:27.282 DEBUG:boto:Data: 2024-04-12T04:32:27.282 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.282 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.282 DEBUG:boto:Params: {} 2024-04-12T04:32:27.283 DEBUG:boto:Token: None 2024-04-12T04:32:27.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-75/obj8 2024-04-12T04:32:27.283 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:4VmI/sXVVJOYAJpAWkpxoU8L6dQ= 2024-04-12T04:32:27.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4VmI/sXVVJOYAJpAWkpxoU8L6dQ=', 'Content-Length': '0'} 2024-04-12T04:32:27.284 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000ce720f300ac2e20-006618b95b-2d9e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.284 DEBUG:boto:path=/dgbpxg-74/ 2024-04-12T04:32:27.284 DEBUG:boto:auth_path=/dgbpxg-74/ 2024-04-12T04:32:27.285 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.285 DEBUG:boto:Path: /dgbpxg-74/ 2024-04-12T04:32:27.285 DEBUG:boto:Data: 2024-04-12T04:32:27.285 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.285 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.285 DEBUG:boto:Params: {} 2024-04-12T04:32:27.285 DEBUG:boto:Token: None 2024-04-12T04:32:27.285 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/ 2024-04-12T04:32:27.285 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:5DdNnHgymUXd8iBY/Jxwmny+2a4= 2024-04-12T04:32:27.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5DdNnHgymUXd8iBY/Jxwmny+2a4=', 'Content-Length': '0'} 2024-04-12T04:32:27.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071a84344f5117dc5-006618b95b-2d9e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.288 DEBUG:boto:path=/dgbpxg-74/obj7 2024-04-12T04:32:27.288 DEBUG:boto:auth_path=/dgbpxg-74/obj7 2024-04-12T04:32:27.288 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.288 DEBUG:boto:Path: /dgbpxg-74/obj7 2024-04-12T04:32:27.288 DEBUG:boto:Data: 2024-04-12T04:32:27.288 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.288 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.288 DEBUG:boto:Params: {} 2024-04-12T04:32:27.289 DEBUG:boto:Token: None 2024-04-12T04:32:27.289 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj7 2024-04-12T04:32:27.289 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uTV/maCC57AICrFoCso+qacxnm4= 2024-04-12T04:32:27.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uTV/maCC57AICrFoCso+qacxnm4=', 'Content-Length': '0'} 2024-04-12T04:32:27.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb4fc8cff8d54972-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.293 DEBUG:boto:path=/dgbpxg-74/obj7 2024-04-12T04:32:27.293 DEBUG:boto:auth_path=/dgbpxg-74/obj7 2024-04-12T04:32:27.294 DEBUG:boto:Method: GET 2024-04-12T04:32:27.294 DEBUG:boto:Path: /dgbpxg-74/obj7 2024-04-12T04:32:27.294 DEBUG:boto:Data: 2024-04-12T04:32:27.294 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.294 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.294 DEBUG:boto:Params: {} 2024-04-12T04:32:27.294 DEBUG:boto:Token: None 2024-04-12T04:32:27.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj7 2024-04-12T04:32:27.294 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oQFmCRJzkMcZW3tCa0qlguViQdo= 2024-04-12T04:32:27.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oQFmCRJzkMcZW3tCa0qlguViQdo=', 'Content-Length': '0'} 2024-04-12T04:32:27.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cd26c74025314be-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.338 DEBUG:boto:path=/dgbpxg-74/obj8 2024-04-12T04:32:27.339 DEBUG:boto:auth_path=/dgbpxg-74/obj8 2024-04-12T04:32:27.339 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.339 DEBUG:boto:Path: /dgbpxg-74/obj8 2024-04-12T04:32:27.339 DEBUG:boto:Data: 2024-04-12T04:32:27.339 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.339 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.339 DEBUG:boto:Params: {} 2024-04-12T04:32:27.339 DEBUG:boto:Token: None 2024-04-12T04:32:27.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj8 2024-04-12T04:32:27.340 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ac2pN+O6K7oMIr0HSD47IilNWes= 2024-04-12T04:32:27.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ac2pN+O6K7oMIr0HSD47IilNWes=', 'Content-Length': '0'} 2024-04-12T04:32:27.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e066448cb2a76e6c-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.342 DEBUG:boto:path=/dgbpxg-74/obj8 2024-04-12T04:32:27.342 DEBUG:boto:auth_path=/dgbpxg-74/obj8 2024-04-12T04:32:27.342 DEBUG:boto:Method: GET 2024-04-12T04:32:27.342 DEBUG:boto:Path: /dgbpxg-74/obj8 2024-04-12T04:32:27.342 DEBUG:boto:Data: 2024-04-12T04:32:27.342 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.342 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.342 DEBUG:boto:Params: {} 2024-04-12T04:32:27.343 DEBUG:boto:Token: None 2024-04-12T04:32:27.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj8 2024-04-12T04:32:27.343 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sAx6YJQL3Dl1AP81EKlABl6s/A= 2024-04-12T04:32:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sAx6YJQL3Dl1AP81EKlABl6s/A=', 'Content-Length': '0'} 2024-04-12T04:32:27.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064e11de75b61897d-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.345 DEBUG:boto:path=/dgbpxg-74/obj7 2024-04-12T04:32:27.345 DEBUG:boto:auth_path=/dgbpxg-74/obj7 2024-04-12T04:32:27.345 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.345 DEBUG:boto:Path: /dgbpxg-74/obj7 2024-04-12T04:32:27.345 DEBUG:boto:Data: 2024-04-12T04:32:27.345 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.345 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.345 DEBUG:boto:Params: {} 2024-04-12T04:32:27.346 DEBUG:boto:Token: None 2024-04-12T04:32:27.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj7 2024-04-12T04:32:27.346 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:uTV/maCC57AICrFoCso+qacxnm4= 2024-04-12T04:32:27.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uTV/maCC57AICrFoCso+qacxnm4=', 'Content-Length': '0'} 2024-04-12T04:32:27.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000254e8b3779f24e1b-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.348 DEBUG:boto:path=/dgbpxg-74/obj7 2024-04-12T04:32:27.348 DEBUG:boto:auth_path=/dgbpxg-74/obj7 2024-04-12T04:32:27.348 DEBUG:boto:Method: GET 2024-04-12T04:32:27.348 DEBUG:boto:Path: /dgbpxg-74/obj7 2024-04-12T04:32:27.349 DEBUG:boto:Data: 2024-04-12T04:32:27.349 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.349 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.349 DEBUG:boto:Params: {} 2024-04-12T04:32:27.349 DEBUG:boto:Token: None 2024-04-12T04:32:27.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj7 2024-04-12T04:32:27.349 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:oQFmCRJzkMcZW3tCa0qlguViQdo= 2024-04-12T04:32:27.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oQFmCRJzkMcZW3tCa0qlguViQdo=', 'Content-Length': '0'} 2024-04-12T04:32:27.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e963a40873c1857d-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.351 DEBUG:boto:path=/dgbpxg-74/obj8 2024-04-12T04:32:27.351 DEBUG:boto:auth_path=/dgbpxg-74/obj8 2024-04-12T04:32:27.351 DEBUG:boto:Method: HEAD 2024-04-12T04:32:27.352 DEBUG:boto:Path: /dgbpxg-74/obj8 2024-04-12T04:32:27.352 DEBUG:boto:Data: 2024-04-12T04:32:27.352 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.352 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.352 DEBUG:boto:Params: {} 2024-04-12T04:32:27.352 DEBUG:boto:Token: None 2024-04-12T04:32:27.352 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj8 2024-04-12T04:32:27.352 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ac2pN+O6K7oMIr0HSD47IilNWes= 2024-04-12T04:32:27.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ac2pN+O6K7oMIr0HSD47IilNWes=', 'Content-Length': '0'} 2024-04-12T04:32:27.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa480644b40af1ce-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.354 DEBUG:boto:path=/dgbpxg-74/obj8 2024-04-12T04:32:27.354 DEBUG:boto:auth_path=/dgbpxg-74/obj8 2024-04-12T04:32:27.354 DEBUG:boto:Method: GET 2024-04-12T04:32:27.355 DEBUG:boto:Path: /dgbpxg-74/obj8 2024-04-12T04:32:27.355 DEBUG:boto:Data: 2024-04-12T04:32:27.355 DEBUG:boto:Headers: {} 2024-04-12T04:32:27.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:32:27.355 DEBUG:boto:Port: 8001 2024-04-12T04:32:27.355 DEBUG:boto:Params: {} 2024-04-12T04:32:27.355 DEBUG:boto:Token: None 2024-04-12T04:32:27.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:32:27 GMT /dgbpxg-74/obj8 2024-04-12T04:32:27.355 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:8sAx6YJQL3Dl1AP81EKlABl6s/A= 2024-04-12T04:32:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:27 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8sAx6YJQL3Dl1AP81EKlABl6s/A=', 'Content-Length': '0'} 2024-04-12T04:32:27.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:30:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-75:b7d2416c-9d6f-4780-9e14-1f66614caba8.10851.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068489a93cc17dc58-006618b95b-2d9e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:27.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:27.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:27.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:27.483 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:32:27.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:27.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:27.484 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:27.530 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:27.530 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:27.608 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:32:27.609 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2024-04-12T04:32:27.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.611 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896347.458070_2268.1", 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:27.458070Z" 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.715 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:32:27.716 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896347.458070_2268.1'} 2024-04-12T04:32:27.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:32:27.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:27.717 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:27.827+0000 7f2f7e40a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:27.828+0000 7f2f7e40a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:32:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:23.159827Z", 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:32:27.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:27.849 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:27Z" 2024-04-12T04:32:27.849 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:27.850 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896223.159827_2264.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:23.159827Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:32:27Z" } 2024-04-12T04:32:27.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896223.159827_2264.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:23.159827Z', 'realm_epoch': 2}}] 2024-04-12T04:32:27.850 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896347.458070_2268.1'} 2024-04-12T04:32:27.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:32:27.850 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896347.458070_2268.1 target=1_1712896223.159827_2264.1 2024-04-12T04:32:32.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:32.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:32.851 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:32.909 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:32.910 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:32.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:32.958+0000 7f1720547780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:32.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:32.959+0000 7f1720547780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:32:32.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:32.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:32.975 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896223.159827_2264.1", 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:32.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:23.159827Z", 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:32Z" 2024-04-12T04:32:32.977 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:32.978 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896223.159827_2264.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:23.159827Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:32:32Z" } 2024-04-12T04:32:32.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896223.159827_2264.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:23.159827Z', 'realm_epoch': 2}}] 2024-04-12T04:32:32.979 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896347.458070_2268.1'} 2024-04-12T04:32:32.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896223.159827_2264.1'} 2024-04-12T04:32:32.979 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896347.458070_2268.1 target=1_1712896223.159827_2264.1 2024-04-12T04:32:37.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:37.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:37.980 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:38.093+0000 7f9586238780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:38.094+0000 7f9586238780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:38.113 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:38.113 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.114 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:38.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:32:38.117 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896347.458070_2268.1", 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.118 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:27.458070Z", 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:38Z" 2024-04-12T04:32:38.119 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:38.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896347.458070_2268.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:27.458070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:32:38Z" } 2024-04-12T04:32:38.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896347.458070_2268.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:27.458070Z', 'realm_epoch': 2}}] 2024-04-12T04:32:38.121 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896347.458070_2268.1'} 2024-04-12T04:32:38.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896347.458070_2268.1'} 2024-04-12T04:32:38.121 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:32:38.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.122 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.240 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2024-04-12T04:32:38.240 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-76 2024-04-12T04:32:38.240 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:32:38.240 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:32:38.241 DEBUG:boto:Method: PUT 2024-04-12T04:32:38.241 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:32:38.241 DEBUG:boto:Data: 2024-04-12T04:32:38.241 DEBUG:boto:Headers: {} 2024-04-12T04:32:38.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:38.241 DEBUG:boto:Port: 8000 2024-04-12T04:32:38.241 DEBUG:boto:Params: {} 2024-04-12T04:32:38.241 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:32:38.242 DEBUG:boto:Token: None 2024-04-12T04:32:38.242 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:32:38 GMT /dgbpxg-76/ 2024-04-12T04:32:38.242 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:0f63RIFDD/d7MmdZm5+FPIDipAk= 2024-04-12T04:32:38.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0f63RIFDD/d7MmdZm5+FPIDipAk=', 'Content-Length': '0'} 2024-04-12T04:32:38.280 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005550def90b7b3d15-006618b966-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:38.280 INFO:rgw_multi.tests:create bucket zone=a1 name=dgbpxg-77 2024-04-12T04:32:38.280 DEBUG:boto:path=/dgbpxg-77/ 2024-04-12T04:32:38.280 DEBUG:boto:auth_path=/dgbpxg-77/ 2024-04-12T04:32:38.280 DEBUG:boto:Method: PUT 2024-04-12T04:32:38.280 DEBUG:boto:Path: /dgbpxg-77/ 2024-04-12T04:32:38.280 DEBUG:boto:Data: 2024-04-12T04:32:38.281 DEBUG:boto:Headers: {} 2024-04-12T04:32:38.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:38.281 DEBUG:boto:Port: 8000 2024-04-12T04:32:38.281 DEBUG:boto:Params: {} 2024-04-12T04:32:38.281 DEBUG:boto:Token: None 2024-04-12T04:32:38.281 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:32:38 GMT /dgbpxg-77/ 2024-04-12T04:32:38.281 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:zeN4pBJpGuc8xXRwtLl42OgXSes= 2024-04-12T04:32:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zeN4pBJpGuc8xXRwtLl42OgXSes=', 'Content-Length': '0'} 2024-04-12T04:32:38.308 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075d1f137fd212348-006618b966-2d94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:38.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:38.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:38.308 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:38.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.356 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.397 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:32:38.422 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:32:38.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:38.422 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:32:38.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:32:38.423 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:38.307643Z" 2024-04-12T04:32:38.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.424 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:32:38.425 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:38.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:32:38.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:38.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:38.425 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:38.528+0000 7f6fc3f05780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:38.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:38.529+0000 7f6fc3f05780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:38.548 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:32:38.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:32:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:32:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:38Z" 2024-04-12T04:32:38.554 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:38.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:32:38Z" } 2024-04-12T04:32:38.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:32:38.555 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:38.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:38.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:32:38.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.556 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:38.604 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.604 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.664 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:32:38.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.665 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:38.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.778 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:32:38.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.779 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dgbpxg-76 --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-76 --dest-bucket=dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:38.905 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:32:38.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dgbpxg-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:38.906 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dgbpxg-77 --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-76 --dest-bucket=dgbpxg-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:38.960 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:38.960 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:39.018 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:32:39.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dgbpxg-76', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.019 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dgbpxg-76 --dest-bucket=dgbpxg-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:39.072 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:39.072 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:39.153 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:32:39.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.153 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:39.201 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:39.202 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:39.269 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:32:39.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.270 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:39.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:39.317 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:39.386 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22, 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:32:39.387 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:32:39.388 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:32:39.389 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.390 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.393 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-76", 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:32:39.394 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:32:39.395 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-77", 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-77", 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:32:39.397 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:32:39.398 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.399 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-77", 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.400 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.402 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:32:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:32:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:32:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:32:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:32:39.409 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:32:39.411 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:32:39.412 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:32:39.413 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:32:39.416 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:32:39.417 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:32:39.418 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:32:39.419 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:39.420 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:39.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:32:39.422 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:32:39.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:39.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:39.527 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:32:39.527 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:32:39.527 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:32:39.528 DEBUG:boto:Method: HEAD 2024-04-12T04:32:39.528 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:32:39.528 DEBUG:boto:Data: 2024-04-12T04:32:39.528 DEBUG:boto:Headers: {} 2024-04-12T04:32:39.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:39.528 DEBUG:boto:Port: 8000 2024-04-12T04:32:39.528 DEBUG:boto:Params: {} 2024-04-12T04:32:39.529 DEBUG:boto:Token: None 2024-04-12T04:32:39.529 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:39 GMT /dgbpxg-76/ 2024-04-12T04:32:39.529 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XPDnblE1SXNaRx2vV9e2inAUf1s= 2024-04-12T04:32:39.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XPDnblE1SXNaRx2vV9e2inAUf1s=', 'Content-Length': '0'} 2024-04-12T04:32:39.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b5c5a680af7b48d-006618b967-2f4d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:39.790 DEBUG:boto:path=/dgbpxg-76/obj1 2024-04-12T04:32:39.790 DEBUG:boto:auth_path=/dgbpxg-76/obj1 2024-04-12T04:32:39.790 DEBUG:boto:Method: PUT 2024-04-12T04:32:39.790 DEBUG:boto:Path: /dgbpxg-76/obj1 2024-04-12T04:32:39.790 DEBUG:boto:Data: 2024-04-12T04:32:39.790 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:32:39.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:39.790 DEBUG:boto:Port: 8000 2024-04-12T04:32:39.790 DEBUG:boto:Params: {} 2024-04-12T04:32:39.791 DEBUG:boto:Token: None 2024-04-12T04:32:39.791 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:32:39 GMT /dgbpxg-76/obj1 2024-04-12T04:32:39.791 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:ZPH3hE4VV2BPxau6SeJC22wH/bc= 2024-04-12T04:32:39.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:32:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZPH3hE4VV2BPxau6SeJC22wH/bc='} 2024-04-12T04:32:39.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d4145ee3ba7500a-006618b967-2f4d-a1'), ('Date', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:39.829 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:32:39.829 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:32:39.829 DEBUG:boto:Method: HEAD 2024-04-12T04:32:39.829 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:32:39.829 DEBUG:boto:Data: 2024-04-12T04:32:39.829 DEBUG:boto:Headers: {} 2024-04-12T04:32:39.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:39.829 DEBUG:boto:Port: 8000 2024-04-12T04:32:39.829 DEBUG:boto:Params: {} 2024-04-12T04:32:39.829 DEBUG:boto:Token: None 2024-04-12T04:32:39.830 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:32:39 GMT /dgbpxg-76/ 2024-04-12T04:32:39.830 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XPDnblE1SXNaRx2vV9e2inAUf1s= 2024-04-12T04:32:39.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:32:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XPDnblE1SXNaRx2vV9e2inAUf1s=', 'Content-Length': '0'} 2024-04-12T04:32:39.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a86c760a2feaf16-006618b967-2f4d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:39.834 DEBUG:boto:path=/dgbpxg-76/obj2 2024-04-12T04:32:39.834 DEBUG:boto:auth_path=/dgbpxg-76/obj2 2024-04-12T04:32:39.834 DEBUG:boto:Method: PUT 2024-04-12T04:32:39.834 DEBUG:boto:Path: /dgbpxg-76/obj2 2024-04-12T04:32:39.834 DEBUG:boto:Data: 2024-04-12T04:32:39.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:32:39.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:32:39.835 DEBUG:boto:Port: 8000 2024-04-12T04:32:39.835 DEBUG:boto:Params: {} 2024-04-12T04:32:39.835 DEBUG:boto:Token: None 2024-04-12T04:32:39.835 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:32:39 GMT /dgbpxg-76/obj2 2024-04-12T04:32:39.835 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:p5K7D0C8zZomM4NpCpJvOPXdmJQ= 2024-04-12T04:32:39.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:32:39 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:p5K7D0C8zZomM4NpCpJvOPXdmJQ='} 2024-04-12T04:32:39.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000001d1b2a6d5099e7-006618b967-2f4d-a1'), ('Date', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:32:39.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:39.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:39.879 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:39.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:39.952 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:40.066 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:32:40.135 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:32:40.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:32:40.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.137 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:32:40.137 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:38.307643Z" 2024-04-12T04:32:40.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.137 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:32:40.138 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:40.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:32:40.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.138 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:40.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:40.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:40.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:40.303+0000 7f361670d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:40.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:32:40.305+0000 7f361670d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.357 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:40.358 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:32:40.359 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:32:40.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:40Z" 2024-04-12T04:32:40.362 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:40.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:32:40Z" } 2024-04-12T04:32:40.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:32:40.364 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:40.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:32:40.364 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:32:40.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.364 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:40.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:40.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:40.568 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:32:40.569 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.570 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:32:40.571 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:40.571 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:32:40.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:40.572 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:40.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:40.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:40.744 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:40Z" 2024-04-12T04:32:40.750 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:40.751 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:40Z" } 2024-04-12T04:32:40.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:40.751 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:40.751 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:40.751 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:32:45.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:45.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:45.753 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:45.812 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:45.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:45.889 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:45.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:45.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:45.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:45.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:45Z" 2024-04-12T04:32:45.895 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:45.896 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:45Z" } 2024-04-12T04:32:45.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:45.896 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:45.897 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:45.897 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:32:50.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:50.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:50.898 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:32:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:50Z" 2024-04-12T04:32:51.013 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:51.014 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:50Z" } 2024-04-12T04:32:51.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:51.015 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:51.015 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:51.015 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:32:56.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:56.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:32:56.016 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:32:56.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:32:56.076 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:56.141 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.145 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:32:56.146 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:32:56.146 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:32:56.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:32:56.146 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:32:56Z" 2024-04-12T04:32:56.146 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:32:56.147 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:32:56Z" } 2024-04-12T04:32:56.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:32:56.147 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:56.147 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:32:56.147 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:01.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:01.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:01.149 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:01.268 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:01.269 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:01.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:01.272 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.273 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:01.274 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:01.274 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:01.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:01.274 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:01Z" 2024-04-12T04:33:01.274 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:01.275 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:01Z" } 2024-04-12T04:33:01.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:01.275 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:01.275 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:01.275 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:06.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:06.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:06.277 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:06.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:06.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:06.394 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:06.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:06.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:06.397 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:06.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:06.397 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:06.398 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:06.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:06.400 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:06Z" 2024-04-12T04:33:06.401 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:06.401 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:06Z" } 2024-04-12T04:33:06.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:06.402 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:06.402 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:06.402 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:11.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:11.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:11.404 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:11.460 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:11.461 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:11.540 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:11.541 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.543 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:11.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:11Z" 2024-04-12T04:33:11.546 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:11.547 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:11Z" } 2024-04-12T04:33:11.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:11.547 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:11.548 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:11.548 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:16.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:16.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:16.549 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:16.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:16.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:16.685 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:16.685 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:16.686 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:16.687 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:16.688 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:16.689 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:16.691 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:16.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:16.691 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:16Z" 2024-04-12T04:33:16.691 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:16.692 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:16Z" } 2024-04-12T04:33:16.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:16.692 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:16.692 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:16.692 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:21.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:21.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:21.694 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:21.816 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:21.816 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:21.816 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:21.816 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:21.816 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:21.817 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:21.818 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:21.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:21.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:21.818 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:21.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:21.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:21.821 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:21.822 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:21Z" 2024-04-12T04:33:21.823 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:21.824 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:21Z" } 2024-04-12T04:33:21.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:21.824 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:21.825 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:21.825 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:26.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:26.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:26.827 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.944 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:26.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:26.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:26.947 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:26.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:26Z" 2024-04-12T04:33:26.949 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:26.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:26Z" } 2024-04-12T04:33:26.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:26.951 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:26.951 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:26.951 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:31.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:31.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:31.952 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:32.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:32.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:32Z" 2024-04-12T04:33:32.084 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:32.085 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:32Z" } 2024-04-12T04:33:32.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:32.085 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:32.085 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:32.085 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:37.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:37.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:37.087 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:37.226 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:37.226 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:37.227 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:37.228 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:37.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:37.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:37.231 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:37.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:37.232 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:37Z" 2024-04-12T04:33:37.232 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:37.233 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:37Z" } 2024-04-12T04:33:37.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:37.233 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:37.233 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:37.233 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:42.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:42.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:42.235 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:42.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:42.348 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:42.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:42.350 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:42.351 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.352 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:42.353 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:42.353 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:42.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:42.353 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:42Z" 2024-04-12T04:33:42.353 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:42.354 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:42Z" } 2024-04-12T04:33:42.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:42.354 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:42.354 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:42.355 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:47.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:47.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:47.356 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:47.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:47.484 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:47.485 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:47Z" 2024-04-12T04:33:47.486 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:47.487 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:47Z" } 2024-04-12T04:33:47.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:47.488 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:47.488 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:47.488 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:52.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:52.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:52.489 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:52.601 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:52.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:52.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:52.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:52.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:52.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:52Z" 2024-04-12T04:33:52.607 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:52.608 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:52Z" } 2024-04-12T04:33:52.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:52.609 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:52.609 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:52.609 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:33:57.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:57.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:57.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:57.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:57.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:57.747 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:33:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:33:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:33:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:33:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:33:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:33:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:33:57Z" 2024-04-12T04:33:57.752 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:33:57.753 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:33:57Z" } 2024-04-12T04:33:57.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:33:57.754 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:57.754 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:33:57.754 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:02.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:02.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:02.755 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:02.814 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:02.814 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:02Z" 2024-04-12T04:34:02.887 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:02.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:02Z" } 2024-04-12T04:34:02.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:02.888 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:02.888 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:02.888 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:07.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:07.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:07.889 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:07.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:07.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:08.021 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:08Z" 2024-04-12T04:34:08.028 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:08.028 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:08Z" } 2024-04-12T04:34:08.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:08.029 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:08.029 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:08.029 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:13.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:13.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:13.030 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:13.089 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:13.089 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:13.161 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:13.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:13.161 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:13.164 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:13.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:13.166 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:13Z" 2024-04-12T04:34:13.167 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:13.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:13Z" } 2024-04-12T04:34:13.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:13.168 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:13.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:13.169 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:18.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:18.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:18.170 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:18.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:18.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:18.300 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:18.300 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:18.300 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:18.301 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:18.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:18.304 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:18.305 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:18Z" 2024-04-12T04:34:18.306 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:18.307 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:18Z" } 2024-04-12T04:34:18.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:18.307 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:18.307 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:18.308 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:23.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:23.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:23.309 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:23.437 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:23.437 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:23.437 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:23.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:23.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:23.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:23.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:23Z" 2024-04-12T04:34:23.442 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:23.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:23Z" } 2024-04-12T04:34:23.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:23.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:23.444 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:23.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:28.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:28.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:28.445 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:28.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:28.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:28.559 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:28.559 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:28.560 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:28.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:28.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:28Z" 2024-04-12T04:34:28.564 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:28.565 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:28Z" } 2024-04-12T04:34:28.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:28.566 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:28.566 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:28.566 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:33.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:33.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:33.567 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:33.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:33.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:33.700 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:33.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:33.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:33.703 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:33.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:33Z" 2024-04-12T04:34:33.705 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:33.706 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:33Z" } 2024-04-12T04:34:33.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:33.707 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:33.707 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:33.707 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:38.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:38.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:38.709 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:38.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:38.769 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:38.829 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:34:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.364792Z" 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:34:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.376462Z" 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:38.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:38.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:38.835 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:38Z" 2024-04-12T04:34:38.835 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:38.836 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.364792Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:30:54.376462Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:38Z" } 2024-04-12T04:34:38.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.364792Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.376462Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:38.836 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:38.836 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:38.836 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=2 master=00000000000000000000:00000000000000054631 target=00000000000000000000:00000000000000054490 2024-04-12T04:34:43.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:43.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:43.838 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:43.896 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:43.896 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:43.952 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:43.952 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:43.952 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:43.952 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:43.952 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:43.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:34:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:43.955 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:34:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:34:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:34:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:43Z" 2024-04-12T04:34:43.958 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:43.959 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:34:43Z" } 2024-04-12T04:34:43.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:34:43.959 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:43.959 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:34:43.959 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:34:43.959 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:34:43.960 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:34:43.960 DEBUG:boto:Method: HEAD 2024-04-12T04:34:43.960 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:34:43.960 DEBUG:boto:Data: 2024-04-12T04:34:43.960 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.960 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.960 DEBUG:boto:Params: {} 2024-04-12T04:34:43.961 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:34:43.961 DEBUG:boto:Token: None 2024-04-12T04:34:43.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-76/ 2024-04-12T04:34:43.962 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:tPyiLiSwljmTVZC3cS9A6u2c61w= 2024-04-12T04:34:43.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPyiLiSwljmTVZC3cS9A6u2c61w=', 'Content-Length': '0'} 2024-04-12T04:34:43.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f59d7f9c97e208b-006618b9e3-2f4b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.976 DEBUG:boto:path=/dgbpxg-76/obj1 2024-04-12T04:34:43.976 DEBUG:boto:auth_path=/dgbpxg-76/obj1 2024-04-12T04:34:43.976 DEBUG:boto:Method: HEAD 2024-04-12T04:34:43.976 DEBUG:boto:Path: /dgbpxg-76/obj1 2024-04-12T04:34:43.976 DEBUG:boto:Data: 2024-04-12T04:34:43.977 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.977 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.977 DEBUG:boto:Params: {} 2024-04-12T04:34:43.977 DEBUG:boto:Token: None 2024-04-12T04:34:43.977 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-76/obj1 2024-04-12T04:34:43.977 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:JOLrPA8xiLtKIQObZb/Jz/UZPEI= 2024-04-12T04:34:43.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JOLrPA8xiLtKIQObZb/Jz/UZPEI=', 'Content-Length': '0'} 2024-04-12T04:34:43.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d3c3a12c528939-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.979 DEBUG:boto:path=/dgbpxg-76/obj1 2024-04-12T04:34:43.979 DEBUG:boto:auth_path=/dgbpxg-76/obj1 2024-04-12T04:34:43.979 DEBUG:boto:Method: GET 2024-04-12T04:34:43.980 DEBUG:boto:Path: /dgbpxg-76/obj1 2024-04-12T04:34:43.980 DEBUG:boto:Data: 2024-04-12T04:34:43.980 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.980 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.980 DEBUG:boto:Params: {} 2024-04-12T04:34:43.980 DEBUG:boto:Token: None 2024-04-12T04:34:43.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-76/obj1 2024-04-12T04:34:43.980 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:1lkr4sxt9Bub8KrszSJQ2xeRCCo= 2024-04-12T04:34:43.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1lkr4sxt9Bub8KrszSJQ2xeRCCo=', 'Content-Length': '0'} 2024-04-12T04:34:43.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000970b6f180329cb69-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.982 DEBUG:boto:path=/dgbpxg-76/obj2 2024-04-12T04:34:43.982 DEBUG:boto:auth_path=/dgbpxg-76/obj2 2024-04-12T04:34:43.982 DEBUG:boto:Method: HEAD 2024-04-12T04:34:43.982 DEBUG:boto:Path: /dgbpxg-76/obj2 2024-04-12T04:34:43.983 DEBUG:boto:Data: 2024-04-12T04:34:43.983 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.983 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.983 DEBUG:boto:Params: {} 2024-04-12T04:34:43.983 DEBUG:boto:Token: None 2024-04-12T04:34:43.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-76/obj2 2024-04-12T04:34:43.983 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:XOk43KSYlQ/LujKgExvUb3FIzAI= 2024-04-12T04:34:43.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XOk43KSYlQ/LujKgExvUb3FIzAI=', 'Content-Length': '0'} 2024-04-12T04:34:43.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b60e618dfc19b8c-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.985 DEBUG:boto:path=/dgbpxg-76/obj2 2024-04-12T04:34:43.985 DEBUG:boto:auth_path=/dgbpxg-76/obj2 2024-04-12T04:34:43.985 DEBUG:boto:Method: GET 2024-04-12T04:34:43.985 DEBUG:boto:Path: /dgbpxg-76/obj2 2024-04-12T04:34:43.985 DEBUG:boto:Data: 2024-04-12T04:34:43.985 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.986 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.986 DEBUG:boto:Params: {} 2024-04-12T04:34:43.986 DEBUG:boto:Token: None 2024-04-12T04:34:43.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-76/obj2 2024-04-12T04:34:43.986 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Yw43HmNY7DUvEGS+x/0dUHSonzs= 2024-04-12T04:34:43.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Yw43HmNY7DUvEGS+x/0dUHSonzs=', 'Content-Length': '0'} 2024-04-12T04:34:43.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a37286eada38e29-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.988 DEBUG:boto:path=/dgbpxg-77/ 2024-04-12T04:34:43.989 DEBUG:boto:auth_path=/dgbpxg-77/ 2024-04-12T04:34:43.989 DEBUG:boto:Method: HEAD 2024-04-12T04:34:43.989 DEBUG:boto:Path: /dgbpxg-77/ 2024-04-12T04:34:43.989 DEBUG:boto:Data: 2024-04-12T04:34:43.989 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.989 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.989 DEBUG:boto:Params: {} 2024-04-12T04:34:43.989 DEBUG:boto:Token: None 2024-04-12T04:34:43.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-77/ 2024-04-12T04:34:43.990 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HolHItwzLCB8btYTahVdJeoaDaw= 2024-04-12T04:34:43.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HolHItwzLCB8btYTahVdJeoaDaw=', 'Content-Length': '0'} 2024-04-12T04:34:43.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000358cc35682a674ba-006618b9e3-2f4b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.992 DEBUG:boto:path=/dgbpxg-77/obj1 2024-04-12T04:34:43.992 DEBUG:boto:auth_path=/dgbpxg-77/obj1 2024-04-12T04:34:43.992 DEBUG:boto:Method: HEAD 2024-04-12T04:34:43.992 DEBUG:boto:Path: /dgbpxg-77/obj1 2024-04-12T04:34:43.992 DEBUG:boto:Data: 2024-04-12T04:34:43.992 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.993 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.993 DEBUG:boto:Params: {} 2024-04-12T04:34:43.993 DEBUG:boto:Token: None 2024-04-12T04:34:43.993 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-77/obj1 2024-04-12T04:34:43.993 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:Ee+XC3mzsDxkXR32fUTjZt639+A= 2024-04-12T04:34:43.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ee+XC3mzsDxkXR32fUTjZt639+A=', 'Content-Length': '0'} 2024-04-12T04:34:43.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066d61c98ca87ff4e-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:43.995 DEBUG:boto:path=/dgbpxg-77/obj1 2024-04-12T04:34:43.995 DEBUG:boto:auth_path=/dgbpxg-77/obj1 2024-04-12T04:34:43.995 DEBUG:boto:Method: GET 2024-04-12T04:34:43.995 DEBUG:boto:Path: /dgbpxg-77/obj1 2024-04-12T04:34:43.995 DEBUG:boto:Data: 2024-04-12T04:34:43.995 DEBUG:boto:Headers: {} 2024-04-12T04:34:43.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:43.995 DEBUG:boto:Port: 8001 2024-04-12T04:34:43.996 DEBUG:boto:Params: {} 2024-04-12T04:34:43.996 DEBUG:boto:Token: None 2024-04-12T04:34:43.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:34:43 GMT /dgbpxg-77/obj1 2024-04-12T04:34:43.996 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:UUE92sNsVVd4KO/QMposLHwTbIY= 2024-04-12T04:34:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UUE92sNsVVd4KO/QMposLHwTbIY=', 'Content-Length': '0'} 2024-04-12T04:34:43.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1736bcebb8b8e94-006618b9e3-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:44.038 DEBUG:boto:path=/dgbpxg-77/obj2 2024-04-12T04:34:44.038 DEBUG:boto:auth_path=/dgbpxg-77/obj2 2024-04-12T04:34:44.039 DEBUG:boto:Method: HEAD 2024-04-12T04:34:44.039 DEBUG:boto:Path: /dgbpxg-77/obj2 2024-04-12T04:34:44.039 DEBUG:boto:Data: 2024-04-12T04:34:44.039 DEBUG:boto:Headers: {} 2024-04-12T04:34:44.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:44.039 DEBUG:boto:Port: 8001 2024-04-12T04:34:44.039 DEBUG:boto:Params: {} 2024-04-12T04:34:44.039 DEBUG:boto:Token: None 2024-04-12T04:34:44.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:34:44 GMT /dgbpxg-77/obj2 2024-04-12T04:34:44.039 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:xFUDpuL1nUsSwGmR+1jd6nVAIsA= 2024-04-12T04:34:44.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xFUDpuL1nUsSwGmR+1jd6nVAIsA=', 'Content-Length': '0'} 2024-04-12T04:34:44.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba06cc2d407519a8-006618b9e4-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:44.041 DEBUG:boto:path=/dgbpxg-77/obj2 2024-04-12T04:34:44.041 DEBUG:boto:auth_path=/dgbpxg-77/obj2 2024-04-12T04:34:44.041 DEBUG:boto:Method: GET 2024-04-12T04:34:44.041 DEBUG:boto:Path: /dgbpxg-77/obj2 2024-04-12T04:34:44.042 DEBUG:boto:Data: 2024-04-12T04:34:44.042 DEBUG:boto:Headers: {} 2024-04-12T04:34:44.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:34:44.042 DEBUG:boto:Port: 8001 2024-04-12T04:34:44.042 DEBUG:boto:Params: {} 2024-04-12T04:34:44.042 DEBUG:boto:Token: None 2024-04-12T04:34:44.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:34:44 GMT /dgbpxg-77/obj2 2024-04-12T04:34:44.042 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:/Y0l6mrkp6+XajIGQJ3mcRWGYDo= 2024-04-12T04:34:44.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:34:44 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Y0l6mrkp6+XajIGQJ3mcRWGYDo=', 'Content-Length': '0'} 2024-04-12T04:34:44.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:32:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f8d93a8f17d820c-006618b9e4-2f4b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:34:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:34:44.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.045 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.095 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.095 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.170 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:34:44.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.265 INFO:teuthology.orchestra.run.smithi099.stderr:WARNING: cannot find source zone id for name=* 2024-04-12T04:34:44.265 INFO:teuthology.orchestra.run.smithi099.stderr:WARNING: cannot find source zone id for name=* 2024-04-12T04:34:44.282 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:34:44.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.282 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.330 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.330 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.390 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:34:44.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.390 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.505 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 23, 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2024-04-12T04:34:44.506 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:34:44.507 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:34:44.508 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:34:44.509 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.510 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:34:44.512 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-76", 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:34:44.513 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:34:44.514 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.515 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-dgbpxg-77", 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-76", 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "b7d2416c-9d6f-4780-9e14-1f66614caba8" 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "dgbpxg-77", 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "*" 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2024-04-12T04:34:44.518 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "*" 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2024-04-12T04:34:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:34:44.521 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2024-04-12T04:34:44.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.522 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:34:44.523 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:34:44.524 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2024-04-12T04:34:44.525 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "compress-encrypted", 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2024-04-12T04:34:44.526 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.527 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2532217463 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2914034528 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2928306118 2024-04-12T04:34:44.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2226213169 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.530 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:34:44.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:34:44.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2024-04-12T04:34:44.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:44.533 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:44.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.534 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.686 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:34:44.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.686 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.752 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.752 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.891 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:34:44.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:44.892 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:44.941 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:34:44.942+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:34:44.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.186 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:34:45.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-76', '--source-bucket=*', '--dest-bucket=dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-76', '--source-bucket=*', '--dest-bucket=dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.187 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dgbpxg-76 --source-zones=a1 --dest-zones=a2 --bucket=dgbpxg-76 '--source-bucket=*' --dest-bucket=dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:45.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:45.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.398 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:34:45.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-76', '--source-bucket=*', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dgbpxg-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=dgbpxg-76', '--source-bucket=*', '--dest-bucket=dgbpxg-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.398 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dgbpxg-77 --source-zones=a1 --dest-zones=a2 --bucket=dgbpxg-76 '--source-bucket=*' --dest-bucket=dgbpxg-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:45.459 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:45.459 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.603 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:34:45.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.604 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:45.660 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:45.660 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.762 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:34:45.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:34:45.763 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:34:45.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:45.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.888 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-dgbpxg-76","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dgbpxg-77","source":{"bucket":"dgbpxg-76","zones":["a1"]},"dest":{"bucket":"dgbpxg-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:34:45.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:45.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:45.889 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:45.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:45.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:45.996 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:34:46.017 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:34:46.018 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896485.741542_2364.1", 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:34:45.741542Z" 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.019 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:34:46.020 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:34:46.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:34:46.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:46.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:46.020 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:46.108+0000 7f3985084780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:46.109+0000 7f3985084780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:46.120 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:46.120 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:46.120 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:46.120 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:34:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:34:46.122 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:46.123 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:34:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:34:46.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:46.126 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:46Z" 2024-04-12T04:34:46.126 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:46.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:46Z" } 2024-04-12T04:34:46.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:34:46.127 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:34:46.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:34:46.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:34:51.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:51.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:51.129 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:51.237+0000 7f32dc841780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:51.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:51.238+0000 7f32dc841780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:34:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:51Z" 2024-04-12T04:34:51.264 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:51.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:51Z" } 2024-04-12T04:34:51.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:34:51.266 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:34:51.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:34:51.266 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:34:56.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:56.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:56.268 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:56.373+0000 7f19ad6c9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:34:56.374+0000 7f19ad6c9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:34:56.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:34:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:34:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:34:56Z" 2024-04-12T04:34:56.391 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:34:56.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:56Z" } 2024-04-12T04:34:56.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:34:56.392 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:34:56.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:34:56.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:01.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:01.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:01.394 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:01.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:01.499+0000 7f74f8222780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:01.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:01.500+0000 7f74f8222780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:01.519 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:01.519 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:01.519 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:01.520 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:01.521 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:01.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:01.523 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.524 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:01.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:01.525 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:01.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:01.525 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:01Z" 2024-04-12T04:35:01.525 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:01.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:01Z" } 2024-04-12T04:35:01.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:01.526 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:01.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:01.527 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:06.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:06.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:06.528 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:06.629+0000 7f8c8dc5f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:06.630+0000 7f8c8dc5f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:06.645 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:06.645 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:06Z" 2024-04-12T04:35:06.651 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:06.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:06Z" } 2024-04-12T04:35:06.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:06.653 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:06.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:06.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:11.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:11.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:11.654 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:11.710 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:11.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:11.753+0000 7fb449720780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:11.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:11.754+0000 7fb449720780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:11.774 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:11.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:11.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:11.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:11Z" 2024-04-12T04:35:11.779 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:11.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:11Z" } 2024-04-12T04:35:11.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:11.780 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:11.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:11.781 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:16.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:16.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:16.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:16.890+0000 7ff47cfbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:16.891+0000 7ff47cfbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:16.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:16.905 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:16.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:16.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:16.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:16Z" 2024-04-12T04:35:16.909 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:16.910 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:16Z" } 2024-04-12T04:35:16.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:16.911 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:16.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:16.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:21.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:21.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:21.913 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:21.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:22.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:22.014+0000 7fd55bd88780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:22.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:22.015+0000 7fd55bd88780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:22.029 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:22.029 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:22.029 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:22.029 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:22.030 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:22.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:22.032 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:22.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.034 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:22.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:22.035 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:22.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:22.035 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:22Z" 2024-04-12T04:35:22.035 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:22.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:22Z" } 2024-04-12T04:35:22.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:22.036 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:22.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:22.036 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:27.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:27.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:27.038 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:27.149+0000 7fa4135e0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:27.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:27.150+0000 7fa4135e0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:27.171 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:27Z" 2024-04-12T04:35:27.177 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:27.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:27Z" } 2024-04-12T04:35:27.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:27.178 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:27.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:27.178 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:32.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:32.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:32.179 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:32.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:32.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:32.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:32.274+0000 7fb0f08d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:32.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:32.275+0000 7fb0f08d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:32.287 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:32Z" 2024-04-12T04:35:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:32.294 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:32Z" } 2024-04-12T04:35:32.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:32.295 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:32.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:32.295 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:37.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:37.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:37.296 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:37.352 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:37.352 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:37.402+0000 7f880c335780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:37.403+0000 7f880c335780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:37.428 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:37.428 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:37.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:37.428 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:37Z" 2024-04-12T04:35:37.428 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:37.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:37Z" } 2024-04-12T04:35:37.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:37.429 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:37.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:37.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:42.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:42.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:42.431 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:42.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:42.532+0000 7f3fe15d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:42.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:42.533+0000 7f3fe15d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.548 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:42.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:42.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:42.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:42.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:42.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:42.554 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:42.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:42.554 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:42Z" 2024-04-12T04:35:42.554 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:42.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:42Z" } 2024-04-12T04:35:42.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:42.555 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:42.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:42.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:47.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:47.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:47.557 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:47.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:47.657+0000 7fa42f62a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:47.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:47.658+0000 7fa42f62a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:47.679 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:47.679 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:47.680 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:47.681 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:47.682 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:47.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:47Z" 2024-04-12T04:35:47.684 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:47.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:47Z" } 2024-04-12T04:35:47.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:47.686 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:47.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:47.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:52.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:52.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:52.688 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:52.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:52.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:52.797+0000 7f81b2f13780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:52.798+0000 7f81b2f13780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:52.814 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:52.814 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:52.814 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:52.814 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:52.814 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:52.815 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:52.816 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:52.817 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:52.818 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:52Z" 2024-04-12T04:35:52.820 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:52.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:52Z" } 2024-04-12T04:35:52.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:52.821 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:52.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:52.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:35:57.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:57.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:57.823 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:57.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:57.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:57.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:57.932+0000 7ff83286a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:57.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:35:57.933+0000 7ff83286a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:57.952 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:35:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:35:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:35:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:35:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:35:57Z" 2024-04-12T04:35:57.958 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:35:57.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:57Z" } 2024-04-12T04:35:57.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:35:57.959 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:35:57.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:35:57.960 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:36:02.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.961 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:03.018 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.019 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:03.054+0000 7f343da69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:03.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:03.055+0000 7f343da69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.071 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:03.072 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:03.075 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:03.076 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:03.077 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:36:03.078 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:36:03.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:03.078 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:03Z" 2024-04-12T04:36:03.078 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:03.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:03Z" } 2024-04-12T04:36:03.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:36:03.080 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:03.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:36:03.080 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:36:08.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:08.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:08.082 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:08.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:08.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:08.182+0000 7f08910b5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:08.183+0000 7f08910b5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:08.204 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:08.205 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:08.206 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:08.207 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:08.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:36:08.209 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:36:08.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:08.210 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:08Z" 2024-04-12T04:36:08.210 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:08.211 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:08Z" } 2024-04-12T04:36:08.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:36:08.211 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:08.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:36:08.211 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:36:13.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:13.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:13.213 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:13.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:13.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:13.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:13.323+0000 7efc5f654780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:13.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:13.323+0000 7efc5f654780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:13.339 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:13.339 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:13.339 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:13.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:13.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:13.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896358.307643_2276.1", 2024-04-12T04:36:13.343 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:38.307643Z", 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:36:13.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:13.345 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:13Z" 2024-04-12T04:36:13.345 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:13.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896358.307643_2276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:38.307643Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:13Z" } 2024-04-12T04:36:13.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896358.307643_2276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:38.307643Z', 'realm_epoch': 2}}] 2024-04-12T04:36:13.346 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:13.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896358.307643_2276.1'} 2024-04-12T04:36:13.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896485.741542_2364.1 target=1_1712896358.307643_2276.1 2024-04-12T04:36:18.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.348 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:18.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:18.404 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:18.444+0000 7fcfc38ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:18.445+0000 7fcfc38ab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.459 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896485.741542_2364.1", 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:34:45.741542Z", 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:18Z" 2024-04-12T04:36:18.464 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:18.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896485.741542_2364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:34:45.741542Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:18Z" } 2024-04-12T04:36:18.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896485.741542_2364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:34:45.741542Z', 'realm_epoch': 2}}] 2024-04-12T04:36:18.465 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:18.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:18.466 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:36:18.466 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:36:18.466 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:36:18.466 DEBUG:boto:Method: HEAD 2024-04-12T04:36:18.467 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:36:18.467 DEBUG:boto:Data: 2024-04-12T04:36:18.467 DEBUG:boto:Headers: {} 2024-04-12T04:36:18.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:18.467 DEBUG:boto:Port: 8000 2024-04-12T04:36:18.467 DEBUG:boto:Params: {} 2024-04-12T04:36:18.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:18.468 DEBUG:boto:Token: None 2024-04-12T04:36:18.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:18 GMT /dgbpxg-76/ 2024-04-12T04:36:18.468 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bp5Fr4Dfs+m+UJFODklsIwuBVJk= 2024-04-12T04:36:18.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bp5Fr4Dfs+m+UJFODklsIwuBVJk=', 'Content-Length': '0'} 2024-04-12T04:36:18.502 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f24501ab57d6acf-006618ba42-305d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:18.503 DEBUG:boto:path=/dgbpxg-76/obj3 2024-04-12T04:36:18.503 DEBUG:boto:auth_path=/dgbpxg-76/obj3 2024-04-12T04:36:18.503 DEBUG:boto:Method: PUT 2024-04-12T04:36:18.503 DEBUG:boto:Path: /dgbpxg-76/obj3 2024-04-12T04:36:18.503 DEBUG:boto:Data: 2024-04-12T04:36:18.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:18.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:18.504 DEBUG:boto:Port: 8000 2024-04-12T04:36:18.504 DEBUG:boto:Params: {} 2024-04-12T04:36:18.504 DEBUG:boto:Token: None 2024-04-12T04:36:18.504 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:36:18 GMT /dgbpxg-76/obj3 2024-04-12T04:36:18.504 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:B2wVPeBXx/2h8BPIogEZ0fRAPOM= 2024-04-12T04:36:18.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B2wVPeBXx/2h8BPIogEZ0fRAPOM='} 2024-04-12T04:36:18.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059feb4dc5a2e8025-006618ba42-305d-a1'), ('Date', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:18.515 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:36:18.515 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:36:18.515 DEBUG:boto:Method: HEAD 2024-04-12T04:36:18.515 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:36:18.515 DEBUG:boto:Data: 2024-04-12T04:36:18.515 DEBUG:boto:Headers: {} 2024-04-12T04:36:18.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:18.515 DEBUG:boto:Port: 8000 2024-04-12T04:36:18.516 DEBUG:boto:Params: {} 2024-04-12T04:36:18.516 DEBUG:boto:Token: None 2024-04-12T04:36:18.516 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:18 GMT /dgbpxg-76/ 2024-04-12T04:36:18.516 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:bp5Fr4Dfs+m+UJFODklsIwuBVJk= 2024-04-12T04:36:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bp5Fr4Dfs+m+UJFODklsIwuBVJk=', 'Content-Length': '0'} 2024-04-12T04:36:18.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae40f87a9df142d6-006618ba42-305d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:18.518 DEBUG:boto:path=/dgbpxg-76/obj4 2024-04-12T04:36:18.518 DEBUG:boto:auth_path=/dgbpxg-76/obj4 2024-04-12T04:36:18.519 DEBUG:boto:Method: PUT 2024-04-12T04:36:18.519 DEBUG:boto:Path: /dgbpxg-76/obj4 2024-04-12T04:36:18.519 DEBUG:boto:Data: 2024-04-12T04:36:18.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:18.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:18.519 DEBUG:boto:Port: 8000 2024-04-12T04:36:18.519 DEBUG:boto:Params: {} 2024-04-12T04:36:18.519 DEBUG:boto:Token: None 2024-04-12T04:36:18.519 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:36:18 GMT /dgbpxg-76/obj4 2024-04-12T04:36:18.519 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sPEB2CAA2yZY1qEIFRYGI07kHJM= 2024-04-12T04:36:18.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sPEB2CAA2yZY1qEIFRYGI07kHJM='} 2024-04-12T04:36:18.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084cb2c39da5e88df-006618ba42-305d-a1'), ('Date', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:18.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.533 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:18.581 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:18.581 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:18.624 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:18.641 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:36:18.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:18.641 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:18.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896485.741542_2364.1", 2024-04-12T04:36:18.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:34:45.741542Z" 2024-04-12T04:36:18.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.643 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:36:18.643 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:18.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:36:18.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.644 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:18.692 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:18.692 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:18.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:18.727+0000 7f28fc3ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:18.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-04-12T04:36:18.728+0000 7f28fc3ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:18.740 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:18.741 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1712896485.741542_2364.1", 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:34:45.741542Z", 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2024-04-12T04:36:18.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:18Z" 2024-04-12T04:36:18.746 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:18.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712890482.603296_102.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T02:54:42.603296Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712890659.711628_123.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T02:57:39.711628Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896485.741542_2364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:34:45.741542Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:18Z" } 2024-04-12T04:36:18.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896485.741542_2364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:34:45.741542Z', 'realm_epoch': 2}}] 2024-04-12T04:36:18.747 DEBUG:rgw_multi.tests:log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:18.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712896485.741542_2364.1'} 2024-04-12T04:36:18.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:36:18.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.748 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:18.795 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:18.795 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:18.873 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-04-12T04:36:18.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033328", 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:36:18.533168Z" 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054772", 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:36:18.515811Z" 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:18.875 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-04-12T04:36:18.876 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:18.876 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:36:18.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:18.877 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:18.926 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:18.926 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:19.001 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:19.002 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:19.003 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:19.004 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:19.005 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.006 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:19.007 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:19.007 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:19.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:19.007 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:18Z" 2024-04-12T04:36:19.007 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:19.008 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:18Z" } 2024-04-12T04:36:19.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:19.008 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:19.008 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:19.009 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:24.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:24.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:24.010 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:24.068 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:24.068 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:24Z" 2024-04-12T04:36:24.126 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:24.127 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:24Z" } 2024-04-12T04:36:24.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:24.127 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:24.127 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:24.127 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:29.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:29.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:29.129 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:29.188 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:29.189 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:29.267 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:29.267 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:29.267 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:29.267 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:29.267 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:29.268 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:29.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:29.270 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:29.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:29.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:29.273 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:29.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:29.273 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:29Z" 2024-04-12T04:36:29.273 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:29.274 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:29Z" } 2024-04-12T04:36:29.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:29.274 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:29.274 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:29.274 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:34.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:34.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:34.276 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:34.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:34.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:34.401 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:34.401 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:34.401 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:34.401 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:34.402 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:34.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:34.404 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:34.405 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:34.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:34.407 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:34Z" 2024-04-12T04:36:34.407 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:34.407 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:34Z" } 2024-04-12T04:36:34.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:34.408 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:34.408 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:34.408 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:39.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:39.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:39.410 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:39.467 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:39.467 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:39.537 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:39.537 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:39.537 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:39.538 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:39.539 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:39.540 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:39.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:39.542 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:39.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:39.543 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:39Z" 2024-04-12T04:36:39.543 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:39.544 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:39Z" } 2024-04-12T04:36:39.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:39.544 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:39.544 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:39.544 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:44.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:44.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:44.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:44.658 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:44.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:44.660 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:30:54.427543Z" 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:44.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054631", 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.827180Z" 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:44.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:44Z" 2024-04-12T04:36:44.664 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:44.665 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:30:54.427543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054631", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:32:39.827180Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:44Z" } 2024-04-12T04:36:44.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:30:54.427543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054631', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.827180Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:44.665 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:44.665 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000054631', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:44.666 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033328 target=00000000000000000000:00000000000000033187, shard=2 master=00000000000000000000:00000000000000054772 target=00000000000000000000:00000000000000054631 2024-04-12T04:36:49.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:49.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:49.667 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:49.724 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:49.724 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:49.798 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10653437930919767770 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:49.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:32:39.867953Z" 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2024-04-12T04:36:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000033328", 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:36:18.533168Z" 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:49.801 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000054772", 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 57, 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:36:18.515811Z" 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2024-04-12T04:36:49.802 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000046995", 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2024-04-12T04:28:08.091586Z" 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 66, 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 66 2024-04-12T04:36:49.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-04-12T04:36:49.804 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2024-04-12T04:36:49Z" 2024-04-12T04:36:49.804 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-04-12T04:36:49.804 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10653437930919767770 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:32:39.867953Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033328", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:36:18.533168Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054772", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:18.515811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046995", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:08.091586Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:36:49Z" } 2024-04-12T04:36:49.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:32:39.867953Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033328', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:36:18.533168Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054772', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:18.515811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046995', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:08.091586Z'}}] 2024-04-12T04:36:49.805 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:49.805 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033328', 2: '00000000000000000000:00000000000000054772', 3: '00000000000000000000:00000000000000046995'} 2024-04-12T04:36:49.805 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:36:49.805 DEBUG:boto:path=/dgbpxg-76/ 2024-04-12T04:36:49.805 DEBUG:boto:auth_path=/dgbpxg-76/ 2024-04-12T04:36:49.805 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.806 DEBUG:boto:Path: /dgbpxg-76/ 2024-04-12T04:36:49.806 DEBUG:boto:Data: 2024-04-12T04:36:49.806 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.806 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.806 DEBUG:boto:Params: {} 2024-04-12T04:36:49.806 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:49.807 DEBUG:boto:Token: None 2024-04-12T04:36:49.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-76/ 2024-04-12T04:36:49.807 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:rQwnOPXlWyPKsWv+8h3352WMfqA= 2024-04-12T04:36:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rQwnOPXlWyPKsWv+8h3352WMfqA=', 'Content-Length': '0'} 2024-04-12T04:36:49.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c3660dfbb55621-006618ba61-3051-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.839 DEBUG:boto:path=/dgbpxg-76/obj3 2024-04-12T04:36:49.839 DEBUG:boto:auth_path=/dgbpxg-76/obj3 2024-04-12T04:36:49.839 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.840 DEBUG:boto:Path: /dgbpxg-76/obj3 2024-04-12T04:36:49.840 DEBUG:boto:Data: 2024-04-12T04:36:49.840 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.840 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.840 DEBUG:boto:Params: {} 2024-04-12T04:36:49.841 DEBUG:boto:Token: None 2024-04-12T04:36:49.841 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-76/obj3 2024-04-12T04:36:49.841 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:fUrZbWy/NPi0n/GGy2KXmnGXU8o= 2024-04-12T04:36:49.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fUrZbWy/NPi0n/GGy2KXmnGXU8o=', 'Content-Length': '0'} 2024-04-12T04:36:49.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d837821ee7a4e83b-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.843 DEBUG:boto:path=/dgbpxg-76/obj3 2024-04-12T04:36:49.843 DEBUG:boto:auth_path=/dgbpxg-76/obj3 2024-04-12T04:36:49.843 DEBUG:boto:Method: GET 2024-04-12T04:36:49.843 DEBUG:boto:Path: /dgbpxg-76/obj3 2024-04-12T04:36:49.844 DEBUG:boto:Data: 2024-04-12T04:36:49.844 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.844 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.844 DEBUG:boto:Params: {} 2024-04-12T04:36:49.844 DEBUG:boto:Token: None 2024-04-12T04:36:49.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-76/obj3 2024-04-12T04:36:49.844 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yycU/5AAnKsA08cqmSkxkF6LBWE= 2024-04-12T04:36:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yycU/5AAnKsA08cqmSkxkF6LBWE=', 'Content-Length': '0'} 2024-04-12T04:36:49.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3db173dda3cdbb8-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.846 DEBUG:boto:path=/dgbpxg-76/obj4 2024-04-12T04:36:49.846 DEBUG:boto:auth_path=/dgbpxg-76/obj4 2024-04-12T04:36:49.847 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.847 DEBUG:boto:Path: /dgbpxg-76/obj4 2024-04-12T04:36:49.847 DEBUG:boto:Data: 2024-04-12T04:36:49.847 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.847 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.847 DEBUG:boto:Params: {} 2024-04-12T04:36:49.847 DEBUG:boto:Token: None 2024-04-12T04:36:49.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-76/obj4 2024-04-12T04:36:49.848 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:HPDOrRA2o7Y5N59+/CuhSclu3O4= 2024-04-12T04:36:49.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HPDOrRA2o7Y5N59+/CuhSclu3O4=', 'Content-Length': '0'} 2024-04-12T04:36:49.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007be1b6394f7fe06f-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.849 DEBUG:boto:path=/dgbpxg-76/obj4 2024-04-12T04:36:49.849 DEBUG:boto:auth_path=/dgbpxg-76/obj4 2024-04-12T04:36:49.849 DEBUG:boto:Method: GET 2024-04-12T04:36:49.849 DEBUG:boto:Path: /dgbpxg-76/obj4 2024-04-12T04:36:49.850 DEBUG:boto:Data: 2024-04-12T04:36:49.850 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.850 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.850 DEBUG:boto:Params: {} 2024-04-12T04:36:49.850 DEBUG:boto:Token: None 2024-04-12T04:36:49.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-76/obj4 2024-04-12T04:36:49.850 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:KGx7Ij6X+kTgFXsFxrzIxbsX03Y= 2024-04-12T04:36:49.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KGx7Ij6X+kTgFXsFxrzIxbsX03Y=', 'Content-Length': '0'} 2024-04-12T04:36:49.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000588a90137fc9018d-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.852 DEBUG:boto:path=/dgbpxg-77/ 2024-04-12T04:36:49.852 DEBUG:boto:auth_path=/dgbpxg-77/ 2024-04-12T04:36:49.852 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.852 DEBUG:boto:Path: /dgbpxg-77/ 2024-04-12T04:36:49.853 DEBUG:boto:Data: 2024-04-12T04:36:49.853 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.853 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.853 DEBUG:boto:Params: {} 2024-04-12T04:36:49.853 DEBUG:boto:Token: None 2024-04-12T04:36:49.853 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-77/ 2024-04-12T04:36:49.853 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:sIb4jtxzV77UcfLLgm8im1m1m8s= 2024-04-12T04:36:49.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sIb4jtxzV77UcfLLgm8im1m1m8s=', 'Content-Length': '0'} 2024-04-12T04:36:49.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054f763ef27a78dd5-006618ba61-3051-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.855 DEBUG:boto:path=/dgbpxg-77/obj3 2024-04-12T04:36:49.855 DEBUG:boto:auth_path=/dgbpxg-77/obj3 2024-04-12T04:36:49.855 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.856 DEBUG:boto:Path: /dgbpxg-77/obj3 2024-04-12T04:36:49.856 DEBUG:boto:Data: 2024-04-12T04:36:49.856 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.856 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.856 DEBUG:boto:Params: {} 2024-04-12T04:36:49.856 DEBUG:boto:Token: None 2024-04-12T04:36:49.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-77/obj3 2024-04-12T04:36:49.856 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:9QHlWXBYyLvSPx/rju0ZSYcDM68= 2024-04-12T04:36:49.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9QHlWXBYyLvSPx/rju0ZSYcDM68=', 'Content-Length': '0'} 2024-04-12T04:36:49.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf2ba00935844297-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.858 DEBUG:boto:path=/dgbpxg-77/obj3 2024-04-12T04:36:49.858 DEBUG:boto:auth_path=/dgbpxg-77/obj3 2024-04-12T04:36:49.858 DEBUG:boto:Method: GET 2024-04-12T04:36:49.858 DEBUG:boto:Path: /dgbpxg-77/obj3 2024-04-12T04:36:49.858 DEBUG:boto:Data: 2024-04-12T04:36:49.858 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.859 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.859 DEBUG:boto:Params: {} 2024-04-12T04:36:49.859 DEBUG:boto:Token: None 2024-04-12T04:36:49.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-77/obj3 2024-04-12T04:36:49.859 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:jJGUk3bxyoshJkW/N1+X4XmD17U= 2024-04-12T04:36:49.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jJGUk3bxyoshJkW/N1+X4XmD17U=', 'Content-Length': '0'} 2024-04-12T04:36:49.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033ed4a526fa06be8-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.902 DEBUG:boto:path=/dgbpxg-77/obj4 2024-04-12T04:36:49.903 DEBUG:boto:auth_path=/dgbpxg-77/obj4 2024-04-12T04:36:49.903 DEBUG:boto:Method: HEAD 2024-04-12T04:36:49.903 DEBUG:boto:Path: /dgbpxg-77/obj4 2024-04-12T04:36:49.903 DEBUG:boto:Data: 2024-04-12T04:36:49.903 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.903 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.904 DEBUG:boto:Params: {} 2024-04-12T04:36:49.904 DEBUG:boto:Token: None 2024-04-12T04:36:49.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-77/obj4 2024-04-12T04:36:49.904 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:InXEWvRbGMAwPcoCUrD1G5v4nPo= 2024-04-12T04:36:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:InXEWvRbGMAwPcoCUrD1G5v4nPo=', 'Content-Length': '0'} 2024-04-12T04:36:49.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec5f82a1b3782bc6-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.906 DEBUG:boto:path=/dgbpxg-77/obj4 2024-04-12T04:36:49.906 DEBUG:boto:auth_path=/dgbpxg-77/obj4 2024-04-12T04:36:49.906 DEBUG:boto:Method: GET 2024-04-12T04:36:49.906 DEBUG:boto:Path: /dgbpxg-77/obj4 2024-04-12T04:36:49.906 DEBUG:boto:Data: 2024-04-12T04:36:49.906 DEBUG:boto:Headers: {} 2024-04-12T04:36:49.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:49.906 DEBUG:boto:Port: 8001 2024-04-12T04:36:49.907 DEBUG:boto:Params: {} 2024-04-12T04:36:49.907 DEBUG:boto:Token: None 2024-04-12T04:36:49.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:36:49 GMT /dgbpxg-77/obj4 2024-04-12T04:36:49.907 DEBUG:boto:Signature: AWS JB2ZC967UHS0RDZ3J3O2:yb/2AAAGObLw7ZqL52sm3Qpjhq0= 2024-04-12T04:36:49.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:49 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yb/2AAAGObLw7ZqL52sm3Qpjhq0=', 'Content-Length': '0'} 2024-04-12T04:36:49.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-76:b7d2416c-9d6f-4780-9e14-1f66614caba8.11670.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000866536dca45706a2-006618ba61-3051-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:36:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:49.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:49.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dgbpxg-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:49.909 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dgbpxg-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:49.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:49.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.017 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:36:50.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:50.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:50.018 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:50.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.064 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.147 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:36:50.147 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2024-04-12T04:36:50.148 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.148 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:50.148 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-12T04:36:50.148 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:50.149 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:50.149 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:50.149 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dgbpxg-11 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:40 GMT 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1/9SPqIvpsf+z4unCX8zfdbOYBY= 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1/9SPqIvpsf+z4unCX8zfdbOYBY=', 'Content-Length': '0'} 2024-04-12T04:36:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009da5fb3101871c69-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dgbpxg-12 2024-04-12T04:36:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:40 GMT 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0SYL+G8xwa5F3Su8ue2zfoh7BVg= 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0SYL+G8xwa5F3Su8ue2zfoh7BVg=', 'Content-Length': '0'} 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fcc50dc16703d3d8-006618a3d8-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:40 GMT 2024-04-12T04:36:50.156 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI= 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI=', 'Content-Length': '0'} 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd32b42ce2de8f63-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj1 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj1 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj1 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:40 GMT 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj1 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:NYdVQn09C1oguohq4tigtpQhd+s= 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NYdVQn09C1oguohq4tigtpQhd+s='} 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dea6e3bbc2d0217-006618a3d8-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:40 GMT 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI= 2024-04-12T04:36:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:40 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/Gywkt83mtPGrie3leJY3Cwr3NI=', 'Content-Length': '0'} 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004b257621e44aecb-006618a3d8-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj2 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj2 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj2 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.160 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj2 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JjE/TLJYFzDPH1JZnZCu8jULgo4= 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JjE/TLJYFzDPH1JZnZCu8jULgo4='} 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a69c5cc22dd95a-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036c239f237cc9310-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj3 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj3 2024-04-12T04:36:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj3 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.163 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj3 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tOZmRCYMfdj/xFj82+2lUIcFQEU= 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tOZmRCYMfdj/xFj82+2lUIcFQEU='} 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001570df5ef4eec7c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000def3974718d891e-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj4 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj4 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj4 2024-04-12T04:36:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj4 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:XqtAil5lS2ODXPoXYjtbhYpNr6A= 2024-04-12T04:36:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XqtAil5lS2ODXPoXYjtbhYpNr6A='} 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000938d5a1a7bf9fe0d-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.167 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ead05634fc59cd50-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj5 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj5 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj5 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj5 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:nGUEzqaO7Y6OAhGKD9Nh5bnAJtE= 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nGUEzqaO7Y6OAhGKD9Nh5bnAJtE='} 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f35e53c31e24cb22-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.170 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000913e1b2143d85cb5-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj6 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj6 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj6 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj6 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/xGlZg+UFoSQT366Kb+wPCvDbc0= 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/xGlZg+UFoSQT366Kb+wPCvDbc0='} 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d66e0e80cd3fa0ba-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.173 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d657b000d46ccfa3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj7 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj7 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj7 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj7 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:nTnpylJgB0c1y43FVbxx7ByfNOk= 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nTnpylJgB0c1y43FVbxx7ByfNOk='} 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055a72bca4dbca67d-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.176 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef5f0eb856067280-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj8 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj8 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj8 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj8 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:usS/k8mmRnphNOmaI5U28aQcyG8= 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:usS/k8mmRnphNOmaI5U28aQcyG8='} 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019debf5e76510900-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006923fb3f4df9914c-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj9 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj9 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj9 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.180 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj9 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0bzCJa9Okbr4fkXsmL04HhcVPCM= 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0bzCJa9Okbr4fkXsmL04HhcVPCM='} 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b3836d1824d5a37-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bba538ccc9d61fec-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj10 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj10 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj10 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.183 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj10 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/hDnq7jOldxYgkqMmhYqyTJubT8= 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/hDnq7jOldxYgkqMmhYqyTJubT8='} 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a70ad4e4d42a78e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003123ce0d86cb2e1a-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj11 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj11 2024-04-12T04:36:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj11 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.186 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj11 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ej16i8dLYw9c2YA9KuNWS03p/1E= 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ej16i8dLYw9c2YA9KuNWS03p/1E='} 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a7997b5ffc9b211-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f61e6b95b5dca9f3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj12 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj12 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj12 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj12 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KOyfy4FdmGL72mAaDfjYTJ9GLjI= 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KOyfy4FdmGL72mAaDfjYTJ9GLjI='} 2024-04-12T04:36:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027cc2fd1fd7ef4d2-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.190 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8382277f9e028d6-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj13 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj13 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj13 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj13 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:OiqETF7WW1wuPW1vvFPmNZ1avKo= 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OiqETF7WW1wuPW1vvFPmNZ1avKo='} 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec8b7a32bfdceedd-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb9bc0f3eec57028-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj14 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj14 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj14 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.194 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj14 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:A2+AlL1/od+uEdxiiRumHVMhvMU= 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A2+AlL1/od+uEdxiiRumHVMhvMU='} 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005750028f2e74e19-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.196 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4b331f3ce5ec103-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj15 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj15 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj15 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj15 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:CBSgIVnrRnH/3Dw22uwzCqFcZc8= 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CBSgIVnrRnH/3Dw22uwzCqFcZc8='} 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f3dff57c1984652-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b2538dd84dc7814-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj16 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj16 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj16 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.200 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj16 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:YiDXr4PDwCK1/aUGMhFZfrS/FSE= 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YiDXr4PDwCK1/aUGMhFZfrS/FSE='} 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c39c4d5306c32c20-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a32de531a7a61c1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj17 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj17 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj17 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.203 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj17 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gxN2Q3xHacYPGMHoV7kiVbF+n1E= 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gxN2Q3xHacYPGMHoV7kiVbF+n1E='} 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45bd7bd61f5bf97-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf208e3208301566-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj18 2024-04-12T04:36:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj18 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj18 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.206 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj18 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:v+XuogUMCCOLEhoXIIECVZ8usW8= 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v+XuogUMCCOLEhoXIIECVZ8usW8='} 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008595a842326005e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbfa7f35c584d9bd-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj19 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj19 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj19 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj19 2024-04-12T04:36:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HIupO6AyMWF3huzEAFrKZkj3FtU= 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HIupO6AyMWF3huzEAFrKZkj3FtU='} 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000705298fbbc2aa6a7-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.211 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd6307206b488058-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj20 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj20 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj20 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj20 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.213 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:i86t7ac4We05US/aVkcjo83N63E= 2024-04-12T04:36:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i86t7ac4We05US/aVkcjo83N63E='} 2024-04-12T04:36:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d39c327618fef22-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002acd464fea51f253-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj21 2024-04-12T04:36:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj21 2024-04-12T04:36:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj21 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.218 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj21 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:T8H1dzShULjhX4e1WGWws1Ycw1k= 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:T8H1dzShULjhX4e1WGWws1Ycw1k='} 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000205d23ae842f406f-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a821a588d1716c9a-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj22 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj22 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj22 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj22 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jlHEr8vQF9m5TQa9EXsDY4+DYjM= 2024-04-12T04:36:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jlHEr8vQF9m5TQa9EXsDY4+DYjM='} 2024-04-12T04:36:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096f052f61bee4379-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.223 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000baa6b6f143d28b3-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj23 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj23 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj23 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj23 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:64KhN9zPvDOmePW+o4vXzGGzprY= 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:64KhN9zPvDOmePW+o4vXzGGzprY='} 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006890ee63e3e3c868-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.226 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002104865adc0fcad2-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj24 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj24 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj24 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj24 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8PC4VfiXWUtcPUFnwdgYaet6Szk= 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8PC4VfiXWUtcPUFnwdgYaet6Szk='} 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adc80db49ada131e-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.229 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff047f7004873fbd-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj25 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj25 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj25 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj25 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oTuzu8/F8O/mvalUcSSPr55TCJU= 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oTuzu8/F8O/mvalUcSSPr55TCJU='} 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022c4a6cbae330048-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.232 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e3ada157a9dc406-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj26 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj26 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj26 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj26 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tgwVZ8YHwyRu3dIdIkkRqkYvqk4= 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tgwVZ8YHwyRu3dIdIkkRqkYvqk4='} 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a02435eed21353e4-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000db94d1fca07fafa-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj27 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj27 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj27 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj27 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:m7BmwTEVzW65ZFUje6vF5KSOD7Q= 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m7BmwTEVzW65ZFUje6vF5KSOD7Q='} 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4555a998e0c2728-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000397fcf3785fc0ad1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj28 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj28 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj28 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj28 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Dk5scZzQNeJCUU5XLPqslXKMc48= 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Dk5scZzQNeJCUU5XLPqslXKMc48='} 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025684601abf7aa99-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043bacdb9ab46b030-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj29 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj29 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj29 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.242 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj29 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RzaXEGh9GYS5fIivVsHHufr1gVU= 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RzaXEGh9GYS5fIivVsHHufr1gVU='} 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eed8067895c54f75-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000904553afaddf5ec1-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj30 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj30 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj30 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.245 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj30 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:sAGWzrTVp0tjOFN1GWKHT5LAsWw= 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sAGWzrTVp0tjOFN1GWKHT5LAsWw='} 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4c7b5b1d3b8169c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b09894ab2036e10-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj31 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj31 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj31 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.248 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj31 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MheFR3qI96xZYl4H71R/D8GyWS4= 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MheFR3qI96xZYl4H71R/D8GyWS4='} 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007964599670dbadc3-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000108931aefc153c32-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj32 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj32 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj32 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.251 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj32 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RbLXcLwEzElMw2g+KY3qDde/IdE= 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RbLXcLwEzElMw2g+KY3qDde/IdE='} 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edce7301e28b0aa4-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc3eeb5bbca33cf0-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj33 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj33 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj33 2024-04-12T04:36:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj33 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:H/am/ybng9doEmrbDdKpD9RCDmg= 2024-04-12T04:36:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H/am/ybng9doEmrbDdKpD9RCDmg='} 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044005802d5240491-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.255 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000059ff258f93fe025-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj34 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj34 2024-04-12T04:36:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj34 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj34 2024-04-12T04:36:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5shWXGHMeA+hOP5oW1InF7POsPc= 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5shWXGHMeA+hOP5oW1InF7POsPc='} 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005b688a40eafb2ea-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a050f9a9c72c9920-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj35 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj35 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj35 2024-04-12T04:36:50.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.260 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj35 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:mbTFzLq/OjNoJywlYl0aZntXUow= 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mbTFzLq/OjNoJywlYl0aZntXUow='} 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000194423722bd6322a-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e49b541a10f35a24-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj36 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj36 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj36 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj36 2024-04-12T04:36:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:A/+dm21Uo1691jY0FADOLPck2gg= 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:A/+dm21Uo1691jY0FADOLPck2gg='} 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077c793fdc06d8487-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.267 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f316dc22f7557e7-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj37 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj37 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj37 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj37 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fTHnFfOdFx/nx0sJQPDGwbO1RAc= 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fTHnFfOdFx/nx0sJQPDGwbO1RAc='} 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c9803888dc28cc5-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.270 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d0be1dfda3bd8e-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj38 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj38 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj38 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj38 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dJXfNboX/l6dAMUAi83hubtSqiA= 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dJXfNboX/l6dAMUAi83hubtSqiA='} 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000258c312d97b12976-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009456c076c592eb19-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj39 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj39 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj39 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj39 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:H2knZbAtkrNbKbZ7bLWnnRZbOZk= 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H2knZbAtkrNbKbZ7bLWnnRZbOZk='} 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a259bb26d8d0e38-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU= 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MzHJ/aI7HvAM5kIVJUOQqNFsVTU=', 'Content-Length': '0'} 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015aef08849a7b8d9-006618a3d9-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj40 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj40 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj40 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:41 GMT 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj40 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:NNlWfC/5yWBpfrFbE/5Kk393nxM= 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:41 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NNlWfC/5yWBpfrFbE/5Kk393nxM='} 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004903be498575ef3c-006618a3d9-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.279 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a60d6612db44694-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj41 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj41 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj41 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj41 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FFjBR4rpUm6YMHKSpippxVCGOOI= 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FFjBR4rpUm6YMHKSpippxVCGOOI='} 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cccba0c7663827c-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.282 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000361f015c02a93367-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj42 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj42 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj42 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj42 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vnUEfuKf/Yl2GMD5ZwMFnrfi+ww= 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vnUEfuKf/Yl2GMD5ZwMFnrfi+ww='} 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c199d659b1b425-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.285 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008db3e08c81e92b01-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj43 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj43 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj43 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj43 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dMy3JxK8epAhBgFsgeL2XkmAusQ= 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dMy3JxK8epAhBgFsgeL2XkmAusQ='} 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c2be66efbc7911-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.288 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c3472965dcd77c4-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj44 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj44 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj44 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj44 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:i2U7aM3nNezas3lEccCiO8UXZFw= 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:i2U7aM3nNezas3lEccCiO8UXZFw='} 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e639b960968fb5ae-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae8f5431cd86acb2-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj45 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj45 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj45 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj45 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dA6us5rrTPnF0RdzPEHAuKhGXZ4= 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dA6us5rrTPnF0RdzPEHAuKhGXZ4='} 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e87849bbc0840349-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0fe00dc641c3c2-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj46 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj46 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj46 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.295 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj46 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yTX/yfWlmkH7b3/Yy8/HecZa1zU= 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yTX/yfWlmkH7b3/Yy8/HecZa1zU='} 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b1402cef176122-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000798e0e34d456ff3f-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj47 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj47 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj47 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.298 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj47 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:y8kjnBy2Mxsf4VQhSCH1Gs2gj1c= 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y8kjnBy2Mxsf4VQhSCH1Gs2gj1c='} 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000837aa5c7c44b5492-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009dd03e9f2e42b68-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj48 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj48 2024-04-12T04:36:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj48 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj48 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.301 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:4UQfxwsarRTD8RgfpJNU4CapI4c= 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4UQfxwsarRTD8RgfpJNU4CapI4c='} 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d724e30790d9e3d6-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/ 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/ 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/ 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.302 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/ 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4= 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f11qYUun8PH196OrUsa5ti6/Mq4=', 'Content-Length': '0'} 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1b65a70af018016-006618a3da-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-11/obj49 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-11/obj49 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-11/obj49 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:/dgbpxg-11/obj49 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:6DPZqCPJOAwFAYD7UvqlU3bZpH0= 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6DPZqCPJOAwFAYD7UvqlU3bZpH0='} 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a14c42a8a64ccf6-006618a3da-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.305 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d47b1d02eb72564e-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj1 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj1 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj1 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj1 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MKBOZ6r5mT191v4BxQAxG8brfYE= 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MKBOZ6r5mT191v4BxQAxG8brfYE='} 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a05b3249851ad76b-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001da415d18898e31c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj2 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj2 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj2 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj2 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:N1/EvSOtbR94vMqQuxRdKuxkcSk= 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N1/EvSOtbR94vMqQuxRdKuxkcSk='} 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da9ce9f17d2cfb0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.311 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c066e87246d1121-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj3 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj3 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj3 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.312 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj3 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:QHcE2sC6FRBkW7NO0xra7F8/KDM= 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QHcE2sC6FRBkW7NO0xra7F8/KDM='} 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a724c332208be2a-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a52bc89d28f787-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj4 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj4 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj4 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.315 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj4 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:4hzvcaH1uINM5XRAwjF11z6HuGM= 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4hzvcaH1uINM5XRAwjF11z6HuGM='} 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008eccbe4f8c940f7f-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000817190d707f43377-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj5 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj5 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj5 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.318 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj5 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jzSFy+a5okPVXFBBEqpdTulsnzs= 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jzSFy+a5okPVXFBBEqpdTulsnzs='} 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc828099c58ac911-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f4bb21a48077a19-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj6 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj6 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj6 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.321 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.322 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.322 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj6 2024-04-12T04:36:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.322 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MnbZcW5xEgFdTGiuowUJWKxODr8= 2024-04-12T04:36:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MnbZcW5xEgFdTGiuowUJWKxODr8='} 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6fe86d264941d5d-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf8bd9da6eea9a53-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj7 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj7 2024-04-12T04:36:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj7 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj7 2024-04-12T04:36:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HaWVdKaA5JwFRt6A+BfT2c0R0bY= 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HaWVdKaA5JwFRt6A+BfT2c0R0bY='} 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c086d580a6453e6-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.329 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4ab61516c599c91-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj8 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj8 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj8 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj8 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8Aqpwzjjl68yQf/RO29dsKYstuw= 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Aqpwzjjl68yQf/RO29dsKYstuw='} 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0e84387ddb5947f-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.332 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050db539f9b35799-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj9 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj9 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj9 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj9 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ji4a+UnS2/AyXGEHWs70oRdjk0U= 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ji4a+UnS2/AyXGEHWs70oRdjk0U='} 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ecd635ae93679502-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.335 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076a69beb0df414e0-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj10 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj10 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj10 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj10 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fxQu/IxSWMeguV+6WCU348RPyao= 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fxQu/IxSWMeguV+6WCU348RPyao='} 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0cca18cd1d49c4c-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.338 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047955ef6a0c6e6c6-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj11 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj11 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj11 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj11 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uemqwaNovFwpP6aXh1ud9zH8HEc= 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uemqwaNovFwpP6aXh1ud9zH8HEc='} 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063fbfa2157d6df56-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003733e3398246d70a-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj12 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj12 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj12 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.342 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj12 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Z3bavVZPlBaqKCOGuHZYge47pdk= 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Z3bavVZPlBaqKCOGuHZYge47pdk='} 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbcc8e52f5f97168-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006fc8b8bc06901a3-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj13 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj13 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj13 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.345 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj13 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:83MoXt4qgs69Xy6gtdGnkJVkUlo= 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:83MoXt4qgs69Xy6gtdGnkJVkUlo='} 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fcec73b61bb68c0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000014ebafaaa707ee6-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj14 2024-04-12T04:36:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj14 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj14 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.348 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj14 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:22r2tOEHUA+Q6mHqbE3jznGkzcc= 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:22r2tOEHUA+Q6mHqbE3jznGkzcc='} 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c64c90327fff8d2-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e4599acfde9a05-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj15 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj15 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj15 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj15 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.351 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:mjqTY8nna7KLxlZCj6qw/5yv2gg= 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mjqTY8nna7KLxlZCj6qw/5yv2gg='} 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fca26cd2a4a74344-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.352 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001989a1894f726a17-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj16 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj16 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj16 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj16 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8SIDLvw1vt6C9Lb+Qib4xtJrIec= 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8SIDLvw1vt6C9Lb+Qib4xtJrIec='} 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064dddb9d2f848216-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.355 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4fc182d51421314-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj17 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj17 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj17 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj17 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:VpMB6FlsuF6s/qwlTlRaPmNciIQ= 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VpMB6FlsuF6s/qwlTlRaPmNciIQ='} 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b628837d630202d0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.358 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d23006395520aea-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj18 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj18 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj18 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj18 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ohR8izBcktUWBew1XJ6Bv1fapY8= 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ohR8izBcktUWBew1XJ6Bv1fapY8='} 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005905fb954b799ba0-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.361 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044aa880b9cea82da-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj19 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj19 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj19 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj19 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1RPStJq6t4DhUtTY2oOaFFmhlXQ= 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1RPStJq6t4DhUtTY2oOaFFmhlXQ='} 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091b18d74f8b823ce-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cec405d693a9d814-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj20 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj20 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj20 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj20 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.366 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:QQzPMik5J4pboDEyJsn882U8PNA= 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QQzPMik5J4pboDEyJsn882U8PNA='} 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0768b962bb5e425-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1b485759e7c6785-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj21 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj21 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj21 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.370 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.370 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj21 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:YXzXsblAn/33ad1bSg0tvYKYW94= 2024-04-12T04:36:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YXzXsblAn/33ad1bSg0tvYKYW94='} 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000414dc7b1adacd763-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.373 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.373 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.373 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.373 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034f65e16615d6fa0-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj22 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj22 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj22 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj22 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KNm5ZVYT8il+BU4NY2gm2BCfdQ4= 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KNm5ZVYT8il+BU4NY2gm2BCfdQ4='} 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009300aebb579ed707-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.376 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e59b58625d256ffa-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj23 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj23 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj23 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj23 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yqFctJJoTXuTBnBQBUKFNht1cPc= 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yqFctJJoTXuTBnBQBUKFNht1cPc='} 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000684b46e12b095342-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.379 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa031cecbccc6cd-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj24 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj24 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj24 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj24 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:pLWj9b3kknADciRfG027mRYMnu0= 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pLWj9b3kknADciRfG027mRYMnu0='} 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cdf17b1219e2414-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.382 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dbc5bbb3b643952-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj25 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj25 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj25 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj25 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:glXa+MR0LbvNb61mD/SLVEmb28U= 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:glXa+MR0LbvNb61mD/SLVEmb28U='} 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001acb0ee6a752e0e8-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a5ebdcf9a86b803-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj26 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj26 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj26 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj26 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:wWji12qCwLwi4YZUmChn2sPPbdY= 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wWji12qCwLwi4YZUmChn2sPPbdY='} 2024-04-12T04:36:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000794131ca6954c8a7-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.388 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c2084a40574a9c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj27 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj27 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj27 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj27 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rSGnsnIDXWzBzfc/bj6F1a/X8sE= 2024-04-12T04:36:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rSGnsnIDXWzBzfc/bj6F1a/X8sE='} 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e0eb472369827ac-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.391 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee3217b27cc4eb8-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj28 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj28 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj28 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj28 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8UlYIGELU4lJ7qSxmuMqsSke3b4= 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8UlYIGELU4lJ7qSxmuMqsSke3b4='} 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a8cdc80618e2d96-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.394 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034c97c27dcc71501-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj29 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj29 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj29 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj29 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:H4kevjP9aqQvmg1c3HfRmGs7oQw= 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:H4kevjP9aqQvmg1c3HfRmGs7oQw='} 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5d768e04af300b3-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.397 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2f674cf99075b5c-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj30 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj30 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj30 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj30 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JGBPbglgoUfAMpJf6TCw4Of/VO0= 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JGBPbglgoUfAMpJf6TCw4Of/VO0='} 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008c1df1e3d37234d-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.400 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE= 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vKxyFBYpL59ZkEqYy9IEcPI/mXE=', 'Content-Length': '0'} 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000927e0e9ed9edf8c4-006618a3da-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj31 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj31 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj31 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:42 GMT 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj31 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:OboIowbcpQP1ZHCijmXuoRuezkY= 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:42 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OboIowbcpQP1ZHCijmXuoRuezkY='} 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000987aa84857942667-006618a3da-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000519b1dddf7209679-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj32 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj32 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj32 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj32 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:egPuEagRrUag04ZhF2ySTfHEJGc= 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:egPuEagRrUag04ZhF2ySTfHEJGc='} 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b487cac52c06324f-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.406 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000037e774939991b55-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj33 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj33 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj33 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj33 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1Ndif9qjKO3MMQOtwGyhoxloF/I= 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1Ndif9qjKO3MMQOtwGyhoxloF/I='} 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043cef1157f30fcdd-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa75f57522b68edb-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj34 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj34 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj34 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.410 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj34 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C/Ar3f41rfm7OxIhBhL8dg1L+v0= 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C/Ar3f41rfm7OxIhBhL8dg1L+v0='} 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b5552b52ffe3da8-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003db96a3438f25349-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj35 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj35 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj35 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.413 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj35 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:VvcKi5+YVhjRcTj5yh3nbfCAh+I= 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VvcKi5+YVhjRcTj5yh3nbfCAh+I='} 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006edfacff82307a53-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006704af4a2d0f3b4d-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj36 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj36 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj36 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.416 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj36 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:QegDawR8aXocfM3wirIpoaPGK+4= 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QegDawR8aXocfM3wirIpoaPGK+4='} 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005115d1fd9ac4c58b-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f312a54fcdff8497-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj37 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj37 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj37 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.419 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj37 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aj8Dw5K6QfEdlOAfTGKd3fCewoQ= 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aj8Dw5K6QfEdlOAfTGKd3fCewoQ='} 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c2b7917e38ab4ab-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000028af2c7d666886c-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj38 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj38 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj38 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.422 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj38 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MZrTYmzWMYkzv0qH+wxf0diJbPs= 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MZrTYmzWMYkzv0qH+wxf0diJbPs='} 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bb711c431051b91-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.424 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba63ec1902b5d69a-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj39 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj39 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj39 2024-04-12T04:36:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj39 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.426 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:nmmKXBSlK98MQ+QvYx8vzyTPy9U= 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:nmmKXBSlK98MQ+QvYx8vzyTPy9U='} 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edab207d453fccc-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.427 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061f44a40d29e24dc-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj40 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj40 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj40 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj40 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FkvoPd5sLbpJGyJaanP015SBq0M= 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FkvoPd5sLbpJGyJaanP015SBq0M='} 2024-04-12T04:36:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094f80f6e7f9c65c9-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.430 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d538300762de1d8-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj41 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj41 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj41 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj41 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:PTAm6ibmBjkIFcPfq0ANLGGaKV0= 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PTAm6ibmBjkIFcPfq0ANLGGaKV0='} 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02e8bb7bef60b72-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.434 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d768ad18d62286-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj42 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj42 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj42 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.435 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj42 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hfaijiiVBOhcaPHWdOrrt2uiroo= 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hfaijiiVBOhcaPHWdOrrt2uiroo='} 2024-04-12T04:36:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e97dd088e018243d-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002763b4ab6846a7c6-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj43 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj43 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj43 2024-04-12T04:36:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj43 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gyN8exwLa9H1DRZ+vdMxOHv1rFQ= 2024-04-12T04:36:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gyN8exwLa9H1DRZ+vdMxOHv1rFQ='} 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c74601776cffefef-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.442 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b3b6cd167de5f4b-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj44 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj44 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj44 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj44 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lw4+dxevUFlpnr20Ed3jVaZ+tUM= 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lw4+dxevUFlpnr20Ed3jVaZ+tUM='} 2024-04-12T04:36:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f938e835a94a9362-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.445 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021578669ed157c7d-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj45 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj45 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj45 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj45 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:UeKOVpzQoDA47P18wqYlQPGR2Oc= 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UeKOVpzQoDA47P18wqYlQPGR2Oc='} 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0f74757b0ff2880-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.448 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6e00f13047ee3db-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj46 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj46 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj46 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj46 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:TD4aXzIltSvZzIV0BkAI+z+ykQw= 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TD4aXzIltSvZzIV0BkAI+z+ykQw='} 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d94d8a897e63061-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.451 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006983d882f7e95357-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj47 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj47 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj47 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj47 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:O5GIfTdTpatRSaIdDc9TBO774NA= 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O5GIfTdTpatRSaIdDc9TBO774NA='} 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b2891839c30f9aa-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bd5416fcc22af94-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj48 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj48 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj48 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.455 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj48 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:op1tJmrKH9mbkMKRPPlpYJ/Ls18= 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:op1tJmrKH9mbkMKRPPlpYJ/Ls18='} 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd1d7031baefda38-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/ 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/ 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/ 2024-04-12T04:36:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/ 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8= 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+hpDFFIHJxvTJjzjWRM4TTwR6s8=', 'Content-Length': '0'} 2024-04-12T04:36:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e592d9c101e47268-006618a3db-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-12/obj49 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-12/obj49 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-12/obj49 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:50.458 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:00:43 GMT 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:/dgbpxg-12/obj49 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:j50a27Lv8G3A+Op7RcOXia+lw+g= 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:00:43 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:j50a27Lv8G3A+Op7RcOXia+lw+g='} 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ff6dcafb04b5451-006618a3db-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:00:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:50.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890840.928479_199.1", 2024-04-12T04:36:50.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T03:00:40.928479Z" 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:43.565+0000 7f321ad6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:43.566+0000 7f321ad6d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890840.928479_199.1", 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:00:40.928479Z", 2024-04-12T04:36:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:00:43Z" 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.466 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.467 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.468 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.469 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890840.928479_199.1", 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:00:40.928479Z", 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.470 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:00:43Z" 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712890840.928479_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:00:40.928479Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712890840.928479_199.1'} 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:43.678+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:48.679+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:53.681+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:00:58.681+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:03.682+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:08.683+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:13.683+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:18.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:23.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:28.684+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:33.685+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:38.686+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:43.687+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:48.687+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:53.688+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:01:58.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:03.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:08.689+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:13.690+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:18.691+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:02:20.289+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:23.692+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:28.693+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:33.693+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:38.694+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:43.695+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:48.696+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:53.695+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:02:58.696+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:03.697+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:08.698+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:13.699+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:18.699+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:23.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:28.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:33.700+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:38.701+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:43.702+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:48.703+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:53.704+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:03:58.705+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:03.705+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:08.706+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:13.707+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:18.706+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:23.707+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:28.709+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:33.710+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:38.710+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:43.711+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:48.711+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:53.712+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:04:58.712+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:03.713+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:08.713+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:13.715+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:18.715+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:23.716+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:28.716+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:33.717+0000 7f2b112c0780 1 waiting to reach incremental sync.. 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:38.718+0000 7f2b112c0780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:38.718+0000 7f2b112c0780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:05:38.718+0000 7f2b112c0780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:36:50.477 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:50.478 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.478 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:50.478 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-04-12T04:36:50.478 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:50.480 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dgbpxg-33 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-33/ 2024-04-12T04:36:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-33/ 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-33/ 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.482 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:10:47 GMT 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:/dgbpxg-33/ 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1O1+atcPhqAKrfwBKs1jrqkWwaU= 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1O1+atcPhqAKrfwBKs1jrqkWwaU=', 'Content-Length': '0'} 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c4b872f4402770cd-006618a637-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dgbpxg-34 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-34/ 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-34/ 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-34/ 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:10:47 GMT 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:/dgbpxg-34/ 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lTOIjBKvxVw4jww2xxGkXRu1dEg= 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:10:47 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lTOIjBKvxVw4jww2xxGkXRu1dEg=', 'Content-Length': '0'} 2024-04-12T04:36:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c94475fdb62e32b6-006618a637-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891447.229209_425.1", 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T03:10:47.229209Z" 2024-04-12T04:36:50.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:47.418+0000 7f4b08a69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:47.419+0000 7f4b08a69780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:10:47Z" 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.493 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.494 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.496 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:10:47Z" 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:52.823+0000 7fb8964fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:52.824+0000 7fb8964fd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:10:52Z" 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.502 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.504 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.506 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:10:52Z" 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:57.968+0000 7fe9c279a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:10:57.969+0000 7fe9c279a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:10:57Z" 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.514 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:10:57Z" 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:03.070+0000 7f873c7b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:03.071+0000 7f873c7b9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:03Z" 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.523 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.526 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.528 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.529 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.529 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:03Z" 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:08.182+0000 7f4d1bf46780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:08.183+0000 7f4d1bf46780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:08Z" 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.541 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:08Z" 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:13.310+0000 7fd4af38f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:13.311+0000 7fd4af38f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:13Z" 2024-04-12T04:36:50.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:50.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:13Z" 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.560 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:18.425+0000 7f1fa8cf4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:18.426+0000 7f1fa8cf4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:18Z" 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.570 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:18Z" 2024-04-12T04:36:50.574 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:23.553+0000 7ff77c236780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:23.555+0000 7ff77c236780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:23Z" 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.580 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.583 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.584 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:23Z" 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:28.670+0000 7f5ee3e30780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:28.671+0000 7f5ee3e30780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:28Z" 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.593 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.594 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:28Z" 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:33.780+0000 7fb5f5999780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:33.781+0000 7fb5f5999780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:33Z" 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.635 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.636 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:33Z" 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:38.882+0000 7f432dd27780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:38.883+0000 7f432dd27780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:38Z" 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:38Z" 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:43.983+0000 7fc56a993780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:43.984+0000 7fc56a993780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:43Z" 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.653 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.654 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.655 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.656 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:43Z" 2024-04-12T04:36:50.657 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:49.095+0000 7fa501a8a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:49.096+0000 7fa501a8a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:49Z" 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.663 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.667 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:49Z" 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:54.202+0000 7f4b7f78a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:54.203+0000 7f4b7f78a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:54Z" 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.674 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.677 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.678 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:54Z" 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:59.310+0000 7f2dc0a16780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:11:59.311+0000 7f2dc0a16780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:11:59Z" 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.686 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.687 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.688 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.690 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:11:59Z" 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:04.418+0000 7f39f6057780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:04.419+0000 7f39f6057780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:04Z" 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.697 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.698 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.701 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:04Z" 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:09.529+0000 7f78c895b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:09.530+0000 7f78c895b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:09Z" 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.708 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:50.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:09Z" 2024-04-12T04:36:50.712 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:14.663+0000 7f5a8f4d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:14.665+0000 7f5a8f4d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:14Z" 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.718 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.720 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.721 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:14Z" 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:19.774+0000 7fa4002dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:19.775+0000 7fa4002dd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.759416_368.1", 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.759416Z", 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:19Z" 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.728 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.729 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.730 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.731 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.733 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.759416_368.1", 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.759416Z", 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.734 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:19Z" 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.759416_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.759416Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.759416_368.1'} 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.759416_368.1 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:24.880+0000 7f1d99885780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:24.881+0000 7f1d99885780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.961337_393.1", 2024-04-12T04:36:50.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.961337Z", 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:24Z" 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.741 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.743 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.961337_393.1", 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.961337Z", 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:24Z" 2024-04-12T04:36:50.745 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.961337_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.961337Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.961337_393.1'} 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.961337_393.1 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:29.991+0000 7fd1a0107780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:29.992+0000 7fd1a0107780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:29Z" 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.751 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.754 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.755 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:29Z" 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:12:34.213+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:35.102+0000 7f90a0fd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:35.103+0000 7f90a0fd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:35Z" 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.763 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.764 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:35Z" 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:40.211+0000 7f1e09c2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:40.212+0000 7f1e09c2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:40Z" 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.772 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.774 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.776 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:40Z" 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:45.315+0000 7fccd28e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:45.316+0000 7fccd28e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:45Z" 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.786 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.787 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:45Z" 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:50.423+0000 7f2b319a5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:50.424+0000 7f2b319a5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:50Z" 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.794 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.795 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.798 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:50Z" 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:55.533+0000 7f8951433780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:12:55.534+0000 7f8951433780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:12:55Z" 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:12:55Z" 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:00.640+0000 7f3ffb8b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:00.641+0000 7f3ffb8b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:00Z" 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.815 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.816 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.817 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.818 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.819 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:00Z" 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:05.749+0000 7f59b6086780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:05.750+0000 7f59b6086780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:05Z" 2024-04-12T04:36:50.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.829 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:05Z" 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:10.857+0000 7f10ad476780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:10.858+0000 7f10ad476780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:10Z" 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.836 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.837 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:10Z" 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:15.967+0000 7fdc0f17c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:15.968+0000 7fdc0f17c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:15Z" 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.846 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.849 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.850 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:15Z" 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:21.075+0000 7fa358aa3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:21.076+0000 7fa358aa3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:21Z" 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:21Z" 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:26.183+0000 7fedb5208780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:26.184+0000 7fedb5208780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:26Z" 2024-04-12T04:36:50.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.869 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:26Z" 2024-04-12T04:36:50.872 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:31.293+0000 7fbeb79cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:31.294+0000 7fbeb79cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:31Z" 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:31Z" 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.883 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:36.403+0000 7fae1a659780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:36.404+0000 7fae1a659780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:36Z" 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.889 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.892 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.893 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:36Z" 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:41.512+0000 7fa29c30f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:41.513+0000 7fa29c30f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:41Z" 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.901 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.902 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:41Z" 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:46.623+0000 7fcfe6636780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:46.625+0000 7fcfe6636780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:46Z" 2024-04-12T04:36:50.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:46Z" 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.915 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:51.737+0000 7f3e82c12780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:51.738+0000 7f3e82c12780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:51Z" 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.921 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.924 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:51Z" 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:56.844+0000 7f6e0fc1b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:13:56.846+0000 7f6e0fc1b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:13:56Z" 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.932 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.933 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.934 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:13:56Z" 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.936 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:01.954+0000 7fac088c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:01.955+0000 7fac088c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:01Z" 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.948 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:01Z" 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:07.064+0000 7f69994cc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:07.065+0000 7f69994cc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:07Z" 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.960 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:07Z" 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:50.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:12.174+0000 7fd47cfe2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:12.175+0000 7fd47cfe2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:12Z" 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.968 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.972 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:12Z" 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:17.293+0000 7f0c7c936780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:17.294+0000 7f0c7c936780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:17Z" 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.985 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:17Z" 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:22.404+0000 7fb637d8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:22.405+0000 7fb637d8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:50.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:50.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:50.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:22Z" 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:50.993 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.994 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.996 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.997 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:22Z" 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:50.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.558140_357.1 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:27.516+0000 7f4550679780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:27.517+0000 7f4550679780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:50.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:27Z" 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.005 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.006 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.009 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:27Z" 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:32.630+0000 7f83e0f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:32.632+0000 7f83e0f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:32Z" 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.021 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.025 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:32Z" 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:37.750+0000 7fe9cb720780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:37.751+0000 7fe9cb720780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:37Z" 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.034 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.037 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.038 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:37Z" 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:42.862+0000 7f684fd7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:42.863+0000 7f684fd7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:42Z" 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.046 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:42Z" 2024-04-12T04:36:51.051 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:47.971+0000 7f5ae9cef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:47.972+0000 7f5ae9cef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:47Z" 2024-04-12T04:36:51.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.059 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.060 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.061 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.062 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:47Z" 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:53.078+0000 7f9797686780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:53.079+0000 7f9797686780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:53Z" 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.071 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.072 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:53Z" 2024-04-12T04:36:51.076 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:58.189+0000 7f1823119780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:14:58.190+0000 7f1823119780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.637900_362.1", 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.637900Z", 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.083 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.084 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.637900_362.1", 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.637900Z", 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.088 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.637900_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.637900Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.637900_362.1'} 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.637900_362.1 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:03.300+0000 7f1d3d902780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:03.301+0000 7f1d3d902780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:03Z" 2024-04-12T04:36:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.096 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.097 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.098 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.099 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.100 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:03Z" 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:08.412+0000 7fc4e27ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:08.413+0000 7fc4e27ae780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:08Z" 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.108 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.110 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.112 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:08Z" 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:13.523+0000 7f38de8be780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:13.524+0000 7f38de8be780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:13Z" 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.120 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.121 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.122 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.124 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.125 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:13Z" 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:18.634+0000 7fe8a7e92780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:18.635+0000 7fe8a7e92780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:18Z" 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.135 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:18Z" 2024-04-12T04:36:51.138 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:23.744+0000 7f743365c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:23.745+0000 7f743365c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:23Z" 2024-04-12T04:36:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.146 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.147 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.148 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.149 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.150 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:23Z" 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:28.857+0000 7fe9d8861780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:28.858+0000 7fe9d8861780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:28Z" 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.157 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:28Z" 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:33.971+0000 7f04d1255780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:33.972+0000 7f04d1255780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:33Z" 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.169 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.170 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.171 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.173 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.174 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:33Z" 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:39.081+0000 7fd4e2a70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:39.082+0000 7fd4e2a70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:39Z" 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.184 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.185 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.186 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:39Z" 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:44.195+0000 7f0ce5dd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:44.196+0000 7f0ce5dd1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:44Z" 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.194 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.196 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.197 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.198 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:44Z" 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:49.308+0000 7f6839bd5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:49.309+0000 7f6839bd5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:49Z" 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:49Z" 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.211 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891447.229209_425.1'} 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891447.229209_425.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:51.212 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-04-12T04:36:51.213 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-12T04:36:51.214 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dgbpxg-35 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-35/ 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-35/ 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-35/ 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:/dgbpxg-35/ 2024-04-12T04:36:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:84PkzjT2im0qstSP109FuY8l928= 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:84PkzjT2im0qstSP109FuY8l928=', 'Content-Length': '0'} 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d678c193c9fa1d43-006618a76a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dgbpxg-36 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-36/ 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-36/ 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-36/ 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:/dgbpxg-36/ 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.218 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aWK5uJMBnMQqflXWXh2SasgxLZg= 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aWK5uJMBnMQqflXWXh2SasgxLZg=', 'Content-Length': '0'} 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000caab7bb155e0f62f-006618a76a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-35/ 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-35/ 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-35/ 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:/dgbpxg-35/ 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ptVlmH46P4ctydhvvsra4wagkQ4= 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ptVlmH46P4ctydhvvsra4wagkQ4=', 'Content-Length': '0'} 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afd562a8852d7fc1-006618a76a-110e-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-35/key 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-35/key 2024-04-12T04:36:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-35/key 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.221 INFO:tasks.rgw_multisite_tests:/dgbpxg-35/key 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:TAA4tTy3BjfJyCsDJZy9xQVRZtk= 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TAA4tTy3BjfJyCsDJZy9xQVRZtk='} 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a8e9d3de7b1f3e0-006618a76a-110e-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-36/ 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-36/ 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-36/ 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:/dgbpxg-36/ 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:L8dZDokjU4DFNl4JmRSYk7C9HRQ= 2024-04-12T04:36:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L8dZDokjU4DFNl4JmRSYk7C9HRQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1f61640be882376-006618a76a-1112-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-36/key 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-36/key 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-36/key 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:15:54 GMT 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:/dgbpxg-36/key 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:IVmhKlSbH5xeYm8DUN0A82xSaVs= 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:54 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IVmhKlSbH5xeYm8DUN0A82xSaVs='} 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072746f72aab18c3d-006618a76a-1112-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891754.433386_468.1", 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T03:15:54.433386Z" 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:54.647+0000 7f76e4f59780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:54.649+0000 7f76e4f59780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:54Z" 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.234 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.239 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:54Z" 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:59.754+0000 7f01b62ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:15:59.755+0000 7f01b62ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:15:59Z" 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.251 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:15:59Z" 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:04.865+0000 7fefa4845780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:04.866+0000 7fefa4845780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:04Z" 2024-04-12T04:36:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:04Z" 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:09.970+0000 7fd024634780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:09.971+0000 7fd024634780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:09Z" 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.268 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:09Z" 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:16:14.221+0000 7fdb33d4a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:15.078+0000 7fe71ebcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:15.079+0000 7fe71ebcb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:15Z" 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.280 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.283 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:15Z" 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:16:15.371+0000 7fdb29d36700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:20.197+0000 7f3f46045780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:20.199+0000 7f3f46045780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:20Z" 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.289 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.290 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.291 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.292 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.293 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:20Z" 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:25.312+0000 7f93a132e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:25.313+0000 7f93a132e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:25Z" 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.300 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:25Z" 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:30.426+0000 7fa3c71a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:30.427+0000 7fa3c71a6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:30Z" 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.310 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.311 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.312 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.313 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.314 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:30Z" 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:35.533+0000 7f35abe52780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:35.534+0000 7f35abe52780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:35Z" 2024-04-12T04:36:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:35Z" 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:40.641+0000 7f5d3f7f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:40.643+0000 7f5d3f7f7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:40Z" 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.331 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:40Z" 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:45.752+0000 7f2d64423780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:45.753+0000 7f2d64423780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:45Z" 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.341 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.343 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.346 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.347 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:45Z" 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:50.862+0000 7f9fa10d9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:50.863+0000 7f9fa10d9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:50Z" 2024-04-12T04:36:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.354 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.357 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:50Z" 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:55.971+0000 7fc1a338c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:16:55.972+0000 7fc1a338c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:16:55Z" 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.364 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:16:55Z" 2024-04-12T04:36:51.368 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:01.083+0000 7fb961563780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:01.084+0000 7fb961563780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:01Z" 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.374 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:01Z" 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:06.194+0000 7fce28ac0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:06.195+0000 7fce28ac0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:06Z" 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.385 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.387 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.388 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.389 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:06Z" 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:11.296+0000 7fa618e97780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:11.297+0000 7fa618e97780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:11Z" 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.396 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.398 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:11Z" 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:16.404+0000 7f30a9bc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:16.405+0000 7f30a9bc0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:16Z" 2024-04-12T04:36:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.409 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:16Z" 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:21.514+0000 7fe7859e5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:21.516+0000 7fe7859e5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:21Z" 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.417 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.418 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:21Z" 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:26.629+0000 7f197b5b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:26.630+0000 7f197b5b7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:26Z" 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.427 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.428 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:26Z" 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:31.742+0000 7f3884e08780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:31.743+0000 7f3884e08780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:31Z" 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.437 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.439 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.440 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.441 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:31Z" 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:36.883+0000 7fcfe4b87780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:36.884+0000 7fcfe4b87780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:36Z" 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.450 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:51.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:36Z" 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.558140_357.1 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:41.991+0000 7f5b5d18d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:41.992+0000 7f5b5d18d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:41Z" 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.460 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.462 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:41Z" 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:47.170+0000 7fe4a6849780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:47.171+0000 7fe4a6849780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:47Z" 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.471 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.472 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.473 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:47Z" 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:17:48.179+0000 7fdb1cd1c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:52.280+0000 7efd2d522780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:52.282+0000 7efd2d522780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:52Z" 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.480 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.481 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:52Z" 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.484 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:57.401+0000 7fbc0b078780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:17:57.402+0000 7fbc0b078780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:17:57Z" 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.490 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.491 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.492 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.493 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:17:57Z" 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:02.523+0000 7f6586502780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:02.524+0000 7f6586502780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:02Z" 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.500 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.501 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.502 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:02Z" 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:07.637+0000 7fedb86c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:07.638+0000 7fedb86c8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:07Z" 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.511 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:07Z" 2024-04-12T04:36:51.514 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:12.749+0000 7fa297883780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:12.750+0000 7fa297883780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:12Z" 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.520 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.521 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:12Z" 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:17.853+0000 7f572e122780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:17.854+0000 7f572e122780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:17Z" 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.530 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.532 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.533 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.534 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:17Z" 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:22.966+0000 7f196c882780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:22.967+0000 7f196c882780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:22Z" 2024-04-12T04:36:51.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.544 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:22Z" 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:28.077+0000 7fdbfd0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:28.078+0000 7fdbfd0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:28Z" 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.551 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.552 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:28Z" 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:33.196+0000 7f7fd82c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:33.197+0000 7f7fd82c0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:33Z" 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.561 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.562 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.563 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.565 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:33Z" 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:38.493+0000 7ff6e6ae4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:38.494+0000 7ff6e6ae4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:38Z" 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:38Z" 2024-04-12T04:36:51.576 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:43.608+0000 7fa2f6065780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:43.609+0000 7fa2f6065780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:43Z" 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:43Z" 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:48.720+0000 7f96797e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:48.721+0000 7f96797e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:48Z" 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.595 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:48Z" 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:53.830+0000 7ff261ed4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:53.831+0000 7ff261ed4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:53Z" 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.601 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.602 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.604 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.494539_350.1", 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.494539Z", 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.605 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:53Z" 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.494539_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.494539Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.494539_350.1'} 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.494539_350.1 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:58.956+0000 7f11be899780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:18:58.957+0000 7f11be899780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:18:58Z" 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.611 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.613 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.558140_357.1", 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.558140Z", 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:18:58Z" 2024-04-12T04:36:51.616 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.558140_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.558140Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.558140_357.1'} 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.558140_357.1 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:04.065+0000 7fafb4289780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:04.066+0000 7fafb4289780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:04Z" 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.622 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.625 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:04Z" 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.626 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:09.176+0000 7fc455fb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:09.177+0000 7fc455fb1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:09Z" 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.633 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:09Z" 2024-04-12T04:36:51.636 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:14.285+0000 7f269975f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:14.286+0000 7f269975f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:14Z" 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.642 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:14Z" 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:19.396+0000 7f826a26a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:19.397+0000 7f826a26a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:19Z" 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:19Z" 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:24.508+0000 7fdf887db780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:24.509+0000 7fdf887db780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:24Z" 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.662 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.664 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.665 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.666 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:24Z" 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:29.621+0000 7f8f67d30780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:29.622+0000 7f8f67d30780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:29Z" 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.672 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.673 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:29Z" 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:34.758+0000 7f332ecff780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:34.759+0000 7f332ecff780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:34Z" 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.686 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:34Z" 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:39.870+0000 7f161e841780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:39.870+0000 7f161e841780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:39Z" 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.692 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.694 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:39Z" 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:44.976+0000 7f741eb18780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:44.977+0000 7f741eb18780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:44Z" 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.703 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.706 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:44Z" 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:50.093+0000 7f9c0a36c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:50.094+0000 7f9c0a36c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:50Z" 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.712 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.714 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.716 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:50Z" 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:55.208+0000 7f8a7c689780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:19:55.209+0000 7f8a7c689780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:19:55Z" 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.722 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.723 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.724 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.725 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.726 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:55Z" 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:00.320+0000 7f734fb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:00.321+0000 7f734fb37780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:00Z" 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.733 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:00Z" 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.737 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:05.427+0000 7fd576122780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:05.428+0000 7fd576122780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:05Z" 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.743 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.744 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.745 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.746 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.747 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:05Z" 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:10.535+0000 7fda943d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:10.536+0000 7fda943d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:10Z" 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.753 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.754 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.755 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.757 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:10Z" 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:15.640+0000 7f2ef83cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:15.641+0000 7f2ef83cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:15Z" 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.764 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:15Z" 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.768 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:20.751+0000 7f0a4b592780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:20.752+0000 7f0a4b592780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:20Z" 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.774 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.775 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:20Z" 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:25.863+0000 7f7f09091780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:25.864+0000 7f7f09091780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:25Z" 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.784 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.785 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.786 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.787 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:25Z" 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:30.974+0000 7f208067b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:30.975+0000 7f208067b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.552617_356.1", 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:30Z" 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.794 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.795 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.796 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.552617_356.1", 2024-04-12T04:36:51.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.798 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:30Z" 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.552617_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.552617Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.552617_356.1'} 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.552617_356.1 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:36.086+0000 7f51fb361780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:36.088+0000 7f51fb361780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.552617_356.1", 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:36Z" 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.804 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.552617_356.1", 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.552617Z", 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:36Z" 2024-04-12T04:36:51.808 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.552617_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.552617Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.552617_356.1'} 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.552617_356.1 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:41.199+0000 7fe83dfbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:41.200+0000 7fe83dfbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:41Z" 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.814 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.816 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.612433_358.1", 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.612433Z", 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:41Z" 2024-04-12T04:36:51.818 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.612433_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.612433Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.612433_358.1'} 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.612433_358.1 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:46.313+0000 7ff03f263780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:46.314+0000 7ff03f263780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:46Z" 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.824 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.826 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:46Z" 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:51.428+0000 7fd154327780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:51.429+0000 7fd154327780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:51Z" 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.834 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:51Z" 2024-04-12T04:36:51.838 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:56.535+0000 7f1d0ce07780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:20:56.536+0000 7f1d0ce07780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:20:56Z" 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.844 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891443.549294_354.1", 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:10:43.549294Z", 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.848 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:20:56Z" 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891443.549294_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:10:43.549294Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891754.433386_468.1'} 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712891443.549294_354.1'} 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712891754.433386_468.1 target=1_1712891443.549294_354.1 2024-04-12T04:36:51.849 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:51.850 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.850 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:51.850 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-12T04:36:51.850 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1451, in test_bucket_reshard_index_log_trim 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2024-04-12T04:36:51.851 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dgbpxg-57 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.852 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:e4v1XOijxVgXmNgpq4SJIxGogoA= 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:e4v1XOijxVgXmNgpq4SJIxGogoA=', 'Content-Length': '0'} 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004037f59eb2de63c8-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T04:36:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071cb7fc40735f79f-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:51.855 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tj7A4q8PdxeKfm50lbwG82ab+Yo= 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tj7A4q8PdxeKfm50lbwG82ab+Yo='} 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca03e5c4e3b799a6-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ac10aa68442b1e8-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.858 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FS1BTjo61tBhy5GqPGSMQiVDIAA= 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FS1BTjo61tBhy5GqPGSMQiVDIAA='} 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a84969b8365784c-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T04:36:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e89639259aae760c-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.861 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:SmZWzRkhC4ECskLe/idaxx0zfQg= 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SmZWzRkhC4ECskLe/idaxx0zfQg='} 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626a93277c1d6d1c-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E= 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:m2rzGXXhbmCJL80j06ljoEE/M6E=', 'Content-Length': '0'} 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c69473375de3dc4-006618ab34-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:04 GMT 2024-04-12T04:36:51.864 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:pi4gbSvwkhU2zDS3gSHQsumjjkg= 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:04 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pi4gbSvwkhU2zDS3gSHQsumjjkg='} 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e05ce567e441feb0-006618ab34-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712892724.035697_1004.1", 2024-04-12T04:36:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T03:32:04.035697Z" 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:04.328+0000 7f6f715f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:04.329+0000 7f6f715f3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712892724.029550_1001.1", 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.029550Z", 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:32:04Z" 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.872 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.874 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1712892724.029550_1001.1", 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:32:04.029550Z", 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.876 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:32:04Z" 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892724.029550_1001.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:32:04.029550Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.029550_1001.1'} 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712892724.035697_1004.1 target=1_1712892724.029550_1001.1 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:09.439+0000 7f145cfd9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:09.440+0000 7f145cfd9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712892724.035697_1004.1", 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.035697Z", 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T03:32:09Z" 2024-04-12T04:36:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:51.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:51.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1712892724.035697_1004.1", 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:32:04.035697Z", 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:51.886 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:32:09Z" 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892724.035697_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:32:04.035697Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712892724.035697_1004.1'} 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:51.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:09.552+0000 7f422433b780 1 waiting to reach incremental sync.. 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:14.553+0000 7f422433b780 1 waiting to reach incremental sync.. 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:19.555+0000 7f422433b780 1 bucket sync caught up with source: 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000001.51.6, , , , 00000000001.28.6, , , 00000000001.26.6, , 00000000001.61.6] 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000001.51.6, , , , 00000000001.28.6, , , 00000000001.26.6, , 00000000001.61.6] 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:19.555+0000 7f422433b780 0 bucket checkpoint complete 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:19.675+0000 7f22f2dfd780 1 waiting to reach incremental sync.. 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:24.677+0000 7f22f2dfd780 1 bucket sync caught up with source: 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000001.1211.6, , , , 00000000001.610.6, , , 00000000001.1248.6, , 00000000001.1327.6] 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000001.1211.6, , , , 00000000001.610.6, , , 00000000001.1248.6, , 00000000001.1327.6] 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:24.677+0000 7f22f2dfd780 0 bucket checkpoint complete 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU= 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU=', 'Content-Length': '0'} 2024-04-12T04:36:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea23468e3b50d158-006618ab48-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.891 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU= 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JH8ItUBEE+7Nd1ZrGJ5JiQd1JqU=', 'Content-Length': '0'} 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000deb92efba495fdca-006618ab48-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T04:36:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6e1984a2e5e417-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.895 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f51e48855d3913-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.897 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600047c6e4d0fd97-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg= 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iDFcZ+uBST9TjEvXmAQY6zAOleg=', 'Content-Length': '0'} 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e55892b09eae75e7-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.899 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.900 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae26717403340ee-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.902 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb31f4248c7daed-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI= 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI=', 'Content-Length': '0'} 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62ceb95058c0fb6-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.905 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI= 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ty/iEn9URum8P+Ver3uZ9n+E3nI=', 'Content-Length': '0'} 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7c80d1fdea682c-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead69c97b42e8d4f-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.908 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ= 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:79qtP4lB5kxqMRvL0pvQNJ6tFmQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ef6ea554d61764-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776bd00bb0a20a35-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.911 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5db47dd672c3851-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI= 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI=', 'Content-Length': '0'} 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbda03e8dd2b8db-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.914 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI= 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9jAq0bUhO6lFaP0JkKmpuY/C2eI=', 'Content-Length': '0'} 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4458528ef67e43-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU= 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU=', 'Content-Length': '0'} 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048845d2331ed2618-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU= 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1iP1HXhUsGsNqdgGhghmDSKlsGU=', 'Content-Length': '0'} 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001131951757a07294-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.918 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcaaa9f7de51d866-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c24909a70910970c-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd13413821fb967-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.923 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029878ae7dd438cf5-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY= 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY=', 'Content-Length': '0'} 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e7d60bb78dc4a7-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.926 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY= 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AmxFuqMQQYPAoufU8Z84DyMILhY=', 'Content-Length': '0'} 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004611e8246af91685-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256bb3466ddf8f8f-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.929 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U= 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DJIp/3hBSFn9Y/rctZT0tgvvI/U=', 'Content-Length': '0'} 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f918c983b04c9dcd-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273fcfad7fb87c35-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.932 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2690f1f351defbd-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk= 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk=', 'Content-Length': '0'} 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b4444aa70e31de-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.935 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk= 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C8b6mW0zZwCAt5cETI4mJh9JLPk=', 'Content-Length': '0'} 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1bc833657a69b59-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0= 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0=', 'Content-Length': '0'} 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcec7687dd054e69-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.938 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0= 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y4Z9+0T1jcARtDHsa/jNl02zfb0=', 'Content-Length': '0'} 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005918ac9461734269-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009606439912693aa7-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544a6bb7f2a3a83e-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43fca59beff5218-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.944 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015fa2d505f81a2f6-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.946 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI= 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI=', 'Content-Length': '0'} 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3158f8f2cddebd-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI= 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Q/Zeg5+JM2WYCF1Dz3MLFbfQfjI=', 'Content-Length': '0'} 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fea6091c92b54a6-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703a1ac52c80474d-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU= 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/VTIKQ4w9zMpQ3axw3234zbnFaU=', 'Content-Length': '0'} 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2043bc84006bdb-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.952 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b5b07e9fd91b5a-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a749daf2a6661a61-006618ab48-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.955 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo= 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo=', 'Content-Length': '0'} 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000247103755d70877e-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo= 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8NMcagOQL6nZXIIv5dqY03pDIdo=', 'Content-Length': '0'} 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf1b568a3b3379b-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ= 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f1504667d8b5de-006618ab48-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.959 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ= 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Lyil/OSfiLg0E2+Bx9MVNBfc6UQ=', 'Content-Length': '0'} 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5bb3f9d39b17829-006618ab48-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:24 GMT 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M= 2024-04-12T04:36:51.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:24 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:13oJ8prMAZnt2YKLxTDFu3LB97M=', 'Content-Length': '0'} 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028fb5278f3da6f4f-006618ab48-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83ef1ad57e1235d-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.963 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.964 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4944adbf534503-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.966 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744973b933751fa5-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8= 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8=', 'Content-Length': '0'} 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050299d447782dc1f-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.969 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8= 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0A0sStPfDuiSG5EKdlxPKi0x8/8=', 'Content-Length': '0'} 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a80a22b7c2c8680-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444135719933d3ea-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.972 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU= 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:B/ZezkHpRX5TMw0rsbYzlN/v4cU=', 'Content-Length': '0'} 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867434c1d49899e3-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.974 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53c3832942cccc6-006618ab49-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T04:36:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf7bcaeaf2dee3a-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.977 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE= 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE=', 'Content-Length': '0'} 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a46968db79a4b2-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE= 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mco1ZvdRJ+zKUpIL9mcWxrBasHE=', 'Content-Length': '0'} 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faf7e18b6430971-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.980 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng= 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng=', 'Content-Length': '0'} 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b595ce40275293-006618ab49-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng= 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6fRhW54ZPLypvwUFdcSo7hEEXng=', 'Content-Length': '0'} 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033270db4c54e4d16-006618ab49-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.983 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080fd46c0646d0359-006618ab49-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:51.985 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8= 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DEfOF5VdztCuCwEZrTL8GuQafk8=', 'Content-Length': '0'} 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf64ab5fd6e0f1f-006618ab49-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "1#00000000001.51.6", 2024-04-12T04:36:51.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.14556731054388386214", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "a", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.044816674Z", 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 5 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:36:51.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "5#00000000001.28.6", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.16167048996727160496", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "d", 2024-04-12T04:36:51.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.112463396Z", 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 4 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "8#00000000001.26.6", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.7091642361043578140", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "c", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.103008525Z", 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2024-04-12T04:36:51.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 3 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "10#00000000001.61.6", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "b7d2416c-9d6f-4780-9e14-1f66614caba8.6433.1910817736580495436", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "b", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T03:32:04.093278075Z", 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 9 2024-04-12T04:36:51.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20" 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:51.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-57 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 4 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2024-04-12T04:36:51.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:51.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dgbpxg-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dgbpxg-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [] 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.001 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7157626c7ac9333-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:R1woxug4lRbXkPJ/AKTcJZzXiiQ= 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R1woxug4lRbXkPJ/AKTcJZzXiiQ='} 2024-04-12T04:36:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff475f2d5059a3a3-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.004 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f176b4980abdc5c0-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:kqlp2zfxJjODQbUnL1urTJG60CU= 2024-04-12T04:36:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kqlp2zfxJjODQbUnL1urTJG60CU='} 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013dd7c9e94cb7757-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.007 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000d36cfb6de2340c-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1vKVsGuUTSTLFx9+kieiNL611Xo= 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vKVsGuUTSTLFx9+kieiNL611Xo='} 2024-04-12T04:36:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3a6d7ad2e9b9aa7-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.010 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg= 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oOAHgscMj0PTEx1D864ha+46aIg=', 'Content-Length': '0'} 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030ba0a654919a71e-006618ab49-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:32:25 GMT 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:GBOIFbuzRxug25lt6szcF6bwIHM= 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:32:25 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GBOIFbuzRxug25lt6szcF6bwIHM='} 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8994fff5044af7d-006618ab49-1921-a1'), ('Date', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:25.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:30.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:35.852+0000 7f6ac1450780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:40.855+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.42.6, , ] 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:45.855+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.42.6, , ] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:50.856+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.42.6, , ] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:32:55.857+0000 7f6ac1450780 1 waiting for incremental sync to catch up: 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.42.6, , ] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:00.858+0000 7f6ac1450780 1 bucket sync caught up with source: 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.42.6, 00000000002.66.6, 00000000002.52.6] 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:00.858+0000 7f6ac1450780 0 bucket checkpoint complete 2024-04-12T04:36:52.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:00.973+0000 7fd90ca99780 1 waiting for incremental sync to catch up: 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000001.1211.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000001.1248.6, , 00000000002.1331.6] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:05.974+0000 7fd90ca99780 1 bucket sync caught up with source: 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T04:36:52.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:05.974+0000 7fd90ca99780 0 bucket checkpoint complete 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.016 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:05 GMT 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss= 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss=', 'Content-Length': '0'} 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005883d9650e69ba-006618ab71-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:05 GMT 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss= 2024-04-12T04:36:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:05 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:O+sewEOW+zjtu9Zj9cj2iB7I5Ss=', 'Content-Length': '0'} 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa14c91dcceef6ac-006618ab71-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646030a0bcb1df18-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.020 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a227a87b2c711c-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.024 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f1cc5a757b2cfe-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64= 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qKMEgWDKvaqjqR1Wek9rumU3s64=', 'Content-Length': '0'} 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f7e33980f0ddd5-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7919b151280636-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.029 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ce483a2898e181-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY= 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY=', 'Content-Length': '0'} 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9e0bf7fbd00db9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.032 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY= 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xCMDVl7zRKVzhpF+CBnwSCdHjcY=', 'Content-Length': '0'} 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d064a4adbd5fc8e5-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T04:36:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e8c20e7097e900-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc= 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:anaYGVR6/iVPOuYaCDvnF2CTHZc=', 'Content-Length': '0'} 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225f294b21a3f73b-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.037 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8fa941e923d26d5-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001151f6e1f31678-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.040 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y= 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9100e3115e9e7c4-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y= 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:QdO7bx1YaWwOaPPM7GM5OqCz94Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a323f1b68d16f6c8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.043 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI= 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI=', 'Content-Length': '0'} 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059092285e94feea8-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI= 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peT/waCPA0dPNdLNLaSStVmF8pI=', 'Content-Length': '0'} 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a91e32f556f6452-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.046 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d3786a25bfb6bc-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebd142f8bfc6495a-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745487cde1ef6b1d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000758d036da385c6f8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc= 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc=', 'Content-Length': '0'} 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e5ffb0f4537d16-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.054 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc= 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:OM9EKpdr5rm7jMZXYzNfkexdPoc=', 'Content-Length': '0'} 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585976525169cc91-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T04:36:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d0cf3da66b1aa0-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.057 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc= 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:yMyD4u4g6isE/tmZFXqX7r5dyXc=', 'Content-Length': '0'} 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437b5ee9a33c37cf-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a324cde70169c9-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e1449f633b7fc43-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk= 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk=', 'Content-Length': '0'} 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baa61815f2590500-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.063 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk= 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:V8Aos/1v2zo0kgnBQbYSCfOoEBk=', 'Content-Length': '0'} 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af2604cb1584f15-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.065 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI= 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI=', 'Content-Length': '0'} 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed0c8167ec13183-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI= 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:35l47yrtlp5N6XK6ilJB6EJ7CVI=', 'Content-Length': '0'} 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d4e609834308dd-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbaa0526dfcbd3ab-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000789e90160192ac53-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.071 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a3fe0af8675590-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb069f6ff38439ff-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.074 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck= 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck=', 'Content-Length': '0'} 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af1306f25810da5e-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck= 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HkjSYuR9CzIjblrE6mMi4zKwAck=', 'Content-Length': '0'} 2024-04-12T04:36:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ad7e8939ea6732-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008032eee19ef8c0b9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.079 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM= 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WTbsu82Wz0TYQmF0jXaWIHCcRlM=', 'Content-Length': '0'} 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac6dfaa41f2acf7-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766b694c31f05753-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e47ca4ce5460ca0c-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.084 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA= 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA=', 'Content-Length': '0'} 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720dd30c8b002458-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA= 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ZpbEEaBkM1yyr91UlF1RQQDlUcA=', 'Content-Length': '0'} 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49a4790473ab9c6-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80= 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80=', 'Content-Length': '0'} 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049527890658a1d13-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.088 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80= 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:y+Iy5s5iqLBVa94qGDpPuhg8U80=', 'Content-Length': '0'} 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523c3cc098816c17-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.090 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d3bed081da2db4-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004df45d8e14840b9a-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.093 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf936d7671d93f1-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a561e820ea5cc4-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE= 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE=', 'Content-Length': '0'} 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ebf0133210dc77-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.098 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE= 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hf3ftcxAHMUW02l+IhkUKBv5WwE=', 'Content-Length': '0'} 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25105dc416eef75-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d84db5204dd337-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.101 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s= 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:66OkH/MXUOVnK8dJ1E0iRQprF6s=', 'Content-Length': '0'} 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b6107c591137fb-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.104 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebd7599d5f43ae2-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a707ed4156c783b-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I= 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I=', 'Content-Length': '0'} 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517ceacc7a43decb-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.109 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I= 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oSYhsKbCESVrWGc3Gr9JwihwH2I=', 'Content-Length': '0'} 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcffdcd00ff9620-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI= 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI=', 'Content-Length': '0'} 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec84ec7a637e182-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.112 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI= 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+ydB4u1AVAA6gTaD0eC2nd9bJI=', 'Content-Length': '0'} 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e47ecd5eefe5558-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9262e01eeb9288-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb6d7695316018b-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.117 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebfea66986cd7f7-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.119 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ffdd4f3e36b648-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0= 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0=', 'Content-Length': '0'} 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62ba5b97e6e9c2b-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.122 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0= 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hWKXU3DcrZr7pB5/Yv3vTn3iRo0=', 'Content-Length': '0'} 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f14f26694199b3-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.124 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf4c064d336be4d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw= 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:peYi+4SFfWqovkW4WwVFbPwSCTw=', 'Content-Length': '0'} 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014377a3dbb97664f-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.127 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d804cda0e6b1b1b-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e476e1b9dfe2e899-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.130 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4= 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4=', 'Content-Length': '0'} 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ad2f0f0328050f-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4= 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KUZgpIvI/ZY9tvLo2uVDJbqgNB4=', 'Content-Length': '0'} 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008795f2ebeca952d6-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.133 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U= 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U=', 'Content-Length': '0'} 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afeca178710a17f9-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U= 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:CE9l8xw2IMeuRoEUDY3RogtXX4U=', 'Content-Length': '0'} 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e454475320dfd355-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.136 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001273ed892810192d-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.138 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e09991f7827144-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T04:36:52.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db8d5905cb6d3f6d-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.141 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180aa4496525740e-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA= 2024-04-12T04:36:52.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA=', 'Content-Length': '0'} 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce75ca45f49a668-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA= 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u4ggcSkxGMpATVxkPtB6C0oRiDA=', 'Content-Length': '0'} 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b0d8086bac2e8d-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.146 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ff32f29ada8357-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc= 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/eTy4GEMJQpWHA9CI+a3hiYVXHc=', 'Content-Length': '0'} 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468b052187d0ef23-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.149 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db436830dfed5b2-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.151 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a6cf47204983e4-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM= 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM=', 'Content-Length': '0'} 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012102413fb20fdf5-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM= 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:MdETWpmMTqtXItzEurw85UQLeTM=', 'Content-Length': '0'} 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008768e9e2540497c3-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY= 2024-04-12T04:36:52.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY=', 'Content-Length': '0'} 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002651764143101c63-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.157 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY= 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:v63k6QdytVcQ4rvVRIKLc3S++GY=', 'Content-Length': '0'} 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026bde55a5f42050c-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.159 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000558319a6ce073d95-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b89e746e7388a32-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55773f4eba14f24-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.164 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dc2b1c23b56e82-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.166 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc= 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc=', 'Content-Length': '0'} 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2eb1beb677f0f8-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc= 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:N6FodgA21wjbp6PHXdoC21Pg7yc=', 'Content-Length': '0'} 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb206a582dda29e-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca89cccf0fa68113-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.171 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg= 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fqLuD6ImtDQWAKgkPARM9gFbnKg=', 'Content-Length': '0'} 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c7877173fe9b5b-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e7d9e4a585a216-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b5ffa9de6fd98fd-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.176 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8= 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8=', 'Content-Length': '0'} 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d0815d9e793cbf-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8= 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rYB+Yh18KZeNENl7J0HP/CXekR8=', 'Content-Length': '0'} 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ca4f424b0b5ba0-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE= 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE=', 'Content-Length': '0'} 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041bca1d107560370-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE= 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:obXnOEbx7amica4N8u1MZuMegKE=', 'Content-Length': '0'} 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4b04209acee3e5-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.182 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f7efe983702f34-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e01c6a6d530c52-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.185 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7042769e7730ce-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.187 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008505cfcd6d068de4-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A= 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A=', 'Content-Length': '0'} 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098be36d9fd0fd13b-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.190 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A= 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5OLePVrkzg5idGYMP2c+gQUGv+A=', 'Content-Length': '0'} 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a220ed7fb70f98d8-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63ad86849c2cfd1-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M= 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:dt/9k0NFqWLZm40znUd0gKzdc9M=', 'Content-Length': '0'} 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d764510098761911-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c02d2f00ac227c1-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004459fc344e12cfbc-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM= 2024-04-12T04:36:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM=', 'Content-Length': '0'} 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3d6b7703bf51f6-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM= 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:sG+jPpWw89ERdMr/yHu0qiquRNM=', 'Content-Length': '0'} 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab8a91ab7ec1d0d-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI= 2024-04-12T04:36:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI=', 'Content-Length': '0'} 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979ca4d87ba4d723-006618ab72-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.202 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI= 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3iHnrABbxIbFMypUTi6LBDrSldI=', 'Content-Length': '0'} 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45543ac6249ed97-006618ab72-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db6fde09ad09ab4-006618ab72-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:06 GMT 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI= 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:06 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aU4KFod+F2ncjBlYP2qgsDeQ3SI=', 'Content-Length': '0'} 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f069bbef19bf68d2-006618ab72-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dgbpxg-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dgbpxg-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-57 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 8 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-57 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:07.167+0000 7f6ab427b780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:12.167+0000 7f6ab427b780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:17.168+0000 7f6ab427b780 1 bucket sync caught up with empty source 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:17.168+0000 7f6ab427b780 0 bucket checkpoint complete 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-57 2024-04-12T04:36:52.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:17.283+0000 7fec211b6780 1 bucket sync caught up with source: 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1215.6, , , 00000000001.1585.6, 00000000001.610.6, , , 00000000002.1252.6, , 00000000002.1331.6] 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T03:33:17.283+0000 7fec211b6780 0 bucket checkpoint complete 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU= 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU=', 'Content-Length': '0'} 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d52ea03efb7dfa5e-006618ab7d-192d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/ 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU= 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HFJzCjY6eL6/L4ofIIeWuPwz5eU=', 'Content-Length': '0'} 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037b07e1c698dc628-006618ab7d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c361faa91d9a48c0-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:36:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.219 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000303592e159a68e6e-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:36:52.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.221 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000986b2d2ee71c8478-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?versions&max-keys=999 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/?versions 2024-04-12T04:36:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I= 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s01ZvT02AQdZ4SnR+GTo2p0vl1I=', 'Content-Length': '0'} 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f843ada06589db4-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57999falseanulltrue2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a147889bb997c211-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b8c26f92eb10e9-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI= 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI=', 'Content-Length': '0'} 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021daa681ac690bf4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.229 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI= 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1tt/9AfYlfQW+G4tP8oAiSniowI=', 'Content-Length': '0'} 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3b49b97ddffd0d-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.231 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a349ec900924c5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a?versionId=null 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y= 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:0I/29qpkrVO5zLgd68bEEc8wx8Y=', 'Content-Length': '0'} 2024-04-12T04:36:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541747d5b87ea2b6-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.234 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064425f5571f9fe4c-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.236 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9b9383b778b251-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4= 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4=', 'Content-Length': '0'} 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009beb913e1d6d630b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.239 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4= 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ddoOwYRLOVwjoMXkwixFDy5vXK4=', 'Content-Length': '0'} 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29458cd0855569f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.240 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ= 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c4bba6ace7dc5c-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/a 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/a 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/a 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/a 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ= 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Z/4PB/IoirzLaSMeTXybWqoAyQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da883f52e3ee0aa-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.243 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717c4005e2af5062-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=a 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=a 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92e1a281eb91458-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57a1000falsea2024-04-12T03:32:04.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.247 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002347a427dbba2a2-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91d32f47386a33e-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.250 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk= 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk=', 'Content-Length': '0'} 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4395db9538e7eb5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk= 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GrKGqOyPsWyNQc7e4MdsYD5zXtk=', 'Content-Length': '0'} 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b76706c6bc45245-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.253 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af81f775bd3569ad-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b?versionId=null 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg= 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:soO2RchWdGysrJzi3a9romFEINg=', 'Content-Length': '0'} 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d00313abb402dec-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.256 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b970153bf55c2d3-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.258 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a01d3572bacdbc05-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.259 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY= 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY=', 'Content-Length': '0'} 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea48cfd0fb6dadc-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY= 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8127+4EwTe+EzuKqV6rhjTjXuNY=', 'Content-Length': '0'} 2024-04-12T04:36:52.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194e86f1ffc1338f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ= 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3f2fb149af113d-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/b 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/b 2024-04-12T04:36:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/b 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/b 2024-04-12T04:36:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ= 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HSjKMsgehYRjNdA/03YqOZ9oZtQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001070d28cfc5b5645-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051386634569bc295-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=b 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=b 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.267 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9531ccc39c615f6-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57b1000falseb2024-04-12T03:32:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T04:36:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004981884d883260e5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020880f8387456b2d-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs= 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs=', 'Content-Length': '0'} 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d84e8a80b79da23-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs= 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FmDZb5kyxh1EKpvBjcq1vY13UKs=', 'Content-Length': '0'} 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef323e55da7323c-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.275 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adac000b37c607c-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c?versionId=null 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.277 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU= 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Eqd4/aqqREOD2G0+H5AGcZhd1iU=', 'Content-Length': '0'} 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c862322fac142e42-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff65eda47c15b728-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.280 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04f4985498df740-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.282 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4= 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4=', 'Content-Length': '0'} 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ea5e045bdd37e4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4= 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FMqd7qIU3N7pOQ0RPs9VpSi+sM4=', 'Content-Length': '0'} 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b97b2e93a92b74-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ= 2024-04-12T04:36:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40b6b44a334794a-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/c 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/c 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/c 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/c 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ= 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:q2CnXVwHA72thcyIpFYDEmWGqhQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fdd40e00e423d4-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.288 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006054f88e53c22e19-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=c 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=c 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c0eba41a593021a-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57c1000falsec2024-04-12T03:32:04.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae976bd5f4207c7-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.293 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a267a89b7d410339-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98= 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98=', 'Content-Length': '0'} 2024-04-12T04:36:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca3aad887bc8b1b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98= 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K8fESMvvo9+GaJ0tahhJJ6C3z98=', 'Content-Length': '0'} 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daffa953a1905f68-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.298 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d7a2070641d9ef-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d?versionId=null 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA= 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:C40zwVPyBayOBJiLFEASeTEm1mA=', 'Content-Length': '0'} 2024-04-12T04:36:52.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ac630bd8873435-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.301 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba2e205d6c1d983-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000150dc1281fd1a4ce-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA= 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA=', 'Content-Length': '0'} 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fa03a4b823844b-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.306 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA= 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F5KL34gmYU7WTG8PLVM5t2QHlmA=', 'Content-Length': '0'} 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eeb1ca4dfa070b6-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw= 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw=', 'Content-Length': '0'} 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7a8b310ea96268-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/d 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/d 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/d 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/d 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.309 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw= 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:L9bbr0ri75+/Ly8N75LxQAWfVNw=', 'Content-Length': '0'} 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e2eb85c0216ff5-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010542cc94b406125-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=d 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=d 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.312 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee54774b9d79e4ec-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57d1000falsed2024-04-12T03:32:04.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T04:36:52.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea5f354a77ff861-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.315 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7130a3c29c0b3f7-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI= 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI=', 'Content-Length': '0'} 2024-04-12T04:36:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd4a004e3599223-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI= 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qYg3LhVVXfRTtXkjkKYzpW3XGzI=', 'Content-Length': '0'} 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af9fdcad923f3a7-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.320 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617b1e592e1ffcf5-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e?versionId=null 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg= 2024-04-12T04:36:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r6DvSg4MTLwNKCqojjrYdWzVRvg=', 'Content-Length': '0'} 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6485b3bb79cdb8-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc94ad3317b447c-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.325 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542ad4f9bf5edb32-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.327 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g= 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g=', 'Content-Length': '0'} 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5f2e44b2310eb9-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.328 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g= 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:IKJo+8ZsxY95u406gKMTYT8E21g=', 'Content-Length': '0'} 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d961fd1bfadbed8f-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU= 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU=', 'Content-Length': '0'} 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d275bf6b18b991-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/e 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/e 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/e 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/e 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU= 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ygEDX6ARBZHkuvbzi/5jAskg9HU=', 'Content-Length': '0'} 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5fd95189ecf99e-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.333 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08b7a78e8011ad9-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=e 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=e 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041eb86a313fb9e54-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57e1000falsee2024-04-12T03:32:25.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f572d654b7ad12-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.338 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d52668fe44f1e6a-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.340 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU= 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU=', 'Content-Length': '0'} 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad539d25e3989df6-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU= 2024-04-12T04:36:52.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:PMHiScTlXsc89dyKXG0B65dbqXU=', 'Content-Length': '0'} 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fab76948555cb5-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621cee8faa7210dc-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.345 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f?versionId=null 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk= 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uojVPpzPX3EfapOxPbE8P03/ltk=', 'Content-Length': '0'} 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b288cfa99dd8f10-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749149480a80c69f-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfc83de3879699c-006618ab7d-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.350 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE= 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE=', 'Content-Length': '0'} 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7f9e2a40d3d5a4-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE= 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/BpaF9PL7auJxTW0fuSYdaZ6yYE=', 'Content-Length': '0'} 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5ac67dfbd08a30-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.352 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.353 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk= 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk=', 'Content-Length': '0'} 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e347219318c1a8c1-006618ab7d-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/f 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/f 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/f 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/f 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk= 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGluo8lpKvIdZErG042ehXFWLMk=', 'Content-Length': '0'} 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002067403bb3b7a83b-006618ab7d-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.356 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:17 GMT 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k= 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:17 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:oyFEfKc7vX4IeYu55GsAiSD1e1k=', 'Content-Length': '0'} 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccf9598ed90e534-006618ab7d-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=f 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=f 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004749028ef76b3a0d-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57f1000falsef2024-04-12T03:32:25.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e04de658b379076-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.361 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b16330a708ae708-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M= 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M=', 'Content-Length': '0'} 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cab4e34728ade53-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M= 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eSotqnKzqq+4QYFzHfq49vEnV4M=', 'Content-Length': '0'} 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c25ce97dba6d4a4-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.366 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846b0df10f16ae08-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g?versionId=null 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8= 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:w/OD8oaUNI2uBoa4UI0kwHt80G8=', 'Content-Length': '0'} 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379e1da1fcdf332c-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.369 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3cf22e038bda86-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d35eec86ac8f540-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.372 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU= 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU=', 'Content-Length': '0'} 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf8e34cf2427999-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU= 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aja/pXFxobjeXbemboVu8TAAyMU=', 'Content-Length': '0'} 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da70a7d14a74279c-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.374 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.375 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU= 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU=', 'Content-Length': '0'} 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006027aafb97f7ca12-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/g 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/g 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/g 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/g 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU= 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Sa/n30yJ3L61h1NylDZ/26ybEoU=', 'Content-Length': '0'} 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf03f4fedd86b38b-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.378 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea855cce890e41e2-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=g 2024-04-12T04:36:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=g 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.380 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc5dd89a78fa7a90-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57g1000falseg2024-04-12T03:32:25.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T04:36:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a67425af8ce1c09-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.383 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0795c94dd786866-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo= 2024-04-12T04:36:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo=', 'Content-Length': '0'} 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fbc1b8e6e7a44e4-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo= 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HANkyeKFaHhqXmk3VIner6Ym0Fo=', 'Content-Length': '0'} 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002345bff6e365483a-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.388 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb922a65e420873-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h?versionId=null 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU= 2024-04-12T04:36:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hr6DFXTGrnvPnC6WkNAiCqjR/SU=', 'Content-Length': '0'} 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3036a4e7e66b198-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8302c7944b17fae-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.393 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736d965af19fc5f2-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE= 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE=', 'Content-Length': '0'} 2024-04-12T04:36:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b963cb6aa7f35948-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.396 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE= 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:DTfH15Y449F+ztC9V6ebz39mHSE=', 'Content-Length': '0'} 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b62200780c26c96-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo= 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo=', 'Content-Length': '0'} 2024-04-12T04:36:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-57:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f927933d45ccd9fb-006618ab7e-192d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/h 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/h 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/h 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.399 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/h 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo= 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qY0MvMjIM0Wvy4dYWxYU2ldbZEo=', 'Content-Length': '0'} 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:32:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d973a7b90bc224-006618ab7e-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009277745277103920-006618ab7e-192d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/ 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/ 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-57/?prefix=h 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-57/?prefix=h 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.402 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:33:18 GMT 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:/dgbpxg-57/ 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU= 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:18 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rHtlIV1MGABh0Hz0aSH/YGapbZU=', 'Content-Length': '0'} 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78589a4620bc07c-006618ab7e-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-57h1000falseh2024-04-12T03:32:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dgbpxg-57 zones={a2, a1} 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.317+0000 7fdabbf98700 -1 req 13533205992890308817 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-28:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.28,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.324+0000 7fdadc7d9700 -1 req 3357766871182794852 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-39:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.329+0000 7fdacc7b9700 -1 req 18321812564828740639 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-30:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.30,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.367+0000 7fda436a7700 -1 req 12000507695627104626 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-40:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.368+0000 7fda40ea2700 -1 req 14230048013344398153 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-5:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.5,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.371+0000 7fda5ceda700 -1 req 11522024088187569265 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-26:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.26,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.409+0000 7fda73707700 -1 req 14664376769477074816 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-27:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.27,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.413+0000 7fdae9ff4700 -1 req 7644102589494978202 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-1:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.1,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.408 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.686+0000 7fdb0e03c700 -1 req 5212105698036019351 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-3:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.3,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.686+0000 7fda39693700 -1 req 8275274726849920892 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-25:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.25,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.693+0000 7fda1ae56700 -1 req 16794667982912389458 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-33:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.33,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.738+0000 7fdad6fce700 -1 req 15620137926626652178 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-4:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.4,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.740+0000 7fdaa1f64700 -1 req 16042163566763317451 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-2:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.2,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T03:33:18.783+0000 7fda86f2e700 -1 req 12496473277467897360 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=b7d2416c-9d6f-4780-9e14-1f66614caba8,az=0},d={b=dgbpxg-23:b7d2416c-9d6f-4780-9e14-1f66614caba8.4368.23,z=d800b3e1-d3eb-4250-a3c6-cce8e78e25b6,az=0}} returned ret=-2 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dgbpxg-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dgbpxg-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2024-04-12T04:36:52.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.412 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:52.413 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.413 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:52.413 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-12T04:36:52.413 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:52.413 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:36:52.414 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:46 GMT 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+pwJqKgH2A49Zu+jTTlafJW2t1I= 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:46 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+pwJqKgH2A49Zu+jTTlafJW2t1I=', 'Content-Length': '0'} 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000040c1de4f6ccdf8da-006618b2de-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dgbpxg-65 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:52.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712894686.658536_1657.1", 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:04:46.658536Z" 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:46.831+0000 7f704f586780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:46.832+0000 7f704f586780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712894686.652046_1654.1", 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:04:46.652046Z", 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:04:46Z" 2024-04-12T04:36:52.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1712894686.652046_1654.1", 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:04:46.652046Z", 2024-04-12T04:36:52.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:04:46Z" 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894686.652046_1654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:46.652046Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.652046_1654.1'} 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712894686.658536_1657.1 target=1_1712894686.652046_1654.1 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:51.943+0000 7f725bf50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:51.944+0000 7f725bf50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712894686.658536_1657.1", 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:04:46.658536Z", 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:04:51Z" 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.435 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.436 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1712894686.658536_1657.1", 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:04:46.658536Z", 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.438 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:04:51Z" 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894686.658536_1657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:04:46.658536Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712894686.658536_1657.1'} 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:51 GMT 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:F6E90/xgIepmHKXSRD0IrbyiUrQ= 2024-04-12T04:36:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F6E90/xgIepmHKXSRD0IrbyiUrQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039f42862b13f145c-006618b2e3-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj11 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj11 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj11 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.441 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:51 GMT 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj11 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:qboygkJZjuKKzRG0PFw3WOIcyTQ= 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:51 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:qboygkJZjuKKzRG0PFw3WOIcyTQ='} 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5f0b8ed863a6916-006618b2e3-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027a9eb146a6a1d68-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj22 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj22 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj22 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.444 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj22 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:7QiQ2AKLx7IowlBU3DPbIZThc5Q= 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7QiQ2AKLx7IowlBU3DPbIZThc5Q='} 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3f0fb2c54342cb8-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064898fa2f4459d7c-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj33 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj33 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj33 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.447 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj33 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FGuzPy1CkygeKG6w8Ym1hmbOres= 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FGuzPy1CkygeKG6w8Ym1hmbOres='} 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026fb33a15008fad1-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022dba9c27b5e40c6-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj44 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj44 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj44 2024-04-12T04:36:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj44 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:G2ODJxrpz5g7b8ADSUzwHsjuQko= 2024-04-12T04:36:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:G2ODJxrpz5g7b8ADSUzwHsjuQko='} 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003038dbfc09023f9a-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.451 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b08537ab3208058-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj55 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj55 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj55 2024-04-12T04:36:52.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj55 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f3XgLyakhzgaT6JBZj0NC47Rm2c= 2024-04-12T04:36:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f3XgLyakhzgaT6JBZj0NC47Rm2c='} 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4b6f8147dd98db1-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 5 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000199d8105808d7803-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj17 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj17 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj17 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj17 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JxUuLhg+QyS5Ina7w81CoR6qP1o= 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JxUuLhg+QyS5Ina7w81CoR6qP1o='} 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a79c5d4776f14648-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006efadaea728b015d-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj34 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj34 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj34 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.459 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj34 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:n1ImiBzxzgKU4KMRoOrFJEkgceY= 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:n1ImiBzxzgKU4KMRoOrFJEkgceY='} 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8f72bb4d2d3c0c5-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006c745dfee5c1b5a-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj51 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj51 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj51 2024-04-12T04:36:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj51 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.462 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:xYocXufHz78vLXu6wTvI/Juqbk0= 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:xYocXufHz78vLXu6wTvI/Juqbk0='} 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007565682bd3a06161-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.463 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e81d75642c52bf-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj68 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj68 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj68 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj68 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/s3kSLEgxd5f35ZR7PIRZqVwL/o= 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/s3kSLEgxd5f35ZR7PIRZqVwL/o='} 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac50f57e3d092e5c-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.466 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18b3714af70c070-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj85 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj85 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj85 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj85 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:s9wjo+ZgUIdJooKalphEPYFLZJw= 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:s9wjo+ZgUIdJooKalphEPYFLZJw='} 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcadd799bf97935d-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 10 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003054a51048efdd24-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj19 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj19 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj19 2024-04-12T04:36:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj19 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+xsSAieSYp7k0hdCP6bm6rda8eE= 2024-04-12T04:36:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+xsSAieSYp7k0hdCP6bm6rda8eE='} 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efe98f2fddccbc19-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.472 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d440918b6527df07-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj38 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj38 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj38 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj38 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:eGeYJZWZpOvf/XrU7Rt6mUqqs18= 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:eGeYJZWZpOvf/XrU7Rt6mUqqs18='} 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d1456659017ed6f-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.475 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c0a2a76aeb0dc0-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj57 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj57 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj57 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj57 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uPebCRJTfRLMM3m02/3+ooYgFoA= 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uPebCRJTfRLMM3m02/3+ooYgFoA='} 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc67baed8aa65f98-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006c0165cbef814cf-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj76 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj76 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj76 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj76 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:S6q8Fg1S7zqLpZHH+Et1nZ2x4rI= 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:S6q8Fg1S7zqLpZHH+Et1nZ2x4rI='} 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e042cb5f81646046-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.481 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a6b58e6f186341c-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj95 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj95 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj95 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj95 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:UK5oxe3mnH8OhM76MsEEToqD/DU= 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:UK5oxe3mnH8OhM76MsEEToqD/DU='} 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b62c3c6d4bdc70a-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 15 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a8df35890906708-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj23 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj23 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj23 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.486 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj23 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zTI6adg9Ny9Tikr66uMcYLZVg58= 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zTI6adg9Ny9Tikr66uMcYLZVg58='} 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af31bbae976d4cd5-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb94f18a10aad133-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj46 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj46 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj46 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.489 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj46 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:vF1Yo0NtkxcPLTIyGaGBnx2uhPI= 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:vF1Yo0NtkxcPLTIyGaGBnx2uhPI='} 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e29a109457232e6-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988c5ca5ebfe2200-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj69 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj69 2024-04-12T04:36:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj69 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.492 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj69 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:u+SQGhReEcoPSYFOLCdFmFfY/BM= 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:u+SQGhReEcoPSYFOLCdFmFfY/BM='} 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adf8a3e485c00d30-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051bc2837e8f9a29f-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj92 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj92 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj92 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.495 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj92 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:VZC3yQQcqNzeD8IISOkB3XkNIx0= 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:VZC3yQQcqNzeD8IISOkB3XkNIx0='} 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083518037aa2dc8c8-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ= 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:LFOUs8tYMvw/W/f1xlhxqA83BoQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514e636f1d8be514-006618b2e4-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj115 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj115 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj115 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:52 GMT 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj115 2024-04-12T04:36:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:X4dtaPKLzQuemllOa+EOUE7aRXk= 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:X4dtaPKLzQuemllOa+EOUE7aRXk='} 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5339bc5a9a4d4d3-006618b2e4-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 20 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7cd0fc8905df9c5-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj29 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj29 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj29 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.501 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj29 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:hdusGen5xw+ealOK06A1YdI1l34= 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:hdusGen5xw+ealOK06A1YdI1l34='} 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c41e2c14215a39a-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.503 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f326a8cd11b6298-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj58 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj58 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj58 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj58 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JGH6giSCLJaoeheodg+DjyFZsV0= 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JGH6giSCLJaoeheodg+DjyFZsV0='} 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d32bbe65ed16a494-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.506 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fe0cf43210b844d-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj87 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj87 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj87 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.507 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj87 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:iGMBzuo2V87hUSw0xSXN9RVZheE= 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:iGMBzuo2V87hUSw0xSXN9RVZheE='} 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028852dbfdce8e4b6-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd3373c4c520b954-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj116 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj116 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj116 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.510 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj116 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ga4KeekOtdsXAbicbjfySnx7a68= 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ga4KeekOtdsXAbicbjfySnx7a68='} 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbd12025edea1e89-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.512 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079949862aef07a92-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj145 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj145 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj145 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.513 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj145 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:7yYVD+SvTSqgRwh45lZ4khvi5e4= 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:7yYVD+SvTSqgRwh45lZ4khvi5e4='} 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db8613aa90d56072-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 25 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.515 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c19feb32cd982b31-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj31 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj31 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj31 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj31 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:R7LWZhMKyO61vKmtzsWq25lqWvY= 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:R7LWZhMKyO61vKmtzsWq25lqWvY='} 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000937414ac06246082-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.518 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddeadc3fd49bde9e-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj62 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj62 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj62 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj62 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:TpLK7n3uWdfWBT3tqGm4gTwwlhc= 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:TpLK7n3uWdfWBT3tqGm4gTwwlhc='} 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018eef364fb9f656b-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.521 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f5ffae58f245572-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj93 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj93 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj93 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj93 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:YA0TMm3SPLCcOTpMtBdyvmwjWGI= 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:YA0TMm3SPLCcOTpMtBdyvmwjWGI='} 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5fb41142cf0d3a-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.524 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092922d2aab602fab-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj124 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj124 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj124 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj124 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:/uFTaaiViUi5y6oVUZuBuG/spLw= 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:/uFTaaiViUi5y6oVUZuBuG/spLw='} 2024-04-12T04:36:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1f02b51959aa1e3-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.527 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000885c5f0c2f58f4e1-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj155 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj155 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj155 2024-04-12T04:36:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj155 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:9yNGjbbUG0TTEWkU/SHU2CCh7Po= 2024-04-12T04:36:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9yNGjbbUG0TTEWkU/SHU2CCh7Po='} 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056321dd58fc23f39-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-65 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 30 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a98d8845efa7f157-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj37 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj37 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj37 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj37 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1VnYHbF1o/8dBUAQBwBdvXVZUog= 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1VnYHbF1o/8dBUAQBwBdvXVZUog='} 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056eef232e7037b82-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f31ac122769446f8-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj74 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj74 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj74 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj74 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fn0/gy8SZGr6VMTrrCn8AV8R+9I= 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fn0/gy8SZGr6VMTrrCn8AV8R+9I='} 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f91b2f3ceb14a9f-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b8226b7ed9afd3-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj111 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj111 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj111 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj111 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:INjG+F92xFDGbPrJ2Io86c5UMmA= 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:INjG+F92xFDGbPrJ2Io86c5UMmA='} 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003063e5739b4b22d1-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.540 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb63bd7654cbc8de-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj148 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj148 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj148 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj148 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:SrV00NV/58yNtc9QYdk1X+WMNGc= 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:SrV00NV/58yNtc9QYdk1X+WMNGc='} 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e4b201f935456b-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/ 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/ 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/ 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.543 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/ 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk= 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:RtgHpL37b0A9xsAXb/wOkbhJuVk=', 'Content-Length': '0'} 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff23c4850e24eb7-006618b2e5-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-65/obj185 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-65/obj185 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-65/obj185 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:04:53 GMT 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:/dgbpxg-65/obj185 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:+WXK9/669oASWS4AQ9Oe0bVb1dk= 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:04:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:+WXK9/669oASWS4AQ9Oe0bVb1dk='} 2024-04-12T04:36:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031b8e06efa4a6362-006618b2e5-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:04:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-65 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:54.026+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:04:59.027+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:04.027+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:09.028+0000 7efc9436f780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:14.032+0000 7efc9436f780 1 bucket sync caught up with source: 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.121.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.104.6, , , , 00000000001.124.6] 2024-04-12T04:36:52.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.121.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.104.6, , , , 00000000001.124.6] 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:14.032+0000 7efc9436f780 0 bucket checkpoint complete 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-65 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:14.150+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:19.151+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:24.151+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:29.152+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:34.153+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:39.154+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:44.154+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:49.155+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:54.156+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:05:59.157+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:04.156+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:09.157+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:14.158+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:19.159+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:24.160+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:29.160+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:34.161+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:39.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:44.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:49.162+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:54.163+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:06:59.164+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:04.165+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:09.166+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:14.166+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:19.167+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:24.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi099.stdout: INFO: 2024-04-12T04:07:28.544+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:29.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:34.168+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:39.169+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:44.170+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:49.171+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:54.171+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:07:59.172+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:04.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:09.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:14.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:19.173+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:24.175+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:29.176+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:34.176+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:39.177+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:44.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:49.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:54.178+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:08:59.179+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:04.180+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:09.182+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:14.182+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:19.183+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:24.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:29.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:34.185+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:39.184+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:44.185+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:49.187+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:54.188+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:09:59.188+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:04.189+0000 7f52fcf85780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.190+0000 7f52fcf85780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.190+0000 7f52fcf85780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.190+0000 7f52fcf85780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:52.552 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1937, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-12T04:36:52.553 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Hw5YpAR5wmMu1THEyphTB19Vieo= 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Hw5YpAR5wmMu1THEyphTB19Vieo=', 'Content-Length': '0'} 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000679b976c9177fd61-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dgbpxg-66 2024-04-12T04:36:52.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:52.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895009.257409_1720.1", 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:10:09.257409Z" 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.444+0000 7f6491aeb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.445+0000 7f6491aeb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895009.257409_1720.1", 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:10:09.257409Z", 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:10:09Z" 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.562 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.564 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.565 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895009.257409_1720.1", 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:10:09.257409Z", 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.566 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:10:09Z" 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895009.257409_1720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:09.257409Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895009.257409_1720.1'} 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b47caf59f9ceabe-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:st99Bhht4r3RbbHkwSDqNOZwzn0= 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:st99Bhht4r3RbbHkwSDqNOZwzn0='} 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000265dfd2f2fbae73c-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:36:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b68cd78ad70be27-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.572 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:wFDnbU4NzR3vQ9n0wRI31htfhDM= 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:wFDnbU4NzR3vQ9n0wRI31htfhDM='} 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000551bb34dc6c41c7-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:36:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba5e1a5c86b3c097-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.575 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gaQbNZ7G+DwWISJj7ZL+TzywMcA= 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gaQbNZ7G+DwWISJj7ZL+TzywMcA='} 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059eeb279a455681c-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.578 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000810f01e0fc4efdf6-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5lM2xZpSMxFWkCil3doZ6yvdsg8= 2024-04-12T04:36:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5lM2xZpSMxFWkCil3doZ6yvdsg8='} 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b9868b4f7a34146-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.581 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY= 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zMwVhGvHc8JYm6ZB/5OWt0+2MPY=', 'Content-Length': '0'} 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d5e16ca42ddacd0-006618b421-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55 2024-04-12T04:36:52.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:09 GMT 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:GYR8ng6EBLmcK7Tg4gdAHBBpyeY= 2024-04-12T04:36:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:09 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:GYR8ng6EBLmcK7Tg4gdAHBBpyeY='} 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061aacd8a228eb450-006618b421-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-66 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:09.689+0000 7f1199207780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:14.691+0000 7f1199207780 1 bucket sync caught up with source: 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.98.6, 00000000002.148.6, 00000000002.147.6] 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.98.6, 00000000002.148.6, 00000000002.147.6] 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:14.691+0000 7f1199207780 0 bucket checkpoint complete 2024-04-12T04:36:52.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dgbpxg-66 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:14.803+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:19.803+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:24.804+0000 7fc0de56c780 1 waiting to reach incremental sync.. 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:29.805+0000 7fc0de56c780 1 bucket sync caught up with source: 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.3243.6, 00000000002.1972.6, 00000000002.1467.6] 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.3243.6, 00000000002.1972.6, 00000000002.1467.6] 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:29.805+0000 7fc0de56c780 0 bucket checkpoint complete 2024-04-12T04:36:52.585 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dgbpxg-66 zones={a2, a1} 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.586 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE= 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE=', 'Content-Length': '0'} 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e20dae285818d86-006618b435-2289-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE= 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KjgMU83hMMw52PNn+60ZWSvIosE=', 'Content-Length': '0'} 2024-04-12T04:36:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cea0678dcf5efd7b-006618b435-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/?versions 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d55d587c7145efe-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-12T04:36:52.590 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/?versions 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d89c5dd0a13ea7a-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-12T04:36:52.592 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/?versions 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756086cad2e29852-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?versions&max-keys=999 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/?versions 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ= 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:viGeXSr4umX2C3hcGBuKKlgjGvQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2267355cc256ed9-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66999falseobj11nulltrue2024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab57b3fbe876b7c9-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.599 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d1a73aeb3051cb-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo= 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo=', 'Content-Length': '0'} 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b3a7c5e2c9346d-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.602 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo= 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:cUqujm8dNUSOi/Pg1c9yHGGmQeo=', 'Content-Length': '0'} 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5fc3d3e0e7ea3c-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc89b53d778faadb-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.605 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11?versionId=null 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA= 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ynt+0lpnpuyD5AWsi1mBMfnl+lA=', 'Content-Length': '0'} 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c91233106f69e1d-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7398519c5c0452e-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb70bb30be7a423-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.610 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI= 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI=', 'Content-Length': '0'} 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000963e7c156790c11-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI= 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:AaUnwrWwIGxKIx6HyDWtCzVKhqI=', 'Content-Length': '0'} 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097cd2a71de090234-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:36:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.613 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38= 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38=', 'Content-Length': '0'} 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009501a8936bc09985-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj11 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj11 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj11 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj11 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38= 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:3NXm0h2zRFPvhG3Jb2437EyPz38=', 'Content-Length': '0'} 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc34f9cd835bbe1-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.616 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b728d684d7b2093-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj11 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:36:52.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000461569c1374fa345-006618b435-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj111000falseobj112024-04-12T04:10:09.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3abd3425cc9e4cd-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.621 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab75a4cb435840bd-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.623 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM= 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM=', 'Content-Length': '0'} 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000486202be93bd9768-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM= 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XCIoUBM78vWVhMnaP0D6H1rtlpM=', 'Content-Length': '0'} 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47184937fab3789-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.625 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df74829936a75a8-006618b435-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.628 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22?versionId=null 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4= 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Gk5/KwEgH4XwTF1lFW+iHDT3IF4=', 'Content-Length': '0'} 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3b73bfd36c0d5e-006618b435-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:29 GMT 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA= 2024-04-12T04:36:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:29 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:jmSUtVEfI7jwqUOMeHGsZrgfVUA=', 'Content-Length': '0'} 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5363bf862729c4-006618b435-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428e9e17b8210428-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.633 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY= 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY=', 'Content-Length': '0'} 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1b2b6f6716f7fc-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY= 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bBIYxIDC+RJFXYPld3buZivQBzY=', 'Content-Length': '0'} 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004887da127bdf9d6-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.635 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.636 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY= 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY=', 'Content-Length': '0'} 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa8ed57d84b7538-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj22 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj22 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj22 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj22 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY= 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:31H7u+cJHs49f1kc30EeYuMpUxY=', 'Content-Length': '0'} 2024-04-12T04:36:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fd1567b31dd63f-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.639 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b76f02f2d868ea-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj22 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.641 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000261ea897b522ecb0-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj221000falseobj222024-04-12T04:10:09.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c2dbc2feca0311-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.644 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a651a36ada3b7412-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM= 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM=', 'Content-Length': '0'} 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b225625ca026468-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM= 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Bh4p92wt2Hi/UjVWb7QDwAtn4wM=', 'Content-Length': '0'} 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538004dc257f57d6-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.648 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.649 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f927793338723517-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33?versionId=null 2024-04-12T04:36:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ= 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:HzHiT1ZWGDRsE3HFss165xuQDCQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0709b91802c24e9-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce10d58f2e294c31-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e164bb0732d65e5a-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.656 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q= 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q=', 'Content-Length': '0'} 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001774901aed6fd283-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q= 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gY0brGG7LpHd3eZB14vYl1Qt/9Q=', 'Content-Length': '0'} 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3617ef93e2b25cf-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:36:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.659 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI= 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI=', 'Content-Length': '0'} 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007106be7b9f6dfe60-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj33 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj33 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj33 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj33 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI= 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Ma1q5c2X4WchXQT5O51BS0ZY0VI=', 'Content-Length': '0'} 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c50106c878ea054-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.662 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce01f2ec8bed253-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj33 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd1e92148fc5814-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj331000falseobj332024-04-12T04:10:09.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.666 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49ebf4fd1cbb186-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71cd9569191d5b3-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.669 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw= 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw=', 'Content-Length': '0'} 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6ad901e3ef1f60-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw= 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KmmhRcwZfaobYpFNAxy/PAwaesw=', 'Content-Length': '0'} 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983b995480b2f79e-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.671 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.672 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebe0e01441e564f-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44?versionId=null 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.674 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w= 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1vlHmrJO57aYi0F8GfvG7wZSL9w=', 'Content-Length': '0'} 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a6640e5b4c5f8a-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54e2b8188df32ce-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.677 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ba670f549fc67a-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44 2024-04-12T04:36:52.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo= 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo=', 'Content-Length': '0'} 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049792ab73de34fdb-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo= 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:fiIDiapq7S03w5LJhGUHJdkc6Eo=', 'Content-Length': '0'} 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff4fd3a5a66e74f-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.682 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc= 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc=', 'Content-Length': '0'} 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d308110a225935-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj44 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj44 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj44 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj44 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc= 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JRG2XND5yVahyfDu6v6agDQ34Wc=', 'Content-Length': '0'} 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4337e4d10b257e-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bacc5ca692a3742-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj44 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a17a947069cc52-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj441000falseobj442024-04-12T04:10:09.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8d9be9ebb98948-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e200658308131ad2-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE= 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE=', 'Content-Length': '0'} 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ef435734a25289-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.693 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE= 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:gJbo5CCHO29RQ4LZvBQs/SkdFZE=', 'Content-Length': '0'} 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbed53554442abc3-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:36:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000495935e5da5f985-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55?versionId=null 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs= 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ldDTPtUmNQxHe2jw2RousRToXgs=', 'Content-Length': '0'} 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973ffc78f6b34ca9-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.698 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce67c8d145c6639e-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4589d60da4b33b-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55 2024-04-12T04:36:52.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM= 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM=', 'Content-Length': '0'} 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecae9f7046142b48-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM= 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5AuAalgpgIulRxiwt5W8Gxk75iM=', 'Content-Length': '0'} 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2de37a56abce70-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.704 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI= 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI=', 'Content-Length': '0'} 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-66:b7d2416c-9d6f-4780-9e14-1f66614caba8.6441.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a29b2bcc6846c0f-006618b436-2289-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj55 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj55 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj55 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.705 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj55 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI= 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lLShxwbrDiiETvHtWp1WoyjMuhI=', 'Content-Length': '0'} 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004980eff2ba0fc596-006618b436-1921-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01f2c215c9aa807-006618b436-2289-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/?prefix=obj55 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:30 GMT 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk= 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:2dkIqzu/JNMOQk8O4u5zMSp41Bk=', 'Content-Length': '0'} 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3c12b824f1e276-006618b436-1921-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dgbpxg-66obj551000falseobj552024-04-12T04:10:09.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dgbpxg-66 zones={a2, a1} 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-12T04:36:52.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T04:10:30.364+0000 7fc17bf9f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 112588) UID: 0 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T04:10:30.364+0000 7fc1811747c0 -1 shutting down 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 5 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061bdec6070dd203f-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj17 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj17 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj17 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj17 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:uqr2jhTFyf+TQprgX15ua4KOdTE= 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:uqr2jhTFyf+TQprgX15ua4KOdTE='} 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb695bdb9678ef05-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1394b976c41d7cd-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj34 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj34 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj34 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.715 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj34 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:pEy1fx+Sgu7e/xxnYWy9akq/xwY= 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:pEy1fx+Sgu7e/xxnYWy9akq/xwY='} 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9834779ad899747-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c686d7b6643ef2-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj51 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj51 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj51 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.718 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj51 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:apOJYnItgo3FoIXIUMQQQTN6HWo= 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:apOJYnItgo3FoIXIUMQQQTN6HWo='} 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040cb8925e2051e65-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5f258c77b7e7793-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj68 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj68 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj68 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.721 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj68 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:bK3d4x7MvzQ1Gwb5gpSA9aUAwhw= 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:bK3d4x7MvzQ1Gwb5gpSA9aUAwhw='} 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cca05c7cb5b1d3de-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e55f34c7058ad3ef-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj85 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj85 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj85 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj85 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rQRWhOcqmQsb38YpN6CgrcAzb9w= 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rQRWhOcqmQsb38YpN6CgrcAzb9w='} 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009acaaab8f3f46ecb-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 10 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af970a8d0c11f50e-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj19 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj19 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj19 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.728 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj19 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:FYbnQbMIVPgapU+FKhP0v+nCFgQ= 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:FYbnQbMIVPgapU+FKhP0v+nCFgQ='} 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa2adeb5205d37c-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e73e68df47b82cca-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj38 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj38 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj38 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj38 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:unABRh5bTOl1w8TPqoG5c/T4awM= 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:unABRh5bTOl1w8TPqoG5c/T4awM='} 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000738dd4687ab29e7a-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad241aac06187881-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj57 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj57 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj57 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.734 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj57 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:88Hmh0M1Gy0CHxGa8y/eybsAhF0= 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:88Hmh0M1Gy0CHxGa8y/eybsAhF0='} 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008428d6cd19bb515c-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002df48289e77b0c53-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj76 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj76 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj76 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.737 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj76 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:1SrFk0DQ9YayBxGt2kMLxf7qUCs= 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:1SrFk0DQ9YayBxGt2kMLxf7qUCs='} 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a46f6c248dca53-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA= 2024-04-12T04:36:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:tPwpvIag7sPvaF4qobNTxgRTTTA=', 'Content-Length': '0'} 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fea01e8789ee917-006618b43c-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj95 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj95 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj95 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.740 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:36 GMT 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj95 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:4+1pZqfV01VJK5OGfHEGb/STL9g= 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:36 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:4+1pZqfV01VJK5OGfHEGb/STL9g='} 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ece6dfd09f4828e7-006618b43c-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 15 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed70d8d4e1977242-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj23 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj23 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj23 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.744 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj23 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5e9elyVxMGYCRkvIIcS7MYnaSgY= 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5e9elyVxMGYCRkvIIcS7MYnaSgY='} 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026a9932c7354fb51-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000512805d1f08d7e8b-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj46 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj46 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj46 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.747 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj46 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:D47hnpgTuGWH7yHDDJgpLMhwHik= 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:D47hnpgTuGWH7yHDDJgpLMhwHik='} 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9a82e5eba5bd6ae-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dab56b520e5fefd-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj69 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj69 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj69 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.750 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj69 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:rwBBJygixEXzAL0CjgHE5RDPosw= 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:rwBBJygixEXzAL0CjgHE5RDPosw='} 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016250765597d6c34-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7f05196a53aa483-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj92 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj92 2024-04-12T04:36:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj92 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.753 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj92 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:r7mr2Mi/NI4F88wuPuGORfpkc/I= 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:r7mr2Mi/NI4F88wuPuGORfpkc/I='} 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c097850834e2e259-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f46aa8103ad8d31d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj115 2024-04-12T04:36:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj115 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj115 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.756 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj115 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:o9G4Cm6Ea3y4hylE0z5VHM+zxOI= 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:o9G4Cm6Ea3y4hylE0z5VHM+zxOI='} 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff4ada091939bb3f-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 20 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.758 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002504a63f946b361f-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj29 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj29 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj29 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj29 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Qz/qg6KUhlnbap8uF52v/nOqomo= 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Qz/qg6KUhlnbap8uF52v/nOqomo='} 2024-04-12T04:36:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a6f0b71b6f66ac8-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.761 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021a7c34b936ebc24-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj58 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj58 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj58 2024-04-12T04:36:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj58 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.763 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:JqYfvpM7Cq++4lw701aUdRK6GpY= 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:JqYfvpM7Cq++4lw701aUdRK6GpY='} 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e2cadcb55db28fd-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fd5eb20cb288ee7-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj87 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj87 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj87 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.766 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj87 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:649sxG1smdjDaXqneK3ATr5yzNs= 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:649sxG1smdjDaXqneK3ATr5yzNs='} 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c8e7c388ac32d8c-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001477ecf848edfffa-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj116 2024-04-12T04:36:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj116 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj116 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.769 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj116 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:psnAcLcgvVrZi/x5QvnLwVisuYc= 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:psnAcLcgvVrZi/x5QvnLwVisuYc='} 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037b318073f118519-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001230c3cb3649cdfc-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj145 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj145 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj145 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.772 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj145 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Es56RVxGOU0J9mbArFKDSJAp4/8= 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Es56RVxGOU0J9mbArFKDSJAp4/8='} 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035740b3bd54bad78-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 25 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000013a1c90976a762d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj31 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj31 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj31 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj31 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.776 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zLHRNvVDRlBQn3jsaa+pC4FRTi0= 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zLHRNvVDRlBQn3jsaa+pC4FRTi0='} 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e16302e9d1200d51-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.777 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a55d3e7887b991-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj62 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj62 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj62 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj62 2024-04-12T04:36:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:5/LLMLxE3CFJhMdghaxciLPbs+8= 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:5/LLMLxE3CFJhMdghaxciLPbs+8='} 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040059dc5615cb235-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3305f97cce9d30d-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj93 2024-04-12T04:36:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj93 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj93 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.782 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj93 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:9zjVWyHaapzFgdzeCZewmU6BWwQ= 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9zjVWyHaapzFgdzeCZewmU6BWwQ='} 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ed559ce16a20910-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f5e110b93ea213c-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj124 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj124 2024-04-12T04:36:52.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj124 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.785 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj124 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:kiSSjjRP48dv4Nrqrm/JvucxQgE= 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:kiSSjjRP48dv4Nrqrm/JvucxQgE='} 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbe21b0f43383303-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk= 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:lAEkXfAG5RECr2WBganKiHmyBNk=', 'Content-Length': '0'} 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031b7b3185fc004e0-006618b43d-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj155 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj155 2024-04-12T04:36:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj155 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:37 GMT 2024-04-12T04:36:52.788 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj155 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:b6PjyC0CKKVmZGJ7Ut3Md+U+FLY= 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:37 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:b6PjyC0CKKVmZGJ7Ut3Md+U+FLY='} 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004759fe3fad22a21-006618b43d-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dgbpxg-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dgbpxg-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: dgbpxg-66 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 30 2024-04-12T04:36:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.790 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa5609ae43d59234-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj37 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj37 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj37 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj37 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:WmDLsvUOD7A2zhszllLz6OHAcX8= 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:WmDLsvUOD7A2zhszllLz6OHAcX8='} 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058df0aabd2f07b79-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.793 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f151d1451fc0969c-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj74 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj74 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj74 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj74 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:Rd3GLe1qNSCaCqKb7XQSR95dQUM= 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:Rd3GLe1qNSCaCqKb7XQSR95dQUM='} 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a088c4a06442065-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005545d66562e747e7-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj111 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj111 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj111 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj111 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:zxN8Zcq61D69+Mmf3WIqbPxUb7w= 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:zxN8Zcq61D69+Mmf3WIqbPxUb7w='} 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a913c0487b1d6913-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.799 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006316d501d39b04d0-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj148 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj148 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj148 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.800 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj148 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:K7meYlMiznH9EE2Bj94GCJIfx+I= 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:K7meYlMiznH9EE2Bj94GCJIfx+I='} 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba54cf9742c1549c-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/ 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/ 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/ 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/ 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k= 2024-04-12T04:36:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:ft1cuCF8H5PDUcPRVtZ8Ypa1Q7k=', 'Content-Length': '0'} 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd7085ae970c4a0-006618b43e-1921-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-66/obj185 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-66/obj185 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-66/obj185 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.803 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:10:38 GMT 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:/dgbpxg-66/obj185 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:mvpIjQrZmRW0z4g76yS3tiLV9oI= 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:10:38 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:mvpIjQrZmRW0z4g76yS3tiLV9oI='} 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3ea89bb8649886f-006618b43e-1921-a1'), ('Date', 'Fri, 12 Apr 2024 04:10:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:38.255+0000 7fdb2b0f8780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:36:52.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:38.257+0000 7fdb2b0f8780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > curl http://smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:36:52.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: anonymous 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dgbpxg-66 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dgbpxg-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dgbpxg-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:39.456+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:44.457+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:49.458+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:54.458+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:10:59.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:04.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:09.459+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:14.460+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:19.461+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:24.462+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:29.463+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:34.463+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:39.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:44.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:49.464+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:54.465+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:11:59.466+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:04.467+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:09.467+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:14.468+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:19.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:24.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:29.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:34.469+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:39.471+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:44.472+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi099.stdout: INFO: 2024-04-12T04:12:45.994+0000 7f38bfecc700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:49.473+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:54.473+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:12:59.474+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:04.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:09.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:14.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:19.475+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:24.477+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:29.477+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:34.478+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:39.478+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:44.479+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:49.480+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:54.479+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:13:59.480+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:04.481+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:09.482+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:14.483+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:19.483+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:24.484+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:29.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:34.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:39.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:44.485+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:49.487+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:54.487+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:14:59.488+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:04.489+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:09.489+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:14.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:19.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:24.490+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:29.491+0000 7f22bbbf5780 1 waiting to reach latest gen 6, on 4.. 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:34.493+0000 7f22bbbf5780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:34.493+0000 7f22bbbf5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:15:34.493+0000 7f22bbbf5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:36:52.811 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-04-12T04:36:52.812 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:52.812 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:36:52.812 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:36:52.812 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:36:52.812 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2213, in test_sync_flow_symmetrical_zonegroup_all 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[1], content) 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:52.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:52.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:17:30.171194Z" 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:51.832+0000 7f7fb73dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:51.835+0000 7f7fb73dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:17:51Z" 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.824 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:17:51Z" 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:52.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[]} 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 15, 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "predecessor_uuid": "49749ba5-837f-416d-b464-cc0e38ff4676", 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": [], 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_map": { 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zonegroups": [ 2024-04-12T04:36:52.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a", 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a", 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": true, 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a1", 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2024-04-12T04:36:52.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "compress-encrypted", 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a2", 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8001" 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2024-04-12T04:36:52.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "compress-encrypted", 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [ 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-group", 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "data_flow": { 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "symmetrical": [ 2024-04-12T04:36:52.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-flow1", 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pipes": [ 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-pipe", 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2024-04-12T04:36:52.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "*", 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": { 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "*", 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6" 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "params": { 2024-04-12T04:36:52.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "filter": { 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [] 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": {}, 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "priority": 0, 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "mode": "system", 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user": "" 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "enabled" 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "b", 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b", 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": false, 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b2", 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi181.front.sepia.ceph.com:8001" 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:36:52.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "compress-encrypted", 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b1", 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi181.front.sepia.ceph.com:8000" 2024-04-12T04:36:52.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "compress-encrypted", 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:36:52.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [] 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "short_zone_ids": [ 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "79a72dd1-625f-4b14-843f-f3ed404d65dc", 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 2532217463 2024-04-12T04:36:52.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "79aa477c-57c9-443d-814c-33bacc0229a0", 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 2914034528 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 2928306118 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "d800b3e1-d3eb-4250-a3c6-cce8e78e25b6", 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 2226213169 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zonegroup": "a", 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "b7d2416c-9d6f-4780-9e14-1f66614caba8", 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_config": { 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_quota": { 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_quota": { 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2024-04-12T04:36:52.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_ratelimit": { 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_ratelimit": { 2024-04-12T04:36:52.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "anonymous_ratelimit": { 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "ff072c13-de01-44da-babe-80b386c4a5cf", 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_name": "test-realm", 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dgbpxg-68 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/ 2024-04-12T04:36:52.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/ 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/ 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.848 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:52 GMT 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/ 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:9Mph+az2Hr6KsjLddB7GmlPS1Js= 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:9Mph+az2Hr6KsjLddB7GmlPS1Js=', 'Content-Length': '0'} 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ddcc5cc7b0909dcf-006618b5f0-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/ 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/ 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/ 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:52 GMT 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/ 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:XQwe9IO8KAeXmPbkMVr3Jlu8zYo= 2024-04-12T04:36:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:XQwe9IO8KAeXmPbkMVr3Jlu8zYo=', 'Content-Length': '0'} 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a448229c66da58-006618b5f0-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/obj1 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/obj1 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/obj1 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.851 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:52 GMT 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/obj1 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:45Aa49m0Oz3dZnQUteL5Zv1sbnk= 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:45Aa49m0Oz3dZnQUteL5Zv1sbnk='} 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000847efa78c6e263c7-006618b5f0-25b5-a1'), ('Date', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dgbpxg-69 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-69/ 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-69/ 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-69/ 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:52 GMT 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:/dgbpxg-69/ 2024-04-12T04:36:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:KixI0QcTNQ8i3DsVkaj+H3bwYXQ= 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:52 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:KixI0QcTNQ8i3DsVkaj+H3bwYXQ=', 'Content-Length': '0'} 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000bcb036023695b2f5-006618b5f1-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-69/ 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-69/ 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-69/ 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:53 GMT 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:/dgbpxg-69/ 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:f7Bvs4aZ2r50vv9w7XFhPYYdvco= 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:17:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:f7Bvs4aZ2r50vv9w7XFhPYYdvco=', 'Content-Length': '0'} 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000367bb8380f08a35e-006618b5f1-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-69/obj2 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-69/obj2 2024-04-12T04:36:52.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-69/obj2 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:17:53 GMT 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:/dgbpxg-69/obj2 2024-04-12T04:36:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:F7dcn+SSe1DYPErdWjaHAq3nJ4k= 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:17:53 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:F7dcn+SSe1DYPErdWjaHAq3nJ4k='} 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f652871d6f02f5d3-006618b5f1-25c1-a2'), ('Date', 'Fri, 12 Apr 2024 04:17:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=3ee804bf-b01f-4af0-895f-822e923b43dd 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:54:42.603296Z" 2024-04-12T04:36:52.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T02:57:39.711628Z" 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:17:53.109714Z" 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:53.442+0000 7f41a6d2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:53.444+0000 7f41a6d2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:17:53Z" 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.864 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.868 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:17:53Z" 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:58.556+0000 7fdbb9005780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:17:58.557+0000 7fdbb9005780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:17:58Z" 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.875 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.879 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:17:58Z" 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:03.668+0000 7fb8332ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:03.669+0000 7fb8332ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:03Z" 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.888 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.889 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.890 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:03Z" 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:08.777+0000 7fbb5b4fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:08.778+0000 7fbb5b4fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:08Z" 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.896 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.898 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.899 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.900 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:08Z" 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:13.888+0000 7f705842f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:13.889+0000 7f705842f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:13Z" 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.909 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.910 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.911 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:13Z" 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:19.000+0000 7f0fbc0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:19.001+0000 7f0fbc0ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:19Z" 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.917 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.918 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.920 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.921 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:19Z" 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:24.113+0000 7f4696e21780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:24.114+0000 7f4696e21780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:24Z" 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.928 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.929 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.930 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.932 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:24Z" 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:29.225+0000 7f1ef517c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:29.226+0000 7f1ef517c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:29Z" 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.939 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.942 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:29Z" 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:34.338+0000 7fdc54566780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:34.339+0000 7fdc54566780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:34Z" 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.949 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.951 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.952 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:34Z" 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:39.451+0000 7f7897db0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:39.452+0000 7f7897db0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:39Z" 2024-04-12T04:36:52.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.960 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.963 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:39Z" 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:44.563+0000 7f09db0ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:44.564+0000 7f09db0ac780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:44Z" 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.969 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.972 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.973 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:44Z" 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:49.676+0000 7fa450256780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:49.677+0000 7fa450256780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:49Z" 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.980 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.983 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:49Z" 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:54.786+0000 7fe625d31780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:54.787+0000 7fe625d31780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:54Z" 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:52.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:54Z" 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:52.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:59.896+0000 7f503d915780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:18:59.897+0000 7f503d915780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:52.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:52.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:52.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:18:59Z" 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.003 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.004 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.005 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.006 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:18:59Z" 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:05.009+0000 7f428a2c3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:05.010+0000 7f428a2c3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:05Z" 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.012 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.015 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.016 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:05Z" 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:10.121+0000 7f3011060780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:10.122+0000 7f3011060780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:10Z" 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.024 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.025 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.026 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:10Z" 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:15.235+0000 7f92d24bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:15.236+0000 7f92d24bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:15Z" 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.033 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.034 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.035 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.037 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:15Z" 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:20.345+0000 7f9fd7cde780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:20.346+0000 7f9fd7cde780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:20Z" 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.044 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.045 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.046 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.048 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:20Z" 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2024-04-12T04:19:24.133+0000 7ff19438b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:25.457+0000 7f49cd338780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:25.458+0000 7f49cd338780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:25Z" 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.055 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.056 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895450.171194_1845.1", 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:30.171194Z", 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:25Z" 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895450.171194_1845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:30.171194Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895450.171194_1845.1'} 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895473.109714_1924.1 target=1_1712895450.171194_1845.1 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:30.569+0000 7f12f4838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2024-04-12T04:19:30.570+0000 7f12f4838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:53.109714Z", 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2024-04-12T04:36:53.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "period": "3ee804bf-b01f-4af0-895f-822e923b43dd", 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.066 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890482.603296_102.1", 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:54:42.603296Z", 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.067 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1712890659.711628_123.1", 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T02:57:39.711628Z", 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895473.109714_1924.1", 2024-04-12T04:36:53.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:53.109714Z", 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.070 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712890482.603296_102.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T02:54:42.603296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712890659.711628_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T02:57:39.711628Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895473.109714_1924.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:53.109714Z', 'realm_epoch': 2}}] 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712890482.603296_102.1', 1: '', 2: '1_1712890659.711628_123.1', 3: '1_1712895473.109714_1924.1'} 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2024-04-12T04:36:53.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 10653437930919767770 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:36:53.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 57, 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 66, 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 66 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "instance_id": 10653437930919767770 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.078 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047354", 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:10:38.115982Z" 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032764", 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:10:39.150359Z" 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:36:53.080 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053085", 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:17:52.955658Z" 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046713", 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:36:53.081 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:10:38.051655Z" 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:19:30Z" 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:36:53.082 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047354', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.115982Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:10:39.150359Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053085', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:17:52.955658Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046713', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:10:38.051655Z'}}] 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000047354', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000053085', 3: '00000000000000000000:00000000000000046713'} 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/ 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/ 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/ 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:19:30 GMT 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/ 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:aD8NnjQAPwH6HY3SkB7yQ+cb8pM= 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:aD8NnjQAPwH6HY3SkB7yQ+cb8pM=', 'Content-Length': '0'} 2024-04-12T04:36:53.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dd769ccbb688a75-006618b652-25c1-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/obj1 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/obj1 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/obj1 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.085 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:19:30 GMT 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/obj1 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8rvvpH/Bs6x98+4PcH3b0MsHtP8= 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8rvvpH/Bs6x98+4PcH3b0MsHtP8=', 'Content-Length': '0'} 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-68:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b7115a52cc3fe14-006618b652-25c1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-68/obj1 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-68/obj1 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-68/obj1 2024-04-12T04:36:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:19:30 GMT 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:/dgbpxg-68/obj1 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:6guTUZqQHsZMrnsBP+UIPhXM+gI= 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:6guTUZqQHsZMrnsBP+UIPhXM+gI=', 'Content-Length': '0'} 2024-04-12T04:36:53.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:17:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b7d2416c-9d6f-4780-9e14-1f66614caba8:dgbpxg-68:b7d2416c-9d6f-4780-9e14-1f66614caba8.9655.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d92f63b207bada0-006618b652-25c1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-69/ 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-69/ 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-69/ 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:53.088 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:19:30 GMT 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:/dgbpxg-69/ 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:NBmw9rd5Tu5fvwVq75Bp9yjfUJk= 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:NBmw9rd5Tu5fvwVq75Bp9yjfUJk=', 'Content-Length': '0'} 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000339e9357113cd464-006618b652-25b5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dgbpxg-69/obj2 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dgbpxg-69/obj2 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dgbpxg-69/obj2 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:36:53.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:19:30 GMT 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:/dgbpxg-69/obj2 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:AWS JB2ZC967UHS0RDZ3J3O2:8Vcz+TeT10ROJNpcIj10623OB20= 2024-04-12T04:36:53.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:30 GMT', 'Authorization': 'AWS JB2ZC967UHS0RDZ3J3O2:8Vcz+TeT10ROJNpcIj10623OB20=', 'Content-Length': '0'} 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008d079799a6c7bb12-006618b652-25b5-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 6008.908s 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:36:53.091 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=7) 2024-04-12T04:36:53.094 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-12T04:36:53.532 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=aab955982cad4989b8b197673dafa86f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-12T04:36:53.533 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-12T04:36:53.546 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-12T04:36:53.558 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-12T04:36:53.567 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-12T04:36:53.568 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:36:53.583 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:36:53.569+0000 7f38de709700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 47766) UID: 0 2024-04-12T04:36:53.583 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2024-04-12T04:36:53.569+0000 7f38e38de7c0 -1 shutting down 2024-04-12T04:36:54.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-12T04:36:59.670 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-12T04:36:59.671 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-12T04:36:59.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:36:59.780 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:36:59.780 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:36:59.781 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:36:59.782+0000 7ff1a5bae700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 116282) UID: 0 2024-04-12T04:36:59.781 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2024-04-12T04:36:59.782+0000 7ff1aad837c0 -1 shutting down 2024-04-12T04:36:59.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2024-04-12T04:37:05.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:37:05.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-12T04:37:05.882 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:37:05.883 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-12T04:37:05.904 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:37:05.976 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-12T04:37:05.977 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:05.978 INFO:tasks.rgw.c2.client.0.smithi181.stdout:2024-04-12T04:37:05.979+0000 7fc3d222c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 48224) UID: 0 2024-04-12T04:37:05.979 INFO:tasks.rgw.c2.client.0.smithi181.stdout:2024-04-12T04:37:05.979+0000 7fc3d74017c0 -1 shutting down 2024-04-12T04:37:10.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-12T04:37:10.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:37:12.078 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-12T04:37:12.078 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-12T04:37:12.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:37:12.282 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-12T04:37:12.283 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:12.283 INFO:tasks.rgw.c2.client.1.smithi181.stdout:2024-04-12T04:37:12.284+0000 7fe77a89c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 49492) UID: 0 2024-04-12T04:37:12.284 INFO:tasks.rgw.c2.client.1.smithi181.stdout:2024-04-12T04:37:12.284+0000 7fe77fa717c0 -1 shutting down 2024-04-12T04:37:16.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-12T04:37:16.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-12T04:37:18.384 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-12T04:37:18.385 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-12T04:37:18.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:37:18.478 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-12T04:37:18.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-04-12T04:37:19.092 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-12T04:37:19.093 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-12T04:37:19.093 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-12T04:37:19.093 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:19.173 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-12T04:37:19.173 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-12T04:37:19.173 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:19.251 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-12T04:37:19.252 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-12T04:37:19.252 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:19.323 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-12T04:37:19.323 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-12T04:37:19.323 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-12T04:37:19.323 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:19.348 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-12T04:37:19.348 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-12T04:37:19.348 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-12T04:37:19.348 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:19.407 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-12T04:37:19.407 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-12T04:37:19.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-12T04:37:19.525 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T04:37:19.526 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-12T04:37:19.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T04:37:19.926 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-12T04:37:20.046 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T04:37:20.046 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-12T04:37:20.157 INFO:tasks.ceph:Archiving mon data... 2024-04-12T04:37:20.159 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/data/c2.mon.a.tgz 2024-04-12T04:37:20.159 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-04-12T04:37:20.176 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.Yy1757vm0u 2024-04-12T04:37:20.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.Yy1757vm0u 2024-04-12T04:37:20.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.Yy1757vm0u 2024-04-12T04:37:20.459 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.Yy1757vm0u is 447KB 2024-04-12T04:37:20.555 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.Yy1757vm0u 2024-04-12T04:37:20.572 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-12T04:37:20.572 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:37:20.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:37:20.638 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-12T04:37:20.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-12T04:37:21.001 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-12T04:37:21.001 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-12T04:37:21.002 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-12T04:37:21.002 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:21.049 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-12T04:37:21.049 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-12T04:37:21.049 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:21.103 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-12T04:37:21.104 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-12T04:37:21.104 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:21.158 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-12T04:37:21.158 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-12T04:37:21.158 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-12T04:37:21.158 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:21.183 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-12T04:37:21.183 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-12T04:37:21.183 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-12T04:37:21.183 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:37:21.243 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-12T04:37:21.243 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-12T04:37:21.243 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-12T04:37:21.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-12T04:37:21.312 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T04:37:21.312 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-12T04:37:21.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T04:37:21.729 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-12T04:37:21.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T04:37:21.826 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-12T04:37:21.906 INFO:tasks.ceph:Archiving mon data... 2024-04-12T04:37:21.906 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/data/c1.mon.a.tgz 2024-04-12T04:37:21.906 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-04-12T04:37:21.923 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.ZnYsFUbrrc 2024-04-12T04:37:21.923 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.ZnYsFUbrrc 2024-04-12T04:37:22.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.ZnYsFUbrrc 2024-04-12T04:37:22.258 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.ZnYsFUbrrc is 495KB 2024-04-12T04:37:22.356 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.ZnYsFUbrrc 2024-04-12T04:37:22.373 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-12T04:37:22.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:37:22.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:37:22.444 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-12T04:37:22.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-12T04:37:22.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-12T04:37:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-12T04:37:22.511 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-12T04:37:22.572 INFO:tasks.ceph:Archiving crash dumps... 2024-04-12T04:37:22.576 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi099/crash 2024-04-12T04:37:22.577 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-12T04:37:22.604 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi181/crash 2024-04-12T04:37:22.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-12T04:37:22.639 INFO:tasks.ceph:Compressing logs... 2024-04-12T04:37:22.639 DEBUG:teuthology.orchestra.run.smithi099:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T04:37:22.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T04:49:57.585 INFO:tasks.ceph:Archiving logs... 2024-04-12T04:49:57.587 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi099/log 2024-04-12T04:49:57.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T04:50:20.763 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi181/log 2024-04-12T04:50:20.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T04:51:52.320 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-12T04:51:52.332 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-12T04:51:52.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T04:51:52.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T04:51:52.457 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T04:51:52.469 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T04:51:52.470 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T04:51:52.473 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T04:51:52.488 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-04-12T04:51:52.489 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-04-12T04:51:52.493 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T04:51:52.494 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-04-12T04:51:52.494 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv01.front.sepia.ceph.com 3 8 377 157 +1299us[+1280us] +/- 51ms 2024-04-12T04:51:52.494 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 160 -844us[ -863us] +/- 69ms 2024-04-12T04:51:52.494 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 93 -723us[ -723us] +/- 71ms 2024-04-12T04:51:52.494 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:51:52.544 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T04:51:52.544 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-04-12T04:51:52.544 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 7 377 26 +1227us[+1208us] +/- 53ms 2024-04-12T04:51:52.544 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -956us[ -956us] +/- 71ms 2024-04-12T04:51:52.545 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -845us[ -845us] +/- 72ms 2024-04-12T04:51:52.545 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:51:52.546 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T04:51:52.558 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T04:51:52.559 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T04:51:52.580 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T04:51:52.655 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T04:51:52.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T04:51:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:51:52.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T04:51:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:51:52.799 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T04:51:52.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T04:51:52.827 INFO:teuthology.task.internal:Duration was 8007.685872 seconds 2024-04-12T04:51:52.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T04:51:52.841 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T04:51:52.842 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T04:51:52.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T04:51:52.890 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T04:51:52.933 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T04:51:53.295 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T04:51:53.295 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-04-12T04:51:53.295 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T04:51:53.317 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-04-12T04:51:53.318 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T04:51:53.355 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T04:51:53.356 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T04:51:53.359 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T04:51:53.469 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T04:51:53.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T04:51:53.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T04:51:53.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T04:51:53.566 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T04:51:53.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T04:51:53.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T04:51:53.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T04:51:53.660 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T04:51:53.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T04:51:53.686 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-04-12T04:51:53.710 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-04-12T04:51:53.732 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T04:51:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:51:53.788 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T04:51:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:51:53.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T04:51:53.816 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T04:51:53.817 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi099 2024-04-12T04:51:53.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T04:51:53.869 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588/remote/smithi181 2024-04-12T04:51:53.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T04:51:53.908 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T04:51:53.908 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T04:51:53.910 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T04:51:53.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T04:51:53.974 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T04:51:53.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T04:51:53.987 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T04:51:53.987 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T04:51:53.990 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T04:51:54.004 INFO:teuthology.orchestra.run.smithi099.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 04:51 /home/ubuntu/cephtest 2024-04-12T04:51:54.016 INFO:teuthology.orchestra.run.smithi181.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 04:51 /home/ubuntu/cephtest 2024-04-12T04:51:54.018 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T04:51:54.030 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T04:51:54.054 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T04:51:54.088 DEBUG:teuthology.nuke:shortname: smithi099 2024-04-12T04:51:54.089 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T04:51:54.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:32:59.595267', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTwpORqxF9iz3chL4tw4yrbtah//4i57koBpG6bhfbnaBL2JrJPBf4di8jSm+YAH/YD8q536515s1peMJjqEzg='} 2024-04-12T04:51:54.145 DEBUG:teuthology.nuke:shortname: smithi181 2024-04-12T04:51:54.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T04:51:54.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:01-rgw-reef-release-distro-default-smithi/7652588', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:32:59.593702', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/9ghsepfvA6W427iDW4ZktMrfXGxKfbxzC0J3HL+ULgMBV5nXlZzI6yDzw85DCviyvol6Y+ZF+JXcH/Mlg7oY='} 2024-04-12T04:51:54.230 INFO:teuthology.orchestra.console.smithi099:Power off 2024-04-12T04:51:54.231 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T04:51:54.285 INFO:teuthology.orchestra.console.smithi181:Power off 2024-04-12T04:51:54.286 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T04:51:54.297 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-04-12T04:51:54.298 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:51:54.332 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-04-12T04:51:54.340 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-04-12T04:51:54.340 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:51:54.391 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-12T04:51:58.333 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:51:58.393 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:51:58.446 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-04-12T04:51:58.505 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-12T04:52:02.448 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:52:02.506 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:52:02.561 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-04-12T04:52:02.622 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-12T04:52:06.562 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:52:06.623 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T04:52:06.675 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-04-12T04:52:06.735 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-04-12T04:52:06.776 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-04-12T04:52:06.836 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-04-12T04:52:06.941 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} duration: 8007.6858723163605 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=aab955982cad4989b8b197673dafa86f status: fail success: false 2024-04-12T04:52:06.941 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T04:52:07.031 INFO:teuthology.run:FAIL